Are time series foundation models ready to ship?

On 23 August 2023, Typhoon Saola formed southeast of Taiwan and later approached Hong Kong. This prompted the Hong Kong port authorities to suspend pilotage services for a couple of days. Naturally, closing a busy port like Hong Kong, even for a few days, can cause tremendous economic damage. Once operations restarted, the port workers faced a backlog of ships that had built up during the storm. Ships waiting outside the port could then arrive close together, leading to a peak in traffic as operations caught up.

A little over a month later, Koinu brought severe weather to Hong Kong on 8–9 October. Container-terminal services were suspended again. Two disruptions, close together, make an interesting test: can a forecasting model use the earlier event to anticipate the response to the next one?

An interesting use case for time series foundation models is predicting port traffic. These time series often look fairly flat, with irregular noise on top. In this case, we use data from PortWatch, an IMF–Oxford platform that tracks shipping at ports around the world. The Koinu case mentioned above can be clearly seen. During the typhoon, arrivals drop to zero, followed by a peak as operations resume.

Time series foundation models attempt to replicate the success of scaling in language foundation models for time series. Recently, Mistral released a new family aptly named Timestral. We use Timestral-209M. Here, we show that, in these examples, it can fail quite spectacularly to extrapolate patterns that are present, even where a human could probably guess the pattern reasonably well.

Knowing the story at the beginning of this post, could you guess what Koinu did to Hong Kong? Probably yes. Nevertheless, Timestral seems less confident when provided with this information and an indicator of a future disruption. Its predicted median stays close to the usual daily level (remember, arrivals per day are often a fairly flat time series). This is a reasonable guess for normal operations, yet it mainly predicts more uncertainty during the typhoon. This is especially interesting because the event indicator marks only one earlier disruption: Saola.

Another question is to what extent one could learn to predict the noise patterns in the data. At some level of granularity, the noise is not random but just the result of shipping decisions made by thousands of ships on the open sea, moving goods from one country to another. Hence, for the next experiment, we take the port of Rotterdam and put all the port and chokepoint series in our PortWatch dataset into the model’s context. This includes every tracked port, every recorded disruption, and every tracked chokepoint.

The only thing it seems to pick up is the weekend reduction in shipping arrivals. This is not particularly useful, but it is interesting. We also see that all this extra data makes little difference in this run: switching it off (try it yourself!) barely changes forecast accuracy. We do feed the model the day of the week, which seems to be the only clear pattern it finds.

Finally, we can explore a counterfactual scenario. In late 2023, Houthi attacks in the Red Sea raised insurance premiums and led ships to take a longer detour around the Cape of Good Hope. This reduced traffic through Suez. What would happen if the Houthis stopped threatening to attack ships? Surely, time series foundation models could be useful for exploring such counterfactuals.

So we simply set a flag for whether Houthi activity continues after 1 April 2024. Intuitively, one would predict a return towards the baseline. Timestral does predict a recovery. Interestingly, based on my own research and experience, I would expect it to be a bit faster and perhaps include a peak above the average, similar to the port examples. Of course, this is still a guess. It is also interesting that Timestral’s predicted recovery remains well below the pre-Houthi level.

What is missing? Are time series foundation models doomed? In these reasonably simple examples, the model seems to have missed even some of the simpler patterns, while someone with the shipping context could quickly guess what the next scenarios might look like.

I think what is missing is a metadata layer that would allow time series models to reason and incorporate contextual data beyond correlations between time series. Otherwise, the model risks failing to predict anything relevant.

If I were to show you just the time series without the shipping context, it would be hard to predict where they are going in most cases. The context helps us reason about what happens next: if a port is closed one day, you might work harder the next day to catch up. I guess we will see what the future brings.

I'm Mitja. I work on large-scale simulations of shipping and economic systems. Comments are welcome: devetak.github.io