Home

The Fox lab had our first “lab” paper come out this year that was started and finished during my tenure here at UGA! In the paper, myself and my collaborators (Lauren Meyers from UT, and Evan, Nick, and Minsu from UMass Amherst) investigated how the size of ensembles impacted their performance using data from all of the recent collaborative forecast hub data: https://wwwnc.cdc.gov/eid/article/30/9/24-0026-f2

Background

The background for this project came as Nick, Evan, and I were chatting at the CSTE Forecast conference in 2022 about forecasting. There was a question that came up repeatedly that year if we really needed dozens of teams to be producing weekly forecasts for COVID-19, because in the end we tend to only broadcast a single ensemble forecast. Since all of the collaborative hubs have publicly available data, we mapped out a project to actually investigate it.

Methods

We gathered all of the forecasts for all recent collaborative forecast efforts (millions and millions of data points), constructed random ensembles of varying sizes, and scored their performance. A key limitation is that we chose to run the analysis with a subset of models that had regularly forecasted during the time periods we chosen, meaning that our sample size wasn’t that large for many of the hubs AND that our analysis likely focuses on some of the better forecast models (many models that don’t perform well come in and out, so the consistent models tend to be some of the better ones submitted).

Results

We have three results from our primary analysis: (1) adding models always improves and reduces variability of the average forecast performance, (2) performance gains level off after 4-5 models are included, and (3) you want to include at least four models in your ensemble to beat baseline forecasts. All of these conclusions can be seen in Figure 2 of the manuscript (Panel B of this version):

After running these analyses, one can see that there are some randomly assembled ensembles for almost all of these efforts that beat the published ensemble that is composed of all submitted forecasts (some of the grey shaded regions are below the horizontal purple line and lower numbers are better in these). We set out to see if we could develop a simple way to select these models and found that generally ensembles of models that perform well in one season perform well in others (The Ensemble rank line in this figure):

While I think this is a really amazing finding, I didn’t feel comfortable having this as the key finding in the paper, for two reasons:

1. We didn’t investigate more sophisticated ensemble weighting methodologies that likely will outperform these ones
2. This approach will not work if the submitted models change season to season. For example, this past season the Flusion model performed amazingly, but would not have been included in this ensemble, because we were basing everything on performance from the previous season. I would love to pursue a method to make a real-time version of this ensemble that would address this issue and see if it outperforms other approaches.

Conclusions

Hub organizers should target a minimum of 4 validated forecast models to ensure robust performance compared with baseline models. Adding more models reduces the variability in expected ensemble performance but might come with diminishing returns in average forecast skill. As public health officials and researchers look to expand collaborative forecast efforts, and as funding agencies allocate budgets across methodological and applied forecast efforts, our results can be used to identify target participation rates, assemble appropriate forecast models, and further improve ensemble forecast performance.

The project was largely built on infrastructure made available through the hubverse ecosystem, and I think these data have a lot of potential for project you’re interested in the topic or methodology, you can find the code to recreate the analysis here: https://github.com/sjfox/ensemble-size