Assessing the performance of multiple influenza forecasting models

AMHERST, Mass. – In what the authors believe is the first documented comparison of several real-time infectious disease forecasting models by different teams across many seasons, five research groups report this week that a majority of models consistently showed higher accuracy than historical baseline models.

Led by biostatistician Nicholas Reich at the University of Massachusetts Amherst, the research teams formed a consortium called the FluSight Network and compared the forecast accuracy of 20 models compared to a historical baseline seasonal average. Accurately predicting the size and timing of infectious disease outbreaks helps public health officials in planning appropriate responses, they say.

Reich, at the School of Public Health and Health Sciences at UMass Amherst, says this paper, the first of several planned from this research consortium, offers “a survey of which models do well, when and why, plus a sort of meta-analysis of the state of the field right now. We have brought together some of the top flu forecasting teams in the world, and through this collaboration have enabled an apples-to-apples comparison of our different methods and results.” Details appear in the current online edition of Proceedings of the National Academy of Sciences.

He and colleagues write, “Across all regions of the United States, over half of the models showed consistently better performance than the historical baseline when forecasting incidence of influenza-like illness one, two and three weeks ahead of available data and when forecasting the timing and magnitude of the seasonal peak.”

Influenza infects an estimated 9 million to 35 million individuals each year in the United States and is a contributing cause for between 12,000 and 56,000 deaths annually, the researchers point out. More than 6 million Americans have already caught influenza this flu season, Reich adds, and as many as 80,000 have ended up in the hospital, federal health officials reported most recently last Friday.

In addition to comparing the contributed models, the efforts of the FluSight Network has enabled the creation of an “ensemble” model. Each group uses their own advanced analytical techniques, Reich explains, to come up with a forecasted flu trajectory for the year. These individual forecasts are combined into a single “ensemble” forecast that is sent to the CDC each week.

When compared to historical trends, these forecasts help the CDC and other public health officials more effectively plan and respond to the evolving seasonal epidemic. The FluSight Network ensemble was one of the top-performing real-time models in the 2017/2018 season. The CDC uses the FluSight Network ensemble approach in their internal and external communication and planning for the influenza season. Peer-reviewed research on the ensemble model is forthcoming, says Reich.

Overall, the authors state, “Our collaborative, team science approach highlights the ability of multiple research groups working together to uncover patterns and trends of model performance that are harder to observe in single-team studies.”

“The field of infectious disease forecasting is in its infancy and we expect that innovation will spur improvements in forecasting in the coming years,” they add. “Public health officials are still learning how best to integrate forecasts into real-time decision making. Close collaboration between public health policy-makers and quantitative modelers is necessary to ensure that forecasts have maximum impact and are appropriately communicated to the public and the broader public health community.”

Reich worked with colleagues at Carnegie Mellon University, the University of Texas at Austin, the U.S. Centers for Disease Control and Prevention (CDC), Columbia University, Los Alamos National Laboratory and Mount Holyoke College on the project.

The researchers note that in the past 15 years, increased interest in forecasting infectious disease has been fueled in part by the promise of “big data,” and the idea that new data streams could lead to measurable improvements in how disease transmission is measured, forecast and controlled. “While multi-model comparisons exist in the literature for single-outbreak performance, here we compare a consistent set of models over seven influenza seasons,” 2010 through 2017.

###