Large language models can now shoulder one of the most labor-intensive jobs in public health research: reading news coverage of contested policies and identifying the rhetorical frames used to argue for or against them. A new study published in Archives of Public Health reports that when researchers prompted GPT-5 to extract quotations from German newspaper articles and classify them into issue-specific frames, the model recovered 75.2 percent of the frames that a domain expert had identified manually, achieving a precision of 79.8 percent and a recall of 79.8 percent, for an F1-score of 79.8 percent. A second frontier model, Gemini 3.1 Pro Preview, traded some recall for markedly higher precision, reaching an F1-score of 79.0 percent. The authors, Johanna Wenderoth, Peter von Philipsborn, Anna Leibinger and Philipp Cimiano, present the work as a systematic workflow for using large language models in framing analysis while evaluating the outputs transparently against an expert reference standard.
Framing analysis is a well-established method for monitoring policy debates. It seeks to identify the patterns, or frames, that media and policy actors use to argue for or against a topic, making some issues prominent while omitting others. As the researchers note, Entman formally defined framing as selecting some aspects of a perceived reality and making them more salient in a communicating text, and Boydstun and colleagues emphasize that framing can significantly influence public attitudes toward important policy issues. Framing studies in health policy have examined meat taxes, sugar taxes, tobacco control and spirits taxes, and a scoping review by Koon and colleagues covering 52 framing analysis articles from 12 countries concluded that framing research is an important vehicle for understanding human behavior in the health policy process.
The problem is that framing analyses usually rely on manual, qualitative coding, in which domain experts read relevant articles and identify frames. According to the study, this is time-consuming and labor-intensive, which limits the amount and recency of data a study can include. Frame identification is also somewhat subjective and may be influenced by a coder’s perspective and beliefs, and a continuous stream of new content brings potential new frames. These constraints matter because policymakers need timely evidence about public debates to inform evidence-based policymaking, particularly when new health policy proposals are controversially discussed.
The new workflow builds directly on a previous qualitative framing analysis of the German media debate about a tax on animal-based foods. That earlier study had collected 381 news articles from 10 national German newspapers and eight trade press outlets, all published between February 2019 and July 2023 and all written in German. A domain expert had manually identified 2,700 relevant quotes and coded them into frames, using a codebook developed through an iterative, consensus-based approach, with a second author checking 30 percent of all coded text segments for quality control. Because arguments against a policy are of particular interest to policymakers, the new study focused on quotes framing the tax negatively.
The researchers narrowed the task to the ten most frequent negative frames, which together accounted for 47 percent of all coded quotes. These included tax as amplifying inequalities, tax money as not going to stated aims, tax as not improving animal welfare, tax as hurting domestic production, and earmarking as not possible, among others. From the 202 articles containing one of these frames, they excluded 18 articles used for training and randomly sampled 25 percent of the remainder, giving 46 articles with an average length of 6,763 characters, or 917 words. Using in-context learning with 20 examples drawn from the training articles in a few-shot setting, they built prompts with the open-source LangChain framework. Each prompt contained a system message, a description of the desired output format, the 20 examples, and the full text of one news article, and the script sent one request per article.
Evaluation was deliberately rigorous. The team compared every automatic result against the human-coded reference, treating a result as correct when the same frame had been identified manually for the same article. To compare quotations, they applied a heuristic: two quotes counted as equivalent if they were more than 70 percent alike by a similarity ratio, if one was a subset or superset of the other, or if they shared at least one sentence. Frames the model found that the manual coder had not were judged by two authors independently as fitting or non-fitting, with differences resolved by consensus. Precision was calculated as correct or fitting frames divided by all frames extracted, while recall was defined against all human results plus the additionally verified results, an adjustment the authors made to avoid underestimating true recall. Because outputs varied between runs, they performed three repetitions per setting and reported the median run.
The headline numbers show a strong but imperfect performance. GPT-5 extracted 129 quote/frame pairs, 22.9 percent more than the domain expert’s 105, and matched 79 of the manually identified frames, with the quote itself also matching in 94.9 percent of those cases. Of 50 additional frames the model proposed beyond the manual coding, 48.0 percent were judged fitting after review. In nearly half of these additional cases, the human coder had extracted a quote from the same text segment but assigned a different frame, sometimes a related one outside the top ten, such as earmarking as against EU law rather than earmarking as not possible. There was one instance of hallucination, where the returned quote did not exist in the article, and other errors included overly broad frame assignments and misread frame definitions.
Sensitivity analyses revealed how the setup shapes performance. Reducing the examples from two per frame to one produced the most results, 143, and identified 78.1 percent of manual frames, the highest of any run, but at the lowest precision. A German-language prompt slightly outperformed the standard English run, with an F1-score of 80.3 percent. Gemini 3.1 Pro Preview delivered the fewest results, 93, identified 67.6 percent of manual frames, but achieved by far the highest precision at 89.2 percent, yielding a comparable F1-score of 79.0 percent versus 79.8 percent for GPT-5. The authors say these findings suggest the approach is generalizable across different frontier models. In an exploratory post-hoc analysis with a more complex set of 20 frames representing 77 percent of human-coded instances, however, GPT-5 identified only 66.7 percent of human frames, an 8.5 percentage-point drop, indicating that recall declines as frame frequency decreases and prompt complexity increases.
The practical implications are resource-related rather than clinical. The authors estimate that validating a frame takes a domain expert roughly 60 to 90 seconds, with an average of three frames to check per article, which is significantly less than the 20 to 40 minutes needed to read and code an entire article from scratch. On that basis they argue the time savings outweigh the cost of human-in-the-loop validation, positioning the model as a second coder that can confirm manually extracted frames, flag classifications that might need review, or extract frames from new articles quickly and at scale. The workflow could, they suggest, increase the representativeness and timeliness of data and help discover trends such as shifts in public opinion, though the method remains configurable toward higher recall for discovery or higher precision for confidence.
Important limitations temper the enthusiasm. The model saw only the top ten frames and two examples per frame, without general frame descriptions, so it worked with less differentiated information than the domain expert, making the comparison not perfectly like-for-like. The top ten frames covered just 47 percent of human-coded negative-frame instances, and the approach can only identify previously defined frames, not newly emerging ones. The authors also note the subjective nature of qualitative research, meaning differences can be expected even between human coders, and they caution that ongoing quality testing and prompt maintenance will be necessary over time. Their conclusion is measured: the large language model does not reach the quality of a human coder and cannot replace human coding, but it can improve the overall results of a framing analysis when used in addition, with future work planned on adding frame descriptions to prompts, splitting extraction and classification into separate prompts, and expanding beyond the top ten frames.
Subject of Research: Using large language models to extract and evaluate policy frames in public health news coverage
Article Title: A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis
Article References: Wenderoth, J., von Philipsborn, P., Leibinger, A., & Cimiano, P. (2026). A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis. Archives of Public Health, 84(1), Article 225. https://doi.org/10.1186/s13690-026-02089-6
Image Credits: AI Generated
DOI: 10.1186/s13690-026-02089-6
Keywords: large language models, framing analysis, public health policy, GPT-5, Gemini, news media, qualitative coding, precision and recall, animal-based food tax, natural language processing, Germany, prompt engineering
News Source: Timothy Lambert. (October 29, 2026). AI Reads the News: Large Language Models Decode Public Health Policy Debates. Scienmag.
