In a comparative assessment of artificial intelligence tools and human reviewers in systematic review screening, the agentic software Loon Lens outperformed Catchii when evaluated against human decisions. The study, published in Health and Quality of Life Outcomes by Marco Gorici, Abel Torres-Espin, and Mark Oremus, evaluated how automated screening technologies function within evidence synthesis workflows. Systematic reviews are essential for informing healthcare policy and practice, yet screening vast citation libraries demands substantial labor. While artificial intelligence promises considerable time savings by automating citation filtering, the screening performance of agentic tools relative to traditional human reviewers has remained uncertain.
To assess screening capabilities under realistic conditions, the researchers formulated a de novo review question investigating potential associations between ambient air pollutants and the onset or worsening of Parkinson’s disease. The pollutants analyzed comprised carbon monoxide, ozone, and nitrogen dioxide. A health sciences librarian designed the bibliographic search strategy, establishing the literature base for testing. The evaluation spanned two operational phases: initial title and abstract screening comparing both artificial intelligence tools to humans and to each other, and subsequent full-text screening evaluating Loon Lens directly against human reviewers.
The methodological framework used human screening as the primary reference standard to evaluate the accuracy of both artificial intelligence platforms. Catchii was additionally employed as a reference standard to benchmark Loon Lens. The research team evaluated performance using sensitivity, specificity, positive predictive value, negative predictive value, concordance, kappa, and the F1 score, contrasting these observed statistics against results obtained through naïve guessing to establish empirical baselines.
At the title and abstract stage, Loon Lens demonstrated superior diagnostic accuracy compared to Catchii when measured against human screening decisions. Loon Lens attained a sensitivity of 0.66, a kappa coefficient of 0.74, and an F1 score of 0.76. In contrast, Catchii achieved a sensitivity of 0.49, a kappa of 0.46, and an F1 score of 0.50 against the same human baseline. These divergent findings illustrate notable performance differences between two agentic systems performing the identical screening task on the same citation library.
During the full-text screening phase, comparative analysis between Loon Lens and human reviewers identified a distinct pattern of discordance. Most screening disagreements occurred when Loon Lens elected to include candidate articles that human screeners ultimately excluded. This tendency toward broader inclusion highlights how algorithmic screening decisions can diverge from human expert judgments when examining comprehensive study texts rather than abbreviated bibliographic abstracts.
The study also examined the utility of automated confidence metrics by regressing screening concordance onto confidence ratings provided by Loon Lens. The software assigns a discrete confidence level of Very High, High, Medium, or Low to each screening decision. Across both title and abstract screening and full-text screening, higher assigned confidence scores were associated with lower odds of disagreement with human screeners, demonstrating that internal tool confidence correlates with classification agreement.
Given the expanding selection of artificial intelligence tools, their divergent screening performance, and rapid advancements across technology, the authors recommended that systematic review teams pilot test their selected software before commencing any review. Furthermore, the researchers identified sensitivity, kappa, and F1 score as the optimal performance statistics for evaluating screening tools. They observed that raw concordance and negative predictive value can become misleadingly inflated during title and abstract screening, where severe imbalance between small numbers of included citations and vast volumes of excluded citations distorts overall accuracy measures.
Subject of Research: Performance of artificial intelligence versus human screening in systematic reviews
Article Title: Comparing artificial intelligence versus human screening in systematic reviews
Article References: Gorici, M., Torres-Espin, A., & Oremus, M. (2026). Comparing artificial intelligence versus human screening in systematic reviews. Health and Quality of Life Outcomes. https://doi.org/10.1186/s12955-026-02642-5
Image Credits: AI Generated
DOI: 10.1186/s12955-026-02642-5
Keywords: artificial intelligence, systematic reviews, Loon Lens, Catchii, title and abstract screening, full-text screening, human screening, sensitivity, kappa coefficient, F1 score, concordance, negative predictive value, Parkinson's disease, air pollution, literature search, confidence scores, health policy
News Source: Ophelia Keating. (November 19, 2026). Agentic AI Tool Loon Lens Outperforms Catchii in Systematic Review Screening. Scienmag.



