Survey maps how LLM web agents turn browsing into verified datasets

Large language model agents can now click through websites, drive graphical interfaces and call software tools with impressive fluency. But a new systematic survey argues that getting the job done is not the same as getting the data right, and it maps the field around a harder goal: assembling trustworthy, structured datasets from the open web without site-specific redevelopment.

The study, published in the International Journal of Data Science and Analytics, is a PRISMA-informed systematic mapping review with a narrative synthesis covering work from January 2017 to April 2026. Led by Ismail Khayoub, Mohamed-Amine Chadi, Hajar Mousannif and Firdaous Ait Mohamed, the team screened 805 unique studies, of which 454 were archival published or formally accepted. After reconciling publication status and full texts, and moving one already-mapped, publication-eligible KDD 2025 survey into the core corpus during scope re-adjudication, the final qualitative synthesis comprised 386 studies, all read in full.

The authors frame the problem as generalized schema-guided extraction: discovering relevant sources, traversing unfamiliar sites, populating typed records with field-level provenance, and validating the resulting dataset end to end. Their synthesis connects seven research threads — observation, grounding, planning, memory, learning, security and evaluation — to a six-stage pipeline that runs from raw source to finished dataset. This structure lets them compare how different agent designs handle each stage, from perceiving a page to certifying that every extracted field can be traced back to where it came from.

A distinctive feature of the survey is that it does not stop at cataloguing papers. The authors ground their proposed multi-level evaluation framework in a controlled 108-run case study that crossed three agent architectures, three LLM backends, four frozen commerce sites and three repetitions.

The case study’s headline numbers carry an uncomfortable lesson. The strongest observed configuration reached a macro Field F1 of 0.9440, indicating highly accurate extraction at the field level. Yet a Playwright-MCP configuration posted a respectable macro Field F1 of 0.7872 while achieving task success of only 0.1667 — succeeding at barely one in six navigation tasks. Extraction quality and task completion, in other words, can diverge sharply, and a leaderboard that scores only task success would misjudge which agent actually produces usable data.

That divergence anchors the survey’s central conclusion: progress in web agents should be assessed by verifiable outputs, not navigation success alone. The proposed evaluation framework is designed to make such distinctions measurable across multiple levels rather than collapsing them into a single success flag.

The review also consolidates the field’s scope honestly. Of the 805 mapped studies, only 454 met archival publication standards, and the historical synthesis-candidate set of 403 records was trimmed to 385 before the final adjudication brought it to 386 — figures that signal how much web-agent literature exists outside peer-reviewed venues. The 114 highest-priority P0/P1 studies received deep critical analysis, while the remaining 272 P2/P3 studies underwent complete structured reading.

For a field racing toward autonomous web assistants, the message is a sobering one: benchmarks that reward browsing behavior may be measuring the wrong thing. What ultimately matters, the authors argue, is whether an agent can hand over a validated dataset — typed fields, documented provenance, no site-specific redevelopment — and whether the community builds evaluation tools rigorous enough to tell the difference.

Subject of Research: A systematic survey of LLM-based agents for generalized web automation and schema-guided data extraction

Article Title: LLM-based agents for generalized web automation and schema-guided data extraction: a survey

Article References: Khayoub, I., Chadi, M.-A., Mousannif, H., & Ait Mohamed, F. (2026). LLM-based agents for generalized web automation and schema-guided data extraction: a survey. International Journal of Data Science and Analytics, 22(1), Article 365. https://doi.org/10.1007/s41060-026-01335-y

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01335-y

Keywords: large language models, web agents, data extraction, systematic review, schema-guided extraction, evaluation framework, task success, field-level provenance, web automation, benchmarking, LLM-based, agents