AI Turns Tsunami Evacuation Logs into Stories, But a New Audit Reveals Its Flaws

Imagine reading a first-person account of a tsunami evacuation—”I left home at zero seconds, reached the crossing at 93 seconds, and found the road congested”—generated automatically from the raw output of an agent-based simulation. That vision is now a step closer to reality, but so is a sobering understanding of what can go wrong when large language models are asked to turn dense simulation logs into human-readable narratives. In a study published in Discover Artificial Intelligence, Yuto Hirahata and Fumihiro Sakahira of the Osaka Institute of Technology present a quality-gate framework designed to audit exactly this kind of AI-generated storytelling, and their results reveal both the promise and the persistent weaknesses of the approach.

Agent-based evacuation simulations have long been a staple of disaster planning. Researchers model thousands of virtual residents moving through a real street network toward shelters, incorporating details such as reduced walking speed in dense crowds to reproduce bottlenecks and congestion. Traditionally, the results of such models are reported through aggregate statistics: how many agents evacuated, how long it took on average, where the choke points formed. These numbers are indispensable for modelers, but they say little about the lived, second-by-second experience of any individual evacuee. As computational power has grown and simulation outputs have become larger and higher-dimensional, the gap between what the models produce and what humans can readily interpret has widened.

Large language models, combined with retrieval-augmented generation, offer a way to close that gap. In a retrieval-augmented generation, or RAG, pipeline, the model does not rely on its internal knowledge alone; instead, relevant external documents—in this case movement logs, route maps, hazard maps, and population attribute data—are retrieved and supplied as context before the model writes its output. Multimodal RAG extends this to images, which matters here because route diagrams and disaster-prevention maps carry spatial information that a plain CSV table cannot easily convey. The Japanese research team built their pipeline around a tsunami-evacuation simulation of the Sanbo district of Sakai City, Osaka Prefecture, where agents follow shortest paths on a GIS-derived road network toward sites such as Asakayama Hospital.

The central methodological innovation is a three-layer decomposition borrowed from prior work on experience-oriented social simulation. The event layer contains the objective facts output directly by the simulation: the sequence of locations each agent passed, the arrival times, and the route. The context layer holds interpretations derived from those facts—in this study, estimates of whether congestion was present at each key location, obtained by comparing a congestion-free travel time with the agent’s actual passage time. The value layer is the only part allowed to vary: a first-person narrative expressing the agent’s subjective viewpoint, emotions, and decision-making rationale. The design goal is explicit boundary management—subjective color may shift, but the factual skeleton of times, locations, and route order must remain invariant.

To make this auditable rather than merely plausible, the researchers fixed every variable they could. The reference corpus, the vector store built with FAISS, the search query templates, and the retrieved documents were all frozen as snapshots, so that across 100 iterative generations per condition the model received identical evidence. Embeddings were produced with the Sentence-Transformers model all-MiniLM-L6-v2, image captioning used the BLIP model supplemented by analysis from a large vision-language model, and narrative generation ran on GPT-5-nano at temperature zero. Even at temperature zero, deployed language models are not fully deterministic across API calls, and the 100-trial design was built precisely to characterize that residual variability rather than to hide it behind a single cherry-picked example.

The quality gate itself is a set of fully automatic, rule-based checks with no tunable thresholds. An output passes only if it mentions all seven key locations, includes all eight expected timestamps in exact form, preserves the chronological order of locations, contains no prohibited internal representations such as agent identifiers or TRUE/FALSE flags, and avoids formatting deviations like bullet lists that would be jarring in a continuous first-person account. The gate is explicitly framed as a diagnostic instrument, not a certification: it detects and quantifies failure modes rather than guaranteeing that a narrative is fit for public deployment.

The results are instructive. For three agents following the same route, location coverage was high, at 95 to 99 percent, and sequence inconsistency was nearly absent, at 0 to 1 percent—evidence that the factual skeleton of the route survives generation. Timestamp coverage, however, lagged at 83 to 91 percent, and a programmatic check confirmed that the missing times were genuinely absent from the text in any form, not merely reformatted. More strikingly, prohibited terms or internal representations leaked into 7 to 23 percent of outputs, with strings like AgtID and TRUE/FALSE appearing in the prose. Overall quality-gate pass rates ranged from 62 to 81 percent across the three agents—far below what unsupervised public-facing use would require, and the authors state this plainly rather than downplaying it.

The team also tested whether the value layer could be steered without disturbing the factual skeleton. Holding the event and context layers fixed, they substituted household-composition data for one agent across three conditions: a baseline five-person household including a teenager, a household replacing the teenager with a member in their 90s, and an all-elderly household of five members in their 80s. The lexical indicators responded in the expected directions—child-related terms fell when the teenager was removed, elderly-related and mutual-aid terms rose with the older households—while location coverage, time coverage, and sequence consistency remained broadly comparable across conditions. No clear disruption of the factual skeleton was detected under any tested substitution, suggesting that the layered architecture can separate what must stay fixed from what is allowed to vary.

The authors trace the dominant leakage path to a specific engineering choice: the raw log vocabulary, including Boolean congestion flags and column names, was exposed to the model without normalization. Their proposed remedy is a label-normalization layer inserted before value-layer retrieval, which would rewrite TRUE/FALSE flags into natural phrases and mask identifiers at the source. For the timestamp bottleneck, they outline interventions including structured skeleton generation validated programmatically before rendering, prompt constraints requiring every event sentence to begin with its timestamp, and post-generation verify-and-repair loops that regenerate only the affected sentences. Notably, prompt-level prohibitions on meta-commentary proved largely effective, appearing in at most 1 percent of outputs—evidence that instruction-following works when the forbidden content is not simultaneously present in the input data.

The study’s limitations are acknowledged with unusual candor. It examines a single tsunami scenario in one district, three agents on an identical non-branching route, and a narrow set of household substitutions within the same speed class and household size. The narratives were generated in Japanese, so the lexical indicators are tied to Japanese dictionaries and would need re-instantiation for other languages. What the authors argue generalizes is not the specific numbers but the audit protocol itself: the three-layer decomposition, the fixed-retrieval iterative-generation design that isolates model variability, and the rule-based quality gates are all portable to other simulation domains. For now, the framework stands as pre-deployment diagnostic evidence—a way to measure, systematically and repeatably, where AI-rendered simulation narratives break down before anyone entrusts them to the public.

Subject of Research: Quality-gated auditing of LLM-based narrative rendering of agent-based tsunami evacuation simulation logs using multimodal retrieval-augmented generation

Article Title: Quality gate framework for auditing narrative rendering of evacuation simulation logs by multimodal retrieval augmented generation

Article References: Hirahata, Y., & Sakahira, F. (2026). Quality gate framework for auditing narrative rendering of evacuation simulation logs by multimodal retrieval augmented generation. Discover Artificial Intelligence, 6(1), Article 1366. https://doi.org/10.1007/s44163-026-02075-5

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02075-5

Keywords: large language models, retrieval-augmented generation, agent-based simulation, tsunami evacuation, quality gates, data-to-text generation, multimodal RAG, hallucination, synthetic population, disaster preparedness, narrative rendering, simulation explainability