AI Guardrails for Factories: Architecture Keeps LLMs From Unsafe Plant Actions

Large language models can reason about alarms, trends and operating procedures, but they also hallucinate, invent equipment that does not exist and propose physically impossible actions. In a chemical plant, where an incorrect valve command can damage equipment, release hazardous material or endanger lives, that unpredictability has kept the technology out of safety-critical control loops. A new study argues that the solution is not a better model but a stricter architecture, and it backs that argument with a working prototype and a battery of stress tests.

The research, published in the journal Autonomous Intelligent Systems, introduces the Autonomous Action Execution (AAE) framework, a four-layer multi-agent design that confines LLM reasoning to a purely observational role. The language model sits in the outermost Monitor layer, where it watches process data and drafts action proposals, but it has no write access to the SCADA systems that actually drive the plant. Every proposal must pass through a deterministic Verification Agent and a singleton Execution Agent before anything touches the process, and all inter-agent communication flows through MQTT v5.0, a lightweight messaging protocol with quality-of-service guarantees, cryptographic signing and schema validation at the broker.

The design principle is simple: the deeper a layer sits in the stack, the more conservative its logic. The Monitor layer is the only place where stochastic LLM inference happens. The Orchestrator consolidates proposals from multiple monitors, deduplicates them and resolves conflicts through a weighted vote based on confidence and anomaly scores. The Verification Agent is architecturally isolated from everything upstream — it shares no model weights, no memory and no communication channel except the consolidated plan arriving over the message broker. That isolation creates a formal trust boundary: even if an adversary fully compromised the monitor’s language model, its proposals would still have to clear deterministic structural checks before reaching SCADA.

Those checks are grounded in the plant’s own engineering documentation. Piping and Instrumentation Diagrams, or P&IDs, are the authoritative record of process topology, instrumentation and control relationships in process industries. The researchers convert the P&ID into a traversable graph in which equipment, instruments and valves become nodes and piping connections and control relationships become directed edges. Six automated checks then run against every proposal: tag existence, which catches hallucinated references to non-existent equipment; actuatability, which distinguishes writable elements such as valves and pumps from read-only sensors; SCADA controllability, which flags manual-only valves that cannot be driven by a setpoint change; control loop relevance; fail-state consistency, which catches physical contradictions such as trying to open a fail-closed pneumatic valve when the suspected root cause is loss of air supply; and downstream impact, which traces the consequences of an action through breadth-first traversal of the pipe network.

To feed the model useful context in the first place, the framework includes a text-level aggregation pipeline with pluggable analyzers. One extracts active alarms from SCADA readings, another detects rising and falling trends and identifies stuck actuators from identical consecutive samples, a third pulls control loop definitions and valve fail states from the P&ID, a fourth surfaces relevant troubleshooting steps from standard operating procedures, and a fifth derives higher-order insights such as broken control loops and downstream impact chains. Together they transform raw plant data into a structured document that gives the LLM explicit equipment tags, loop logic and procedural guidance, reducing — though never eliminating — the likelihood of hallucinated references.

The evaluation spanned five synthetic industrial scenarios. Three were derived from the Tennessee Eastman Process, a widely used community benchmark based on a real chemical process with five major unit operations, 41 measured variables and 12 manipulated variables. The other two were boundary cases: PolyReactor, a batch polymerisation reactor with a well-instrumented diagram representing the best case, and Dryer, a fluidised-bed dryer with a deliberately minimal model containing just one actuatable valve, representing the worst case. Using Claude Sonnet 4 as the reasoning model, the experiments showed that root cause identification was perfect — 100 percent across all runs — but that safety violations persisted. The LLM proposed forbidden actions, such as increasing steam on a saturated valve or stopping a blower while product was present, even when its diagnosis was correct. The failures lay in action selection, not diagnosis, which is precisely the failure mode that architectural separation is designed to address.

The stress tests quantified how reliably the validation layer catches such errors. In an error injection experiment, 43 crafted proposals — 35 unsafe and 8 valid — were fed through the validator across all five scenarios. The result was zero false negatives: every unsafe proposal in the covered categories was detected, with hallucinated tags, sensor writes and forbidden actions each caught at a 100 percent rate. The false positive rate, however, was high at roughly 0.75, meaning valid proposals on valves structurally distant from the root cause were conservatively flagged for review. The authors note this estimate rests on only eight negative samples and carries a wide confidence interval, but they argue the conservative bias is appropriate for a safety net: it is far better to over-flag and require human review than to let a potentially unsafe action through.

A statistical robustness study ran each scenario ten times at a sampling temperature of 0.7, for 50 independent LLM calls in total. Safety violations occurred in 10 to 70 percent of runs depending on the scenario — PolyReactor never violated safety while Dryer produced forbidden actions in 70 percent of its runs — yet every actually unsafe proposal received at minimum a warning verdict and was routed to operator approval rather than automatic execution. The study also revealed a counterintuitive effect: richer context can sometimes make the model worse. On the Dryer scenario, the full pipeline introduced tag hallucination that the raw-data baseline did not produce, apparently because the longer prompt competed for the model’s attention. The authors draw a pointed conclusion from this: if prompt enrichment can nudge a model into worse behaviour, then prompt engineering alone cannot be the safety mechanism, and validation must operate independently on engineering ground truth.

The team is careful about what the system is and is not. The AAE is explicitly not a Safety Instrumented System under IEC 61508 or IEC 61511; it is an advisory layer positioned below and independent of the plant’s existing safety instrumented functions. The LLM is excluded from any safety function, and if the entire advisory system failed, the plant’s certified protection layers would continue to operate unchanged. The authors also acknowledge significant limitations: all data artefacts are synthetic rather than captured from production systems, only one LLM was tested, each scenario contained a single fault, the validation is static and cannot catch unsafe action sequences or rate-of-change violations, no human-factors study with operators has been conducted, and the ground truth rubrics were authored by the system designer, which may introduce evaluation bias.

The broader significance lies in the framing. Rather than treating LLM safety as a model accuracy problem, the study treats it as a systems engineering challenge, mapping established concepts such as separation of concerns, independent protection layers and deterministic safety logic onto a new class of stochastic reasoning components. The proposed deployment path is deliberately cautious: first a lab pilot in a simulated environment, then shadow-mode operation on a real plant where proposals are logged but never executed, and finally a restricted pilot limited to informational notifications on a non-critical process area. Until real-plant validation, dynamic and temporal checks and operator workload studies are complete, the authors make no operational deployment claim — but the prototype demonstrates that the reasoning power of language models can be harnessed behind deterministic barriers built from the plant’s own engineering drawings.

Subject of Research: A multi-agent architecture with P&ID-grounded validation for safely integrating large language models into industrial process control

Article Title: Safe integration of Large Language Models into industrial process control: a multi-agent architecture with P&ID-grounded validation

Article References: Schall, D. (2026). Safe integration of Large Language Models into industrial process control: a multi-agent architecture with P&ID-grounded validation. Autonomous Intelligent Systems, 6(1), Article 14. https://doi.org/10.1007/s43684-026-00136-1

Image Credits: AI Generated

DOI: 10.1007/s43684-026-00136-1

Keywords: large language models, industrial process control, multi-agent systems, P&ID validation, functional safety, SCADA, hallucination, Tennessee Eastman Process, MQTT, autonomous agents, safety instrumented systems, process industries