AI Models Flunk Engineering Simulation Test in Massive New Benchmark

Artificial intelligence systems that can ace medical imaging quizzes and describe photographs in astonishing detail may be far less capable than they appear when confronted with the colorful contour plots and meshed geometries of engineering simulations. That is the sobering conclusion of a new study published in Communications Engineering, in which researchers at Carnegie Mellon University built one of the largest benchmarks ever assembled for evaluating how well vision-language models can interpret engineering simulation outputs. The result: ten state-of-the-art models, many of them celebrated for their general visual reasoning skills, performed at or near random chance when asked questions about simulation results, with effect sizes so small the researchers describe them as negligible.

The benchmark, called OpenSeeSimE, consists of more than 200,000 question-answer pairs spanning 10,000 parametrically varied simulations. According to the authors, Jessica Ezemba, Jason Pohl, Conrad Tucker, and Christopher McComb of Carnegie Mellon’s Department of Mechanical Engineering, the dataset represents an 850-fold scale increase over what had previously been possible in this niche, enabling statistically robust evaluation across diverse simulation configurations and question types. That scale matters enormously. Small benchmarks can produce results that look impressive but dissolve under statistical scrutiny; with hundreds of thousands of question-answer pairs, the researchers could measure model performance with enough statistical power to detect even incremental progress, and to show confidently when there was none.

The motivation behind the work stems from a genuine bottleneck in engineering design cycles. Interpreting simulation outputs, whether from structural analyses, fluid dynamics studies, or thermal models, requires expensive domain expertise. Engineers must validate complex outputs to ensure safety and performance, and that validation step can slow product development considerably. Large language models have been proposed as assistants in this interpretation task, but they face a fundamental scalability limitation: even modest simulations produce data that exceed the context windows of the best-in-class LLMs. A simulation that generates millions of data points simply cannot be fed into a model that processes text sequentially, no matter how large its context window has grown.

Vision-language models offer a promising alternative, and the reasoning behind that promise is elegant. Rather than ingesting raw simulation data, a VLM can process a visualization of the results, a contour plot, a deformed geometry, a field map, as a compressed representation of the underlying information. VLMs have already demonstrated success across technical visual reasoning domains, from medical imaging to materials characterization. If a model can identify anomalies in radiographs or classify microstructures in electron micrographs, why not read a stress distribution from a finite element plot? The problem, the Carnegie Mellon team found, was that nobody had rigorously tested this assumption, largely because large-scale evaluation frameworks did not exist and expert annotation of simulation outputs was prohibitively expensive.

OpenSeeSimE addresses both obstacles through parametric generation. By systematically varying simulation parameters across 10,000 simulations, the researchers could produce an enormous volume of question-answer pairs without relying on scarce expert annotators to label each example individually. The parametric approach also ensures diversity: the benchmark covers a wide range of simulation configurations and question types, so a model cannot succeed by memorizing a handful of visual patterns. The team built the simulation pipeline using PyAnsys, and the authors acknowledge support and guidance from Raunak Borker, Sanjay Ranganayakulu, Chris Hawkins, and the broader Ansys team in utilizing and understanding the software.

The evaluation itself covered ten state-of-the-art vision-language models, and the findings were striking in their consistency. Models that demonstrate strong performance on general visual reasoning benchmarks, the very tasks that have fueled excitement about multimodal AI, scored between 29 and 47 percent on the engineering simulation questions. For context, random chance on multiple-choice questions typically falls in that range, meaning the models were effectively guessing. The effect sizes were negligible, indicating that the differences between models were not meaningful and that none of them had developed even a rudimentary grasp of simulation interpretation.

This disconnect between general visual competence and domain-specific failure carries a broader lesson about the current generation of AI systems. The pattern recognition abilities that allow a VLM to caption a photograph or answer questions about natural images do not automatically transfer to the stylized, information-dense visual language of engineering analysis. Contour plots encode quantitative information through color scales that must be read precisely; meshed geometries convey structural information through conventions that differ sharply from natural scenes. A model trained predominantly on internet images and text has little exposure to these conventions, and the benchmark results suggest that exposure to general visual data provides essentially no foundation for simulation literacy.

The practical implication, according to the authors, is that deploying vision-language models for simulation interpretation will require domain-specific training rather than reliance on general-purpose models. Off-the-shelf multimodal assistants, however impressive their demo performances, cannot be trusted to read a simulation output today. But the benchmark also provides the infrastructure needed to change that. Because OpenSeeSimE is large enough to measure incremental progress with statistical confidence, researchers developing domain-adapted models can use it to track whether fine-tuning and specialized training data are actually working, rather than relying on small evaluation sets that produce noisy, unreliable signals.

The study also establishes critical baselines for the field. Knowing that current models perform at chance levels gives future researchers a clear starting point against which to measure improvement. The benchmark’s reusable framework, with its 850-fold scale increase over prior evaluation efforts, is designed to serve as a lasting resource for the community. As AI systems are increasingly proposed for safety-critical engineering tasks, from validating structural designs to checking thermal performance, rigorous evaluation of their actual capabilities becomes not just scientifically interesting but essential for responsible deployment.

Published open access in Communications Engineering on 23 September 2026, the paper arrives at a moment when the gap between AI hype and AI capability in specialized domains is under intense scrutiny. The Carnegie Mellon work adds engineering simulation to the growing list of technical areas, alongside medicine, law, and scientific research, where general-purpose models stumble without targeted training. For engineers hoping that AI might soon ease the burden of interpreting complex simulation outputs, the message is clear: the tools are not ready yet, but for the first time, there is a rigorous, large-scale yardstick to measure how close they are getting.

Subject of Research: Benchmark evaluation of vision-language models for engineering simulation question answering

Article Title: A large-scale benchmark to assess vision-language model question answering capabilities in engineering simulations

Article References: Ezemba, J., Pohl, J., Tucker, C., & McComb, C. (2026). A large-scale benchmark to assess vision-language model question answering capabilities in engineering simulations. Communications Engineering. https://doi.org/10.1038/s44172-026-00786-2

Image Credits: AI Generated

DOI: 10.1038/s44172-026-00786-2

Keywords: vision-language models, engineering simulations, benchmark, OpenSeeSimE, question answering, machine learning, mechanical engineering, Carnegie Mellon University, visual reasoning, finite element analysis, AI evaluation, Communications Engineering