Large Language Models Struggle to Convert Infrastructure Condition Data into Knowledge Graphs

A comparative evaluation of leading commercial and open-source artificial intelligence systems has revealed that large language models are not yet dependable enough to autonomously convert transportation infrastructure condition data into standardized semantic knowledge graphs. In an investigation conducted by engineering researchers Lea Höltgen, Sven Zentgraf, Philipp Hagedorn, and Markus König, published in AI in Civil Engineering, four advanced models—OpenAI’s gpt-4o and gpt-4-turbo, alongside Meta’s open-source llama-3-70b-instruct and llama-3.1-405b-instruct—were benchmarked on their capacity to translate relational database records of bridge inspections into formal Resource Description Framework statements using the W3C RDB to RDF Mapping Language standard. The experimental findings demonstrated that while models can reliably craft basic structured queries and adhere to template structures, their persistent misapprehension of complex mapping mechanics, relational joins, and domain ontologies produces erratic errors that preclude unassisted real-world deployment in the architecture, engineering, and construction sector.

Relational databases have long served as the digital cornerstone of structured information management across infrastructure agencies, ensuring tabular data integrity, record consistency, and storage efficiency through strict relational schemas and constraints. Within the construction industry, historical records are often segregated across isolated stakeholder silos, proprietary software tools, or obsolete legacy formats. Transforming these disparate records into interconnected knowledge graphs structured via the Resource Description Framework provides a flexible environment where data entities and their interrelationships are explicitly defined as subject-predicate-object triples. This formal semantic enrichment facilitates automated cross-domain reasoning, location-based analyses, and advanced querying via SPARQL that surpasses standard SQL capabilities, yet established manual transformation protocols like R2RML remain notoriously labor-intensive, demanding rare dual expertise in database administration and formal ontologies.

To rigorously determine whether modern natural language processors can automate this translation hurdle, the research team developed a four-stage sequential pipeline designed to isolate and assess intermediate transformation stages. The experimental workflow began with SQL query generation, where models converted natural language prompts into executable database queries against a provided schema. The second phase, SQL-to-ontology mapping, directed the language models to align extracted database tables and attributes with corresponding conceptual classes and properties defined in standardized construction vocabularies, outputting structured JSON schemas. In the third phase, the models generated comprehensive R2RML mapping documents in Turtle syntax based on the established ontology alignments. In the terminal phase, specialized external mapping engines, namely the Morph-KGC processor and the RDFLib framework, executed the generated mapping rules against the relational database to synthesize final RDF triples. This staged architecture deliberately allowed the research framework to validate intermediate outputs, execute programmatic self-correction routines upon detecting syntax failures, and supply continuation prompts whenever model outputs approached token generation ceilings.

The benchmark environment was constructed using Python and configured around the Bridge Inspection Support System relational database, an empirical repository detailing structural degradation, component hierarchies, inspection dates, and repair plans across bridge networks, augmented with structural building entries. To anchor the semantic targets, the researchers selected the Building Topology Ontology, which standardizes core spatial structures, functional building zones, and physical construction elements, alongside the Damage Topology Ontology, which provides schemas for classifying structural deteriorations, affected zones, and adjacent degradation networks. The testing apparatus queried each model at a deterministic temperature setting of zero across 20 distinct transformation runs per configuration, systematically varying prompting approaches across four paradigms: zero-shot brief, zero-shot detailed, few-shot brief, and few-shot detailed.

The experimental trials were arranged across three escalating tiers of computational and relational complexity. The first experiment served as a baseline to examine elementary R2RML syntax generation from a single database table containing no join operations and two basic filtering conditions, requiring models to retrieve damage identifiers, names, and orientations for cracks exhibiting a stability score below three. The second experiment introduced computational complexity by requiring the models to handle SQL aggregation, prompting them to retrieve inspection records for a targeted bridge structure and compute the associated count of recorded damages using nested subqueries. The third and most rigorous experiment evaluated complex multi-table relational joins, incorporating four separate database tables and one condition. This scenario demanded that models construct multiple triples maps linking Buildings, Components, DamageData, and a bridge component join table, requiring precise explicit join conditions to map one-to-many and many-to-many associations into coherent graph representations.

Across the baseline trial of Experiment 1, all four language models demonstrated stable syntactic competence and adherence to general R2RML conventions across most configurations. OpenAI’s gpt-4o and Meta’s llama-3.1-405b-instruct achieved perfect 100 percent syntax correctness across all prompt varieties. Nevertheless, qualitative discrepancies surfaced. When tested under the zero-shot brief technique, llama-3-70b-instruct completely failed to produce valid mappings because it attempted to map table columns that were never retrieved by the underlying SQL source query, reflecting a fundamental disconnection regarding how downstream R2RML conversion engines interpret logical sources. Furthermore, both GPT models displayed a recurring tendency to insert extraneous, factually flawed elements into their generated triples maps, such as fabricating constant property values for damage stability.

In the aggregation challenges of Experiment 2, the syntactic validity of generated mappings improved markedly when models received few-shot examples, culminating in flawless syntactic compliance across all four systems under the few-shot detailed configuration. Conversely, in the zero-shot detailed setting, neither gpt-4-turbo nor llama-3-70b-instruct succeeded in generating a single valid mapping document. Semantic analysis uncovered widespread systematic deficiencies across all configurations. Most models habitually generated redundant triples maps for damage entities rather than handling the calculated damage count as a consolidated attribute within the parent building triples map. This structural inflation rendered the underlying SQL queries insufficiently restrictive, causing the resulting mappings to inadvertently ingest and convert database rows outside the user’s intent. Both gpt-4-turbo and llama-3-70b-instruct repeatedly misconfigured join conditions by supplying incorrect primary or foreign key identifiers, or authored SQL logic that calculated a collective sum across all buildings rather than aggregating damages per unique structure. Among the evaluated pool, gpt-4o produced the highest semantic accuracy, generating mappings that closely mirrored the target benchmark.

The multi-table join demands of Experiment 3 exposed pronounced operational thresholds between generations of language model architectures. Only gpt-4o and llama-3.1-405b-instruct proved capable of maintaining valid R2RML output across every prompting format, whereas gpt-4-turbo produced valid mappings under only two prompting styles, and llama-3-70b-instruct succeeded in only a single configuration, the few-shot brief technique. Error classifications demonstrated a heavy clustering of mapping rule violations, predominantly driven by models attempting to map target columns that were entirely omitted from the SQL queries powering the logical table definitions. In several instances, models renamed columns using SQL alias statements but failed to propagate those modified identifiers into the logical source definition, causing the Morph-KGC processor to terminate execution. Llama-3-70b-instruct routinely generated logical queries that referenced attributes belonging to unrelated tables, while gpt-4o, though successful overall, suffered from data over-inclusion issues during zero-shot runs. All models frequently defaulted to implicit joins using uniform resource identifier templates rather than declaring explicit join condition statements.

Beyond mapping syntax, the investigation identified persistent semantic vulnerabilities within the intermediate SQL-to-ontology matching phase. Rather than declaring an unmapped field or instantiating a designated fallback namespace, the models fabricated novel class and property names and erroneously registered them directly under the authentic namespace identifiers of the Building Topology Ontology or Damage Topology Ontology. In multiple runs, models inappropriately bound conceptually incompatible properties to database attributes, such as assigning a topological adjacency property to describe spatial orientation data. Incorporating an automated self-correction mechanism provided tangible stability gains, allowing models to rectify minor syntax errors upon receiving feedback messages from the validation module, with second-attempt success rates occasionally surpassing the volume of initial valid outputs.

Pipeline timing telemetry gathered across the 20 iterative runs illustrated that overall system execution latency expanded significantly with task complexity and the necessity of self-correction loops. The mean duration of internal non-LLM program routines remained consistent at approximately 15 seconds, and R2RML processor conversion runs consistently executed in under one second. The core language model operational phases—SQL query generation, ontology mapping, and R2RML mapping generation—accounted for nearly all variance, pushing cumulative pipeline execution times from approximately 30 seconds during baseline single-table queries up to over 160 seconds during multi-table join iterations. Across all benchmarks, gpt-4o recorded the fastest operational execution. The researchers cautioned that while runtime measurements reflect pipeline overhead, definitive real-time capability, token expenditure metrics, operational computational expenses, and associated environmental impacts require broader investigation under standardized network conditions.

Given that minuscule syntactic oversights or faulty foreign key references cause total execution failure in downstream RDF processors, the study authors concluded that fully automated language model pipelines cannot currently be recommended as standalone production systems for infrastructure condition data management. Unmonitored deployment risks creating compromised, erroneous graph databases that could distort automated structural assessments or mislead subsequent predictive maintenance algorithms. However, the study emphasized that language models offer substantial value when embedded in hybrid, human-in-the-loop workflows. In such assistive configurations, models rapidly assemble preliminary query drafts, draft candidate ontology alignments, and sketch R2RML mapping blueprints, sparing human specialists the burden of authoring complex Turtle syntax from scratch while retaining domain experts to supervise schema mappings and intercept hallucinations. Future research directions will explore integrating deterministic semantic reasoners to automatically flag ungrounded ontological terms, testing fine-tuned domain-specific architectures, expanding mapping targets to generic multi-format standards like RML, and connecting transformation pipelines directly to open semantic vocabularies.

Subject of Research: Evaluation of large language models for converting bridge condition databases into semantic RDF knowledge graphs

Article Title: Utilizing large language models for semantic enrichment of infrastructure condition data: a comparative study of GPT and Llama models

Article References: Höltgen, L., Zentgraf, S., Hagedorn, P., & König, M. (2025). Utilizing large language models for semantic enrichment of infrastructure condition data: a comparative study of GPT and Llama models. AI in Civil Engineering, 4(1), Article 14. https://doi.org/10.1007/s43503-025-00055-9

Image Credits: AI Generated

DOI: 10.1007/s43503-025-00055-9

Keywords: Large Language Models, Knowledge Graphs, Civil Engineering, Infrastructure Maintenance, R2RML, Semantic Web, Relational Databases, Data Integration, Utilizing, large, language, models