Scientific research environments routinely generate massive collections of heterogeneous records, ranging from inpatient electronic medical files, radiological scans, and machine sensor readings to administrative university transcripts and historical economic indicators. To address the persistent operational barrier of building custom data analysis infrastructure for every unique dataset, researchers Leon Liman, Georg Fette, and Frank Puppe have developed an adaptable research data warehouse platform named PaDaWaN. Designed to eliminate the steep technological learning curves typically associated with complex multi-source evaluations, the framework unifies flexible entity-attribute-value storage, ontology-based hierarchical navigation, dual specialized indexing engines, and graphical query construction. By validating the system across five distinct operating environments encompassing medical centers, higher education administration, industrial manufacturing, and historical macroeconomics, the authors demonstrate that a unified, domain-agnostic architecture can fulfill complex user reporting demands across varied structural paradigms without requiring users to write custom programming code or manage underlying relational database schemes.
The structural demands placed on a research data warehouse differ fundamentally from those encountered in conventional business intelligence and enterprise data warehousing. While conventional business systems rely on predetermined dimensional designs such as Online Analytical Processing cubes that aggregate structured transactional data along fixed axes like customer locations and purchase dates, research platforms must ingest highly varied data from disparate systems without enforcing rigid schemas. Research contexts require systems capable of managing unstructured textual reports from which structured values have not yet been extracted, linking multimodal records like diagnostic imaging studies, navigating deep conceptual ontologies, and defining virtual attributes dynamically during runtime analysis. In contrast to enterprise setups where data models are fixed and reporting routines are largely predetermined, research exploration requires investigating intricate logical interactions across multiple granularities of observation while shielding investigators from the technical intricacies of storage layers and lower-level query languages.
At the structural core of PaDaWaN lies an entity-attribute-value relational database design paired with an ontology navigation layer and supported by two synchronized external indexing technologies, Apache Solr and Neo4j. The central schema isolates concepts into two primary structures: a Catalogue table defining supported attributes along with their data types and hierarchical ontological classifications, and a Facts table recording measured observations, timestamps, and mapping linkages. To accommodate nested relational hierarchies, the Facts structure supports up to three hierarchical entity tiers identified through numeric keys. In a medical data warehouse, these three levels typically correspond to unique individual patients, distinct inpatient or outpatient hospital encounters, and discrete diagnostic documents or examinations generated within a given visit.
PaDaWaN implements a dual indexing methodology that automatically transforms imported database facts into optimized retrieval representations, maintaining synchronization to avoid query overhead. Apache Solr indexes data as documents representing entity tiers and recorded facts, powering high-speed textual retrieval, regular expression parsing, wildcard searches, and proximity filtering where users specify maximum word distance and sequence constraints. Concurrently, the graph database Neo4j organizes entity levels and factual connections as interconnected nodes, allowing investigators to harness its Cypher graph query language for on-the-fly computational operations. Although PaDaWaN supports running both indices in parallel, individual implementations select the indexing technology aligned with their functional requirements, with Solr driving text-intensive and large-scale filtering tasks while Neo4j underpins complex relationship evaluations and mathematical feature generation.
Data entry into PaDaWaN is designed around accessible tabular formats, primarily standard Excel and comma-separated value tables, supplemented by dedicated metadata tables that define attribute labels, hierarchical paths, and human-readable descriptions. For advanced workflows, data can be ingested from XML, JSON, or domain standards like HL7 FHIR and OMOP CDM provided the records can be flattened into tabular arrangements. Beyond standard graphical imports and command-line loaders that automatically log data discrepancies and filter duplicate facts, PaDaWaN incorporates two programmatic interfaces for pre-calculating features during data ingestion. The first interface operates on individual table rows, facilitating automated code translation or pre-computing metrics such as body mass index from paired height and weight values.
The system provides an intuitive graphical user interface that translates drag-and-drop actions into its native underlying query language, MXQL. The visual interface is partitioned into three functional workspaces: an expandable attribute catalogue on the left displaying hierarchical ontology trees and total entity counts; a query assembly zone at the upper right; and an execution results pane across the bottom. Users formulate two distinct classes of queries without manual script authoring: individual data queries that output granular entity rows displaying specific attribute values, and statistical queries that generate multi-dimensional matrix counts cross-tabulating selected row and column criteria. Attributes dropped adjacently into query rows are automatically linked through logical operators that toggle between conjunctions and disjunctions, while contextual pop-up menus allow researchers to apply numerical thresholds, specify negation filters to find entities lacking certain records, or inspect full highlighted textual records.
PaDaWaN expands exploratory capacity through advanced analytical features directly embedded within the graphical builder, most notably ad-hoc information extraction, dynamic attribute calculations, correlation evaluations, and deep learning plugin connectivity. Extending Lucene’s regular expression tools within Solr, the extraction mechanism isolates specific numeric values embedded in free text—such as left ventricular ejection fraction percentages from echocardiogram narratives or serum creatinine concentrations from clinical notes—and treats these extracted values as standard queryable attributes. Through Neo4j integration, users can click an interface button to define calculated attributes on the fly using mathematical formulas, such as adjusting historical wages for inflation or evaluating dimensional ratios between manufacturing components. Furthermore, the statistical builder can execute automated correlation analyses by querying intersecting entity values across configured rows and columns to compute Spearman rank correlation coefficients, while dedicated plugin interfaces in Java, Kotlin, and Python permit streaming query tables directly into analytical pipelines such as scikit-learn.
The practical utility and operational scalability of PaDaWaN have been substantiated across five distinct production use cases. The largest operational deployment manages clinical data from a university medical center, housing more than 100,000 distinct attributes and nearly one billion factual observations accumulated through late 2024. This medical deployment undergoes nightly incremental updates that complete within one to three hours, enabling medical researchers to screen patient cohorts using intricate inclusion and exclusion criteria, evaluate discrepancies between standardized ICD-10 diagnostic codes and physician documentation, populate dedicated clinical research registries, and extract diagnostic radiology images linked directly to clinical text reports via Picture Archiving and Communication System integration. A second medical deployment compares performance metrics across radiology departments in four separate hospitals, evaluating examination volumes, patient waiting times, and multi-modality follow-up imaging sequences across X-ray, computed tomography, and magnetic resonance imaging.
Beyond healthcare settings, PaDaWaN has demonstrated flexibility in higher education, heavy manufacturing, and macroeconomic history. In an academic administration use case, the software processes student records to generate 1,440 annual performance reports across faculties and degree programs, tracking enrollment milestones, study duration, dropout rates, and course grade distributions while maintaining stability even when underlying campus database systems were replaced. In industrial manufacturing, PaDaWaN analyzes telemetry from metal casting straightening plants where hydraulic cylinders apply localized bending forces to correct thermal cooling distortions. Investigators deployed correlation analyses and ad-hoc calculated attributes across 102 requested reports involving two straightening facilities and five distinct component geometries to train predictive machine learning models. In a fifth deployment analyzing historical macroeconomic tables digitised from the German Institute for Economic Research, the system models longitudinal trends across multiple countries, linking timelines of inflation, unemployment figures, and democratic election outcomes.
Comprehensive performance benchmarks indicate that PaDaWaN achieves practical responsiveness while operating on standard computing hardware, requiring primarily sufficient random-access memory and solid-state storage to maintain index performance. Standard logical queries across small and medium cohorts execute in under a few seconds, with specialized regular expression evaluations across thousands of textual notes completing rapidly through native Solr indexing. Similarly, ad-hoc attribute derivations inside Neo4j and Python-driven machine learning plugins for clustering or regression execute in under ten seconds, including data retrieval. When evaluating operational user feedback across all deployments, project teams successfully fulfilled all requested reporting specifications whenever participating institutions provided the necessary underlying data, encountering no internal processing bottlenecks or performance complaints apart from external image transfer delays caused by production hospital imaging server bandwidth throttling.
In comparative evaluations against free data tools and enterprise business intelligence platforms—including i2b2, DHIS2, Hue, Apache Superset, KNIME, Pentaho, DBeaver, Metabase, Power BI, Tableau, and spreadsheet software—PaDaWaN demonstrates distinct advantages in multi-entity querying, combined textual-image retrieval, and zero-code ad-hoc attribute generation. However, the authors delineate specific technical constraints: PaDaWaN lacks native in-application charting, relying instead on exporting tabular outputs to external spreadsheet tools for graph visualization. Furthermore, the platform mandates dedicated data extraction and indexing pipelines rather than querying live source production databases directly, requires disk and memory allocations for external indexing engines, and lacks automated built-in de-identification mechanisms, requiring sensitive personal information to be pseudonymized before ingestion. Future development plans aim to replace the relational database backend entirely with Neo4j to streamline data persistence, refine deep learning interfaces for bidirectional ground-truth training, and expand deployments into archaeological archive classification and remote workplace health analytics.
Subject of Research: Research data warehousing architectures and multi-domain data integration tools
Article Title: Padawan: a tool for creating research data warehouses for heterogeneous use cases and requirements
Article References: Liman, L., Fette, G., & Puppe, F. (2025). Padawan: a tool for creating research data warehouses for heterogeneous use cases and requirements. Discover Data, 3(1), Article 40. https://doi.org/10.1007/s44248-025-00082-y
Image Credits: AI Generated
DOI: 10.1007/s44248-025-00082-y
Keywords: research data warehouse, PaDaWaN, entity-attribute-value model, Apache Solr, Neo4j, clinical data integration, information extraction, ad hoc attribute calculation, PACS integration, Spearman rank correlation, academic reporting, manufacturing analytics, historical economic data, MXQL query language, unstructured text search, multilevel entity modeling
News Source: Denise Maddox. (November 19, 2026). PaDaWaN System Streamlines Heterogeneous Research Data Warehouses Across Diverse Disciplines. Scienmag.



