LLM-Powered Search Engine Unifies China’s Fragmented Public Data Portals

Researchers have built a search system that brings together hundreds of thousands of public datasets scattered across China’s decentralized open data portals, using large language models to clean up messy metadata, rerank results and explain to users why a dataset matches their query. The system, called CN-PDS (China Public Data Search), is described in a paper published in the journal Data Science and Engineering on 29 August 2026, and its code and prompts have been open-sourced on GitHub.

The problem the team set out to solve is structural. Unlike the United States, where data.gov offers a centralized national portal, China’s public datasets are distributed across provincial and city-level portals with no unified national infrastructure. Each portal uses its own metadata schema, and because metadata is largely constructed manually it often suffers from incompleteness, inconsistency and poor structure. That heterogeneity prevents any comprehensive search across portals, leaving valuable open data effectively invisible to anyone who does not know exactly which local website to visit.

The scale of the underlying ecosystem is substantial. By July 2024, 243 provincial and city-level open data platforms had been established in China, a fourteen-fold increase since 2017, covering 24 of 27 province-level regions. More than 370,000 datasets were available at that point, 44 times more than in 2017, spanning domains from urban governance to finance. Some provinces, such as Zhejiang and Shandong, have integrated high-value datasets like healthcare records and enterprise registries and deployed real-time interfaces for weather forecasts and parking availability. Yet without a central search layer, this growing repository remains fragmented.

CN-PDS attacks the fragmentation in two stages. First, it consolidates metadata: an iterative algorithm generates a standard metadata schema from the heterogeneous source schemas, then maps each original schema onto that standard, and finally aligns the field values of every metadata record. Large language models perform the schema understanding and semantic alignment at each step, with a small number of sample records fed into the model to help it grasp what each field means. After the standard schema is generated, manual intervention defines value ranges and format requirements, for instance standardizing date fields to a uniform time format.

Because language models can hallucinate, the consolidation step is wrapped in a multi-agent quality control framework with three distinct agents: one generates predictions, a second assesses their accuracy and consistency, and a third proposes corrections that are fed back for regeneration. The cycle repeats until the output meets quality standards or a predefined iteration limit is reached. In tests of standard schema generation using the GLM-4 model across ten runs with different seed schemas, direct generation produced an average of 2.60 field errors per round, while the multi-agent setup reduced that to 0.63. The errors counted included duplication, omission and redundancy. The revision agent also caught failure cases where the model collapsed an entire schema into a single trivial field such as Keywords.

The second stage handles the fact that even unified metadata is often too sparse to match complex, task-oriented queries. CN-PDS therefore adopts content-based search built on retrieval-augmented generation, or RAG. Metadata and data files are segmented into chunks, each chunk is embedded into a vector by a pre-trained text embedding model, and the vectors are stored in a vector index. When a task requires context, the system retrieves the top-ranked chunks and feeds them to the language model. The same mechanism powers three functions: enriching sparse metadata with descriptions drawn from actual file content, reranking retrieved datasets by semantic relevance rather than metadata keywords alone, and generating natural language explanations grounded in the retrieved chunks.

The deployed system is organized in three layers. A data layer runs portal-specific crawlers that periodically collect dataset-level metadata and lightweight descriptive content from local portals, downloading raw files only when portals explicitly label them as unconditionally accessible. An application layer performs a two-stage retrieval: a computationally cheap BM25 filter generates candidates from indexed metadata, and an LLM-based reranker refines the ranking of only the small candidate set, limiting computational overhead and API usage. An interaction layer offers keyword and faceted search, letting users filter by province, city, data domain and access conditions, alongside a relevance-explanation feature and links back to the original portals rather than locally hosted copies.

At the time of writing, the team had connected to 148 local public data portals covering 25 provincial administrative regions and aggregated 619,718 public datasets, a number still rising. Between October 2023 and June 2024 the collection grew by more than 125,000 datasets. Shandong province leads with 199,412 datasets, 32.18 percent of the total, followed by Sichuan with 120,217 datasets, or 19.40 percent, so two provinces account for more than half the collection. Among cities, Ningbo in Zhejiang ranks first with 34,128 datasets, ahead of Beijing with 17,324 and Taizhou in Jiangsu with 12,892. Regional statistics reveal an imbalance: East China holds 328,281 datasets and the Northeast, Northwest and Central regions, despite having comparable numbers of portals, hold only 4,447, 2,890 and 5,184 respectively, suggesting infrastructure exists where published data depth is still thin.

The researchers evaluated their methods on a new test collection, CNPD, built from 489,882 datasets collected from 138 Chinese open data platforms as of November 28, 2023, with 100 LLM-generated queries and 2,000 manually annotated query-dataset pairs, and on the English subset of the NTCIR dataset search collection, which contains 46,615 datasets from data.gov. Three models were compared: the proprietary GLM-4 and QWEN-max accessed via API, and the open-source LLaMA3-8B run locally on a single NVIDIA A100 GPU. In reranking experiments, all three LLM-based methods substantially improved retrieval performance over pre-reranking baselines on CNPD and outperformed a dense ranking baseline, BGE-M3, when reranking original metadata. On NTCIR, GLM-4 and QWEN-max showed significant improvements, but LLaMA3-8B performed poorly, with all metrics falling below the pre-reranking baseline, indicating that reranking poses greater challenges for smaller models.

Metadata enrichment showed a similar pattern. On CNPD, enrichment with GLM-4 and QWEN-max improved all measured metrics, with GLM-4 using RAG-based enrichment achieving the highest NDCG and MAP scores, while LLaMA3-8B-driven enrichment actually degraded performance. On NTCIR, where metadata quality was already higher, only the GLM-4 RAG enrichment improved on original metadata. For relevance explanations, QWEN-max with RAG outperformed other methods across all evaluation metrics on both collections, and RAG-based generation beat direct generation for all three models. The team also reported practical deployment figures: total disk usage under 6.6 gigabytes, an average retrieval latency of 1.57 seconds over 200 sampled queries, and an average of 1.24 seconds to generate a relevance explanation. The trade-off is latency at scale, since reranking with enriched metadata through the GLM-4 API often exceeds four seconds, which is why the authors are exploring distilling large-model capabilities into smaller locally deployable models as future work.

Subject of Research: An LLM-based framework for unified search over China's decentralized public data portals

Article Title: CN-PDS: Facilitating Dataset Search with LLMs and its Application to China’s Public Data

Article References: Chen, Q., Chen, J., Xu, C., Feng, Y., Shan, B., Li, H., Zhou, X., Shi, Q., & Cheng, G. (2026). CN-PDS: Facilitating Dataset Search with LLMs and its Application to China’s Public Data. Data Science and Engineering. https://doi.org/10.1007/s41019-026-00355-9

Image Credits: AI Generated

DOI: 10.1007/s41019-026-00355-9

Keywords: dataset search, large language models, open data, retrieval-augmented generation, metadata consolidation, multi-agent systems, China public data, information retrieval, reranking, relevance explanation, CN-PDS, data governance