Instruction Tuning and Simple Formats Improve Generative Models in Clinical Entity Extraction

A comparative evaluation of open-access generative large language models in zero-shot clinical named entity recognition demonstrates that instruction-tuned architectures significantly outperform standard chat-tuned and base models, particularly when directed to produce simple output structures rather than complex schemas. Conducted as part of the 8th Biomedical Linked Annotation Hackathon, the study assessed how different architectural configurations and fine-tuning specializations influence an artificial intelligence system’s capacity to detect medical terms without requiring prior task-specific training data. Across standardized evaluations, systems optimized specifically to follow prompt instructions retrieved clinical concepts with noticeably higher accuracy than counterparts tuned merely for conversational dialogue or raw next-token prediction.

Extracting relevant information from electronic health records manually remains a heavily resource-intensive and time-consuming undertaking for healthcare institutions and biomedical researchers. While clinical named entity recognition methodologies have historically achieved strong results, most high-performing traditional pipelines depend on predefined lexical data, supervised machine learning, or rule-based parsing. The grammatical inconsistencies inherent to narrative clinical prose, coupled with an evolving medical lexicon and a persistent scarcity of richly annotated training corpora, have historically made supervised models difficult and expensive to construct and maintain. Consequently, recent expansions in generative model availability have motivated investigators to explore zero-shot extraction frameworks that operate without dedicated labeled corpora.

To investigate the viability of open-access generative architectures, the research team focused on two widely adopted foundational model families, Llama 2 and Mistral, alongside specialized derivatives fine-tuned for code generation, conversational interaction, and instruction compliance. The evaluation specifically contrasted six distinct open-access configurations: the foundational llama-2-7b base architecture, the human chat-aligned llama-2-7b-chat model trained on approximately 27,000 conversational prompt-answer pairs, the code-focused codellama-7b-instruct system trained on code-heavy corpora, the baseline mistral-7b-v0.1 model, the instruction-following mistral-7b-instruct-v0.2 variant, and the larger mixtral-8x7b-instruct-v0.1 network. This diverse model selection allowed the authors to systematically evaluate how structural pre-training, parameter scaling, and adherence tuning influence clinical term identification.

Among the examined systems, mixtral-8x7b-instruct-v0.1 incorporates a Sparse Mixture of Experts architecture designed to augment generation capacity while controlling computational overhead. In this structural paradigm, the feed-forward layers within the network are scaled into multiple specialized subnetworks known as experts, which are added directly to the model layers. During token generation, an internal gating mechanism selectively determines which subset of experts processes each given output. The authors incorporated this framework to analyze whether routing clinical entity recognition queries through dynamically activated expert modules provides measurable performance advantages over monolithic parameter arrangements of comparable computational footprint.

The experimental evaluation utilized a unified silver-standard corpus compiled from three existing overlapping datasets: DisTEMIST, SympTEMIST, and MedProcNER. These corpora originally featured clinical case histories written and annotated in Spanish, which were subsequently translated into English and other languages using machine translation to foster multilingual biomedical text mining. Because each original dataset captured only a single entity class, the researchers merged the overlapping documents to establish an integrated target space covering diseases, symptoms, and medical procedures simultaneously. While an initial attempt was made to incorporate the LivingNER corpus containing living organism annotations, translation discrepancies restricted the shared overlap to merely 90 documents, prompting the authors to exclude LivingNER annotations to maintain rigorous annotation consistency.

The resulting unified corpus comprised 1,000 clinical cases exhibiting an average document length of 333 words, totaling 21,867 annotated mentions mapped according to the standard inside-outside-beginning tagging scheme. To prevent classification ambiguities across conflicting category spans, the researchers eliminated inter-category overlapping spans by exclusively retaining the longest entity mention within any overlapping cluster. The researchers allocated the 90 overlapping cases as an isolated test subset dedicated to benchmarking the zero-shot generative language models, while reserving the remaining 910 documents to train and fine-tune traditional encoder-based transformer networks for comparative performance baselines.

To rigorously test the influence of input phrasing on model behavior, the investigators engineered three prompt variations presenting escalating levels of structural specification. The first prompt described the extraction objective and defined the entity categories in natural language without prescribing any formal output formatting, thereby probing raw task comprehension in free-form generation. The second prompt delivered the category guidelines in a code-like list accompanied by an explicit demonstration formatted as a standard Python dictionary, testing adherence to simple key-value structures. The third prompt instructed models to format responses as complex, fully valid JSON structures, complete with a comprehensive syntax demonstration embedded directly into the prompt instructions.

Model outputs were evaluated through two distinct validation strategies executed on cloud-hosted NVIDIA A40 hardware equipped with 48 gigabytes of memory. The first evaluation strategy assessed pure extraction capacity irrespective of structural compliance: each target term from the silver-standard annotations was cross-referenced against the raw model generation, and any full or partial match was projected back onto the original document words as inside-outside-beginning entity tags. Precision, recall, and F1 metrics were then computed across all clinical tokens. The second strategy utilized regular expressions to strictly enforce syntax validation on prompts two and three, verifying whether the outputs formed valid dictionaries or JSON schemas before determining if extracted items corresponded to valid document text.

Across initial unconstrained evaluations, precision remained consistently high across all tested generative models, indicating that when these systems transcribed candidate terms, the entities were overwhelmingly present within the source clinical cases. However, recall values remained generally low across the board, reflecting the fundamental limitation of zero-shot prompting where models lack task-specific optimization and inevitably overlook entities not explicitly emphasized by the prompt context. Despite low baseline recall, models possessing instruction-following fine-tuning proved distinctly more proficient at retrieving valid terms than conversational or foundation baselines, while systems exposed to code corpora demonstrated slight advantages in boundary consistency.

When assessing structured outputs under the second evaluation strategy, precision was markedly superior under the second prompt variation requiring Python dictionaries compared to the third variation enforcing strict JSON schemas. Increasing instruction complexity and demanding intricate hierarchical serialization frequently caused models to lose focus on the primary extraction task, precipitating a drop in identified entities. Furthermore, categorical performance varied substantially across diagnostic domains: models recognized diseases and symptoms with relative consistency, whereas extracting medical procedures exhibited lower accuracy across all systems, an outcome attributed to the broader semantic boundaries and diverse operational vocabulary defining clinical procedures.

Among instruction-tuned architectures, mixtral-8x7b-instruct-v0.1 exhibited the strongest zero-shot performance, outperforming the specialized codellama-7b-instruct baseline. The researchers noted that while codellama possesses deep training on code syntax, its smaller monolithic architecture specializes narrowly, whereas the mixture-of-experts routing in Mixtral provides broader semantic versatility. Furthermore, qualitative inspection of failed codellama runs under complex JSON prompts revealed that the model frequently abandoned output constraints entirely, instead generating unsolicited medical diagnoses and clinical explanations. This behavioral drift occurred because the system lacked fine-tuning safeguards against generating diagnostic prose, unlike conversational models whose safety training occasionally led them to refuse prompts containing personal health details or injury descriptions.

To contextualize generative capabilities against traditional discriminative approaches, the authors benchmarked seven fine-tuned encoder models trained across three and five epochs on the 910-document training split using dual NVIDIA RTX A5000 graphic processors. General-domain architectures including BERT, RoBERTa, and XLM-RoBERTa were evaluated alongside domain-adapted biomedical variants including Clinical-BERT, BIOMED-RoBERTa, Clinical-XLM-RoBERTa, and BIO-BERT. Domain-adapted encoders consistently surpassed general-domain baselines, with BIO-BERT securing the highest overall micro-average F1 score after five epochs. Notably, this optimized BIO-BERT benchmark achieved performance comparable to zero-shot mixtral-8x7b-instruct-v0.1 under the unconstrained initial prompt, illustrating that general-purpose generative models can approach specialized supervised baselines in zero-shot contexts.

Despite these promising capabilities, the authors emphasize that generative language models exhibit fundamental operational limitations, most notably non-deterministic generation where identical inputs yield variable responses over repeated executions. Traditional discriminative encoder architectures continue to deliver greater reliability, syntactic consistency, and predictable entity boundaries when sufficient annotated training data is accessible. Consequently, the researchers conclude that zero-shot generative models using concise, simple output formats are best positioned not to fully displace supervised pipelines, but to serve as effective complementary tools for clinical information extraction when extensive labeled corpora cannot be acquired.

Subject of Research: Zero-shot clinical named entity recognition across open-access generative large language models

Article Title: Comparative analysis of generative LLMs for labeling entities in clinical notes

Article References: del Moral-González, R., Gómez-Adorno, H., & Ramos-Flores, O. (2025). Comparative analysis of generative LLMs for labeling entities in clinical notes. Genomics & Informatics, 23(1), Article 3. https://doi.org/10.1186/s44342-024-00036-x

Image Credits: AI Generated

DOI: 10.1186/s44342-024-00036-x

Keywords: Clinical Named Entity Recognition, Generative Language Models, Zero-Shot Evaluation, Instruction Fine-Tuning, Electronic Health Records, Biomedical NLP, Llama 2, Mistral, Mixtral SMoE, Code-Llama, Prompt Formatting, Entity Extraction, Encoder Transformers, BioBERT Comparison, DisTEMIST, SympTEMIST, MedProcNER