Multitask AI Model Enhances Tuberculosis Diagnosis from Ugandan Chest X-Rays

Researchers have developed PaliGemma-CXR, a multi-task multimodal artificial intelligence model designed to interpret chest X-rays for tuberculosis diagnosis, lesion segmentation, object detection, visual question answering, and automated medical report generation. Built by fine-tuning the PaliGemma vision-language foundation architecture on clinical radiographs gathered in Uganda, the model processes visual and textual inputs within a unified framework, allowing multiple diagnostic operations to be executed simultaneously. Across comprehensive evaluations on Ugandan patient radiographs, PaliGemma-CXR demonstrated that joint multi-task learning significantly improved diagnostic accuracy and spatial localization compared to identical architectures trained separately on individual interpretation tasks. By leveraging shared representations across clinically interdependent functions, the unified system achieved 90.32 percent accuracy in identifying tuberculosis infection categories, while outperforming task-specific baselines across all evaluated tasks. This multi-task framework addresses the computational and practical challenges associated with deploying multiple disconnected machine learning models in resource-constrained medical environments.

The development of PaliGemma-CXR responds to severe operational bottlenecks in the global and regional management of tuberculosis, an infectious disease that remains among the primary causes of morbidity and mortality worldwide, accounting for an estimated 10.6 million infections in 2021. In Uganda, where the incidence of tuberculosis is high, chest radiography serves as a rapid, low-cost frontline diagnostic tool. However, the accurate interpretation of chest radiographs and the subsequent drafting of clinical reports require certified radiologists, who remain in acute shortage across the country. Although automated machine learning systems have demonstrated diagnostic accuracy matching or exceeding certified radiologists in several international evaluations, existing systems depend on extensive annotated datasets. The authors observed that despite numerous open-source medical imaging databases, labeled chest radiography data from Uganda is almost non-existent. Consequently, algorithms trained on foreign open-source datasets exhibit diminished diagnostic efficacy when applied to local patient images.

To address the lack of African clinical data and build a localized diagnostic model, the research team conducted a year-long imaging collection across four health facilities in Uganda. The selected sites encompassed both major urban referral centers—specifically Mulago National Referral Hospital and Mengo Hospital—and rural healthcare facilities, including Amai Hospital and Kisiizi Hospital, each of which serves over one hundred tuberculosis patients weekly. Prior to initiating data collection, the study secured ethical clearance from the Mengo Hospital Research and Ethics Committee, encompassing patient consent frameworks and data-sharing agreements, alongside regulatory approval from the Uganda National Council for Science and Technology. Certified radiographers and medical officers reviewed all collected digital chest radiographs. In total, the collection yielded 1,835 radiographs across the four facilities, comprising 902 images from the Ernest Cook Ultrasound Research and Education Institute at Mengo Hospital, 716 images from Mulago Hospital, 319 images from Kisiizi Hospital, and 91 images from Amai Hospital.

Following data acquisition, the investigators established an annotation protocol to categorize image quality, disease presentation, and fine-grained anatomical features. Three radiologists with over ten years of clinical experience annotated the images, with each radiograph evaluated by at least two specialists. Image quality was first screened into good for defect-free images, average for interpretable images with minor defects, and poor for uninterpretable radiographs. Radiologists segmented radiological features indicative of tuberculosis, including consolidation, pleural effusion, abnormal lymph nodes, and miliary tuberculosis typically associated with primary infection, alongside cavities, ring shadows, and pleural thickening indicating post-primary disease, as well as pneumonia and hydropneumothorax. Diagnostic labels covered tuberculosis positive, sub-categorized as active or inactive; normal; and non-tuberculosis infections. Selecting radiographs where both radiologists agreed and where quality was good or average produced a final dataset of 1,149 images, comprising 715 active tuberculosis cases, 69 inactive cases, 332 normal studies, and 33 non-tuberculosis infections.

Because no existing dataset provided unified labels for five diagnostic operations, the team curated specialized multimodal subsets directly from the 1,149 annotated images. For report generation, reports comprising clinical findings and impressions were structured using routine radiology templates, marking no findings for negative cases, with radiologist verification on selected samples. For visual question answering, the researchers used GPT-4o to generate 3,117 question-answer pairs derived from the medical reports across abnormality, presence, location, and counting categories, encompassing 1,902 close-ended and 1,215 open-ended questions verified by GPT-4o, manual review, and radiologist audit. The object detection dataset was constructed across 784 images by extracting bounding box coordinates from segmentation masks, normalizing them by image dimensions, and scaling by 1,000. For segmentation, feature masks on 784 images were converted into 16 discrete tokens using a VQ-VAE model. The disease diagnosis dataset incorporated all 1,149 images paired with classification prompts.

Architecturally, PaliGemma-CXR utilizes the open PaliGemma framework, integrating a SigLIP vision encoder, a Gemma-2B autoregressive language model, and a linear projection layer. Under this framework, input radiographs are converted to three-channel RGB, normalized according to training image statistics, and resized to 224 by 224 pixels before being processed by the SigLIP vision encoder. The linear projection layer maps visual embeddings into the identical dimensional space of Gemma-2B vocabulary tokens. Text prompts, including instructional prefixes and questions, are converted into language tokens via the Gemma SentencePiece tokenizer and embedded alongside projected visual tokens. The combined sequence is processed by the language model to predict target tokens autoregressively. The final training objective minimizes the negative log-likelihood of target response tokens, applying cross-entropy loss exclusively to suffix tokens. By formulating bounding coordinates and segmentation masks as discrete tokens within the text sequence, the model performs localization, segmentation, and clinical text generation in a unified output space.

To train the unified model without causing catastrophic forgetting or task dominance, the researchers implemented a specialized multi-task training procedure. Radiographs were randomly split into training, validation, and testing sets in an 8:1:1 ratio prior to generating task-specific datasets, preventing test leakage across tasks. Preliminary experiments indicated that sequential training on separate datasets triggered catastrophic forgetting, while mini-batch mixing required excessive batch sizes without improving performance. Therefore, within each epoch, data was sampled using the inverse ratio of task dataset sizes, establishing sampling ratios of 6:9:6:1:8 for disease diagnosis, object detection, report generation, visual question answering, and segmentation. The vision and multimodal projection layers were frozen, while language model weights were fine-tuned. Training used the AdamW optimizer with beta parameters of 0.9 and 0.999, a learning rate of 1e-5, weight decay of 0.01, a batch size of 16 with 4 gradient accumulation steps, and uniform task loss weighting across 10 epochs on one NVIDIA A40 GPU.

Experimental evaluations demonstrated substantial performance gains for PaliGemma-CXR across disease diagnosis and object detection compared to task-specific and vision-only models. In disease diagnosis, PaliGemma-CXR reached 90.32 percent accuracy, surpassing both task-specific PaliGemma and unimodal baselines ViT-H and SigLIP trained for 50 epochs. While SigLIP achieved the highest macro-precision and F1 score, it recorded the lowest overall accuracy, whereas PaliGemma-CXR secured the highest accuracy and macro-recall. In object detection evaluated at an intersection-over-union threshold above 0.5, PaliGemma-CXR achieved a mean average precision of 19.4. This localized detection performance more than doubled the mean average precision of task-specific PaliGemma and surpassed DETA, a dedicated detection transformer with a ResNet-50 backbone trained for 80 epochs that nearly matched PaliGemma-CXR. In contrast, zero-shot PaliGemma failed to produce valid bounding boxes, resulting in an object detection precision of zero.

The multi-task model demonstrated equally marked improvements across visual question answering, medical report generation, and feature segmentation. In close-ended visual question answering, PaliGemma-CXR achieved 98.95 percent accuracy, outperforming task-specific and zero-shot models across abnormality, presence, location, and counting questions. Notable performance gains occurred in position and counting queries, which the authors attributed to spatial features acquired through concurrent object detection and segmentation training. In report generation, PaliGemma-CXR recorded a BLEU score of 41.3 and a ROUGE-L score of 72.2, exceeding task-specific fine-tuning while zero-shot performance hovered near zero. In segmentation, PaliGemma-CXR achieved a mean average precision of 16.0, more than doubling the task-specific score, while zero-shot models generated no valid masks. Compared to widely used external benchmarks, PaliGemma-CXR surpassed reported results on VQA-RAD and MIMIC-CXR, although direct comparisons remain limited by differing clinical scopes and dataset targets.

Qualitative evaluations and error analyses revealed both the model’s clinical strengths and specific failure modes under data scarcity. Loss curve analysis demonstrated rapid convergence for classification and report generation, while segmentation and object detection converged more slowly due to smaller sample sizes, though all tasks eventually attained stability. Qualitative examination confirmed accurate comprehension, such as correctly answering that pleural effusion was absent and generating medical reports with correct pathology impressions. However, the model produced errors on features with fewer training examples, occasionally omitting pleural effusion or predicting consolidation in both lung fields when ground truth identified involvement in only the left lung. Additionally, normalized confusion matrix evaluations revealed lower diagnostic accuracy on non-tuberculosis infections. The authors determined that while inverse-ratio resampling mitigated imbalance during training, it could not completely overcome the limited representation of non-tuberculosis cases, which numbered only 33 compared to 715 active tuberculosis radiographs.

The authors noted several limitations in their investigation while outlining concrete deployment strategies for clinical environments in Uganda. Key limitations include the modest size of the 1,149-image dataset, the fact that only a subset of generated question-answer pairs received radiologist review, and computational barriers that prevented full comparative fine-tuning of resource-intensive baseline models like Med-SAM and BiomedParse. To deploy the system, the research team received funding from the Uganda Ministry of Science and Technology to build a clinical decision support system for tuberculosis. The platform will be deployed directly on server infrastructure at Mulago National Referral Hospital and Mengo Hospital in urban centers, incorporating patient de-identification and clinical feedback mechanisms to facilitate continuous model improvement. For rural healthcare facilities with internet connectivity but limited computing hardware, including Kisiizi and Amai hospitals, clinical access will be provided through a dedicated web application. Future research aims to evaluate larger multimodal cohorts and explore expanding multi-task architectures to other imaging modalities such as computed tomography, ultrasound, and magnetic resonance imaging.

Subject of Research: Multitask multimodal artificial intelligence for tuberculosis chest radiograph interpretation

Article Title: PaliGemma-CXR: a multi-task multimodal model for tuberculosis chest X-ray interpretation

Article References: Musinguzi, D., Katumba, A., Murindanyi, S., Nakatumba-Nabende, J., Mwikirize, C., Mbabazi, M., Kisembo, H., Malumba, R., & G.Kawooya, M. (2025). PaliGemma-CXR: a multi-task multimodal model for tuberculosis chest X-ray interpretation. BMC Artificial Intelligence, 1(1), Article 7. https://doi.org/10.1186/s44398-025-00008-3

Image Credits: AI Generated

DOI: 10.1186/s44398-025-00008-3

Keywords: Tuberculosis, Artificial Intelligence, Chest X-Ray, Multimodal Learning, Medical Image Interpretation, PaliGemma, Global Health, Uganda, PaliGemma-CXR, multi-task, multimodal, model