MCGE-Net joins image, text and labels to enhance imaging

A research team has developed MCGE-Net, a neural-network framework that enhances images by jointly using a reference image, a text description and pixel-level semantic labels as conditions. The study, published in 2026, reports that the system outperformed comparison methods on a newly assembled dataset spanning medical images, natural scenes and industrial inspection, as well as on the public MS-COCO 2017 benchmark. On the 5,000-image test set of MIID-50K, MCGE-Net produced a peak signal-to-noise ratio of 32.1 decibels, a structural similarity index of 0.903, a Fréchet inception distance of 24.8 and semantic consistency of 89.5%. These results describe controlled machine-learning experiments, not clinical performance or deployment readiness.

The task differs from both conventional image fusion and pure text-to-image generation. Image-fusion systems generally combine paired sensor images without text or semantic guidance, whereas text-to-image models synthesize an image from a prompt without retaining a reference image as its content anchor. MCGE-Net instead uses all three inputs at once: the reference image supplies the original visual structure, the description expresses semantic intent, and the labels provide spatial priors. The objective is to generate an enhanced image that remains faithful to the source pixels, agrees with the text and labels, and looks perceptually realistic. This distinction is important because the reported gains come from a specific three-way conditioning setup rather than from a general claim that one generative model is superior for every imaging task.

MCGE-Net contains a multimodal encoder, three purpose-built processing modules and a decoder. A ResNet-50 backbone extracts visual features at four levels, while the pretrained BERT-base model converts text into token embeddings. The final ResNet stage produces 256 visual tokens with 768 channels, matching the text embedding dimension. Semantic labels are compressed into a 256-dimensional vector, expanded through a two-layer multilayer perceptron and later added across the token sequence. A PyTorch implementation was trained on four NVIDIA A100 graphics processors, using 512-by-512-pixel inputs, the Adam optimizer, an initial learning rate of 2 × 10⁻⁴, a batch size of 32 and cosine annealing over 200 epochs. Gradient clipping was maintained at a maximum norm of 1.0.

The system’s first major component, the Hierarchical Cross-Modal Attention mechanism, combines two scales of information. Fine-grained attention calculates correspondences between individual visual tokens and text tokens, allowing local regions to receive information from semantically related words. Coarse-grained attention uses globally pooled descriptors to represent broader associations across the image and description. A small gating network produces a value between zero and one for each sample, determining how much weight to place on the fine and coarse representations. The authors argue that this adaptive balance is more flexible than a fixed weighting strategy. In an attention ablation, the combined design reached 31.5 decibels PSNR and an FID of 21.8, while the learned gate in a finer experiment produced 31.9 decibels, compared with 30.8 for fine attention alone and 30.1 for coarse attention alone.

A second module, the Dynamic Feature Alignment mechanism, addresses the distribution differences among visual tokens, text embeddings and label features. An MLP examines concatenated visual and textual features and generates per-token, channel-wise scaling values. These dynamic weights control depthwise separable convolutional layers that transform the two feature sets before they are combined with the label embedding through residual addition. The transformation is therefore tailored to each input rather than governed by fixed routing or projection rules. According to the reported ablations, dynamic feature alignment increased semantic consistency by 9.3 percentage points relative to a version without alignment and by 3.5 percentage points relative to a learnable but non-adaptive alternative. The model’s label pathway was also informative: removing the semantic-label features reduced PSNR by 1.0 decibel and semantic consistency by 2.7 percentage points.

The third component, the Multi-Scale Enhancement Network, restores and strengthens information at three spatial resolutions. It reshapes the aligned 16-by-16 feature map into a pyramid through successive upsampling and convolution, producing feature maps of 32-by-32 and 64-by-64 pixels. At each scale, a residual convolutional block and squeeze-and-excitation channel attention emphasize informative channels. The scale-specific outputs are then upsampled, concatenated and passed to a decoder with four transposed-convolution blocks, producing a 512-by-512 image. Unlike a feature pyramid that operates only on visual data, this design conditions every scale on the already aligned image, text and label representation. The module-level ablation attributed the largest visual-quality improvement to this network, including a 2.6-decibel PSNR gain, while the hierarchical attention module produced the largest reported gain in semantic consistency.

MIID-50K contains 50,000 high-resolution image-text-label triplets, with approximately 16,667 samples from each of three domains. Medical data came from NIH ChestX-ray14, natural-scene data from MS-COCO 2017 and industrial data from MVTec AD. Descriptions for the medical and industrial subsets were written independently by a board-certified radiologist and a senior industrial engineer, who achieved a Cohen’s kappa of 0.87 before disagreements were resolved by a third expert. Semantic masks were generated with a domain-specific Mask R-CNN and manually corrected when prediction confidence was below 0.85. Blurry and near-duplicate images were removed, and the source images were divided into 40,000 training, 5,000 validation and 5,000 test examples. All experiments were repeated three times with fixed seeds, and reported results were means with standard deviations.

Against ControlNet, the closest multi-input baseline, MCGE-Net improved PSNR by 1.4 decibels, SSIM by 0.045, FID by 2.8 points and LPIPS by 0.023, while raising semantic consistency by 3.6 percentage points. Paired tests across the 5,000 test samples gave Bonferroni-corrected p-values below 0.001 for the five comparisons. The comparison is not perfectly equivalent, however: the baseline systems used official pretrained weights without fine-tuning on MIID-50K, while MCGE-Net was trained from scratch on the new dataset’s training split. The researchers therefore included additional fine-tuned Stable Diffusion and ControlNet results in their supplementary analysis. On the 5,000-pair MS-COCO 2017 validation set, the model reached an FID of 20.4 and a CLIP-Score of 33.8, reported as 2.3 FID points and 0.9 CLIP-Score points better than ControlNet.

MCGE-Net required 1.23 seconds per sample, making it slower than DALL-E 2 at 1.15 seconds but faster than Stable Diffusion at 2.34 seconds, Imagen at 2.67 seconds and GLIDE at 2.01 seconds. It contains 110.3 million parameters and uses 187.6 billion floating-point operations for a 512-by-512 sample measured on one A100. The authors attribute part of the speed advantage to the single-pass architecture, which avoids the iterative denoising used by diffusion generators. A double-blind study involving 50 professionals also favored MCGE-Net on visual realism, detail richness, semantic consistency and overall quality. Krippendorff’s alpha ranged from 0.68 to 0.74, indicating substantial inter-rater agreement, and differences from ControlNet were significant at p-values below 0.01. The evaluators included 18 medical imaging specialists, 17 photographers or visual designers and 15 image-processing engineers.

The results nevertheless carry several constraints. Manual review of all 5,000 MIID-50K test examples identified recognizable problems in about 18% of outputs, including semantic drift in long-tail categories at 8.3%, scale-boundary artifacts at 5.1% and label-boundary blur at 4.7%. The model was designed for static images, has no explicit mechanism for attributing the final output to individual input modalities, and remains too large for many edge devices without compression. MIID-50K also combines images from established datasets under a new annotation and training protocol, so its aggregate results do not constitute independent validation in a clinical or industrial production environment. The paper presents preliminary potential for medical-image enhancement, inspection and creative-content production, but it explicitly states that domain-specific validation remains necessary before use in safety-critical settings.

Subject of Research: Evaluation of a three-input neural network for multimodal-conditioned image enhancement

Article Title: Multimodal content generation and enhancement for smart imaging

Article References: Zhang, K., Wu, D., Song, Y., & Ren, H. (2026). Multimodal content generation and enhancement for smart imaging. Discover Artificial Intelligence, 6(1), Article 1175. https://doi.org/10.1007/s44163-026-02108-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02108-z

Keywords: multimodal image generation, image enhancement, cross-modal attention, semantic labels, neural networks, medical imaging, industrial inspection, MS-COCO, ControlNet, deep learning, Multimodal, content