AI Reads Protein Alignment Images Well by Name, Fumbles Residue Boundaries

Vision–language models, the AI systems that can look at pictures and talk about them, have passed a new and revealing test of their scientific literacy — and the results are a striking study in uneven competence. In a study published in Genomics & Informatics, Koya Sakuma of the PDB-Descriptome project asked two commercial AI models to read synthetic images of multiple sequence alignments, the colorful, grid-like figures that structural biologists use to flag which stretches of a protein matter. The models turned out to be surprisingly good at reading the labels attached to those flagged regions. They were strikingly bad at working out where, in terms of residue numbers, those regions actually began and ended.

The work sits within an ambitious effort called PDB-Descriptome, an ongoing project introduced at the BLAH9 meeting to systematically link three-dimensional biomolecular structures in the Protein Data Bank with the textual descriptions that scientists write about them. Structural biology papers are inherently multimodal: they combine conventional prose with molecular coordinate data, and they constantly refer to specific segments of molecules — a catalytic domain, an active site, the fifth alpha-helix. Yet, as the study notes, the linkage between the words in the literature and the precise structural definitions behind them has never been systematically standardized. A paper might say “the catalytic domain” without ever stating that it denotes residues 41–125 and 199–230.

To give this gap a precise vocabulary, the project defines two complementary concepts. Structural biological entities, or SBEs, are the things being referred to — substructures of a molecule that, in atomic coordinate data, can be defined as sets of atom indices. Structural biological referring expressions, or SBREs, are the words and phrases that do the referring — spans of text like “Glu81,” “H5,” or “the carbonyl group.” The crucial point is that these expressions are nearly meaningless as isolated text: “Glu81” appears in countless papers about unrelated structures, and nickname-like labels are defined only locally. Linking SBREs to their SBEs is therefore essential, and that linkage is exactly what the new evaluation probed.

The study focused on a simplified case: figures that define SBE–SBRE pairs through sequence-related images, specifically multiple sequence alignments. Because MSAs are often published as raster images rather than machine-readable files, extracting their content is genuinely difficult. Rather than wrestling with real published figures, the researchers generated synthetic data from scratch. A random 180-residue amino acid sequence was created, nine child sequences were derived from it by introducing mutations and gaps with probabilities of 0.1 and 0.05, and the resulting ten-sequence alignment was visualized with the pyMSAviz library. Between two and five regions were randomly assigned per alignment, each named with labels like “Mutation Hotspot” or “Binding Site” plus a unique letter, each 10 to 30 residues long, placed with at least four-residue gaps to prevent overlap.

Across 100 datasets this procedure produced 377 annotated regions, exported as PNG images at 1006 × 867 pixels, with residue numbers displayed at intervals of 10 and sequences broken into lines every 60 residues. The synthetic setup, the author acknowledges, is not realistic compared with the heterogeneous figures found in actual papers — but that was deliberate. With ten fixed-length sequences and standardized rendering, the experiment estimates an approximate upper bound on what the models can do; performance would be unlikely to improve with messier, more diverse data.

Two vision–language models were put to the test: gemini-2.5-flash and gemini-2.5-pro. Each received the images under two prompt styles. A naïve prompt simply asked the model to identify any structural region defined in the figure and report it as a Python list of region names and start and end positions. A strict prompt added an explicit demand for one-residue precision and explained that residue indexes appear only at sparse intervals, so the model would need to interpolate between visible labels to estimate actual boundaries. The flash model received both prompts; the pro model received only the strict one. A single trained structural biologist annotated the same 100 images as a human reference, working without a time limit, permitted to use a ruler without tick marks and text autocompletion in Excel.

The evaluation scored two things separately. For SBRE extraction, extracted region names were compared with ground truth as sets, yielding precision, recall, and F1-scores. For SBE boundaries, mean absolute errors were computed independently for start and end residue indices, counting only correctly and uniquely named regions. The pattern that emerged was sharply asymmetric. Both models showed high precision and recall in extracting region names — gemini-2.5-flash sometimes extracted the same name multiple times, but its scores remained high. Boundary extraction was another matter entirely: the mean absolute errors of VLM-extracted boundaries were large, effectively spanning an entire row of the MSA. The strict prompt slightly reduced these errors for the flash model, and the pro model did better on names, but neither came close to the human.

The human annotator showed slightly lower precision and recall than the machines — a deficit attributed to typographic errors — but the mean absolute errors of the human-extracted boundaries were, in the author’s words, strikingly smaller, nearly two orders of magnitude better than the VLMs. Histograms of the raw errors make the failure mode vivid. While VLM-based boundary estimates were broadly distributed around zero, additional peaks appeared around −60 residues, which the study attributes to the models misassigning the colored region-indicating bars to the wrong rows of the alignment — typically one row above the ground truth. In one documented example, residues 111–119 that belonged to a region called “Low Conservation B” were misread as an independent region and given an improvised name, “Unlabeled Region.”

Oddly, although the prompt specified an exact output format, some of the pro model’s failed responses contained Python code seemingly designed to solve the extraction task or print the answer, code the downstream parser was never built to handle. On speed, the flash model needed roughly 20 minutes for the full set, the pro model 2 hours and 30 minutes (including mandatory pauses to avoid exceeding API quota), and the human annotator about 2 hours and 10 minutes. The annotator also reported that only the first 20 or so images used the ruler, that certain bar colors had low contrast against the white background, and that some loss of focus occurred near the end.

Informal tests hinted the problem is not unique to Gemini. ChatGPT 5.1 and Claude Sonnet 4.5, tried on a handful of real MSA images from structural biology papers through their chat interfaces, showed similar tendencies: reasonable name extraction, poor boundary detection. ChatGPT 5.1 Thinking attempted a promising strategy — iteratively cropping images, reasoning about x–y coordinates, and mapping them to residue numbers with custom Python code — but accuracy improved only marginally and a single image took more than 25 minutes. The author suggests the struggles may reflect intrinsic limitations of current model architectures and the fundamental challenge of mapping visual features to precise numerical coordinates in rasterized images.

The practical conclusion is a division of labor. Because the models excel at exactly what humans find tedious — reading and typing out region names — and humans excel at what the models cannot do — pinpointing residue boundaries — the study supports a human-in-the-loop pipeline in which SBRE extraction is delegated to VLMs and SBE boundary definition is performed by people. Open questions remain: the accuracy of non-expert annotators, inter-annotator consistency, and the cost of professional annotators versus the cost of automation. Possible technical fixes include augmenting synthetic MSAs with gridlines or explicit coordinate labels, better prompts, and secondary LLM instances that validate and reformat responses before parsing. But the author expects worse and more inconsistent results on realistically heterogeneous published figures, and extracting entities from complex three-dimensional structure figures will be harder still. Future VLM advances may reduce the need for human intervention; for now, the hybrid approach is the most reliable path to scaling up structure–text annotation.

Subject of Research: Evaluating vision–language models and a human annotator on extracting structural region names and residue boundaries from synthetic multiple sequence alignment images for the PDB-Descriptome annotation project

Article Title: VLMs struggle to extract definitions of structural segments from MSA images: toward the design of a human-in-the-loop annotation pipeline for the PDB-Descriptome project

Article References: Sakuma, K. (2026). VLMs struggle to extract definitions of structural segments from MSA images: toward the design of a human-in-the-loop annotation pipeline for the PDB-Descriptome project. Genomics & Informatics, 24(1), Article 6. https://doi.org/10.1186/s44342-025-00061-4

Image Credits: AI Generated

DOI: 10.1186/s44342-025-00061-4

Keywords: vision-language models, multiple sequence alignment, PDB-Descriptome, structural biology, Protein Data Bank, image annotation, human-in-the-loop, Gemini, SBRE, SBE, boundary extraction, multimodal AI