Deep learning peptide sequencing stumbles on incomplete spectra, not model design

Deep learning tools that read peptide sequences straight from mass spectrometry spectra, without consulting any protein database, have multiplied rapidly in recent years. A new benchmark of eight state-of-the-art de novo sequencing models now shows that these tools share a common Achilles heel: most of their mistakes stem from incomplete fragmentation spectra rather than from flaws in model architecture. The study, published in Nature Methods, also finds that post-processing tricks designed to polish de novo predictions deliver little or no benefit, and it calls for a fundamental shift in how these tools report their results.

De novo peptide sequencing is attractive because it can identify peptides from fragmentation spectra alone, freeing researchers from dependence on sequence databases. But incomplete spectra create ambiguity, because a spectrum may simply lack the fragment ions needed to distinguish between competing peptide candidates. Recent deep learning advances have produced numerous de novo models that predict sequences and refine peptide-spectrum matches under such conditions, yet their relative strengths, weaknesses and ability to handle spectrum ambiguity remained unclear. The research team, led by Sam van Puyenbroeck, Denis Beslic, Tomi Suomi and colleagues including senior authors Laura L. Elo, Lennart Martens, Robbin Bouwmeester, Tim Van Den Bossche and Tine Claeys, set out to fill that gap.

The team benchmarked eight models—AdaNovo, Casanovo, ContraNovo, InstaNovo, NovoB, PepNet, π-HelixNovo and π-PrimeNovo—on three publicly available proteomics datasets. Two datasets came from a quantification benchmark of a mixture of human, Escherichia coli and Saccharomyces cerevisiae proteins, acquired on an Orbitrap QE-HFX and on a timsTOF Pro instrument, which allowed the researchers to contrast two different fragmentation and mass analysis technologies. The third dataset, from the CAMPI study, spiked in a mock community of eight species to simulate a metaproteomics experiment. Model performance was judged against a gold standard of database search identifications obtained with Sage in chimeric mode and MSGF+ at a q-value below 0.01, and the gold standard deliberately retained multiple peptides per spectrum because a spectrum may be chimeric or the correct peptide may be a close alternative that the search engine did not report.

Performance varied across models, with considerable overlap in correct identifications. On the Orbitrap dataset, the models largely followed a consistent ranking from lowest to highest: PepNet, ContraNovo, AdaNovo, π-HelixNovo, Casanovo, NovoB, InstaNovo and π-PrimeNovo. Precision across all models was strongly affected by the instrument and sample complexity. A noticeable drop in performance occurred for all models on the timsTOF data, probably reflecting training biases toward higher-energy collisional dissociation data such as the nine-species benchmark or the MassiveKB collection of HCD spectra. InstaNovo and AdaNovo additionally showed a remarkable drop in performance on the complex species sample compared with the simpler three-species mix, hinting at potential overfitting on species-specific sequence information. The researchers also noted that π-PrimeNovo and π-HelixNovo showed an enrichment for high-scoring but incorrect peptide-spectrum matches, indicating overconfidence in some predictions.

Although the models performed similarly overall, they diverged substantially in the specific spectra they sequenced correctly. In total, 64.2 percent of all spectra were correctly predicted by at least one model, and 12.6 percent of all correctly identified spectra were correctly sequenced by all eight models. After excluding ContraNovo and PepNet owing to incomplete output and lower performance, that proportion rose to 34.4 percent, suggesting a subset of spectra that are straightforward to sequence. Each model also contributed a unique set of correct identifications: π-PrimeNovo, NovoB and InstaNovo each contributed 4.1, 3.8 and 2.5 percent unique correct identifications, respectively. When the researchers examined incorrect predictions, they found minimal overlap among models, suggesting little agreement when models err. Incorrect predictions aligned across all models for only 0.174 percent of spectra, and 73 percent of those de novo peptides were also present in the protein database, hinting at co-fragmenting peptides.

The team then evaluated three post-processing approaches: Spectralis and InstaNovo+, which can refine candidate peptide-spectrum matches, and MS2Rescore, a rescoring tool widely used in database searches. Across the eight de novo models, a substantial fraction of predictions were refined by Spectralis, between 27.6 and 33.3 percent, and to a lesser extent by InstaNovo+, between 8.19 and 35.3 percent. Despite this frequent refinement, the increase in the number of correct predictions after refinement was limited. Both tools often yielded only minor improvements or even reduced overall performance, and for the more accurate base models such as π-PrimeNovo and Casanovo, refinement actually decreased the number of correct predictions. Where a sequence was genuinely corrected, the correction was limited to fewer than two amino acid substitutions. MS2Rescore consistently underperformed across all models and refinement settings, failing to distinguish correct from incorrect predictions better than the base model scores, a notable result given that it typically improves confident identifications by 20 to 35 percent in database searches with large databases. The researchers suggest that incorrect de novo predictions often show considerably better spectral matches than decoy predictions, making them harder to separate than in traditional database searching.

The error analysis revealed that most sequencing errors were model-independent and driven by limited fragment ion coverage. About a fifth of all errors were localized to a single site, and two subcategories were by construction hard to resolve from fragmentation alone: sequence permutations, where two sequence tags share an amino acid composition, and isobaric tags, where they share a mass. In the majority of cases with transpositions of two amino acids, 68.5 percent, no diagnostic fragment ions were present to distinguish between the two candidates, and around half of the isobaric tags were similarly indistinguishable. These errors were also positionally biased, occurring predominantly at the C-terminus, and most current de novo models start constructing the peptide sequence C-terminally, potentially limiting their use of sequence pattern information in that region. On average, 20 percent of the wrong predictions consisted of peptide sequences completely different from the ground truth, which could indicate co-fragmenting precursor ions, though the limited overlap among tools questioned that hypothesis.

To estimate the upper bound of current de novo sequencing, the team simulated an idealized ensemble in which a spectrum counted as correct if any of the eight models predicted the gold-standard peptide. While the best-performing individual model achieved an accuracy of 50.4 percent, the ensemble reached 64.2 percent. The most striking finding concerned missing complementary b- and y-ion fragments: when all fragmentation sites were covered by at least one of these ions, 92 percent of spectra were correctly predicted by the ensemble. That figure dropped to 85, 70 and 50 percent when one, two or three complementary ion pairs were missing, respectively. Spectral noise, quantified as the inverse of the percentage of explained intensity, was another strong predictor of failure. Only 17.1 percent of spectra with less than 5 percent explained intensity were correctly identified by any model, while this figure rose to over 80 percent for spectra with more than 25 percent explained intensity. The influence of missing complementary ions was much more pronounced than the effect of peptide length, although the two factors appear confounded because longer peptides are more likely to have missing complementary ions.

The study also uncovered a role for information beyond fragmentation itself. Spectra that all models failed to predict showed lower cosine similarity between observed and MS2PIP-predicted fragment intensities, suggesting less predictable or noisier fragmentation patterns. Yet their retention time deviations remained on par with those of well-predicted spectra, meaning that when spectral quality is poor, retention time remains equally and in relative terms even more important for accurate identification. In candidate-selection experiments with Casanovo, retention time error against DeepLC predictions separated the correct candidate from the top-ranked wrong candidate markedly more than cosine similarity, hyperscore or precursor mass error, yet none of the de novo engines currently incorporate retention time as a feature. The researchers argue that future models must better handle incomplete fragmentation and learn to exploit nonfragment-based features such as retention time and ion mobility.

The authors conclude that rather than expecting full-length, high-confidence de novo sequences for every spectrum, a more realistic goal is extracting reliable sequence tags or partial matches where possible, with explicit uncertainty annotation. Only about half of the spectra identified by database search engines were also correctly identified by de novo methods; the rest either lack sufficient signal or represent cases with multiple indistinguishable peptide candidates. In fields such as metaproteomics or antibody sequencing, overconfidence in such predictions can distort downstream interpretation. The team advocates for a centralized, community-driven benchmarking platform, pointing to ProteoBench as a promising candidate, and has integrated a substantial part of the current benchmark into that platform. They also propose shifting focus from the absolute score of the top prediction to the discriminative gap between top-scoring candidates, and returning consensus sequences with variable regions annotated as uncertain where multiple sequences share substantial overlap. The sequence ambiguity problem is expected to grow as de novo engines begin to support post-translational modification prediction, which substantially increases the search space and makes it more likely that multiple modified peptide candidates can explain the same spectrum equally well.

Subject of Research: Benchmarking of deep learning de novo peptide sequencing models and the spectral causes of their sequence ambiguity

Article Title: Limitations of de novo peptide sequencing in resolving sequence ambiguity

Article References: van Puyenbroeck, S., Beslic, D., Suomi, T., Holstein, T., Muth, T., Elo, L. L., Martens, L., Bouwmeester, R., Van Den Bossche, T., & Claeys, T. (2026). Limitations of de novo peptide sequencing in resolving sequence ambiguity. Nature Methods. https://doi.org/10.1038/s41592-026-03301-9

Image Credits: AI Generated

DOI: 10.1038/s41592-026-03301-9

Keywords: de novo peptide sequencing, mass spectrometry, proteomics, deep learning, fragmentation spectra, sequence ambiguity, benchmarking, peptide-spectrum matches, rescoring, retention time, timsTOF, Orbitrap