Correlated errors cap how many voters an LLM ensemble really has

Multi-agent systems built on large language models increasingly decide by committee: several instances propose or rank answers, and their votes are combined by majority, by scoring, or by a further round of discussion. The justification for that design is the Condorcet Jury Theorem, which says that if voters are competent and independent, the probability that their majority is right approaches certainty as their number grows. A new study argues that the fine print of that theorem—independence—is exactly what LLM ensembles fail to satisfy, and that the correlation between agents’ errors is the decisive and most overlooked quantity in ensemble decision-making.

The paper, published in Discover Informatics, works within a simple model of exchangeable binary votes, where each agent is an LLM together with its prompt and decoding settings, and each answer is recorded only as correct or incorrect. Two numbers matter: competence, the probability that an agent answers correctly, and the correlation between the correctness indicators of two agents measured across questions. Correlation answers a simple question: do the two agents tend to get the same questions right and the same questions wrong? It is zero when knowing that one agent failed says nothing about the other, and one when the two always succeed and fail together. Models that share pre-training data, architectures, alignment objectives, or prompts tend to fail on the same questions, so the classical assumption of independent voters is unlikely to hold.

The first main result is a variance identity that survey statisticians know as the design effect of cluster sampling. The variance of the fraction of agents answering correctly splits into an independent part, whose errors cancel as agents are added, and a shared part, the errors the agents make together, which does not cancel. From this identity follows the effective number of independent voters, a variance-based effective sample size: an ensemble of N agents behaves, in how much its average vote wobbles, like N divided by one plus (N minus one) times the correlation independent agents. That number never exceeds the reciprocal of the correlation and approaches it as the ensemble grows. A hundred agents correlated at 0.3 have an effective size of about 3.3; the ceiling for any number of agents at that correlation is 3.33. At a correlation of 0.1 the ceiling is ten; at 0.5 it is two. Beyond the first dozen or so agents at a correlation of 0.3, adding more barely changes the effective size, so most of the ensemble adds cost and apparent agreement rather than independent evidence.

The second result concerns accuracy rather than variance. In the model, each question has a success rate: the fraction of agents of a given kind that would answer it correctly if asked infinitely many times. Agents are correlated because that rate varies from question to question—they succeed together on questions that are easy for their kind and fail together on hard ones. As agents accumulate, each question’s verdict is pushed not toward the truth but toward whatever the agents’ kind leans to, and a wrong verdict becomes more confident, not less. Averaged over many questions, majority accuracy converges to the share of questions on which the agents lean the right way. This share is below one whenever a positive fraction of questions has a success rate below one half, as happens at any positive correlation in the beta-binomial model used for simulation; accuracy then plateaus below certainty and, depending on the distribution of success rates, can lie below the accuracy of a single agent. In simulations with single-agent competence of 0.65, independent agents climbed toward certainty, while at correlations of 0.1 and 0.3 accuracy stalled near 0.83 and 0.71, and at 0.5 it barely cleared the single agent it was meant to improve on. Positive correlation can thus make an enlarged panel less accurate than a smaller one.

The third result distinguishes structural from cosmetic diversity. Adding samples—drawing more answers from one model with a fixed prompt and temperature—cannot raise the accuracy plateau, because the samples share the same success rate on every question, so their plurality converges to the most likely answer of that configuration. Prompt variation, such as assigning personas to instances of one model, can move the plateau only to the extent that it changes which questions the model answers correctly. Structural variation—different base models, corpora, or alignment objectives—has the most room, because different families can lean the wrong way on different questions, and a majority across families can then correct what each family gets wrong. Changing the temperature typically lowers the measured correlation but also lowers each agent’s accuracy, and when the model answers directly without reasoning, rescaling scores by a temperature does not change which answer is most likely, so the plateau cannot move at all. A lower correlation obtained through temperature is therefore not the independence the Jury Theorem needs.

To turn the correlation from a parameter into a measurement, the authors adapt a moment estimator that costs one pass over a labelled evaluation set and three arithmetic operations. With five agents at a correlation of 0.3, its standard error is about 0.06 with 100 questions, 0.03 with 300, 0.02 with 1,000, and 0.01 with 3,000, and bootstrap intervals contained the true value in 93 to 96 percent of simulated evaluation sets. When agents differ in accuracy, the estimator slightly understates the average pairwise correlation, and the direct alternative is to compute the ordinary correlation between the correctness columns of each pair of agents and report the full matrix.

As a proof of concept, the team ran three five-agent ensembles on a stratified sample of the MMLU benchmark: ten random questions from each of its 57 subjects, 570 four-option questions in all, with results reported on the 562 questions that every agent answered after eight were excluded by a content filter. The temperature ensemble sampled gpt-4.1-mini five times at temperature 0.8; the persona ensemble used the same model at temperature 0 with five persona system prompts; the family ensemble used five models from different developers—gpt-4.1-mini, Mistral-Large-3, DeepSeek-V3.2, Kimi-K2.5, and grok-4.3—at temperature 0. All were queried through Azure AI Foundry in September 2026. The two cosmetic ensembles were almost indistinguishable: temperature samples and personas of the base model were both correlated at 0.85, each was worth about 1.1 independent voters, and its plurality vote was no more accurate than its best member. Adding samples barely helped: plurality accuracy rose from 0.836 with one sample to 0.842 with nine.

The family ensemble differed in kind. Its correlation was 0.38, with pairwise values between 0.23 and 0.48; its effective size was 2.0; and its plurality vote reached 0.923, 8.4 points above the base model and 4.5 above the average member. All five temperature samples failed together on 71 questions, or 12.6 percent, while all five families failed together on only 14, or 2.5 percent. Yet the plurality of families remained below its best member, Kimi-K2.5, which scored 0.940: the members differ in accuracy, and simple plurality weighs the weakest like the strongest. The authors note that part of the correlation drop is mechanical, since agents of unequal accuracy cannot be perfectly correlated, but normalized by its maximum the pairwise correlation was still 0.52 among families against 0.87 in both cosmetic ensembles, and the comparison is partly confounded with reasoning, which two family models use and the base model does not.

The study also addresses a natural objection: that counting votes is crude, and that one model reading the others’ proposals and justifications—semantic or cognitive aggregation—might escape the correlation limit. The authors argue it does not automatically. Semantic aggregation changes the objects being combined, arguments instead of labels, but not their provenance. When agents share a pre-training and an alignment, their reasons are plausibly correlated for the same cause as their conclusions, and a synthesizer fed correlated rationales may be compressing variations of one line of thought rather than integrating independent ones. Related evidence points the same way: debate among agents with similar capabilities tends to converge to the majority opinion, and when that opinion stems from a misconception shared through training data, to the misconception. Debate helps when disagreement is deliberately induced, which suggests that independence has to be engineered rather than assumed.

The authors propose a six-item reporting checklist for deployed ensembles, including the estimated correlation with a bootstrap interval, the effective size, and accuracy before and after any diversity intervention, all producible from a single pass over an evaluation set. They caution that the checklist is normative rather than validated, and that the proof of concept is small: one benchmark, one base model, five personas, one temperature, and five model families, all answering with a single letter. That format makes the temperature result close to guaranteed and leaves personas little room to act, so the cosmetic results may not carry over to agents that reason or debate before answering. The broader implication stands: adding a model to the table does not necessarily add a voter. Sometimes it only adds an echo.

Subject of Research: Correlation between errors and its effect on the reliability of ensemble decision making with large language models

Article Title: Correlation limits the reliability of ensemble decision making with large language models

Article References: Morante, S., & García-Rubio, M. R. (2026). Correlation limits the reliability of ensemble decision making with large language models. Discover Informatics, 1(1), Article 35. https://doi.org/10.1007/s44564-026-00040-z

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00040-z

Keywords: large language models, ensemble decision making, correlated errors, Condorcet Jury Theorem, effective sample size, majority voting, MMLU benchmark, semantic aggregation, model diversity, self-consistency, multi-agent systems, reliability