A deceptively simple thought experiment is rattling the foundations of AI ethics. Suppose an artificial intelligence system is designed in democratic fashion, built to follow some measure of the aggregate values of humanity, and suppose further that, according to that very measure, humanity would prefer the system not to exist at all. What should the machine do? The answer, argues researcher Seth D. Baum of the Global Catastrophic Risk Institute in an editorial marking forty years of the journal AI & Society, is unavoidable: the system should shut itself down, because that is precisely what its own values command. From this premise flows a striking logical conclusion — there can be no such thing as an unwanted democratic AI system. The paradox is not merely a philosophical curiosity. Baum contends that it applies with equal force to entire AI development projects, and that the leading laboratories of the current AI boom may already be edging into the territory it describes.
The idea at the heart of the paradox is a familiar one in AI ethics, where researchers have long discussed value alignment, human compatibility, and social choice ethics as guiding frameworks. The democratic character of the approach comes from its structure: just as democracies use voting to aggregate citizen preferences into collective decisions, a democratic AI system aggregates the values of many people into a single measure that steers its behavior. The alternative is what Baum calls authoritarian or dictatorial AI, in which a narrow group — or even a single individual — imposes its values on the machine from the top down. On paper, the democratic model sounds like the obvious moral choice. But the paradox exposes a hidden trapdoor: if the aggregated values of the public turn against the AI itself, a truly democratic system has no principled way to keep running. Its own moral machinery becomes the instrument of its abolition.
Baum extends the logic from individual systems to whole research and development enterprises. Imagine a project that declares its mission to be the advancement of democracy, positioning its AI as a contribution to the global contest between democratic and authoritarian governance — yet the public, whose preferences the project claims to serve, would rather the project did not exist. By the project’s own democratic standard, it ought to disband. This is not a purely hypothetical scenario, Baum argues. The leadership of major AI laboratories, including OpenAI and Anthropic, both based in the United States, have publicly framed their work in explicitly democratic terms, casting advanced AI as a bulwark against authoritarian dictatorship. Meanwhile, American public opinion on AI is mixed and increasingly negative, and civic opposition is mounting against specific aspects of the industry, most visibly the proliferation of energy-hungry data centers. Baum is careful to note that this does not constitute an unambiguous case of unwanted democratic AI, but he argues that the signs point clearly in that direction.
Future scenarios could sharpen the tension dramatically. In a provocative essay titled “Eat your AI slop or China wins,” writer Ross Bellafiore argues that the United States should embrace aggressive AI development to outcompete China, even at the cost of severe harms — mass unemployment, retreat into delusional virtual worlds, and learned helplessness among people who can no longer function without tools like ChatGPT. If public opinion is already souring under comparatively mild conditions, Baum reasons, opposition would likely intensify sharply if conditions deteriorated along those lines. Yet the picture is not uniformly bleak, because AI is an enormously broad category. Surveys show that Americans strongly support AI applications in weather forecasting, drug development, and the detection of fraud and financial crime. It may follow, Baum suggests, that the public would favor narrow AI built for specific, beneficial purposes over the pursuit of artificial general intelligence, the hypothetical technology that dominates the ambitions of the largest laboratories.
That distinction matters because AGI is also heavily associated with catastrophic risk. Scenarios in which AI causes extreme catastrophe are, by definition, unlikely to win public support, and uncertainty about which technologies might trigger such outcomes forces democratic AI to confront hard questions about how to weigh catastrophic risks against other considerations. There is already evidence of where the public stands: strong bipartisan support has emerged for legislation addressing catastrophic AI risks, suggesting a low public appetite for existential gambles — and, by extension, for AGI itself rather than narrow applications. For any project that claims democratic legitimacy while racing toward the most powerful and least predictable forms of the technology, this gap between stated values and public preference represents the paradox in its most acute form.
The paradox also exposes how thin the industry’s conception of democracy really is. In political science, democracy understood simply as the aggregation of citizen preferences through elections is a minimalist, or “thin,” conception. Thicker conceptions add citizen participation in public deliberation and issue advocacy — and it is precisely these richer dimensions of democracy that AI may be undermining. Baum catalogs the mechanisms: AI can generate and spread misinformation at scale, including synthetic deepfake videos, degrading citizens’ ability to form accurate political preferences and hold governments accountable. AI-powered information platforms siphon advertising revenue from news media, further eroding that accountability. AI tools can generate and popularize ballot initiatives, crowding out traditional human-run advocacy organizations. And automation of the economy concentrates wealth and political power in the hands of the narrow elite that owns the technology. There are counterexamples — research has shown that AI chat interventions can make online political conversations less divisive — but Baum concludes that the literature suggests AI’s overall effect on democracy may be strongly negative. He reserves a pointed term for projects that profess democratic values while weakening democracy in practice: “democratic” AI, in scare quotes.
Complicating everything is geopolitics. Bellafiore’s essay articulates what Baum calls the greatest challenge for democratic AI: the prospect that AI becomes so powerful that an advantage in the technology could confer global domination, handing the future to whichever regime — democratic or authoritarian — gets there first. Under that assumption, even a “democratic” AI project that damages democracy at home might still be the lesser of two evils, the only hope of preserving any semblance of democratic governance against authoritarian rivals. But Baum finds this scenario deeply bleak, because it implies democracy is doomed to decline and the only question is how far the fall will go. Crucially, the scenario also hinges on contested assumptions. Current AI technology does not confer global domination, and the International AI Safety Report, led by Yoshua Bengio and colleagues, underscores how deeply uncertain and controversial the prospects for such power remain.
If future AI will not confer world domination, the grim binary dissolves. Competition over AI would then resemble competition in other economic sectors, such as renewable energy and electric vehicles, or in military technologies like drones and stealth aircraft — domains that matter, but that can be balanced against the health of democracy. If the choice is between a stronger democracy and market share in one industry, Baum argues it may be entirely reasonable to choose democracy by heavily regulating or even disbanding “democratic” AI projects. And even if AI could confer domination, he identifies a third path that industry leaders claim does not exist. Sam Altman and Bellafiore have both asserted there is no option between democratic and authoritarian AI, but Baum disagrees: the third option is diplomacy — multilateral or even global agreements to avoid AGI and other extreme AI technologies altogether. Diplomacy guarantees nothing, he concedes, but neither does the aggressive pursuit of advanced AI, and given AI’s potential to be unwanted and corrosive, diplomacy may be the only route to genuinely democratic outcomes.
Baum closes with the conflicts of interest that shadow the entire debate. AI laboratories are not neutral actors within democracies; they are private corporations pursuing market share and profit, and their institutional success can collide with the democratic values they espouse — as when they accept investments from authoritarian governments. Maintaining a pro-democracy image may help such companies dodge regulation and win government support, and if AI ever did become dominant, the incentive to drop the ruse would be overwhelming. From the citizen’s vantage point, Baum argues, this gives democracies strong reason to regulate or even ban AGI pursuits, or to nationalize them under full democratic control. His prescriptions are concrete: AI projects should restrain misinformation-enabling tools, fund independent journalism, support policies that reduce concentrations of wealth and power, back AI diplomacy, and be prepared to shut down if they are unwanted. Citizens, for their part, can organize into advocacy coalitions to push sound AI policy, demand corporate governance in the public interest, and fight the broader corruption of democracy by concentrated wealth. The answer to the paradox, in the end, is conditional: narrow, publicly supported AI can be democratic. Extreme AI pursued in democracy’s name may not deserve to exist at all.
Subject of Research: The paradox of democratic AI systems that must self-destruct if public aggregate values oppose their existence
Article Title: The paradox of unwanted democratic AI
Article References: Baum, S. D. (2026). The paradox of unwanted democratic AI. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03386-y
Image Credits: AI Generated
DOI: 10.1007/s00146-026-03386-y
Keywords: artificial intelligence, AI ethics, democracy, value alignment, social choice, AGI, catastrophic risk, OpenAI, Anthropic, AI regulation, AI diplomacy, public opinion
Cite Scienmag News
APA MLA Chicago
Denise Maddox. (September 24, 2026). When Democracy Says No: The Strange Paradox of AI Built on the Public’s Values. Scienmag. https://scienmag.com/when-democracy-says-no-the-strange-paradox-of-ai-built-on-the-publics-values/
Denise Maddox. “When Democracy Says No: The Strange Paradox of AI Built on the Public’s Values.” Scienmag, 24 September 2026, https://scienmag.com/when-democracy-says-no-the-strange-paradox-of-ai-built-on-the-publics-values/. Accessed 24 September 2026.
Denise Maddox. “When Democracy Says No: The Strange Paradox of AI Built on the Public’s Values.” Scienmag. September 24, 2026. https://scienmag.com/when-democracy-says-no-the-strange-paradox-of-ai-built-on-the-publics-values/
Copy citation Download RIS
Tags: AGIAI development and public opinionAI diplomacyAI ethicsAI regulationAI safety and moralityAI system self-regulationAI value alignmentAnthropicArtificial Intelligencecatastrophic riskdemocracydemocratic AI systemsethical implications of AI shutdownglobal catastrophic risks of AIhuman values in artificial intelligenceinfluence of human preferences on AIOpenAIparadox of democratic AIpublic opinionsocial choicesocial choice ethics in AIvalue alignment
