AI deception detectors stumble when synthetic training meets real-world stress

Detecting a lie from the way someone speaks has long tempted researchers, and modern artificial intelligence has promised to make the task systematic. A new study published in Frontiers in Artificial Intelligence put that promise to a hard test, comparing four classic machine learning classifiers with a knowledge-grounded language model called FranklinAsk on both synthetic interview data and real recordings from a military training exercise. The result was sobering for the statistical models: classifiers that scored above ninety percent accuracy on synthetic text fell to between roughly thirty-five and forty-five percent accuracy when confronted with spontaneous, stress-induced speech from real soldiers.

The real-world data came from a 2019 mock-sabotage exercise run by the National Center for Credibility Assessment with U.S. Army basic training soldiers stationed at Fort Jackson, South Carolina. Participation was voluntary and had no impact on training status. Soldiers were randomly assigned to five-person teams: three truthful “Team Member” participants in Team Alpha and two deceptive “Sabotage” participants in Team Bravo. The Alpha members worked cooperatively to defuse computer-simulated improvised explosive devices while competing against Bravo, whose members had been covertly instructed to mislead teammates into pressing the wrong button during the task.

To raise the stakes, participants were told after the mission that sabotage was suspected within the group and that they would face an investigative interview. Each soldier then answered eight open-ended questions from a computer-generated agent, covering topics such as whom they most and least suspected of sabotage, how honest they considered themselves, and what they wished to clarify about their involvement. Soldiers in the sabotage role had been instructed to lie when necessary to conceal their Bravo affiliation. All interviews were audio-recorded, manually transcribed, and processed with a custom Python pipeline, producing a corpus of seventy-six participants, thirty of them deceptive.

Because this real-world dataset was small, unevenly distributed, and often brief, the researchers also generated a synthetic corpus using the MOSTLYAI platform. A detailed prompt describing the exercise produced one thousand synthetic participants, each answering all eight questions, for a total of eight thousand responses balanced evenly between truthful and deceptive classes. Generation was split into ten independent batches with a temperature of 1.0 and nucleus sampling at top-p 0.95 to reduce repetition. The authors caution that the explicit truthful-versus-deceptive instructions may have introduced systematic class-specific linguistic patterns that do not reflect naturally occurring deception.

On the linguistic side, the team extracted forty-four features from the transcripts, including thirty-five fine-grained part-of-speech tags from the Flair English model, eight broader aggregate categories, and total word count. Sentiment was assessed with a Twitter-RoBERTa model. Group differences were tested with two-sided Welch’s t-tests at a nominal alpha of 0.05, with Benjamini-Hochberg false discovery rate and Bonferroni corrections applied for multiple comparisons, and Cohen’s d used to quantify effect sizes. In the military dataset, three features contained zero counts and were excluded, leaving forty-one tested features.

The findings from the real-world speech were strikingly modest. Only two features, cardinal numbers and particles, reached the unadjusted significance threshold, with nominal p-values of 0.041 and 0.0486 respectively and a moderate effect size of d = 0.55 for each. Neither survived correction: adjusted p-values were 0.668 under the false discovery rate procedure and 1.000 under Bonferroni. Both features occurred more frequently in deceptive responses at the descriptive level, but the authors explicitly label these patterns exploratory and note that, given the number of features tested relative to sample size, the nominally significant results may well be false positives. Sentiment analysis found no meaningful differences, with negative, neutral, and positive polarity all failing to distinguish the groups.

The synthetic dataset told a very different story. Twenty-six linguistic features reached nominal significance before correction, and twenty-five remained significant after false discovery rate correction, with twenty-one surviving the stricter Bonferroni threshold. Effect sizes included large values for singular nouns (d = -0.79) and infinitive-marker “to” forms (d = 0.85), along with moderate effects for determiners, nouns, and regular verbs. Sentiment also separated the classes significantly on all three dimensions, with the largest effect in positive sentiment, which was lower in deceptive responses (d = -0.733). The researchers interpret this widespread separation as evidence that the generative model encoded idealized, stylized distinctions between truth and deception rather than the subtle variation of authentic stressed speech.

Those divergent linguistic landscapes translated directly into classifier performance. Trained on LIWC-22 features with a participant-level seventy-thirty train-test split and ten-fold cross-validation, the supervised models excelled on the synthetic held-out test set of three hundred responses. Logistic regression led with an F1-score of 0.930 and a ROC AUC of 0.978, followed by support vector machine at 0.908 F1, random forest at 0.887, and naive Bayes at 0.878. FranklinAsk, which required no training and instead rated deception likelihood on a one-to-ten scale binarized at a threshold of 4.5, trailed on synthetic data with an F1-score of 0.733.

Applied without retraining to the complete military dataset of seventy-six participants, every supervised model deteriorated sharply. Logistic regression fell to an F1-score of 0.456 and an accuracy of 0.434, with its ROC AUC dropping to 0.417, below the level of a coin flip. Random forest retained the highest recall among the supervised models at 0.733 but collapsed in precision to 0.349 and accuracy to 0.355, reflecting a tendency to label nearly everything deceptive. Naive Bayes fared worst with an F1-score of 0.299. FranklinAsk, by contrast, achieved the highest point estimates among all methods on the real data, with an F1-score of 0.519, precision of 0.412, accuracy of 0.487, and a recall of 0.700, second only to random forest. FranklinAsk’s decline was also the least dramatic, falling from 73.3 percent synthetic accuracy to 48.7 percent on real speech.

The authors are careful about what these numbers do and do not show. Because FranklinAsk incorporated curated deception literature and foundation-model reasoning unavailable to the LIWC-based classifiers, and because the military sample was small, the descriptive differences do not establish statistical superiority or isolate any effect of retrieval augmentation; controlled ablation studies would be needed to separate the contributions of domain knowledge, prompt design, and base-model reasoning. A cross-study comparison with earlier fake-news work using the same classifiers and features reinforced the domain problem: accuracies near seventy to seventy-eight percent on structured misinformation corpora contrast with the thirty-five to forty-five percent range on the stress-induced military speech. The broader lesson is that high performance on synthetic or low-stakes text does not reliably predict real-world effectiveness, that linguistic cues of deception are subtle and context-dependent, and that larger, ecologically valid datasets are required before any of these approaches, statistical or knowledge-grounded, can be trusted in operational credibility assessment.

Subject of Research: Testing machine learning and a knowledge-grounded language model for detecting deception in high-stakes military interview speech

Article Title: Leveraging natural language processing, machine learning, and a knowledge-grounded language model to detect linguistic markers of deception

Article References: Miller, K., Chu, A., Reed, A., McCord, Z., Argentine, E., Allen, J., Coggins, P., Pollina, D., Senter, S., Shazly, T., Uline, M., & Boltin, N. D. (2026). Leveraging natural language processing, machine learning, and a knowledge-grounded language model to detect linguistic markers of deception. Frontiers in Artificial Intelligence, 9, Article 1965099. https://doi.org/10.3389/frai.2026.1965099

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1965099

Keywords: deception detection, natural language processing, machine learning, retrieval-augmented generation, LIWC, synthetic data, psycholinguistics, military training exercise, FranklinAsk, sentiment analysis, credibility assessment, Leveraging