Large language models can classify the cognitive demand of dental specialty examination questions under certain educational frameworks but struggle under others, according to a study evaluating four publicly accessible artificial intelligence models. Researchers found that while systems such as Gemini and DeepSeek demonstrated substantial classification fidelity when evaluating multiple-choice questions under Revised Bloom’s Taxonomy and the SOLO Taxonomy, performance across all evaluated models remained weak when categorizing items according to Miller’s Pyramid. The findings demonstrate that educational taxonomies should not be treated as interchangeable frameworks in automated evaluation. Furthermore, the investigators reported that an artificial intelligence model’s classification fidelity across cognitive frameworks represents an independent capability that cannot be inferred from its answer accuracy alone.
Large language models are finding increasing applications within dental education, yet prior investigations have focused predominantly on whether these systems can correctly answer examination questions. Whether computational models can reliably identify the specific cognitive demand of assessment items has remained unclear.
The research team evaluated a dataset comprising two hundred endodontic multiple-choice questions taken from the Turkish Dentistry Specialty Examination, known as DUS, spanning the years 2012 through 2025. Each examination item was classified independently by four widely accessible models: ChatGPT, Gemini, Kimi, and DeepSeek. To establish an authoritative benchmark for comparison, the researchers developed a three-layer expert consensus reference standard. This rigorous evaluation framework incorporated formal calibration training, a procedural audit, and structured joint deliberation to establish consensus classifications across all three educational taxonomies for the evaluated questions.
To evaluate classification performance and agreement with the expert consensus standard, the investigators employed multiple statistical metrics. Agreement between model classifications and the expert standard was quantified using weighted Cohen’s kappa coefficients, designated as κw, with bootstrap 95 percent confidence intervals. Overall classification performance was measured using accuracy and macro-averaged F1 scores. To assess differences in item-level classification accuracy among the four models, the authors conducted Cochran’s Q tests accompanied by Kendall’s W, followed by pairwise McNemar tests adjusted with Holm–Bonferroni corrections to maintain statistical rigor across multiple comparisons.
The statistical analysis revealed significant inter-model differences in classification accuracy across all three educational taxonomies, with all omnibus comparisons reaching statistical significance at p < 0.001. Under Revised Bloom’s Taxonomy, Gemini and DeepSeek achieved substantial agreement with the expert consensus reference standard, recording weighted kappa values of 0.676 and 0.667, respectively. Under the SOLO Taxonomy, Gemini was the only model to achieve substantial agreement, registering a weighted kappa of 0.622, whereas DeepSeek, Kimi, and ChatGPT demonstrated moderate agreement.
In sharp contrast to the results on the first two taxonomies, classification performance on Miller’s Pyramid remained consistently weak across all evaluated systems, with agreement failing to exceed the fair threshold for any model. In this framework, ChatGPT recorded the highest performance with a weighted kappa of 0.372. This outcome produced a notable reversal in relative model rankings, as ChatGPT ranked lowest among the four models on Revised Bloom’s Taxonomy and the SOLO Taxonomy but highest on Miller’s Pyramid.
Beyond comparing model performance, the researchers identified a significant longitudinal trend within the examination dataset itself. Analysis revealed that the cognitive demand of the endodontic questions increased over time, as Revised Bloom levels assigned by the expert consensus reference standard showed a statistically significant positive correlation across examination years from 2012 to 2025, with a Spearman rank correlation coefficient of ρ = 0.301 (p < 0.001).
In their conclusions, the authors highlighted that artificial intelligence models exhibit meaningful classification fidelity for Revised Bloom’s Taxonomy and the SOLO Taxonomy, particularly when using Gemini or DeepSeek, but remain limited when applied to Miller’s Pyramid.
Subject of Research: Large language model classification fidelity across educational taxonomies in dental examination questions
Article Title: Classification fidelity of large language models on endodontic items from the Turkish dental specialty examination
Article References: Şen, B. H., Yekeler, A., Arpaçay, D. K., Ocak, J., & Akkoçan, O. (2026). Classification fidelity of large language models on endodontic items from the Turkish dental specialty examination. BMC Medical Education. https://doi.org/10.1186/s12909-026-10746-1
Image Credits: AI Generated
DOI: 10.1186/s12909-026-10746-1
Keywords: dental education, large language models, educational taxonomies, Revised Bloom's Taxonomy, SOLO Taxonomy, Miller's Pyramid, ChatGPT, Gemini, Classification, fidelity, large, language
News Source: Courtney Benton. (November 12, 2026). AI Models Differ When Classifying Cognitive Levels of Dental Exam Questions. Scienmag.



