AI Classifies Dental Complications From Clinical Notes

Published in BMC Digital Health, the study analyzed 4,756 clinical notes from return visits occurring within 14 days of dental extractions. The team found that pre-trained language models achieved an F1-score of 0.82 when classifying unplanned returns specifically due to complications, outperforming a simpler bag-of-words logistic regression model which scored 0.71. This work addresses a significant gap in dental quality-of-care metrics, where the reasons for patient returns are typically not recorded in structured data formats.

The motivation for this research stems from the inconsistent measurement and reporting of unplanned returns following dental extractions internationally. While the rate of such returns can serve as a quality-of-care indicator, it is only useful if it can distinguish between visits driven by complications and those for other reasons, such as routine reassurance or treatment of other teeth. In dentistry, standardized diagnostic codes are inconsistently implemented, and procedure codes designed for billing do not adequately capture complications arising from previous visits. Consequently, electronic dental records often lack the structured fields necessary to identify why a patient returned, making it difficult to monitor care quality at scale.

The study utilized electronic dental records from six public dental clinics within the Nepean Blue Mountains Local Health District in Australia. The data covered a period from April 2018 to August 2023. The cohort included 3,835 unique patients aged 18 years and over who had a dental extraction and returned to the clinic within 14 days. This 14-day window was selected because the risk of post-operative complications is highest during the inflammatory and proliferative phases of wound healing. The patient demographic was predominantly female, with 55% of the cohort identifying as such, and the mean age was 52 years. Most patients were born in Australia, with English being the preferred language for 95% of the group.

Developing the dataset required manual annotation of clinical notes to create a ground truth for training and evaluating the models. Two co-authors with expert domain knowledge developed an annotation schema by reviewing clinical notes and refining the categories. The schema classified documents into four classes: complications during a planned visit, complications during an unplanned visit, no complications, and unknown. An unplanned visit was defined as an unscheduled visit initiated by the patient, while a planned visit was a scheduled appointment made by the practitioner. The annotators relied on clinical judgment to determine the presence of complications, synthesizing information about patient symptoms, examination findings, and clinical management, as there is no consensus on standard definitions for post-extraction complications.

The annotation process involved labeling all 4,756 return visits using the Label Studio tool. The inter-annotator agreement, measured by Cohen’s kappa, improved from 0.62 in the initial sample to 0.85 in the remaining evaluation set. Disagreements were often attributable to differences in interpreting the practitioner’s clinical notes, such as overlooking a prescription for antibiotics or a dressing placement. To provide sufficient context, the annotation documents included not just the return visit notes but also the extraction procedure codes, the FDI notation of extracted teeth, and clinical notes from the original extraction visit.

The research team tested four pre-trained language models and a bag-of-words logistic regression model. The pre-trained models included BERT large, RoBERTa, BioBERT, and Bio+Clinical BERT. These models were fine-tuned on the annotated corpus to perform binary classification, distinguishing between unplanned returns due to complications and all other reasons for return. The logistic regression model used bag-of-words vectors, with experiments conducted on tokenization, lemmatization, and the inclusion of structured demographic and medical history data. The inclusion of structured data did not significantly improve the performance of the logistic regression model compared to using text data alone.

The results indicated that the pre-trained language models outperformed the logistic regression model. The best-performing pre-trained language model achieved an F1-score of 0.82 for the class of unplanned returns due to complications. In contrast, the bag-of-words logistic regression model achieved an F1-score of 0.71. Statistical testing using McNemar’s test confirmed that the difference in performance between the logistic regression model and the pre-trained language models was significant, with p-values ranging from 0.000 to 0.002. However, there was no statistically significant difference in performance among the four pre-trained language models themselves, with their mean F1-scores ranging from 0.79 to 0.82.

To understand why the models made their predictions, the researchers conducted a post-hoc explainability analysis using SHapley Additive exPlanations, or SHAP, values. For the logistic regression models, the words with the highest absolute coefficient values were associated with the diagnosis and treatment of alveolar osteitis, commonly known as dry socket. Terms such as “dry,” “socket,” “saline,” “pain,” and “placed” were highly predictive of an unplanned return due to complications. This aligns with clinical practice, where patients often return with pain, and the treatment involves saline irrigation and placement of an intra-socket dressing. The SHAP analysis for the RoBERTa models also highlighted tokens related to the absence or presence of post-operative infections, such as “swelling” and clusters indicating the lack of purulent discharge.

An error analysis revealed that the pre-trained language models frequently made the same prediction errors on specific documents, whereas the logistic regression model made different errors. The manual analysis of errors made by at least three of the four pre-trained models showed that many misclassifications were due to heterogeneous clinical documentation. For instance, some documents described post-operative infections that were treated by a general practitioner or in a hospital emergency department between the extraction and the return visit, which confused the models. Other errors were related to low-incidence complications like paraesthesia or retained root fragments, where the training data was limited. The study also noted that annotation labeling errors contributed to some model mistakes, highlighting the challenges of interpreting ambiguous clinical notes.

The study has several limitations that must be considered when interpreting the results. Only a subset of the annotated corpus was double-annotated, and the error analysis revealed labeling errors that could have been mitigated with more rigorous annotation processes. The results may not generalize to datasets from other public dental hospitals or private practices due to variations in clinical management and documentation practices among different dental practitioners and institutions. Additionally, the explainability of the pre-trained language models using SHAP-based methods was limited, as many tokens with high SHAP values were not clinically interpretable. The uniqueness of dental terminology and abbreviations may also have affected the performance of models pre-trained on general biomedical data.

Despite these limitations, the study demonstrates that natural language processing methods can be used to distinguish unplanned returns due to complications from dental clinical notes. The consistency in documentation of return visit notes was sufficient for multiple NLP approaches to effectively classify the visits. The comparable performance of the different pre-trained language models suggests that factors other than accuracy, such as interpretability and resource requirements, may be prioritized for deployment in clinical settings. Standardized diagnostic codes would be an efficient way to capture common complications, but most electronic dental record systems do not currently record structured diagnostic data. Therefore, NLP methods may remain relevant for accessing this information at scale.

The findings suggest that dental clinical notes could be used to improve measures of unplanned returns as a quality-of-care indicator. By identifying which returns are due to complications, dental practices can better monitor preventable issues and drive improvements in patient outcomes and clinical care efficiency. The ability to distinguish between different types of return visits enhances the fidelity of quality-of-care indicators, allowing for more targeted interventions and accountability. This approach could contribute to better surveillance of post-operative complications and support the development of risk prediction tools for early interception and targeted treatment in dental practice.

Subject of Research: Natural language processing for classifying dental extraction complications

Article Title: Natural language processing methods to classify unplanned returns following dental extractions

Article References: Pethani, F., Kummerfeld, J. K., Yaacoub, A., Dai, X., Conway, M., Kumar, T., Karimi, S., Spallek, H., & Dunn, A. G. (2026). Natural language processing methods to classify unplanned returns following dental extractions. BMC Digital Health, 4(1), Article 47. https://doi.org/10.1186/s44247-026-00273-w

Image Credits: AI Generated

DOI: 10.1186/s44247-026-00273-w

Keywords: natural language processing, dental extractions, clinical notes, quality of care, machine learning, pre-trained language models, alveolar osteitis, electronic dental records, post-operative complications, health informatics, Natural, language