Evaluating the performance of AI-based large language models in radiation oncology

A new study evaluates an artificial intelligence (AI)-based algorithm for autocontouring prior to radiotherapy in head and neck cancer. Manual contouring to pinpoint the area of treatment requires significant time, and an AI algorithm to enable autocontouring has been introduced.  The study is published in the peer-reviewed journal AI in Precision Oncology. Click here to read the article now. 

Nikhil Thaker, from Capital Health and Bayta Systems, and coauthors, evaluated the performance of various LLMs, including OpenAI’s GPT-3.5-turbo, GPT-4, GPT-4-turbo, Meta’s Llama-2 models, and Google’s PaLM-2-text-bison.The LLMs were given an exam comprised of 300 questions, and the answers were compared to Radiation Oncology trainee performance.

The results showed that OpenAI’s GPT-4-turbo had the best performance, with 74.2% correct answers, and all three Llama-2 models under-performed. The LLMs tended to excel in the area of statistics, but to underperform in clinical areas, with the exception of GPT-turbo, which performed comparably to upper-level radiation oncology trainees and superiorly to lower-level trainees. 

“Future research will need to evaluate the performance of models that are fine-tune trained in clinical oncology,” concluded the investigators. “This study also underscores the need for rigorous validation of LLM-generated information against established medical literature and expert consensus, necessitating expert oversight in their application in medical education and practice.”

“The study highlights the potential of generative AI to revolutionize radiation oncology education and practice. OpenAI’s GPT-4-turbo demonstrates that AI can complement medical training, suggesting a future where AI aids in improving patient outcomes. It’s essential, though, to validate these technologies rigorously and involve experts to ensure their reliable and effective use in healthcare,” says Douglas Flora, MD, Editor-in-Chief of AI in Precision Oncology. 

About the Journal
AI in Precision Oncology is the only peer-reviewed journal dedicated to the advancement of artificial intelligence applications in clinical and precision oncology. Spearheaded by Editor-in-Chief Douglas Flora, MD and supported by a diverse and accomplished team of international experts, the Journal provides a high-profile forum for cutting-edge research and frontmatter highlighting important research and industry-related advances rapidly developing within the field. For complete information, visit the AI in Precision Oncology website.

About the Publisher
Mary Ann Liebert, Inc. is a global media company dedicated to creating, curating, and delivering impactful peer-reviewed research and authoritative content services to advance the fields of biotechnology and the life sciences, specialized clinical medicine, and public health and policy. For complete information, please visit the Mary Ann Liebert, Inc. website.