Local AI Tutor Aces Physics Exam but Stumbles in the Classroom

A locally deployed artificial intelligence model can answer introductory physics questions with impressive accuracy, yet it may still fall short as a real classroom tutor. That is the central lesson of a new open-access study published in Discover Artificial Intelligence, in which a team of German researchers put Gemma 3 27B, a compact open-weight language model, through a two-stage stress test. First, they measured how well the model handled a fresh multilingual benchmark of first-year physics questions. Then they let the same model loose as a tutoring chatbot for first-semester engineering students working through a mock exam. The gap between the two results is striking, and it carries a warning for universities hoping that benchmark scores alone can certify an AI system as educationally reliable.

The research was motivated by a practical question facing higher education. Large language models have already woven themselves into student life, with surveys showing that a large majority of university students use generative AI tools beyond formal coursework. Commercial systems such as ChatGPT offer strong performance, but they raise concerns about data privacy, cost, and institutional control. Locally deployable models promise an alternative: a university can run them on its own hardware, keep student conversations off third-party servers, and fix the model version. The catch is that the capabilities of these smaller models, especially in authentic learning interactions rather than tidy one-shot tests, have remained poorly characterized. The team, led by Marcel Völschow of Hamburg University of Applied Sciences together with colleagues at DESY and Helmholtz-Zentrum Dresden-Rossendorf, set out to close that evidence gap.

To measure domain competence, the researchers built mlphys101, a new multiple-choice benchmark of introductory physics questions drawn, with permission, from a publicly available Physics 101 question bank. After removing image-dependent and duplicate items, 731 questions remained. Crucially, the team sorted every question into one of five difficulty categories: replication of definitions, replication of physical facts, conceptual physics and qualitative reasoning, single-step quantitative reasoning, and multi-step quantitative reasoning. This taxonomy matters because it separates what a model merely memorized from what it can actually apply. The questions were translated into German using GPT-4 Turbo and reviewed by a trained physicist and native speaker, and translations into Italian, Polish, and Spanish were also prepared to support future multilingual comparisons.

The model under examination, Gemma 3 27B, was chosen as the best compromise between capability and feasibility for local inference on a 48-gigabyte GPU memory budget. The researchers tested three quantization variants, labeled Q4_K_M, Q6_K, and Q8_0, which compress the model’s numerical weights to different precisions to trade size and speed against quality. Each variant sat for sixteen full exams with different random seeds, served by the llama.cpp inference engine with a 32,768-token context buffer. The prompt instructed the model to think step by step, explain its reasoning in German, and end with a keyword followed by the letter of the correct option. The team deliberately avoided forcing the model into a structured output format, because recent work has shown that format restrictions can impose a substantial accuracy penalty on open-weight models, degrading reasoning even when parsing becomes easier.

The benchmark results were strong. Across all runs and difficulty levels, the model achieved accuracies between 84 and 98 percent. It performed best and most consistently on definitions and single-step quantitative problems, with medians in the mid to high 90 percent range, and dipped to around 90 percent on conceptual questions. Multi-step quantitative reasoning proved the hardest and least consistent category, showing the largest variability across runs. Surprisingly, the quantization level made little difference: higher precision did not yield a uniform improvement, suggesting that even a compressed version of the model retains most of its physics competence. Of more than 34,000 generated responses, the evaluation pipeline failed to extract an answer in only a single case, in which the model concluded that none of the five options was correct.

To put these numbers in context, the researchers also ran the same benchmark against a commercial cloud baseline, OpenAI’s GPT-5.6-Luna. The frontier model achieved consistently near-ceiling accuracy across all five categories, ranging from 0.96 to 0.99, with little variation between repeated runs. The comparison showed that the remaining gap between the local and commercial models was concentrated precisely where tutoring matters most: conceptual understanding and multi-step reasoning. On definitions and straightforward formula application, the compact local model was nearly indistinguishable from its cloud-based rival. But the study’s most important finding came next, when the benchmark champion was asked to do something far messier than answering multiple-choice questions.

In the classroom field experiment, the team deployed Gemma 3 27B locally on a university workstation and exposed it to students through a ChatGPT-like web interface built with Gradio. Thirty-two first-semester electrical and information engineering students took a physics mock exam in which the chatbot, nicknamed Emmy, was their only permitted aid besides a calculator. The system prompt cast the model as an experienced physics teacher who gives hints step by step, reveals full solutions only on request, and remains patient and friendly. Conversations were not saved, protecting student privacy, though students could flag particularly good or bad answers. Afterward, seventeen students completed a detailed questionnaire about their experience.

The survey revealed a sobering picture. Overall helpfulness received a mean rating of only 2.7 out of 5, even though the comprehensibility of explanations scored notably higher at 3.4 and support for understanding fundamental concepts reached 3.5. Fourteen of the seventeen respondents said they had to reformulate their prompts at least once to get a satisfactory answer, and nine had to do so more than once. Fewer than half of the submissions unambiguously agreed that the chatbot understood their questions. Free-text comments pointed to incorrect numerical values and formulas, including one cited error involving the cross-sectional area of a sphere, and to the model losing track of quantities provided earlier in the conversation. Response speed ranked dead last among the system’s features, with fourteen of seventeen students placing it last, reflecting inference delays when many students used the single server simultaneously.

The contrast between the two halves of the study is the paper’s real contribution. On a controlled, single-turn benchmark, the 27-billion-parameter model demonstrated substantial knowledge of introductory physics, approaching commercial performance on many task types. In a genuine multi-turn tutoring setting, that competence did not translate into reliable behavior. Tutoring, the authors argue, demands capabilities that benchmarks never test: interpreting incomplete or conversational student prompts, retaining information across turns, calibrating the level of assistance, and producing explanations that are both physically correct and pedagogically useful. A system that misreads a question, misses a misconception, or offers a plausible but wrong intermediate step may reinforce the very difficulties it is meant to resolve. High benchmark accuracy, in other words, is a necessary but insufficient indicator of tutoring suitability.

The authors are careful about the limits of their work. The field experiment involved a small, single-cohort sample, relied on self-reported perceptions rather than measured learning gains, and deliberately created a worst-case scenario in which students had no other assistance. The benchmark questions were publicly available and could in principle appear in training data, and the reported accuracies measure answer correctness rather than the coherence of the underlying reasoning. Even so, the practical message for universities is clear. Locally deployed language models can genuinely support selected introductory physics activities, offering institutional control over data and infrastructure that commercial services cannot match. But before any model, local or cloud-based, is entrusted with real students, it should be evaluated not only on domain benchmarks but under realistic, multi-turn interaction conditions. Under the conditions studied here, Gemma 3 27B is better described as a potentially useful supportive tool than as a standalone tutor, and the benchmark-to-classroom gap it exposed is one that every educational AI deployment should be tested for.

Subject of Research: Evaluation of a locally deployed large language model for undergraduate physics tutoring using a new benchmark and a classroom field experiment

Article Title: Evaluating Gemma 3 27B for local undergraduate physics tutoring

Article References: Völschow, M., Buczek, P., Riefer, P., Jasko, A., & Steinbach, P. (2026). Evaluating Gemma 3 27B for local undergraduate physics tutoring. Discover Artificial Intelligence, 6(1), Article 1392. https://doi.org/10.1007/s44163-026-02296-8

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02296-8

Keywords: large language models, Gemma 3 27B, physics education, AI tutoring, benchmark evaluation, mlphys101, quantization, local deployment, classroom field experiment, educational technology, chain-of-thought prompting, data privacy