Automated handwritten-answer framework shows 87.4% agreement for text and 85.1% for figures in small validation study

A computer-based framework for evaluating handwritten answer scripts containing text and figures achieved 87.4% mean agreement with examiner marks for textual responses and 85.1% for drawings in separate validation sets of 25 responses each. The study was conducted with undergraduate answer sheets from Noakhali Science and Technology University and evaluated one programming question and one electronics drawing question, so the results provide limited evidence for use across subjects, questions and institutions.

The framework combines OCR-based digitization with several scoring components. Google Docs OCR converted scanned handwriting into text, with the researchers reporting approximately 90% word-level accuracy on legible scripts. Residual recognition errors were manually corrected before scoring, meaning the reported results reflect the downstream evaluation methods rather than performance from a fully automated OCR-to-grade pipeline. The authors describe recognition as an interchangeable front-end that could later use a stronger recognizer.

For text evaluation, the researchers tested two cases on 25 responses to a programming question. The first averaged a writing-quality score, keyword matching, TF-IDF-based cosine similarity and a verbatim reference-overlap measure, producing 83.8% agreement with examiner marks. The second retained three components but replaced TF-IDF cosine similarity with semantic similarity based on the spaCy en_core_web_lg word embeddings, increasing agreement to 87.4%. The improvement was associated with a reduction in paired absolute errors: a Wilcoxon signed-rank test gave p = 0.007, while a paired t-test gave t(24) = 2.75 and p = 0.011.

The semantic approach represented each response as the average of its 300-dimensional GloVe token vectors, whereas the TF-IDF method represented distinct word forms as separate dimensions. In an illustrative response that the examiner awarded 5 out of 5, the semantic method assigned higher similarity because the wording differed from the reference. For the full 25-response text set, the second text case had quadratic weighted kappa of 0.82 and Pearson correlation of 0.90 with examiner marks. These metrics indicate agreement within this limited test, not independent confirmation of broad grading accuracy.

Figure evaluation compared student drawings with a reference image using two methods. The first computed cosine similarity between 1,280-dimensional feature embeddings produced by a pretrained MobileNetV2 network after images were resized to 224 by 224 pixels. The second used Canny edge detection, contour extraction and Hu moments, then calculated Euclidean distance between the descriptors. On 25 electronics drawing responses, the MobileNetV2 cosine-similarity method achieved 85.1% mean agreement, compared with 81.1% for the Hu-moment approach. The 4.0-percentage-point difference did not reach statistical significance in a paired t-test, p = 0.28, although the deep-feature method performed better across the reported error, correlation and kappa measures.

The authors observed that the Hu-moment method was sensitive to alignment, scaling, stroke thickness and scan contrast, which could penalize a visually similar drawing after minor changes in position or proportion. MobileNetV2 embeddings were comparatively less sensitive to those variations. Neither method, however, explicitly verifies the meaning of a diagram or its component relationships. In an example described by the researchers, a clean but conceptually incorrect circuit received 2.4 from cosine similarity and 4.0 from Euclidean distance despite an examiner mark of 1, showing that appearance-based comparisons may reward a well-drawn but incorrect figure.

The study used 100 handwritten scripts collected from students aged 18 or older. Fifty scripts formed the two validation sets: 25 text responses and 25 figure responses, each marked by the relevant course examiner. The remaining scripts supported workflow development and pilot calibration of scoring thresholds and were excluded from the reported validation statistics. The score bands were calibrated heuristically to convert raw component scores into marks out of five. Pilot observations showed that the techniques had different score distributions: TF-IDF similarity for correct answers ranged from 3% to 45%, keyword matching from 8% to 82%, reference overlap from 2% to 17%, and NLP-based similarity from 81% to 92%.

Timing measurements on a standard x86 CPU without GPU acceleration indicated that keyword matching took about 0.1 millisecond per response, TF-IDF similarity 1.6 milliseconds and reference-overlap computation 0.2 milliseconds. The full Euclidean figure pipeline required about 28 milliseconds per image, while spaCy document-vector similarity and MobileNetV2 embedding each took roughly one second per response on CPU. The authors reported that 100 digitized scripts could be evaluated in well under a minute after model loading, although the semi-automated workflow was limited by OCR and manual verification.

Several limitations constrain the findings. Each modality was tested on one question in one course and graded by one examiner, preventing assessment of inter-rater agreement and leaving question-specific effects unresolved. The validation sets contained only 25 responses each, and the scoring thresholds were calibrated on scripts from the same cohort, which may make the observed agreement optimistic. Manual correction of OCR errors also means a fully automated implementation would inherit recognition errors, so the reported agreement should be treated as an upper bound under the evaluated conditions. The authors characterize the framework as assessment support requiring examiner oversight, transparent component scores and an appeal process, rather than as a replacement for human graders.

Subject of Research: Validation of an automated framework for scoring handwritten textual and figure-based student responses

Article Title: AI powered handwritten answer script evaluation framework for automated assessment of text and figure based responses

Article References: Amin, M. N., Khan, M. A. R., & Ahammad, I. (2026). AI powered handwritten answer script evaluation framework for automated assessment of text and figure based responses. Discover Artificial Intelligence, 6(1), Article 1263. https://doi.org/10.1007/s44163-026-02181-4

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02181-4

Keywords: automated grading, handwritten responses, OCR, semantic similarity, figure evaluation, MobileNetV2, spaCy, assessment, validation study, powered, handwritten, answer