Computer Vision Framework Reads Real-Time Indian Sign Language Finger-Spelling

A lightweight computer vision framework combining real-time object detection with sequential letter reconstruction can decipher finger-spelled Indian Sign Language words from live video feeds at fifteen to twenty frames per second. Developed by researchers Nishadha S. G and Dr. Murugeswari R, the system couples a compact YOLOv8 neural network with an automated temporal word-building algorithm to convert raw visual hand gestures into structured words with an overall latency of less than one hundred milliseconds. Operating across thirty-six distinct visual classes that encompass alphanumeric digits zero through nine and English alphabet characters A through Z, the detection model achieved an impressive 99.74 percent precision, 98.43 percent recall, and a mean average precision of 99.40 percent across standard validation splits. Crucially, the researchers emphasize that their architecture specifically targets character-level finger-spelling rather than continuous, sentence-level signing, addressing the fundamental step of transcribing technical terminology, proper nouns, and unfamiliar vocabulary.

Sign language recognition represents an essential bridge between deaf and hearing communities, addressing communicative barriers highlighted by World Health Organization estimates indicating that over five percent of the global population lives with disabling hearing loss. Within India, between five and eight million individuals rely on Indian Sign Language as their primary medium of daily communication, yet computational recognition tools for this language have historically lagged behind resources developed for American Sign Language. Indian Sign Language presents pronounced linguistic hurdles due to extensive regional variations, subtle manual gesture geometries, and a severe shortage of standardized, publicly available visual benchmarks. Early recognition attempts depended heavily on classical hand-crafted feature descriptors such as Histograms of Oriented Gradients and Scale-Invariant Feature Transforms paired with traditional classifiers, but these approaches repeatedly proved brittle when confronted with natural variations in hand morphology, movement speed, and dynamic ambient environments.

To overcome systemic data scarcity, the research team constructed a balanced fifty-thousand-image corpus by combining thirty-five thousand images from an established public Kaggle repository with fifteen thousand frames recorded from custom laboratory sessions. The six total contributors comprised four female and two male signers aged sixteen to fifty-three, each possessing one to two years of active signing experience. Captures took place under controlled indoor conditions using a Logitech C920 high-definition webcam operating at thirty frames per second under regulated illumination of five hundred to seven hundred lux against a neutral gray backdrop. All bounding boxes and alphanumeric labels were manually validated by two independent annotators who maintained a ninety-five percent inter-annotator agreement rate.

Addressing the common pitfall of dataset distortion during synthetic expansion, the authors introduced a size-aware tiered data augmentation protocol that modulated transformation severity according to source volume. Larger data repositories exceeding thirty thousand frames, such as the Kaggle baseline, received moderate geometric and color perturbations with a constrained transformation probability of thirty percent to avert over-augmentation. Conversely, smaller custom recording pools containing fewer than thirty thousand samples underwent aggressive transformations with application probabilities scaled between sixty and ninety percent to maximize behavioral diversity without introducing structural artifacts. To guarantee experimental integrity, the team enforced a strict image-identifier tracking protocol that prevented synthetic derivatives of any training sample from leaking into validation sets, thereby preserving a pristine eighty-twenty training and validation division that maintained approximately one thousand three hundred eighty-nine samples for every alphanumeric class.

Model training centered on the lightweight YOLOv8 nano configuration, chosen alongside the small variant for its balance of high spatial localization accuracy and computational thriftiness. Built upon a CSPDarknet53 feature backbone supported by feature pyramid networks and path aggregation networks, the three-million-parameter nano architecture processed images rescaled to six hundred forty by six hundred forty pixels. Optimization proceeded over one hundred epochs using the AdamW algorithm paired with automatic learning rate scheduling, initiating at a base rate of 0.01 following three initial warm-up epochs. While the comparative small variant triggered early stopping due to plateauing validation loss, the nano variant completed all cycles, reaching near-peak performance convergence as early as the fifth epoch. The resulting validation metrics confirmed minimal false-positive and false-negative classifications across all thirty-six categories, showing only minor classification ambiguities between visually analogous manual postures such as the letter pairs A and E or M and N.

Converting isolated frame-level detections into coherent written text presented substantial temporal challenges, particularly regarding the handling of brief transition states between characters and the accurate transcription of double letters. Rather than relying on simple frame-by-frame majority voting or unweighted temporal smoothing, the authors formulated an algorithmic state machine governed by parameterized consecutive frame thresholds. The procedure establishes a single-letter confirmation threshold requiring three uninterrupted detections before appending a character, whereas identifying intentional double letters demands six consecutive detections under sustained holding postures. This temporal logic successfully filters fleeting misclassifications caused by rapid hand repositioning while accurately transcribing repeated letters in vocabulary words such as ball, tree, or book without requiring complex external linguistic dictionaries or heavyweight recurrent temporal networks that would otherwise compromise real-time responsiveness.

Operational efficiency was maximized through a systematic multi-parameter optimization grid search designed to strike an ideal trade-off between character accuracy and pipeline execution speed. The governing objective function assigned a seventy percent weight to overall detection accuracy and a thirty percent weight to processing speed, systematically testing variations in confidence thresholds, minimum detections per character, frame-skip rates, segment durations, and merge gaps. The optimal operational operating point settled on a confidence threshold of 0.15 combined with a frame skip factor of two, allowing the complete pipeline to sustain throughput rates of fifteen to twenty frames per second. Granular latency profiling revealed that the entire inference cycle consumes approximately one hundred milliseconds per frame, distributed across fifteen milliseconds for intelligent frame extraction, forty-five milliseconds for YOLO forward-pass detection, twenty milliseconds for geometric post-processing, and twenty milliseconds for sequential word building.

Real-world video capture routinely suffers from hardware differences, uneven illumination, and environmental blur, which prompted the design of an adaptive visual enhancement preprocessor preceding neural network inference. The pipeline assesses incoming frames across four objective quality criteria comprising brightness scores, contrast metrics, Laplacian variance sharpness, and high-frequency noise deviations. If measured brightness drops below a calibrated threshold of one hundred, the system executes Contrast Limited Adaptive Histogram Equalization, while low contrast triggers corrective gamma adjustments. Similarly, blurred inputs falling below a sharpness value of fifty receive unsharp masking, and noisy frames exceeding an established variance limit undergo targeted denoising filters. Cross-user evaluation validated this adaptive preprocessing mechanism, showing that raw video inputs yielding an average accuracy of 81.5 percent were elevated to 99.4 percent accuracy once normalized through the automated image correction pipeline.

Despite outstanding performance on validation data drawn from known signers, the researchers uncovered a pronounced performance discrepancy when subjecting the system to strict signer-disjoint testing. Evaluating forty-five dedicated word-spelling test videos performed exclusively by two unseen signers, Diya and Biju, revealed that overall word-level accuracy dropped to 67.78 percent, with individual recognition rates fluctuating widely from 48.89 percent for Diya to 86.67 percent for Biju. The authors attribute this substantial accuracy decline to severe domain shifts stemming from physiological variations, including distinct hand geometries, finger proportions, and skin pigmentation differences, alongside personalized signing cadences varying from 0.5 to 2.0 seconds per character. This stark gap underscores that while high validation scores demonstrate technical feasibility, standard object-detection models inherently risk memorizing signer-specific nuances rather than abstracting purely signer-invariant grammatical features.

Computational scalability and hardware feasibility were rigorously documented across multiple desktop and edge deployment environments. Training the primary YOLOv8 nano model required 14.5 hours on an Ubuntu workstation equipped with an AMD Ryzen 9 5950X processor and an NVIDIA GeForce RTX 3090 graphics card, averaging 8.7 minutes per epoch with a peak graphics memory footprint of 4.2 gigabytes. Edge inference benchmarking performed on an NVIDIA RTX 3060 demonstrated responsive scaling across batch sizes and resolutions, progressing from 15.4 frames per second at single-item batches up to 131.1 frames per second under batch sizes of eight. Statistical reliability was substantiated through three independent experimental runs seeded at 42, 123, and 456, yielding consistent ninety-five percent confidence intervals. Furthermore, paired t-tests confirmed that minor performance variations between model configurations remained statistically non-significant, reinforcing the experimental consistency of the underlying training framework.

Ablation investigations confirmed the tangible performance contributions of each modular component, revealing that tiered augmentation improved baseline precision by 1.24 percent over uniform methods, while the temporal word-construction logic increased precision by 1.64 percent and recall by 1.63 percent. Nevertheless, the authors candidly delineate significant constraints, pointing out that an evaluation cohort of merely two unseen signers cannot substantiate generalized real-world deployment across broader populations. Furthermore, the system remains strictly confined to isolated finger-spelling rather than continuous, full-body sign language discourse incorporating facial expressions, torso orientation, and complex grammatical co-articulation. Future development aims to incorporate multimodal sensor feeds such as optical flow and skeletal pose estimations, while exploring transformer-driven temporal architectures and few-shot domain adaptation algorithms to bridge the persistent performance divide between familiar laboratory contributors and novel everyday signers.

Subject of Research: Real-time finger-spelled Indian Sign Language recognition using object detection and temporal word construction

Article Title: Real-time Indian Sign Language word recognition through visual detection and temporal word construction

Article References: S. G, N., & R, D. M. (2026). Real-time Indian Sign Language word recognition through visual detection and temporal word construction. Discover Artificial Intelligence, 6(1), Article 1857. https://doi.org/10.1007/s44163-026-02502-7

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02502-7

Keywords: Indian Sign Language, computer vision, object detection, YOLOv8, finger-spelling, temporal word construction, assistive technology, deep learning, data augmentation, cross-signer generalization, video preprocessing, real-time processing, human-computer interaction, machine learning, gesture recognition, sign language recognition, CLAHE enhancement