Lightweight hybrid CNN-transformer model detects traffic lights and signs in one pass

A new open-access study in Discover Robotics reports a single neural network that can simultaneously detect traffic lights, detect traffic signs, and recognize the state of a traffic light, while remaining light enough for real-time use. The framework, called YOLOv11–EfficientViT, was developed by Hemant Kumar, Pushpa Mamoria, and Deepak Kumar Dewangan and published on 12 October 2026. On a curated subset of the BDD100K driving dataset, the model reached 99.4% mean average precision at an IoU threshold of 0.5, 92.2% at the stricter 0.5:0.95 range, and 97.7% traffic light state recognition accuracy, with a measured model-level inference time of 3.8 milliseconds per frame.

The motivation comes from a long-standing split in the field. Most existing systems treat traffic light detection, traffic sign detection, and state recognition as separate tasks, often using multi-stage pipelines or independent models. Convolutional neural networks, including the popular YOLO family of one-stage detectors, offer a strong balance between accuracy and speed, but their local receptive fields limit their ability to capture global context. Transformer-based approaches can model long-range dependencies through self-attention, yet they typically carry a high computational cost that makes deployment on resource-constrained vehicles difficult. The authors argue that no existing end-to-end architecture is simultaneously lightweight, unified, and capable of balancing local feature extraction with global contextual reasoning.

The proposed architecture builds on YOLOv11, chosen as the baseline detector for its balance of accuracy, inference speed, and computational efficiency. Two additions define the framework. First, convolutional block attention modules, known as CBAM, are integrated into the backbone and neck of the network. CBAM applies channel attention, which identifies the most informative feature channels using pooled descriptors passed through a shared multilayer perceptron, followed by spatial attention, which emphasizes the most important image regions. This sequential refinement helps the network focus on small, visually similar, or partially occluded traffic elements while suppressing background clutter.

Second, an EfficientViT module captures global contextual dependencies at reduced computational cost. Feature maps from the backbone are reshaped into flattened patches with learned positional encodings, then projected into query, key, and value representations. Instead of conventional softmax-based attention, the module uses a ReLU-based similarity function, and parallel branches with depthwise convolutions of different kernel sizes gather neighborhood information at multiple spatial scales. The attention outputs from all branches are fused by element-wise summation and passed through a linear projection and an MBConv block, which combines depthwise separable convolution with a squeeze-and-excitation module and a residual connection for efficient local feature extraction.

Training uses a multi-task loss that combines a shape-aware intersection over union loss for bounding box regression, a binary cross-entropy classification loss, and a distribution focal loss for finer localization. Notably, the state labels of traffic lights are folded into the classification objective of the traffic-light prediction branch rather than handled by a separate loss term, so detection and state recognition are optimized jointly. The authors also note that this journal study extends their earlier conference work presented at the 2025 IEEE PuneCon, which had used a detection-only YOLOv11 framework with CBAM and SIoU loss; the present version adds the EfficientViT module, the unified multi-task formulation, and much broader experiments.

The experiments relied on curated, task-focused subsets rather than full benchmarks. From BDD100K, the team extracted 2,047 images containing traffic light annotations restricted to the three canonical states of red, yellow, and green, split 70/15/15 into training, validation, and test sets. From TT100K, they extracted 2,093 images with annotated traffic signs across 21 categories, split 70/20/10. All images were resized to 640 by 640 pixels, augmented with horizontal flipping, scaling, resizing, and brightness adjustment, and trained for 200 epochs with an initial learning rate of 0.0001. The model requires approximately 8.9 GFLOPS at that resolution and has only 2.5 million parameters. On the TT100K subset, the framework attained 96.5% mAP@0.5 and 81.2% mAP@0.5:0.95 with a reported inference time of 2.4 milliseconds.

An ablation study on the BDD100K subset traced the contribution of each component. The baseline YOLOv11 achieved 96.8% mAP@0.5 and 88.5% mAP@0.5:0.95. Adding CBAM raised these to 97.9% and 89.9%, while adding EfficientViT alone reached 98.2% and 90.7%. Combining both modules yielded 98.9% and 91.6%, and the full joint-learning configuration reached 99.4% and 92.2%, an overall improvement of 2.6 and 3.7 percentage points over the baseline. State recognition accuracy rose from 89.6% for the baseline to 97.7%, a gain of 8.1 percentage points. The authors interpret the larger gains from combining modules as evidence that local attention refinement and global contextual modeling provide complementary rather than redundant benefits, while cautioning that the small variations in inference time across configurations are empirical measurements rather than evidence that the additions reduce computational complexity.

Cross-dataset testing probed generalization under domain shift. A model trained on the curated BDD100K subset was evaluated without any fine-tuning on a 2,364-image subset of the LISA dataset, and a model trained on the TT100K subset was evaluated on a 1,493-image subset of the Bosch dataset. The cross-dataset runs used a 100-epoch source-domain training configuration, shorter than the 200-epoch in-domain setting, a difference the authors say should be considered when comparing the results. Under domain shift, mAP@0.5 fell only slightly, from 99.4% to 98.5% for traffic lights, but mAP@0.5:0.95 dropped substantially, from 92.2% to 67.7%, indicating that precise localization is considerably more sensitive to domain differences than coarse detection. The authors attribute the larger strict-threshold drop to differences in image characteristics, scene composition, annotation styles, and environmental conditions between datasets.

The study also quantified detection-state consistency, counting a mismatch when a correctly localized traffic light was assigned the wrong signal state. The full jointly trained model showed the lowest mismatch rate at 2.3%, a relative decrease of about 78% compared with the baseline detector, and repeating the analysis across three independent training runs with different random seeds produced standard deviations of 0.3 to 0.6 percentage points. However, the confusion analysis revealed a limitation: the red and yellow states showed higher background confusion, at 0.50, than the green state, at 0.03, which the authors link to the small apparent size of distant lights, low contrast, glare, partial occlusion, and complex urban backgrounds. Because incorrect state predictions may have serious consequences in safety-critical driving, the authors stress that the reported state-recognition performance applies only to the evaluated conditions.

The paper is candid about scope. Comparisons with other published methods were not run under a unified experimental setup, since figures were taken from original publications that differ in dataset scale, splits, and training settings, so the comparisons are indicative rather than direct head-to-head benchmarks. The evaluation used curated subsets rather than complete benchmark splits, covered only three traffic light states and did not test arrows or flashing signals, involved no domain adaptation techniques, and included no formal statistical significance testing for most experiments. Failure cases clustered around state ambiguity under glare and reflections, and around very small or low-contrast traffic elements occupying few pixels. The authors conclude that the hybrid design represents a meaningful step toward unified, deployable traffic perception for autonomous vehicle systems, while emphasizing that the results should not be read as evidence of full robustness under all real-world driving conditions or readiness for safety-critical deployment, and that future work will address temporal modeling, severe weather, full-scale benchmarks, and statistical validation.

Subject of Research: A lightweight hybrid CNN-transformer framework for joint traffic light and traffic sign detection with traffic light state recognition

Article Title: Joint detection and state recognition of traffic lights and signs through a lightweight hybrid CNN–transformer framework

Article References: Kumar, H., Mamoria, P., & Dewangan, D. K. (2026). Joint detection and state recognition of traffic lights and signs through a lightweight hybrid CNN–transformer framework. Discover Robotics, 2(1), Article 34. https://doi.org/10.1007/s44430-026-00049-x

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00049-x

Keywords: autonomous vehicles, traffic light detection, traffic sign detection, state recognition, YOLOv11, EfficientViT, CBAM, computer vision, deep learning, BDD100K, TT100K, real-time inference