Frozen CLIP With Tiny Adapters Boosts Low-Light Human Activity Recognition

Recognising what a person is doing in a darkened room is far harder for a machine than recognising the same action in daylight. Standard video models are trained on brightly lit RGB footage, and when they confront infrared or low-light recordings they suffer from a photometric domain shift: contrast drops, colour statistics change, and noise patterns appear that were never present in their pre-training data. A new open-access study in Discover Artificial Intelligence reports that a frozen CLIP vision-language model, kept entirely unchanged during training, can be adapted to this challenging setting with remarkably few extra parameters, delivering measurable accuracy gains at essentially no additional computational cost.

The research team, led by Md. Ashik Khan with collaborators including Abu Saleh Musa Miah, Fahmid Al Farid, Md. Abdur Rahim and Hezerul Abdul Karim, systematically evaluated three deployment regimes built on the same frozen CLIP ViT-B/32 backbone: zero-shot classification with no parameter updates at all, linear probing that trains only a classifier on top of frozen features, and lightweight parameter-efficient adapters that insert small trainable modules while leaving the backbone untouched. The central question was how far pre-trained vision-language representations can be pushed toward low-light human activity recognition without the expensive and generalization-eroding process of full fine-tuning.

The headline result comes from ARID v1.5, a benchmark of 6,207 videos covering 11 action classes captured with infrared cameras in dark environments. With the backbone frozen and features aggregated by simple statistical pooling, linear probing reached 68.37 ± 0.49% accuracy. Adding a temporal adapter with only 88.8K trainable parameters lifted accuracy to 74.38 ± 0.13%, an absolute gain of 6.01 percentage points. LoRA, a low-rank adaptation method applied to the Query and Value projections of the deeper transformer blocks, matched that same accuracy with just 12.7K parameters, seven times fewer than the temporal adapter. Because these adapters were applied only to the last six transformer blocks, they added under 0.2% extra floating-point operations to the frozen backbone’s budget.

Not every lightweight module succeeded, however, and this failure turned out to be one of the study’s most instructive findings. A temporal attention head with a mere 513 learnable parameters, which had performed competitively on standard RGB data, fell below even the parameter-free mean-pooling baseline on ARID v1.5, scoring 66.06 ± 0.91% against 68.37 ± 0.49%. Diagnostic experiments showed the tiny attention module overfitting, with a persistent gap between training and validation accuracy, and its learned frame weights collapsing toward near-uniform averaging or single-frame selection. The authors conclude that minimal learned components become brittle under appearance shift when only a handful of frames, in this case K ≤ 4, are available as input.

On conventional RGB benchmarks the picture was different. On UCF101, which contains 13,320 YouTube videos across 101 action classes, linear probing on frozen CLIP features achieved 87.26 ± 0.50% top-1 accuracy at 17.44 GFLOPs per video. That places the frozen baseline within roughly 6 percentage points of the fully fine-tuned SlowOnly model, which the literature reports at 93.16% but at 27.3 GFLOPs, meaning the frozen approach used 36% fewer computations. The authors are careful to note that such literature comparisons are contextual references rather than controlled head-to-head experiments, since the baselines differ in backbone architecture, pre-training data and temporal input length. On HMDB51, a harder dataset of 6,766 movie and web videos across 51 classes, the frozen pipeline reached 61.00 ± 2.01% under the official three-fold protocol.

Zero-shot classification, in which CLIP’s pre-trained image-text alignment is used directly with action-oriented prompt templates and no supervision, revealed a wide gap. It trailed linear probing by 14.88 percentage points on HMDB51, 19.84 points on UCF101, and a striking 46.11 points on ARID v1.5, where it managed only 22.26 ± 0.30%. The authors attribute this partly to a semantic mismatch between CLIP’s static image-text training objective and the dynamic nature of video actions, and partly to the compounded effect of infrared imagery departing from both the appearance statistics of CLIP’s pre-training data and the text-aligned visual prototypes used for classification. On ARID, zero-shot errors concentrated among semantically similar motion categories, where nearby text embeddings met ambiguous infrared evidence.

Frame sampling and temporal aggregation choices shaped the efficiency profile considerably. Uniform sampling of K frames proved the most principled strategy, particularly for infrared footage, which often lacks the distinct temporal action peaks that centre-based sampling assumes. Accuracy on ARID v1.5 rose from 63.98 ± 0.49% at a single frame to 71.59 ± 1.21% at 32 frames, but returns diminished sharply beyond K = 4. That configuration, at 17.44 GFLOPs and 22.7 videos per second on the authors’ desktop-class test hardware, emerged as the optimal operating point: moving from K = 4 to K = 16 quadrupled the computational cost for only a 3.40 percentage-point gain, indicating that performance is bounded by how frame features are aggregated rather than by how many are sampled.

A clear pattern emerged across datasets: the value of adaptation scales with the degree of domain shift, not with parameter count. On ARID v1.5, where infrared imagery differs substantially from CLIP’s RGB pre-training distribution, both structured adapters delivered the same 6.01-point improvement. On HMDB51, closer to the source distribution, the temporal adapter improved accuracy by only 0.79 points and LoRA showed no improvement. On UCF101, LoRA actually degraded accuracy sharply, by 15.29 points, consistent with prior evidence that fine-tuning can distort well-calibrated representations on in-distribution data. The authors therefore recommend frozen linear probing as the stable default for standard RGB conditions and structured lightweight adaptation specifically under photometric domain shift.

The study is candid about its limits. Frame-level feature extraction followed by pooling does not explicitly model dense motion dynamics, making the approach unsuitable for fine-grained motion analysis or long-term temporal dependencies, and performance plateaus beyond eight frames. The evaluation focused on object-centric action datasets rather than motion-centric benchmarks such as Something-Something V2, and the low-light evidence rests primarily on ARID v1.5’s 11 classes. The method has not been validated for low-latency deployment on edge hardware, cannot handle untrimmed videos or temporal localisation, and the reported latency of 44.1 milliseconds per video characterises the authors’ local desktop hardware only.

Even with those caveats, the work offers a practical template for deploying activity recognition where lighting cannot be controlled, from fall detection to sign language recognition. By demonstrating that a strictly frozen CLIP backbone, untouched by gradients, can serve as the foundation for competitive video action recognition, and that a few tens of thousands of well-chosen adapter parameters can recover much of the accuracy lost to darkness, the study establishes frozen vision-language models with lightweight adaptation as a scalable and efficient framework for recognition under photometric domain shift. The broader message for the field is equally pointed: when the deployment domain diverges from pre-training data, the structure of a small adapter matters far more than its size, and the cheapest reliable component is sometimes the one with no learned parameters at all.

Subject of Research: Frozen CLIP representations with parameter-efficient adapters for low-light human activity recognition

Article Title: Human activity recognition under low-light conditions using frozen CLIP with lightweight adaptation

Article References: Khan, M. A., Miah, A. S. M., Al Farid, F., Rahim, M. A., & Karim, H. A. (2026). Human activity recognition under low-light conditions using frozen CLIP with lightweight adaptation. Discover Artificial Intelligence, 6(1), Article 1006. https://doi.org/10.1007/s44163-026-02020-6

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02020-6

Keywords: human activity recognition, CLIP, low-light vision, parameter-efficient adaptation, LoRA, temporal adapters, zero-shot learning, domain shift, infrared video, computer vision, linear probing, computational efficiency