Audit Shows Gaze Drives Multimodal Engagement Decoders Over Brain Signals

Gaze signals dominate the predictability of consumer engagement in multimodal decoding systems, revealing that complex neural models may rely primarily on eye tracking rather than brain activity when predicting rule-defined engagement scores. An audit evaluating multimodal decoders on the NeuMa dataset demonstrated that eye-tracking features drove prediction performance, while electroencephalography components failed to surpass simpler reference baselines. The investigation examined how machine learning decoders combine electroencephalography, or EEG, and eye tracking, testing whether reported predictive accuracy against a rule-defined label reflects the decoder’s true reliance on brain data or merely reflects behavioral gaze measurements. Across thirty-seven evaluable engagement folds involving forty-two participants, the analysis showed that multimodal architectures did not outperform unimodal gaze models in their primary evaluations.

The audit evaluated the Interpretable Neuro-Symbolic Hybrid Dynamic Graph Spiking Cross-Modal Transformer, designated as INS-HDGS-CMT, as the primary architecture under examination. To evaluate its decoding capability, the researchers benchmarked INS-HDGS-CMT against three standard reference architectures: EEGNet, ShallowConvNet, and an eye-tracking long short-term memory network, known as an LSTM. Every model was tested against an engagement index constructed by the authors of the study, utilizing a leave-one-subject-out cross-validation framework across the forty-two participants. This testing methodology allowed the investigators to establish whether multimodal integration actually improved engagement decoding compared to unimodal baselines and simpler linear probes.

The first major finding demonstrated that gaze features accounted for almost all predictive success on the constructed engagement index. A linear probe trained strictly on the five gaze terms of the index achieved an area under the receiver operating characteristic curve, or ROC-AUC, of 0.92, whereas the unimodal gaze-only LSTM reached an ROC-AUC of 0.89. In comparison, the complex multimodal INS-HDGS-CMT model attained an ROC-AUC of 0.88, meaning that no multimodal architecture in the principal comparison exceeded the performance of the standalone eye-tracking LSTM. Second, in the principal tuned comparison, the tested EEG encoders yielded ROC-AUC scores between 0.49 and 0.64, scoring lower than a linear probe trained solely on the EEG terms, which reached 0.67. Although the EEG encoders struggled with engagement decoding, both EEGNet and ShallowConvNet successfully separated resting states from browsing behavior with ROC-AUC scores between 0.92 and 0.94. Subsequent retesting of selected models under fold-wise relabelling confirmed that the broad separation between EEG and gaze predictability remained intact.

The audit further revealed that high predictive scores did not substantiate the internal architectural mechanisms claimed for the decoder, including its dynamic connectivity, computational efficiency, or interpretability features. When the researchers retrained density-matched topology surrogates to test the contribution of the decoder’s specific graph connectivity, the surrogates produced no detectable loss in predictive performance. Furthermore, the spiking front-end of the INS-HDGS-CMT decoder operated more slowly than an equivalent dense twin architecture, contradicting expected computational efficiency advantages. The investigation also identified that the final engagement prediction was heavily dominated by the bypass classifier, showing that the intricate neuro-symbolic and graph transformation layers exerted minimal practical influence on the final classification decision.

Finally, retrospective analyses of product selection revealed that predictive performance depended heavily on temporal observation windows rather than rapid engagement detection. Session-level dwell probes, which incorporated eye-tracking data from viewing after product selection had already occurred, achieved mean fold ROC-AUC values between 0.88 and 0.91. Conversely, models evaluated on brief five-second epochs achieved a maximum ROC-AUC of only 0.61, indicating that high session scores reflected retrospective dwelling rather than immediate engagement during choice formation. Alongside these four empirical findings, the authors published code accompanying an eight-step audit protocol, providing a structured methodology for researchers to evaluate label dependence and isolate the individual contributions of components within multimodal consumer decoders.

Subject of Research: Audit of multimodal EEG and eye-tracking consumer engagement decoders

Article Title: What does a multimodal EEG–Eye-tracking engagement decoder learn? A leakage-aware audit on the NeuMa dataset

Article References: Priyadharshini, D., & Shridevi, S. (2026). What does a multimodal EEG–Eye-tracking engagement decoder learn? A leakage-aware audit on the NeuMa dataset. Brain Informatics. https://doi.org/10.1186/s40708-026-00343-9

Image Credits: AI Generated

DOI: 10.1186/s40708-026-00343-9

Keywords: electroencephalography, eye tracking, multimodal decoding, consumer engagement, cross-modal transformer, machine learning audit, NeuMa dataset, neuroinformatics, does, multimodal, Eye-tracking, engagement