ResNet50 Meets Transformer to Push Facial Emotion Recognition to 89.7 Percent on FER2013

A hybrid deep learning model that pairs a ResNet50 convolutional backbone with a Transformer encoder has reached 89.7 percent accuracy on the official test partition of the FER2013 facial emotion benchmark, according to a study published in Discover Computing. The framework, developed by M. Shylaja, Sheshikala Martha and Reem A. Almenweer, was designed to classify facial expressions into seven emotion categories: angry, disgust, fear, happy, sad, surprise and neutral. The result places the model among the competitive entries reported on this widely used benchmark, although the authors themselves caution that comparisons across studies are complicated by differences in preprocessing, augmentation policies, network complexity and evaluation protocols, so the figure should be read as a comparative reference rather than a definitive ranking.

The motivation behind the work lies in a persistent weakness of convolutional approaches. Convolutional layers excel at detecting local patterns such as edges, textures and individual facial features, but facial emotions are often defined by relationships between distant regions of the face, such as the combined configuration of the eyes, eyebrows, nose and mouth. Traditional methods based on handcrafted features and shallow learners have long struggled with subtle expressions, particularly under variations in illumination, pose, occlusion and expression intensity. Even many CNN-based systems perform poorly when faces appear in unfamiliar poses or lighting, and several models show inconsistent results on minority emotion categories such as fear and disgust. The authors therefore sought a balanced framework that combines strong local feature extraction with contextual modeling, while remaining simple enough to reproduce.

The architecture works in a clearly defined sequence. Each grayscale FER2013 image is first converted to a three-channel format and resized to 224 by 224 pixels, with normalization using ImageNet statistics, so that it is compatible with pretrained networks. A ResNet50 model with weights pretrained on ImageNet, and with its classification head removed, serves as the frozen feature extractor. Its final convolutional block produces a feature map of 7 by 7 by 2048, which is reduced through global average pooling. The spatial map is then reshaped into 49 tokens, one per spatial location, each projected through a learnable linear transformation into an embedding dimension of 512. Positional encoding is added so that spatial relationships among facial regions are preserved, since Transformer architectures do not inherently retain spatial information.

At the heart of the model sits a multi-head self-attention mechanism with eight attention heads. Self-attention allows every token to attend to all other tokens, meaning the network can explicitly model how, for example, the shape of the mouth relates to the position of the eyebrows when distinguishing a sad expression from a neutral one. Residual connections and layer normalization follow the attention module to stabilize gradient propagation, and a position-wise feed-forward network with a dimension of 2048 further refines the representation. The contextualized features then pass through a fully connected classification network: a dense layer with 128 neurons, ReLU activations, a dropout rate of 0.5 to guard against overfitting, and a final SoftMax layer that outputs probabilities across the seven emotion classes.

Training followed a reproducible protocol documented in detail. The model was implemented in PyTorch and trained with the Adam optimizer at a learning rate of 0.0001 and a batch size of 64, for a maximum of 50 epochs with early stopping set at a patience of five epochs, alongside weight-decay regularization and a fixed random seed. Data augmentation, including random horizontal flipping, random rotation and random resized cropping, was applied only to the training set, while validation and test images underwent only resizing and normalization to avoid data leakage. The standard FER2013 split was used, with 28,709 images for training, 3,589 for validation and 3,589 for testing. Experiments ran on an NVIDIA Tesla T4 GPU with 16 GB of memory, taking roughly 2.3 hours per experiment, and results were reported as mean accuracy and standard deviation over repeated runs.

During training, the model’s accuracy on the training set rose from 45.78 to 81.11 percent while the training loss fell from 1.4151 to 0.5185, with both training and validation performance improving steadily until convergence. The best checkpoint, selected by validation-based early stopping, was then evaluated on the official test partition. Class-wise results show the model performed strongest on the happy category, achieving a precision of 0.94, a recall of 0.96 and an F1 score of 0.96, with surprise and neutral also classified well. The disgust class proved hardest, with a precision of 0.90 but a recall of only 0.71, meaning many disgust samples went unrecognized, and an F1 score of 0.74; the sad class reached an F1 of 0.79. Macro-average scores of roughly 0.88 to 0.89 indicate broadly consistent performance across classes, with the weighted average slightly higher because larger classes such as happy and neutral are handled well.

The confusion matrix reveals where the model struggles. Sad expressions were misclassified as neutral 251 times and as angry 106 times, while fear was mistaken for sad in 230 cases and for neutral in 126 cases. Angry faces were confused with sad 136 times and with neutral 106 times, and disgust was mistaken for angry in 36 cases. These patterns reflect the genuine visual similarity of these negative emotions, which even contextual feature learning could not fully resolve. Receiver operating characteristic analysis showed the happy class achieving an area under the curve of 0.98, with disgust, surprise and neutral close behind at 0.97 and 0.91, while fear lagged at 0.85. Precision-recall curves told a similar story: the happy class maintained an average precision of 0.93, but fear dropped to 0.45, underscoring the difficulty of distinguishing that emotion.

An ablation study quantified the Transformer’s contribution. A ResNet50 backbone alone, with its extracted features used directly for classification, reached 72.84 percent accuracy. Adding a dense classification layer improved this to 74.63 percent, and the full ResNet50-Transformer hybrid achieved 89.70 percent, an improvement of 16.86 percentage points over the baseline and 15.07 points over the dense-classifier configuration. The same enhancements appeared in precision, recall and F1 score. The authors note that the Transformer’s gains were most visible for emotion categories with similar facial characteristics, particularly fear, sadness and neutral, where relationships between facial regions play a larger role in recognition. A separate set of ablation experiments on the preprocessing pipeline showed that removing data augmentation lowered accuracy to 66.02 percent, removing pretrained weights dropped it to 57.52 percent, and a simplified architecture with fewer neurons in the fully connected layer reached 67.90 percent, compared with 67.65 percent for the baseline configuration, highlighting the importance of both augmentation and transfer learning.

The study is candid about its limitations. Evaluation was confined to FER2013, with no cross-dataset validation on other public facial expression datasets, so generalizability to other imaging modalities, population differences and acquisition settings remains untested. Robustness under specific conditions such as occlusion, illumination variation, pose changes, blur and image noise was not evaluated separately, and the assessment covered offline testing only, with no real-time deployment analysis. A comprehensive evaluation of computational efficiency, including FLOPs, parameter counts, memory consumption and inference latency, was also beyond the scope of the work. The authors additionally point to the class imbalance of FER2013, in which some emotion categories are substantially underrepresented, as a factor affecting the results, and they suggest future exploration of class-weighted loss, focal loss and balanced sampling. Future work is planned on alternative feature-map resolutions, tokenization strategies and embedding dimensions, attention visualization for interpretability, multiple independent runs with confidence intervals and statistical significance testing, and lightweight attention mechanisms for real-time use.

For the broader field, the study offers a concrete demonstration that combining convolutional local feature extraction with Transformer-based contextual modeling can yield measurable gains on a static-image benchmark, while also providing a transparent account of where such systems still fail. The detailed reporting of hyperparameters, hardware, training configuration and error analysis is intended to allow independent researchers to replicate the experiments and compare results directly. The authors position the framework not as an entirely new architecture but as a practical, reproducible reference for facial emotion recognition research, with potential relevance to human-computer interaction, healthcare monitoring, intelligent surveillance, education and affective computing, provided that the remaining questions of real-world robustness and generalization are addressed in future studies.

Subject of Research: A hybrid deep learning framework combining ResNet50 and a Transformer encoder for facial emotion recognition on the FER2013 dataset.

Article Title: Hybrid deep learning architecture for facial emotion recognition on FER2013 dataset

Article References: Shylaja, M., Martha, S., & Almenweer, R. A. (2026). Hybrid deep learning architecture for facial emotion recognition on FER2013 dataset. Discover Computing, 29(1), Article 806. https://doi.org/10.1007/s10791-026-10715-8

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10715-8

Keywords: facial emotion recognition, FER2013, ResNet50, Transformer encoder, deep learning, self-attention, transfer learning, computer vision, ablation study, data augmentation, classification accuracy, affective computing