New AI model reads faces and gender together, with maps that show why

Facial emotion recognition systems have quietly become part of modern life, powering everything from driver-monitoring cameras to tutoring software that senses when a student is confused. Yet most of these systems share two stubborn weaknesses: they classify a whole image with a single emotion label, with no idea where in the frame the face actually is, and they ignore demographic attributes such as gender, which shape both facial structure and how expressions appear. A new open-access study published in Discover Networks on 24 September 2026 tackles both problems at once, introducing a curated dataset called FER2025 and a dual-stream deep learning framework, XFDetNet, that localizes faces, recognizes emotions and gender together, and then explains itself with visual heatmaps.

The dataset is the foundation of the work. The researchers assembled 7,386 facial images from publicly accessible stock and photography platforms, including iStock, Unsplash, Shutterstock and Getty Images, and annotated them with 11,253 bounding boxes, an average of about 1.5 annotated subjects per image. Each box carries one of twelve class labels formed by pairing six emotion categories, namely angry, fear, happy, sad, sleepy and surprised, with binary male or female gender labels. Because the source images are subject to third-party licensing restrictions, the team released only the annotation metadata openly on Zenodo under a CC BY 4.0 license, along with the complete code on GitHub so that others can rebuild the image set under appropriate licenses.

Annotation quality was checked with three independent annotators labeling the same subset of images. Pairwise Cohen’s Kappa values ranged from 0.789 to 0.886, with a mean of 0.8534, and Fleiss’ Kappa and Krippendorff’s Alpha both matched that figure, a range conventionally read as substantial to almost perfect agreement. All three annotators agreed unanimously on 80.3% of images. Where they disagreed, the disagreements clustered between emotionally similar pairs such as sad versus sleepy and happy versus surprised, which the authors interpret as genuine ambiguity in expression rather than careless labeling. Per-class full agreement rates stayed within a narrow band, from 0.78 to 0.82, suggesting no single class was disproportionately hard to annotate.

A particularly careful phase of the work addressed data leakage, a common source of inflated results in machine learning benchmarks. Because images came from multiple commercial websites, near-duplicates could easily land in different splits of the data. Using perceptual hashing with a Hamming distance threshold of ten or less, the team identified 770 multi-image near-duplicate groups, each containing two to four images. Before correction, 342 of those groups, or 44.42%, spanned more than one split, affecting 761 images. The affected groups were consolidated so that every near-duplicate group fell entirely within a single split, verified at 100% after re-splitting. The authors note, however, that without ground-truth identity metadata, subject-level splitting could not be verified and residual identity leakage cannot be fully excluded.

For the models themselves, the researchers fine-tuned YOLOv8s and YOLO11s, small variants of two widely used one-stage object detectors, on 640 by 640 pixel grayscale images for 100 epochs with COCO-pretrained weights, the AdamW optimizer and a fixed random seed for reproducibility. Training ran on an NVIDIA Tesla T4 GPU. The proposed XFDetNet then runs both detectors in parallel as independent streams with no shared weights, queries each with a deliberately permissive confidence threshold of 0.001 to retain a dense pool of candidate predictions, and merges the two streams’ boxes, confidence scores and class labels using Weighted Boxes Fusion, which averages overlapping boxes with confidence-weighted coordinates instead of discarding them as conventional non-maximum suppression does. A final confidence filter of 0.25 yields the consolidated output.

On the FER2025 test set, XFDetNet achieved a recall of 94.56%, an F1-score of 91.46%, an mAP@50 of 94.50% and an mAP@50-95 of 63.91%, outperforming both individual baselines on those measures, while YOLOv11 alone retained the highest precision at 93.16%. An ablation study supported the fusion design: replacing Weighted Boxes Fusion with an NMS-based ensemble dropped mAP@50 from 0.945 to 0.935 and mAP@50-95 from 0.639 to 0.627, and varying the fusion IoU threshold between 0.45 and 0.65 produced only marginal F1 fluctuations, with the configured 0.55 offering the best precision-recall balance.

Collapsing the twelve classes revealed a clear asymmetry between the two sub-tasks. At the gender level, all three models scored F1-scores above 0.93 for both male and female subjects, and gender-level confusion matrices showed near-perfect discrimination of about 0.97 for both categories. Emotion recognition proved harder. XFDetNet led on five of six emotions, but fear remained the weakest class across every model, frequently confused with background, surprised or sad. YOLOv8, for example, misclassified 18% of fear_male and 15% of fear_female instances as background, errors XFDetNet reduced to 5% and 3% respectively, though the fear-surprise and fear-sad ambiguities persisted across all three architectures.

Explainability was built into the pipeline rather than bolted on. Grad-CAM++, a gradient-based technique that highlights the image regions most influential to a predicted class, was computed independently for each stream at the last backbone block before its prediction head, and the two heatmaps were pixel-wise averaged into a single fused explanation. The maps showed attention concentrated on the eyes, nose, mouth and cheeks rather than background. To quantify this, the team computed insertion and deletion AUC faithfulness metrics over a class-balanced sample of 88 images, a subset chosen because the procedure requires repeated forward passes per image. Correctly classified samples scored substantially higher insertion AUC, 0.587 versus 0.390, and deletion AUC, 0.411 versus 0.227, than misclassified ones, indicating the explanations track the model’s decision process most faithfully when the prediction is right. The happy class reached the highest insertion AUC at 0.704, while fear showed the lowest deletion AUC at 0.191.

The speed and generalization results temper the headline numbers. As a dual-stream ensemble plus a fusion step, XFDetNet ran at 98.1 milliseconds per image, about 10.2 frames per second, compared with more than 80 frames per second for either individual model, making it better suited to offline or moderately time-sensitive applications than strict real-time use. Evaluated without any fine-tuning on 2,000 images each from FER2013, RAF-DB and AffectNet, performance dropped sharply: XFDetNet did best on AffectNet with an F1-score of 48.80% and mAP@50 of 26.81%, but fell to an F1 of 22.19% on RAF-DB and 13.47% on FER2013, whose low-resolution, tightly cropped grayscale images differ most from the training distribution. The authors conclude that broader real-world generalization has not yet been established and that practical deployment should currently stay within in-domain or visually similar conditions.

The study is candid about its limits. The dataset is relatively small, covers only six emotions, and uses binary male or female gender labels; demographic sampling was uncontrolled, so imbalances in age, ethnicity, skin tone and expression intensity may exist, and no dedicated fairness metrics such as demographic parity or equalized odds were evaluated. Future work outlined by the team includes expanding the dataset with more diverse, multi-source imagery, adding targeted data for the fear class, moving beyond binary gender categories, collecting subject-level identity metadata to enable identity-disjoint splits, exploring licensing and consent-based collection for a public release, and improving efficiency through lightweight fusion or knowledge distillation. Even so, the combination of bounding-box-level multi-attribute annotation, leakage-aware data curation and quantitatively validated explanations offers a template for facial emotion recognition systems that show not just what they see, but where and, to a measurable degree, why.

Subject of Research: Explainable, gender-aware facial emotion recognition using a hybrid dual-stream deep learning framework and a new annotated dataset

Article Title: Explainable binary gender-aware facial emotion recognition via hybrid deep learning and customized visual dataset

Article References: Haque, M. S., Hossain, M. S., Razzaque, Z. I., Robin, R. A., Afrin, E., Neha, M. M., & Islam, S. (2026). Explainable binary gender-aware facial emotion recognition via hybrid deep learning and customized visual dataset. Discover Networks, 2(1), Article 35. https://doi.org/10.1007/s44354-026-00052-z

Image Credits: AI Generated

DOI: 10.1007/s44354-026-00052-z

Keywords: facial emotion recognition, deep learning, computer vision, XFDetNet, YOLOv8, YOLOv11, Weighted Boxes Fusion, Grad-CAM++, FER2025 dataset, explainable AI, gender-aware recognition, object detection