A Hybrid Deep Learning Framework for Emotion Recognition in Children with Autism During NAO Robot-Mediated Interaction

Understanding emotional responses in children with Autism Spectrum Disorder (ASD) during social interaction remains a critical challenge in both developmental psychology and human-robot interaction. This study presents a novel deep learning pipeline for emotion recognition in autistic children in response to a name-calling event by a humanoid robot (NAO), under controlled experimental settings. The dataset comprises of around 50,000 facial frames extracted from video recordings of 15 children with ASD. A hybrid model combining a fine-tuned ResNet-50-based Convolutional Neural Network (CNN) and a three-layer Graph Convolutional Network (GCN) trained on both visual and geometric features extracted from MediaPipe FaceMesh landmarks. Emotions were probabilistically labeled using a weighted ensemble of two models: DeepFace’s and FER, each contributing to soft-label generation across seven emotion classes. Final classification leveraged a fused embedding optimized via Kullback-Leibler divergence. The proposed method demonstrates robust performance in modeling subtle affective responses and offers significant promise for affective profiling of ASD children in clinical and therapeutic human-robot interaction contexts, as the pipeline effectively captures micro emotional cues in neurodivergent children, addressing a major gap in autism-specific HRI research. This work represents the first such large-scale, real-world dataset and pipeline from India on autism-focused emotion analysis using social robotics, contributing an essential foundation for future personalized assistive technologies.

💡 Research Summary

This research introduces a pioneering hybrid deep learning framework specifically engineered to address the complexities of emotion recognition in children with Autism Spectrum Disorder (ASD) during interactions with the NAO humanoid robot. A significant challenge in Human-Robot Interaction (HRI) for neurodivergent populations is the subtle and atypical nature of emotional expressions, which often eludes conventional emotion recognition systems. To overcome this, the authors propose a sophisticated dual-stream architecture that integrates both visual and structural facial information.

The methodology leverages a fine-tuned ResNet-50 Convolutional Neural Network (CNN) to extract high-level visual features from approximately 50,000 facial frames extracted from 15 children with ASD. Complementing this, a three-layer Graph Convolutional Network (GCN) is employed to process geometric features derived from MediaPipe FaceMesh landmarks. By modeling the facial landmarks as a graph, the system can effectively capture the dynamic, structural changes in facial musculature that constitute micro-expressions. This hybrid approach allows the model to understand not just what the face looks like, but how its structural components move in response to social stimuli, such as the robot calling the child’s name.

A key innovation in this study is the advanced labeling and optimization strategy. Instead of relying on rigid, single-class labels, the researchers implemented a soft-labeling technique using a weighted ensemble of DeepFace and FER models. This produces a probabilistic distribution across seven emotion classes, enabling the model to learn the nuanced and ambiguous boundaries of emotional states. Furthermore, the integration of the CNN and GCN streams is optimized via Kullback-Leibler (KL) divergence, ensuring that the fused embedding effectively reconciles the visual and geometric feature spaces without significant information loss.

The study presents one of the first large-scale, real-world datasets from India focused on autism-specific emotion analysis in a social robotics context. The robustness of this pipeline in capturing micro-emotional cues offers immense potential for the development of personalized, AI-driven therapeutic interventions. Ultimately, this work provides a critical foundation for future assistive technologies, paving the way for more empathetic and effective robotic-mediated therapies for children with ASD.