Affect-driven ordinal engagement measurement from video

User engagement is crucial for the successful completion of education and intervention programs. Automatic measurement of user engagement can provide valuable insights for instructors to personalize the delivery of the program and achieve program objectives. This paper presents a novel approach to automatically measuring users’ engagement in virtual learning programs from their videos. The proposed approach utilizes affect states, continuous values of valence and arousal, along with a new latent affective feature vector and behavioral features extracted from consecutive video frames. Deep-learning sequential models are trained and validated on the extracted features for video-based engagement measurement. Since engagement is an ordinal variable, ordinal versions of the models are also developed to address the problem of engagement level measurement as an ordinal classification problem. The proposed approach was evaluated on two publicly available video-based engagement measurement datasets, Dataset for Affective States in E-Environments (DAiSEE) and Emotion recognition in the Wild-Engagement prediction in the Wild (EmotiW-EW), containing videos of students in virtual learning programs. The experiments demonstrated a state-of-the-art engagement level classification accuracy of 67.4% on the DAiSEE dataset and a regression mean squared error of 0.0508 on the EmotiW-EW dataset. The ablation study demonstrated that incorporating affect states and ordinality of engagement significantly improved engagement measurement.