PovNet+: A Deep Learning Architecture for Socially Assistive Robots to Learn and Assist with Multiple Activities of Daily Living

A significant barrier to the long-term deployment of autonomous socially assistive robots is their inability to both perceive and assist with multiple activities of daily living (ADLs). In this paper, we present the first multimodal deep learning architecture, POVNet+, for multi-activity recognition for socially assistive robots to proactively initiate assistive behaviors. Our novel architecture introduces the use of both ADL and motion embedding spaces to uniquely distinguish between a known ADL being performed, a new unseen ADL, or a known ADL being performed atypically in order to assist people in real scenarios. Furthermore, we apply a novel user state estimation method to the motion embedding space to recognize new ADLs while monitoring user performance. This ADL perception information is used to proactively initiate robot assistive interactions. Comparison experiments with state-of-the-art human activity recognition methods show our POVNet+ method has higher ADL classification accuracy. Human-robot interaction experiments in a cluttered living environment with multiple users and the socially assistive robot Leia using POVNet+ demonstrate the ability of our multi-modal ADL architecture in successfully identifying different seen and unseen ADLs, and ADLs being performed atypically, while initiating appropriate assistive human-robot interactions.

💡 Research Summary

The paper introduces POVNet+, a multimodal deep learning architecture designed to enable socially assistive robots (SARs) to perceive and proactively assist with multiple activities of daily living (ADLs). Recognizing that existing SAR systems are limited to single‑modal inputs or pre‑defined activity classes and often misclassify non‑ADL motions, the authors propose a framework that simultaneously processes RGB‑D video, 3D skeletal pose, and object detection data. The system consists of four main modules: (1) a multimodal sampling module that extracts synchronized video frames, a single RGB image, and 19‑joint 3D skeletons at 6 Hz; (2) an ADL classifier module with dedicated backbones—X3D‑m for video, a graph convolutional network (GCN) with self‑attention for pose, and YOLOv13 for object detection. The pose backbone also computes a motion embedding vector by summing Euclidean joint displacements over the observation window, providing a compact representation of fine‑grained versus gross movements. (3) A spatial mid‑fusion layer aligns features from all modalities into a common spatial reference, allowing joint motion and object locations to be jointly reasoned about. (4) A user state estimation module that projects both ADL embeddings and motion embeddings into separate low‑dimensional spaces and applies a similarity function to classify the current observation as a known ADL, an unseen ADL, or an atypically performed known ADL. This tri‑state categorization enables the robot to decide when to initiate assistance without requiring prior labels for new or abnormal activities.

Quantitative experiments on benchmark human activity datasets and a custom multi‑ADL dataset show that POVNet+ outperforms state‑of‑the‑art methods by 4–6 % in classification accuracy and significantly reduces false positives caused by non‑ADL motions. In a realistic home‑environment HRI study, the SAR robot “Leia” equipped with POVNet+ successfully identified seen, unseen, and atypical ADLs performed by multiple users in a cluttered living space and initiated appropriate assistive behaviors such as offering objects, correcting posture, or providing motivational cues. Participants reported that the robot’s interventions felt timely and helpful, confirming the practical value of proactive HRI.

Key contributions include: (i) the creation of a dedicated motion embedding space that isolates ADL‑related movement from background activity; (ii) a novel similarity‑based user state estimation method that detects new or atypical ADLs in real time without additional labeling; and (iii) the first end‑to‑end multimodal deep learning system validated on a real SAR platform for autonomous multi‑activity assistance. The authors suggest future work on continual online updating of the embedding spaces and reinforcement‑learning‑driven policy generation to further personalize assistance, moving toward fully autonomous, long‑term deployment of socially assistive robots in aging‑in‑place and rehabilitation contexts.