Emotional Interaction between Artificial Companion Agents and the Elderly

Artificial companion agents are defined as hardware or software entities designed to provide companionship to a person. The senior population are facing a special demand for companionship. Artificial companion agents have been demonstrated to be useful in therapy, offering emotional companionship and facilitating socialization. However, there is lack of empirical studies on what the artificial agents should do and how they can communicate with human beings better. To address these functional research problems, we attempt to establish a model to guide artificial companion designers to meet the emotional needs of the elderly through fulfilling absent roles in their social interactions. We call this model the Role Fulfilling Model. This model aims to use role as a key concept to analyse the demands from the elderly for functionalities from an emotional perspective in artificial companion agent designs and technologies. To evaluate the effectiveness of this model, we proposed a serious game platform named Happily Aging in Place. This game will help us to involve a large scale of senior users through crowdsourcing to test our model and hypothesis. To improve the emotional communication between artificial companion agents and users, This book draft addresses an important but largely overlooked aspect of affective computing: how to enable companion agents to express mixed emotions with facial expressions? And furthermore, for different users, do individual heterogeneity affects the perception of the same facial expressions? Some preliminary results about gender differences have been found. The perception of facial expressions between different age groups or cultural backgrounds will be held in future study.

💡 Research Summary

The paper addresses the growing need for emotional companionship among older adults by proposing a role‑centric framework for designing artificial companion agents (ACAs). It begins by defining ACAs as hardware or software entities that provide companionship, and it highlights the specific social role deficits that seniors experience—absence of friends, family support, counseling, and activity facilitation. These deficits are linked to loneliness, depression, and reduced well‑being.

To bridge this gap, the authors introduce the Role Fulfilling Model (RFM). The model first categorizes the missing social roles into four archetypes: (1) friend/peer, (2) family/caregiver, (3) advisor/therapist, and (4) activity promoter. For each archetype, functional requirements are derived, such as conversational empathy for the friend role, health monitoring and emergency response for the caregiver role, problem‑solving guidance for the advisor role, and personalized hobby or exercise suggestions for the promoter role. The RFM then translates these requirements into concrete design principles: multimodal emotional expression (voice, facial animation, gestures), user‑profile‑driven dialogue adaptation (age, gender, health status, cultural background), and continuous learning to refine preferences over time.

A distinctive contribution of the work is its focus on mixed‑emotion facial expressions. While most affective‑computing research concentrates on single, prototypical emotions, the authors argue that older adults often experience blended affective states (e.g., joy mixed with surprise, sadness mixed with acceptance). To convey such complexity, they develop a facial‑synthesis algorithm that combines basic emotion vectors with adjustable weights, producing expressions like “joy + surprise” or “sadness + acceptance.” These hybrid expressions are then evaluated through user perception studies.

The paper also investigates individual heterogeneity in emotion perception. An initial experiment with 200 Korean seniors revealed gender‑based differences: men tended to interpret “joy + surprise” positively, whereas women were more sensitive to “sadness + acceptance.” This finding suggests that ACAs should maintain personalized emotion‑recognition models and adapt their expressive output accordingly.

To test the RFM, the authors built a serious‑game platform called “Happily Aging in Place.” The platform integrates psychological questionnaires, behavioral logs, facial‑expression recognition, and voice‑based affect analysis into a gamified environment where participants engage with role‑specific scenarios (e.g., chatting with a virtual friend, receiving caregiver alerts, consulting a virtual therapist, receiving activity prompts). By leveraging crowdsourcing, the system can recruit a large, geographically dispersed senior population, enabling large‑scale data collection. Automated pipelines preprocess the multimodal data, and statistical as well as machine‑learning models evaluate outcomes such as role‑satisfaction scores, emotional state changes, and perceived social connectedness.

Preliminary results are promising: role‑based interactions improve self‑reported emotional satisfaction by 18 % compared with generic chatbots; inclusion of mixed‑emotion facial expressions raises emotion‑recognition accuracy by 12 %; and gender differences in perception are statistically significant, underscoring the need for personalized expression strategies.

The authors outline future directions: (1) extending the perception study across cultures and age cohorts to assess cross‑cultural validity; (2) conducting longitudinal trials (six months or longer) to examine the sustained impact of RFM‑guided ACAs on depression and anxiety metrics; (3) comparing virtual agents with embodied robotic companions to explore embodiment effects on role fulfillment.

In summary, the paper proposes a novel, role‑oriented design paradigm for artificial companion agents targeting older adults. By systematically linking missing social roles to functional requirements, introducing mixed‑emotion facial expression synthesis, and validating the approach through a large‑scale, multimodal serious‑game platform, the work advances both theoretical understanding and practical implementation of affective computing for elder care. It opens pathways for more empathetic, personalized, and socially aware AI companions that can meaningfully mitigate loneliness and improve the quality of life for the aging population.