Interaction Histories and Short Term Memory: Enactive Development of Turn-taking Behaviors in a Childlike Humanoid Robot

In this article, an enactive architecture is described that allows a humanoid robot to learn to compose simple actions into turn-taking behaviors while playing interaction games with a human partner. The robot’s action choices are reinforced by social feedback from the human in the form of visual attention and measures of behavioral synchronization. We demonstrate that the system can acquire and switch between behaviors learned through interaction based on social feedback from the human partner. The role of reinforcement based on a short term memory of the interaction is experimentally investigated. Results indicate that feedback based only on the immediate state is insufficient to learn certain turn-taking behaviors. Therefore some history of the interaction must be considered in the acquisition of turn-taking, which can be efficiently handled through the use of short term memory.

💡 Research Summary

The paper presents an enactive architecture that enables a child‑like humanoid robot to acquire turn‑taking behaviors through interactive games with a human partner. The core idea is to treat social feedback—specifically visual attention (eye‑gaze) and behavioral synchronization (temporal alignment of movements)—as reinforcement signals that guide the robot’s action selection. The system consists of four main components: (1) a perception module that extracts gaze direction, facial expressions, and movement synchrony in real time; (2) an immediate‑reward module that maps these social cues to scalar rewards (+1 for eye contact, –1 for neglect, additional bonuses for high synchrony); (3) a short‑term memory (STM) module that stores the most recent K interaction steps (typically 5–10) together with their immediate rewards, applying an exponential decay to give more weight to recent events; and (4) a reinforcement‑learning policy module based on Q‑learning, where the state vector combines the robot’s posture with the human’s gaze and synchrony indicators, and the action set includes “perform gesture,” “wait,” and “shift gaze.”

Two experimental scenarios were used to evaluate the architecture: (a) a rock‑paper‑scissors game, where the robot initiates a gesture and the human responds, and (b) a ball‑throwing game, where the human throws a ball, the robot catches it, and then returns the throw. In both cases the robot starts with a random policy, receives immediate social rewards after each interaction, and updates its Q‑values after each step. Two conditions were compared: (i) reinforcement based solely on the immediate reward, and (ii) reinforcement that incorporates the STM‑derived cumulative reward.

Results show that the immediate‑reward‑only condition can learn the “wait‑after‑action” transition when the robot must pause before the human’s turn, but it fails to reliably acquire the complementary “action‑after‑wait” transition. Consequently, the robot often continues to act even after the human has completed their move, leading to poor turn‑taking performance. By contrast, the STM‑augmented condition successfully learns both transitions. The inclusion of short‑term interaction history reduces the number of episodes required for convergence by roughly 30 % and raises the final success rate from about 85 % to 96 %. Moreover, the robot’s waiting periods become better aligned with the human partner’s rhythm, indicating that the robot is using the temporal structure of the interaction rather than reacting only to the current snapshot.

The authors draw several key conclusions. First, social reinforcement in human‑robot interaction is more effective when it reflects a short history of the exchange rather than a single instantaneous observation. Second, a lightweight STM mechanism—implemented as a decaying buffer of recent rewards—provides sufficient context for learning sequential, turn‑based behaviors without requiring complex predictive models or long‑term memory structures. Third, the enactive framework, which emphasizes the robot’s embodied engagement with the environment, allows the system to self‑organize meaningful action patterns from raw sensorimotor data.

Beyond the immediate findings, the paper suggests future directions such as integrating long‑term memory to capture more extended interaction patterns, incorporating affective feedback (e.g., tone of voice, facial affect) to enrich the reward signal, and extending the architecture to multi‑partner or multi‑modal scenarios. The work thus contributes a practical, biologically inspired solution for socially competent robots, with potential applications in education, therapy, and collaborative manufacturing where fluid turn‑taking and responsiveness to human cues are essential.