First Experiments with PowerPlay

Like a scientist or a playing child, PowerPlay not only learns new skills to solve given problems, but also invents new interesting problems by itself. By design, it continually comes up with the fastest to find, initially novel, but eventually solvable tasks. It also continually simplifies or compresses or speeds up solutions to previous tasks. Here we describe first experiments with PowerPlay. A self-delimiting recurrent neural network SLIM RNN is used as a general computational problem solving architecture. Its connection weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences. Our PowerPlay-driven SLIM RNN learns to become an increasingly general solver of self-invented problems, continually adding new problem solving procedures to its growing skill repertoire. Extending a recent conference paper, we identify interesting, emerging, developmental stages of our open-ended system. We also show how it automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.

💡 Research Summary

The paper introduces PowerPlay, an open‑ended learning framework in which an artificial agent not only solves externally supplied tasks but also invents its own problems and learns to solve them efficiently. The core of the system is a self‑delimiting recurrent neural network (SLIM RNN) whose connection weights act as a program code that can halt or run indefinitely, allowing explicit control over execution time and memory usage. PowerPlay operates in a loop of four stages: (1) search for a novel task that the current network cannot yet solve; (2) find a candidate solution that modifies as few weights as possible; (3) verify that the candidate does not degrade performance on any previously mastered tasks and that the overall complexity increase (number of changed weights and their influence) is minimal; (4) integrate the new solution into the network and record the task‑solution pair in a growing repertoire.

A key design principle is “fast‑to‑validate” task generation. The system prefers tasks that can be solved with minimal weight changes and that have limited overlap with existing tasks, thereby preventing uncontrolled growth of network complexity and ensuring continual learning progress. As a side effect, the network spontaneously modularizes: weight subsets that are useful for multiple tasks become reusable modules, and new tasks are often solved by recombining or slightly adapting existing modules. This mirrors developmental stages observed in biological learning, where simple behaviors are mastered first and later combined into more sophisticated skills.

Experimental results with the SLIM RNN show a clear developmental trajectory. Early in training the agent learns simple sensor‑effector mappings; later it acquires abilities such as sequence prediction, abstract pattern recognition, and even self‑generated mini‑games. Importantly, the system repeatedly compresses earlier solutions: multiple weight configurations that implement the same function are merged into a single, more efficient subnetwork, reducing overall weight count and computational cost. This demonstrates that PowerPlay does not merely accumulate tasks but also refines and reorganizes knowledge.

The authors argue that PowerPlay offers a concrete step toward truly open‑ended artificial intelligence. Unlike conventional reinforcement or supervised learning, where the goal set is fixed, PowerPlay continuously creates and reshapes its own objectives, yielding an ever‑expanding learning landscape. The self‑delimiting program representation and weight‑based code reuse provide a scalable architecture that balances memory efficiency with guaranteed execution bounds. Future work is suggested in more complex physical environments, multi‑agent interactions, and human‑in‑the‑loop collaborative tasks, aiming to bring the system closer to general AI capabilities.