LITHE: Bridging Best-Effort Python and Real-Time C++ for Hot-Swapping Robotic Control Laws on Commodity Linux

Modern robotic systems rely on hierarchical control, where a high-level “Brain” (Python) directs a lower-level “Spine” (C++ real-time controller). Despite its necessity, this hierarchy makes it difficult for the Brain to completely rewrite the Spine’s immutable control logic, consequently inhibiting fundamental adaptation for different tasks and environments. Conventional approaches require complex middleware, proprietary hardware, or sacrifice real-time performance. We present LITHE (Linux Isolated Threading for Hierarchical Execution), a lightweight software architecture that collapses the robot control hierarchy onto a commodity single-board computer (Raspberry Pi 4B with pi3hat), while maintaining safe frequency decoupling between the Brain and Spine. LITHE integrates strict CPU isolation (isolcpus), lock-free inter-process communication (IPC), and pipelined execution to meet high-frequency deadlines with minimal jitter. By adding multi-threaded dynamic linking, LITHE enables a Python-based Brain to dynamically evolve the logic of a 1kHz C++ Spine without interruption. We validate “functional real-time” system performance with worst-case execution time (WCET) < 100 $μ$s and maximum release jitter (MRJ) < 4 $μ$s under heavy load. We demonstrate a novel application where a large language model (LLM) supervisor performs online system identification to evolve a real-time controller on-the-fly, without interrupting the 1 kHz control loop. In essence, LITHE eliminates the “immutable compiled code” bottleneck for best-effort Brains to synthesize and inject completely new control laws into the real-time Spine. This bridges a critical gap between high-level AI and low-level real-time control to unlock continuous real-time evolution of embodied intelligence in safe, human-in-the-loop systems.

💡 Research Summary

The paper introduces LITHE (Linux Isolated Threading for Hierarchical Execution), a lightweight software architecture that enables a best‑effort Python “Brain” to dynamically hot‑swap the control law of a hard‑real‑time C++ “Spine” without interrupting a 1 kHz control loop, all on a commodity Raspberry Pi 4B equipped with a pi3hat CAN‑FD expansion board. The authors begin by highlighting the long‑standing bottleneck in hierarchical robot control: the Spine’s compiled C++ code is immutable during operation, forcing costly firmware flashes or system pauses whenever the low‑level controller must be changed. Existing solutions—commercial rapid‑control prototyping hardware, PREEMPT_RT‑patched kernels, or ROS 2 middleware—either impose high financial barriers, require deep kernel expertise, or add latency that jeopardizes sub‑millisecond determinism.

LITHE’s core idea is to achieve functional real‑time performance by strict CPU isolation rather than kernel pre‑emption. Using vanilla Raspberry Pi OS (kernel 6.12.47) with boot parameters such as isolcpus=1-3, nohz_full, and rcu_nocbs=1-3, the four‑core processor is partitioned into dedicated domains: CPU 0 handles housekeeping, CPU 1 runs the real‑time Spine loop, CPU 2 hosts the Python Brain (including heavy AI models), and CPU 3 manages blocking I/O to the motor drivers via the pi3hat library. By pinning the Spine to an isolated core and off‑loading all interrupts, the scheduler’s influence is effectively eliminated, allowing a simple while‑loop to meet a 1 kHz deadline with measured worst‑case execution time (WCET) under 100 µs and maximum release jitter (MRJ) under 4 µs even under heavy computational load.

Inter‑process communication between Brain and Spine is implemented with lock‑free, zero‑copy POSIX shared memory (/dev/shm). A custom schema‑driven code generator produces matching C++ structs and Python ctypes bindings, guaranteeing ABI compatibility. Consistency is enforced with a Seqlock pattern: the Spine reads the shared command buffer only when the sequence counter indicates a stable snapshot, retrying otherwise. This non‑blocking design permits the Python process to pause for garbage collection or GPU inference without ever stalling the real‑time loop. To bridge the frequency gap (Brain typically 50–100 Hz, Spine 1 kHz), the Spine interpolates incoming way‑points using a Catmull‑Rom spline, preserving C¹ continuity at the actuator level.

The Spine’s execution pipeline adopts a double‑buffered (producer‑consumer) architecture. While CPU 3 handles CAN‑FD bus transactions for the previous command, the Spine immediately begins computing the next control output using the most recent sensor feedback. This yields a deterministic one‑cycle latency (Z‑1) but keeps the compute core fully utilized, eliminating idle time during bus waits. A busy‑wait spinlock on an atomic flag keeps the core cache hot, providing tighter timing guarantees than sleep‑based approaches.

Dynamic reconfiguration is achieved through a just‑in‑time hot‑swap loader. The Brain compiles a new control law into a shared object (.so) using gcc on the isolated core, locks the resulting memory pages with mlockall to prevent page faults, and loads the library with dlopen. Crucially, loading and symbol resolution occur on the Brain core, while the Spine thread merely swaps an atomic function‑pointer to the newly loaded entry point at the start of the next cycle. This “load‑then‑activate” separation guarantees that no control cycle is missed and that the Spine’s internal state (e.g., integrators, filters) persists across swaps.

Performance validation includes synthetic stress tests and a concrete experiment with a 1‑DOF robot arm driven by Moteus actuators. In the latter, a decentralized large language model (LLM) – qwen2.5‑coder‑7b – runs on the Brain, performs online system identification, and evolves the controller from a simple PID to an impedance controller within seconds. The LLM‑generated controller is hot‑swapped into the running Spine without any missed deadlines, demonstrating that even heavyweight AI workloads can safely coexist with hard real‑time control on inexpensive hardware.

Compared with related work, LITHE distinguishes itself by (1) avoiding any kernel patches, thereby preserving standard Linux stability and simplifying deployment; (2) eliminating ROS 2‑style DDS overhead through pure shared‑memory communication; (3) providing true hot‑swap of the entire control law (not just parameters) via atomic pointer swaps; and (4) offering an open‑source hardware abstraction layer that unifies CAN‑FD, CANopen, and other bus protocols under a single memory map.

The authors conclude that LITHE bridges the gap between high‑level AI reasoning and low‑level deterministic actuation, enabling continuous, real‑time evolution of embodied intelligence in safe, human‑in‑the‑loop systems. Future directions include scaling to multi‑Spine configurations, deeper analysis of real‑time memory allocation, integration with more complex multi‑degree‑of‑freedom robots, and the addition of security/authentication mechanisms for hot‑swapped code.