NTCCRT: A concurrent constraint framework for real-time interaction (extended version)

Writing multimedia interaction systems is not easy. Their concurrent processes usually access shared resources in a non-deterministic order, often leading to unpredictable behavior. Using Pure Data (Pd) and Max/MSP is possible to program concurrency, however, it is difficult to synchronize processes based on multiple criteria. Process calculi such as the Non-deterministic Timed Concurrent Constraint (ntcc) calculus, overcome that problem by representing multiple criteria as constraints. We propose using our framework Ntccrt to manage concurrency in Pd and Max. Ntccrt is a real-time capable inter- preter for ntcc. Using Ntccrt externals (binary plugins) in Pd we ran models for machine improvisation and signal processing.

💡 Research Summary

This paper presents “NTCCRT,” a novel framework designed to address the significant challenges of managing concurrency and complex synchronization in real-time multimedia interaction systems, particularly within environments like Pure Data (Pd) and Max/MSP.

The core problem identified is the difficulty in expressing and coordinating processes based on multiple, dynamic criteria (e.g., “execute process A until a complex condition C, derived from the confluence of several other processes, is met”) using the graphical patch-based paradigm of Pd/Max. Such logic becomes cumbersome and inflexible.

The proposed solution leverages the Non-deterministic Timed Concurrent Constraint (NTCC) calculus. NTCC is a process calculus where system states are represented as constraints (mathematical relations) on variables. It introduces a discrete notion of time and provides agents like tell (add constraint), when (conditional execution), ∑ (non-deterministic choice), and next (execution in the next time unit). This allows for a declarative and formal specification of concurrent behavior with intricate synchronization conditions.

The primary contribution is the NTCCRT framework itself—a real-time capable interpreter for NTCC models. Its architecture is carefully designed for practicality and performance. NTCCRT is implemented in C++ and utilizes the efficient Gecode library for constraint solving. A key implementation insight is encoding fundamental NTCC agents (e.g., tell, when) as Gecode propagators, enabling effective concurrency control. The framework offers multiple front-ends for model authoring: direct C++ API, Common Lisp macros, and a visual interface using OpenMusic. Crucially, models can be compiled into binary plugins (externals) for seamless integration into Pd or Max patches, or into standalone applications.

To demonstrate its utility, the paper details two substantial applications run as NTCCRT externals in Pd. The first is CCFOMI, a concurrent constraint Factor Oracle model for machine improvisation. This system concurrently performs stylistic learning (building a Factor Oracle automaton from input sequences) and stylistic simulation (generating new sequences). The NTCC model elegantly synchronizes these phases using constraints, with the automaton’s dynamic state represented using a Rational Tree constraint system. In the Pd implementation, the model reacts in real-time to live MIDI input from a human player. The second application involves a signal processing model where an NTCCRT external sends control signals to trigger audio filters within Pd, showcasing its use for event-driven control flow.

In conclusion, NTCCRT successfully bridges the gap between the formal, rigorous world of process calculi and the practical, time-sensitive demands of multimedia programming. It empowers developers to specify complex concurrent interactions declaratively using NTCC’s powerful semantics while maintaining the low-latency, interactive performance required for real-time systems. The framework opens new avenues for building reliable and sophisticated interactive multimedia applications.