Long-range Modeling and Processing of Multimodal Event Sequences

Temporal point processes (TPPs) have emerged as powerful tools for modeling asynchronous event sequences. While recent advances have extended TPPs to handle textual information, existing approaches are limited in their ability to generate rich, multimodal content and reason about event dynamics. A key challenge is that incorporating multimodal data dramatically increases sequence length, hindering the ability of attention-based models to generate coherent, long-form textual descriptions that require long-range understanding. In this paper, we propose a novel framework that extends LLM-based TPPs to the visual modality, positioning text generation as a core capability alongside time and type prediction. Our approach addresses the long-context problem through an adaptive sequence compression mechanism based on temporal similarity, which reduces sequence length while preserving essential patterns. We employ a two-stage paradigm of pre-training on compressed sequences followed by supervised fine-tuning for downstream tasks. Extensive experiments, including on the challenging DanmakuTPP-QA benchmark, demonstrate that our method outperforms state-of-the-art baselines in both predictive accuracy and the quality of its generated textual analyses.

💡 Research Summary

The paper introduces Multimodal Temporal Point Processes (MM‑TPP), a novel framework that extends large‑language‑model‑based point‑process modeling to handle visual information alongside timestamps, event types, and textual descriptions. Building on the recent Language‑TPP, the authors integrate a vision encoder (Qwen2.5‑VL) and design a unified tokenization scheme where each event is represented by a structured template: special tokens encode the 32‑bit timestamp (four byte‑level tokens), event type (e.g., <|type k|>), textual content (standard language tokens), and a placeholder token <|image pad|> that is replaced by visual embeddings during model execution. This design avoids exploding token counts that would arise from patch‑level image tokenization, while still allowing deep fusion of visual and textual modalities.

A central challenge addressed is the dramatic increase in sequence length when multimodal data are concatenated, which makes the O(N²) self‑attention of Transformers prohibitively expensive. To mitigate this, the authors propose an adaptive long‑sequence compression mechanism based on temporal similarity. For consecutive events i and i‑1, if the absolute difference between their inter‑event intervals |τ_i − τ_{i‑1}| is smaller than a predefined threshold Δ, event i is replaced by a single special token <|similar event|>. This token signals that the event follows a similar temporal pattern and allows the model to skip encoding its full multimodal template. The compression dramatically reduces token counts in bursty or periodic streams (e.g., Danmaku comment streams), while preserving distinct events that carry important timing information.

Training proceeds in two stages. First, the model is pretrained on compressed sequences using a standard next‑token prediction objective, enabling it to learn multimodal interactions and the semantics of the compression token. Second, task‑specific fine‑tuning is performed with prompt‑response pairs for three sub‑tasks: (1) predicting the next event’s timestamp, (2) classifying its type, and (3) generating a textual description. This two‑stage paradigm leverages the generalization acquired during pretraining while adapting the model to downstream requirements.

The authors evaluate MM‑TPP on two multimodal TPP benchmarks. TAXI‑PRO augments the classic NYC taxi dataset with map‑patch images and textual stop descriptions, providing a realistic urban mobility scenario. DanmakuTPP‑QA consists of video‑stream comment sequences where each comment is paired with the corresponding video frame, together with a set of question‑answer pairs that require long‑form textual reasoning. Baselines include traditional RNN‑based TPPs, Transformer‑based TPPs, Neural Hawkes processes, and the latest Language‑TPP. Metrics cover time prediction RMSE, type classification F1, and text generation quality (BLEU, ROUGE, human judgments). MM‑TPP consistently outperforms all baselines, achieving 12‑18 % relative improvements on quantitative metrics and notably higher human scores for coherence and informativeness in generated answers. Compression ratios range from 2× to 5×, yet predictive performance degrades by less than 2 %, while memory consumption drops by over 60 % and inference speed improves by ~1.8×.

Key contributions are: (1) a unified multimodal TPP framework that jointly predicts time, type, and generates text conditioned on visual inputs; (2) an adaptive compression strategy that preserves long‑range dependencies while keeping sequence length tractable; (3) the creation of the TAXI‑PRO dataset, enriching TPP research with realistic multimodal covariates; and (4) extensive empirical validation demonstrating superior accuracy and reasoning capability. The work opens avenues for incorporating additional unstructured covariates (audio, sensor streams) and for learning the compression threshold Δ in a data‑driven manner, potentially further enhancing scalability and applicability in real‑time systems such as live comment analysis, traffic incident reporting, and multimodal surveillance.