NEST: Nested Event Stream Transformer for Sequences of Multisets

Event stream data often exhibit hierarchical structure in which multiple events co-occur, resulting in a sequence of multisets (i.e., bags of events). In electronic health records (EHRs), for example, medical events are grouped into a sequence of clinical encounters with well-defined temporal structure, but the order and timing of events within each encounter may be unknown or unreliable. Most existing foundation models (FMs) for event stream data flatten this hierarchy into a one-dimensional sequence, leading to (i) computational inefficiency associated with dense attention and learning spurious within-set relationships, and (ii) lower-quality set-level representations from heuristic post-training pooling for downstream tasks. Here, we show that preserving the original hierarchy in the FM architecture provides a useful inductive bias that improves both computational efficiency and representation quality. We then introduce Nested Event Stream Transformer (NEST), a FM for event streams comprised of sequences of multisets. Building on this architecture, we formulate Masked Set Modeling (MSM), an efficient paradigm that promotes improved set-level representation learning. Experiments on real-world multiset sequence data show that NEST captures real-world dynamics while improving both pretraining efficiency and downstream performance.

💡 Research Summary

The paper introduces NEST (Nested Event Stream Transformer), a foundation model specifically designed for event‑stream data that naturally forms a sequence of multisets (SeqSet). In many real‑world domains—most prominently electronic health records (EHRs)—events are grouped into temporally ordered encounters, while the ordering of events inside each encounter is either unknown or irrelevant. Existing transformer‑based foundation models flatten this hierarchy into a single token sequence, which leads to two major drawbacks: (i) dense self‑attention incurs quadratic computational cost and forces the model to learn spurious relationships among tokens that should be permutation‑invariant, and (ii) set‑level representations are obtained only after pre‑training via heuristic pooling, which degrades downstream performance.

NEST addresses these issues through a hierarchical architecture composed of two alternating modules in each layer: a Set‑Wise Encoder (SWE) and a Cross‑Set Encoder (CSE). The SWE restricts attention to tokens belonging to the same multiset, deliberately omitting positional embeddings so that the model respects the permutation invariance of intra‑set elements. The CSE operates only on the special