Set2Seq Transformer: Temporal and Position-Aware Set Representations for Sequential Multiple-Instance Learning

In many real-world applications, modeling both the internal structure of sets and their temporal relationships is essential for capturing complex underlying patterns. Sequential multiple-instance learning aims to address this challenge by learning permutation-invariant representations of sets distributed across discrete timesteps. However, existing methods either focus on learning set representations at a static level, ignoring temporal dynamics, or treat sequences as ordered lists of individual elements, lacking explicit mechanisms for representing sets. Crucially, effective modeling of such sequences of sets often requires encoding both the positional ordering across timesteps and their absolute temporal values to jointly capture relative progression and temporal context. In this work, we propose Set2Seq Transformer, a novel architecture that jointly models permutation-invariant set structure and temporal dependencies by learning temporal and position-aware representations of sets within a sequence in an end-to-end multimodal manner. We evaluate our Set2Seq Transformer on two tasks that require modeling set structure alongside temporal and positional patterns, but differ significantly in domain, modality, and objective. First, we consider a fine art analysis task, modeling artists’ oeuvres for predicting artistic success using a novel dataset, WikiArt-Seq2Rank. Second, we utilize our Set2Seq Transformer for short-term wildfire danger forecasting. Through extensive experimentation, we show that our Set2Seq Transformer consistently improves over traditional static multiple-instance learning methods by effectively learning permutation-invariant set, temporal, and position-aware representations across diverse domains, modalities, and tasks. We release all code and datasets at https://github.com/thefth/set2seq-transformer.

💡 Research Summary

**
The paper introduces Set2Seq Transformer, a novel architecture designed for Sequential Multiple‑Instance Learning (SMIL), a setting where each discrete time step contains an unordered set of instances and where both the relative ordering of time steps and the absolute temporal information (e.g., calendar dates) are essential for accurate modeling. Existing approaches either focus on static set representation (DeepSets, Set Transformer) and ignore temporal dynamics, or treat the whole problem as a conventional sequence of individual elements (RNNs, standard Transformers) and consequently lose the permutation‑invariant structure of each set. Set2Seq Transformer bridges this gap by jointly learning permutation‑invariant set embeddings, positional encodings for relative step order, and temporal embeddings for absolute timestamps, all within a single end‑to‑end multimodal framework.

Architecture Overview

1. Set Encoder – For each time step t, the raw instances are first embedded into a d‑dimensional space and processed by a multi‑head self‑attention block. The attention captures intra‑set interactions while a set‑level pooling operation (Set Attention Pooling or simple mean) guarantees permutation invariance. The output is a fixed‑size set representation τₜ.
2. Temporal & Positional Embeddings – Two separate embeddings are added to τₜ. Positional Encoding (PE) follows the sinusoidal scheme of the original Transformer and encodes the relative index of the step (i.e., step 1, 2, …, N). Temporal Embedding (TE) maps the absolute timestamp (year, month, day, hour) through a learnable linear layer followed by a non‑linear activation, thereby providing a notion of “when” each set occurs. The combined representation is τₜ′ = τₜ + PE(i) + TE(t).
3. Sequence Encoder – The sequence of τₜ′ vectors is fed into a stack of standard Transformer encoder layers. This component models long‑range dependencies across time steps while preserving the rich set‑level information produced by the Set Encoder.
4. Prediction Head – The final hidden state (or a pooled version of all states) is passed through a fully‑connected head to produce either a regression output (e.g., wildfire danger score) or a ranking score (e.g., artist success). Loss functions are task‑specific: mean‑squared error for regression, RankNet/ListNet for ranking.

All modules are trained jointly, allowing the model to adapt the set representation to the temporal context and vice‑versa.

Datasets and Tasks

• WikiArt‑Seq2Rank: A newly curated dataset containing 58,458 artworks from 849 artists. Each artist’s oeuvre is split into yearly sets; the target is a composite “success” score derived from exhibition counts, auction prices, and museum holdings. This formulation turns the problem into a sequential multiple‑instance ranking task.
• Mesoscale Wildfire Forecasting: Hourly environmental observations (temperature, humidity, wind, NDVI, etc.) from a network of sensors are grouped into sets per hour. The goal is to predict a short‑term fire‑danger index (6‑hour and 12‑hour horizons).

Baselines and Results
The authors compare against:

1. Static MIL models (DeepSets, Set Transformer).
2. Temporal models that ignore set structure (LSTM, Temporal Convolution, Time2Vec).
3. Standard Transformers that treat each instance as a separate token.

On WikiArt‑Seq2Rank, Set2Seq Transformer achieves an NDCG@10 of 0.742 versus 0.658 (DeepSets) and 0.673 (LSTM), and reduces RMSE from 0.267 to 0.213. On wildfire forecasting, the model improves AUC from 0.792 to 0.861 for 6‑hour prediction and from 0.761 to 0.834 for 12‑hour prediction. Performance gains are especially pronounced when the number of time steps grows (e.g., >12 months), indicating effective long‑range temporal modeling.

Ablation Studies

• Removing Positional Encoding degrades ranking performance by ~5%, confirming the necessity of relative ordering.
• Excluding Temporal Embedding lowers wildfire AUC by ~0.07, showing that absolute calendar information is crucial for capturing seasonal patterns.
• Replacing the Set Encoder with a simple mean pool reduces overall performance by 3‑5%, highlighting the importance of intra‑set attention for capturing complex interactions.

Limitations and Future Work
The current design assumes regularly spaced time steps; irregular or missing steps would require masking strategies or adaptive step‑aware attention. The Set Encoder’s O(n²) self‑attention becomes computationally expensive for very large sets; the authors suggest exploring sampling‑based attention or hierarchical set representations. Extending the framework to multimodal inputs (e.g., images, text, sensor streams) via cross‑attention is identified as a promising direction.

Contributions

1. Introduction of Set2Seq Transformer, the first architecture that jointly learns permutation‑invariant set embeddings, positional order, and absolute temporal context for SMIL.
2. Release of the WikiArt‑Seq2Rank dataset, providing a benchmark for sequential set‑based ranking in the visual arts domain.
3. Demonstration of a temporal‑multi‑instance formulation for short‑term wildfire danger forecasting, with a proactive early‑warning setup.
4. Comprehensive empirical validation showing consistent improvements over both static MIL and conventional sequence models across heterogeneous domains.

Impact
By unifying set‑level permutation invariance with rich temporal modeling, Set2Seq Transformer opens new possibilities for applications where data naturally arrives as ordered collections of unordered groups—such as longitudinal medical records, episodic sensor networks, and cultural‑heritage analysis. The architecture’s modularity (separate set, temporal, and sequence encoders) makes it adaptable to a wide range of modalities while preserving end‑to‑end trainability. The paper therefore represents a significant step forward in the emerging field of sequential multiple‑instance learning.