Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models

Representing continuous time is a critical and under-explored challenge in modeling temporal event sequences with large language models (LLMs). Various strategies like byte-level representations or calendar tokens have been proposed. However, the optimal approach remains unclear, especially given the diverse statistical distributions of real-world event data, which range from smooth log-normal to discrete, spiky patterns. This paper presents the first empirical study of temporal tokenization for event sequences, comparing distinct encoding strategies: naive numeric strings, high-precision byte-level representations, human-semantic calendar tokens, classic uniform binning, and adaptive residual scalar quantization. We evaluate these strategies by fine-tuning LLMs on real-world datasets that exemplify these diverse distributions. Our analysis reveals that no single strategy is universally superior; instead, prediction performance depends heavily on aligning the tokenizer with the data’s statistical properties, with log-based strategies excelling on skewed distributions and human-centric formats proving robust for mixed modalities.

💡 Research Summary

This paper tackles a fundamental yet under‑explored problem in applying large language models (LLMs) to event‑sequence modeling: how to represent continuous timestamps as discrete tokens. While prior work has suggested several ad‑hoc solutions—plain numeric strings, calendar‑based tokens, or low‑level byte encodings—no systematic comparison exists, nor is it clear which method works best for different kinds of temporal data.

The authors define five representative temporal tokenization strategies:

1. Numeric String – formats a time interval as a fixed‑precision decimal string (e.g., “0.076”) and feeds it directly to the LLM’s sub‑word tokenizer. This requires no vocabulary changes but often fragments the number into meaningless sub‑tokens, leading to poor token efficiency.

2. Byte Tokenization – treats a 32‑bit floating‑point value as four bytes and maps each byte to a special token (<|byte_0|> … <|byte_255|>). It preserves full float32 precision with a constant four‑token representation, at the cost of extending the vocabulary and providing limited semantic information per token.

3. Calendar Tokenization – decomposes timestamps into human‑readable calendar components (year, month, day, hour, minute, second). Both absolute times and relative intervals can be expressed with tokens such as <|year_2025|> or <|hours_05|>. This approach is semantically rich and robust for mixed‑modal data but can be token‑heavy depending on the chosen granularity (day vs. second).

4. Scale Bin Tokenization – applies a linear or logarithmic transform to all time values, then uniformly partitions the transformed range into K bins (e.g., 256). Each bin receives a unique token, yielding a single‑token representation per time value. The method is data‑driven but assumes a roughly uniform distribution in the transformed space; mismatched binning can cause substantial quantization error.

5. Residual Scalar Quantization (RSQ) – a multi‑stage quantization scheme analogous to residual vector quantization. After a chosen transform (linear or log), the first level runs K‑means to obtain a codebook of K₁ centroids; the residual is then quantized by a second K‑means, and so on for N levels. The final representation is a fixed‑length token sequence ⟨q₁,…,q_N⟩, each token mapping to a centroid. RSQ can achieve high precision with relatively few tokens, but it introduces extra training complexity and requires storing multiple codebooks.

To evaluate these strategies, the authors fine‑tune a 1‑billion‑parameter Llama‑3.2 model using QLoRA on five real‑world event‑sequence datasets that exhibit markedly different temporal distributions:

• Stack Overflow – smooth log‑normal inter‑event times.
• Chicago Crime – also log‑normal but with a longer tail.
• NYC Taxi – a multimodal distribution mixing short and long gaps.
• US Earthquake – log‑normal with occasional large spikes.
• Amazon Review – highly spiky, with many repeated interval values.

All datasets provide textual event types, absolute timestamps, and pre‑computed relative intervals (Δt). The authors keep the event‑type handling identical across experiments (standard sub‑word tokenization) and vary only the temporal tokenization. They measure three metrics: (1) next‑event type accuracy (Acc), (2) root‑mean‑square error of the predicted interval (RMSE), and (3) the average number of tokens required to encode a single time value (efficiency). Each configuration is run five times and averaged.

Key findings:

• Event‑type prediction is largely insensitive to the temporal tokenizer. Across all five datasets, accuracies range narrowly (≈44 % for Stack Overflow to ≈92 % for NYC Taxi) regardless of the tokenization method.

• Time‑prediction performance varies dramatically with the tokenizer and aligns with the underlying temporal distribution.

  • Log‑based strategies (Scale Bin with logarithmic bins, RSQ with log transform) achieve the lowest RMSE on the log‑normal datasets (Stack Overflow, Chicago Crime) and on the spiky Amazon Review data. For example, RSQ(Log, L4) yields RMSE ≈ 0.47 on Stack Overflow, outperforming the baseline TPP‑LLM (RMSE ≈ 0.46) while using far fewer specialized components.
  • For the multimodal NYC Taxi data, the absolute calendar representation at second granularity is the clear winner (RMSE ≈ 0.35), indicating that human‑semantic tokens can capture mixed patterns better than pure quantization.
  • On US Earthquake, the byte‑level encoding attains the best RMSE (≈ 0.51), suggesting that preserving raw floating‑point precision is advantageous when the distribution is log‑normal but not extremely skewed.

• Token‑efficiency trade‑offs:

  • Single‑token log‑scale binning and RSQ(L1) (single‑level) are the most token‑efficient (≈ 1 token per interval) while still delivering competitive RMSE on several datasets.
  • Multi‑token approaches (Byte uses 4 tokens, RSQ(L4) uses 4 tokens) provide higher precision at the cost of increased sequence length and computational overhead.

• Comparison with a dedicated TPP‑LLM baseline: The pure LLM fine‑tuning approach matches or exceeds the baseline in event‑type accuracy and remains competitive in time prediction, despite lacking specialized TPP prediction heads and log‑likelihood loss functions. This demonstrates that a simpler architecture can be viable for many practical scenarios.

Implications and limitations:

The study establishes a clear design guideline: select a temporal tokenization that mirrors the statistical shape of the dataset’s inter‑event times. Log‑scale quantization excels for skewed or spiky distributions; calendar tokens are robust for heterogeneous, multimodal data; byte‑level encoding shines when raw precision matters. Moreover, the findings highlight that token‑efficiency and predictive accuracy can be jointly optimized by choosing an appropriate quantization granularity (e.g., single‑token log bins).

Limitations include the focus on a single model family (Llama‑3.2 up to 3 B parameters), which may not generalize to larger or architecturally different LLMs. The five datasets, while diverse, do not cover the full spectrum of real‑world temporal patterns (e.g., periodic, seasonal, or highly irregular streams). The evaluation is limited to next‑event prediction; other tasks such as long‑horizon forecasting or sequence generation could exhibit different trade‑offs. Finally, the exploration of hyper‑parameters (bin counts, number of RSQ levels) is not exhaustive, leaving room for further optimization.

Conclusion:

This paper delivers the first comprehensive empirical comparison of temporal tokenization strategies for LLM‑based event‑sequence modeling. It demonstrates that while event‑type prediction is relatively robust to the choice of tokenizer, accurate time prediction hinges on aligning the tokenization scheme with the underlying temporal distribution. Log‑based quantization (scale binning, RSQ) is optimal for log‑normal or spiky data, calendar tokens are preferable for mixed‑modal streams, and byte‑level encoding offers the highest raw precision. Importantly, the authors show that a straightforward LLM fine‑tuning pipeline can rival more complex TPP‑specific models, offering a simpler, scalable alternative for practitioners. Future work should explore larger models, broader datasets, and additional downstream tasks to further refine temporal tokenization best practices.