ArcMark: Multi-bit LLM Watermark via Optimal Transport

Watermarking is an important tool for promoting the responsible use of language models (LMs). Existing watermarks insert a signal into generated tokens that either flags LM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of recent multi-bit watermarks insert several bits into text without perturbing average next-token predictions, they largely extend design principles from the zero-bit setting, such as encoding a single bit per token. Notably, the information-theoretic capacity of multi-bit watermarking – the maximum number of bits per token that can be inserted and detected without changing average next-token predictions – has remained unknown. We address this gap by deriving the first capacity characterization of multi-bit watermarks. Our results inform the design of ArcMark: a new watermark construction based on coding-theoretic principles that, under certain assumptions, achieves the capacity of the multi-bit watermark channel. In practice, ArcMark outperforms competing multi-bit watermarks in terms of bit rate per token and detection accuracy. Our work demonstrates that LM watermarking is fundamentally a channel coding problem, paving the way for principled coding-theoretic approaches to watermark design.

💡 Research Summary

The paper tackles the problem of embedding multiple bits into the output of large language models (LLMs) without altering the model’s average next‑token distribution—a requirement the authors term “distortion‑free.” By framing watermarking as a communication problem over a noisy channel with side information available at both encoder (the watermarking LLM) and decoder, the authors derive the first information‑theoretic capacity for multi‑bit LLM watermarking.

Theoretical contribution.
Assuming the LLM’s token‑level conditional distributions are independent and identically distributed (i.i.d.) across time, the generation process can be reduced to a single‑letter channel: a random pair (W, Q) produces a token X, where Q is the original next‑token distribution (a point in the probability simplex) and W encodes the message together with the shared secret key. Under the distortion‑free constraint, the marginal distribution of X must equal Q for every possible message. The capacity is then expressed as

\