The Algorithmic Advantage: How Reinforcement Learning Generates Rich Communication

We analyze strategic communication when advice is generated by a reinforcement-learning algorithm rather than by a fully rational sender. Building on the cheap-talk framework of Crawford and Sobel (1982), an advisor adapts its messages based on payoff feedback, while a decision maker best-responds. We provide a theoretical analysis of the long-run communication outcomes induced by such reward-driven adaptation. With aligned preferences, we establish that learning robustly leads to informative communication even from uninformative initial policies. With misaligned preferences, no stable outcome exists; instead, learning generates cycles that sustain highly informative communication and payoffs exceeding those of any static equilibrium.

💡 Research Summary

The paper embeds a tabular Q‑learning algorithm into the classic Crawford‑Sobel cheap‑talk model to study how algorithmic advice shapes strategic communication. A sender observes a state X (uniformly drawn from a finite grid) and, instead of being a fully rational agent, selects a message m using a soft‑max policy based on its Q‑values. After receiving the message, a rational receiver updates beliefs and chooses an action y that minimizes (y‑x)². The sender’s payoff is –(y‑x‑b)², where b>0 captures a bias toward higher actions (misaligned preferences).

Aligned preferences (b=0). The authors prove that any attracting policy of the learning dynamics must convey a non‑trivial amount of payoff‑relevant information. They derive a tight welfare lower bound: in calibrated simulations the long‑run welfare is at least 98 % of the surplus achievable under full revelation. Importantly, this result does not rely on knife‑edge mixing or favorable initial conditions; even when the algorithm starts with a completely babbling policy, the reward‑driven adaptation forces it to learn an informative mapping. Exploration (ε_t) is essential for learning but simultaneously induces the receiver to treat messages as noisy, leading to more cautious actions and preventing the fully revealing equilibrium from being perfectly stable.

Misaligned preferences (b>0). In this case the classic partition equilibria of Crawford‑Sobel are not stable under Q‑learning, and the learning process never settles at a fixed point. Simulations reveal a characteristic pattern: the state space is split by a threshold θ. Above θ the sender pools all states into a single message, while below θ the policy continually oscillates near full revelation. This creates a stable cycle in which the sender tries to “explain away” lower states using the higher‑state message because of the bias b. Both sender and receiver obtain average payoffs that exceed any static equilibrium payoff identified in the original cheap‑talk literature.

The paper also highlights the dual role of exploration. High exploration rates enable the algorithm to discover better mappings but also dilute the credibility of messages, which can lead to persistent pooling at the extremes of the state space. Conversely, low exploration improves informational efficiency but slows convergence.

Methodologically, the authors choose tabular Q‑learning for its analytical tractability and transparency, arguing that it serves as a benchmark for more complex deep‑RL systems. They discuss external validity, noting that many real‑world recommendation systems expose exploration to users, who then adjust expectations, justifying the assumption of a myopic rational receiver.

In the related‑work discussion, the paper positions itself between economic cheap‑talk theory, evolutionary language emergence, and computer‑science studies of emergent communication. It provides the first rigorous welfare bound for an algorithmic sender in the aligned‑interest case and demonstrates that learning can generate payoffs higher than any static equilibrium when interests diverge.

Overall, the study shows that reinforcement‑learning‑based advice can generate rich, informative communication without explicit modeling of the underlying information, but the same mechanisms that make RL attractive (model‑free learning, exploration, reward‑driven updates) also impose limits: full welfare optimality may be unattainable under aligned interests, and misaligned interests lead to perpetual cycles rather than stable equilibria. These insights are crucial for designing algorithmic recommendation or pricing tools that interact with human decision‑makers.