LLMs and people both learn to form conventions -- just not with each other

Humans align to one another in conversation – adopting shared conventions that ease communication. We test whether LLMs form the same kinds of conventions in a multimodal communication game. Both humans and LLMs display evidence of convention-formation (increasing the accuracy and consistency of their turns while decreasing their length) when communicating in same-type dyads (humans with humans, AI with AI). However, heterogenous human-AI pairs fail – suggesting differences in communicative tendencies. In Experiment 2, we ask whether LLMs can be induced to behave more like human conversants, by prompting them to produce superficially humanlike behavior. While the length of their messages matches that of human pairs, accuracy and lexical overlap in human-LLM pairs continues to lag behind that of both human-human and AI-AI pairs. These results suggest that conversational alignment requires more than just the ability to mimic previous interactions, but also shared interpretative biases toward the meanings that are conveyed.

💡 Research Summary

This paper investigates whether large language models (LLMs) can develop conversational conventions in the same way humans do, using a visual reference game based on tangram figures. Three partnership types were examined: Human‑Human (H‑H), Human‑AI (H‑AI), and AI‑AI. In Experiment 1, each dyad completed 50 turns (five blocks of ten images) in which partners alternated between director (describing a target figure) and matcher (selecting the described figure from a grid). Three quantitative metrics captured convention formation: (1) accuracy (whether the matcher chose the correct figure), (2) utterance length (number of words), and (3) lexical overlap (Jaccard similarity) between successive descriptions of the same figure.

Results showed classic convention signatures for H‑H and AI‑AI dyads. H‑H pairs started with high accuracy (≈81 %) and improved to 93 % by the final block, while their utterances shrank from about 20 words to 6 words and lexical overlap rose from 0.23 to 0.43. AI‑AI pairs began with lower accuracy (≈56 %) but quickly caught up, reaching 99 % by block 4; however, their utterance length remained essentially unchanged (≈58 → 56 words) and lexical overlap was already high from the start, staying above the H‑H level. By contrast, H‑AI pairs performed poorly: accuracy started at 54 % and only rose to about 70 % by the end, utterance length decreased modestly (≈20 → 10 words), and lexical overlap lagged behind both homogeneous dyads (0.23 → 0.30). Qualitative coding revealed that humans relied heavily on analogical (metaphoric) descriptions (68 % of utterances), whereas AI‑AI utterances were predominantly geometric (38 %) or mixed (57 %). H‑AI utterances mixed all three styles, indicating a stylistic mismatch.

A deeper analysis of turn‑by‑turn dynamics showed that successful previous turns led to higher lexical similarity in subsequent turns for all director/matcher role combinations except when an AI acted as director and a human as matcher. This suggests that AI directors do not adapt their descriptions based on human feedback, a key factor behind the H‑AI deficit.

Experiment 2 tested whether prompting the LLM to “behave more like a human” could close the gap. The prompt instructed the model to be concise, use analogies, and cooperate. While this manipulation succeeded in reducing utterance length for both H‑AI and AI‑AI dyads to levels comparable with H‑H pairs, it did not improve accuracy or lexical overlap for H‑AI dyads. AI‑AI dyads still achieved near‑perfect accuracy and high overlap, but their utterance length remained long, confirming that LLMs can form stable, context‑driven conventions without the pressure to be concise.

The authors interpret these findings through the lens of cooperative communication theory. Humans are guided by Gricean maxims and resource‑sensitivity (metabolic and time costs), which drive them toward short, efficient, and mutually understood expressions. LLMs, by contrast, are optimized for token‑prediction accuracy and reinforcement‑learning‑from‑human‑feedback rewards that do not penalize verbosity. Consequently, LLMs can memorize and reproduce prior descriptions (high lexical overlap) but lack the intrinsic incentive to compress information or to align on the same interpretative stance that humans share. The stylistic divergence—human analogical language versus AI geometric language—further reduces the likelihood of shared meaning, leading to lower accuracy and overlap in mixed dyads.

The paper concludes that mere surface‑level prompting is insufficient for LLMs to achieve human‑like conversational alignment. To bridge the gap, future systems must incorporate mechanisms that emulate human resource constraints (e.g., token‑cost penalties), embed shared ontological or conceptual frameworks, and enable genuine adaptation to partner feedback. Only by integrating these deeper cognitive and social biases can LLMs reliably form conventions with humans, moving beyond statistical mimicry toward truly collaborative dialogue.