Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild

Large language models (LLMs) are increasingly acting as dynamic conversational interfaces, supporting multi-turn interactions that mimic human-like conversation and facilitate complex tasks like coding. While datasets such as LMSYS-Chat-1M and WildChat capture real-world user-LLM conversations, few studies systematically explore the mechanisms of human-LLM collaboration in coding scenarios. What tortuous paths do users experience during the interaction process? How well do the LLMs follow instructions? Are users satisfied? In this paper, we conduct an empirical analysis on human-LLM coding collaboration using LMSYS-Chat-1M and WildChat datasets to explore the human-LLM collaboration mechanism, LLMs’ instruction following ability, and human satisfaction. This study yields interesting findings: 1) Task types shape interaction patterns(linear, star and tree), with code quality optimization favoring linear patterns, design-driven tasks leaning toward tree structures, and queries preferring star patterns; 2) Bug fixing and code refactoring pose greater challenges to LLMs’ instruction following, with non-compliance rates notably higher than in information querying; 3) Code quality optimization and requirements-driven development tasks show lower user satisfaction, whereas structured knowledge queries and algorithm designs yield higher levels. These insights offer recommendations for improving LLM interfaces and user satisfaction in coding collaborations, while highlighting avenues for future research on adaptive dialogue systems. We believe this work broadens understanding of human-LLM synergies and supports more effective AI-assisted development.

💡 Research Summary

This paper, “Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild,” presents a comprehensive empirical investigation into the dynamics of collaborative coding between humans and Large Language Models (LLMs) using real-world conversation data.

Background and Motivation: As LLMs evolve into dynamic conversational interfaces (e.g., ChatGPT, Copilot) supporting multi-turn dialogues, understanding their role in complex, iterative tasks like coding becomes crucial. While datasets like LMSYS-Chat-1M and WildChat capture authentic user-LLM interactions, few studies systematically dissect the collaboration mechanics, instruction-following fidelity, and user satisfaction specifically in coding scenarios. This study aims to bridge this gap by analyzing real-world data to answer three core questions: what interaction patterns emerge (RQ1), how well LLMs follow instructions (RQ2), and how satisfied users are (RQ3).

Methodology: The research follows a three-phase, data-driven approach:

1. Data Curation: Coding-related conversations were extracted from the LMSYS-Chat-1M and WildChat datasets using rule-based matching (e.g., programming language keywords, file extensions), resulting in a unified set of 60,949 interactions (LMSYS-WildChat).
2. Data Disentanglement: To isolate single-topic coding conversations from potentially multi-thematic dialogues, an LLM (DeepSeek-V3) was employed for automatic disentanglement. This process was validated through manual review (92% accuracy, Cohen’s Kappa 0.87). Subsequent filtering yielded 66,371 final conversation logs (46,864 single-turn, 19,507 multi-turn).
3. Empirical Analysis: A statistically sampled set of 378 multi-turn conversations (95% confidence level, ±5% margin of error) was subjected to in-depth analysis. Methods included open card sorting for task taxonomy, graph-based mapping for interaction patterns, statistical tests (Kruskal-Wallis, Kolmogorov-Smirnov) for significance, and satisfaction trajectory analysis.

Key Findings:

• RQ1 (Interaction Patterns): Five primary task types were identified: Design-Driven Development, Requirements-Driven Development, Code Quality Optimization, Environment Configuration, and Information Querying (with 13 subtypes). These tasks manifest in three distinct interaction patterns visualized as graph topologies:
  • Linear Pattern: Characterized by sequential turn-by-turn progression, predominantly found in Code Quality Optimization tasks (e.g., iterative refactoring).
  • Star Pattern: Features a central theme with multiple independent branches, typical for Information Querying tasks.
  • Tree Pattern: Exhibits hierarchical expansion with sub-branches, commonly associated with Design-Driven Development, facilitating exploration of complex solutions.
• RQ2 (LLM Instruction-Following): LLMs’ ability to follow user instructions varies significantly by task type. Bug Fixing and Code Refactoring tasks presented substantially higher rates of non-compliance compared to Information Querying tasks. This highlights LLMs’ current limitations in tasks requiring complex diagnostic reasoning, context maintenance, and deep understanding of existing code structures.
• RQ3 (Human Satisfaction): User satisfaction is highly task-dependent and evolves over conversation length.
  • Tasks like Code Quality Optimization and Requirements-Driven Development were associated with lower satisfaction, often due to repetitive corrections and ambiguous initial requirements.
  • Tasks like Structured Knowledge Queries and Algorithm Design yielded higher satisfaction, as LLMs could provide clear, useful information or solutions.
  • A negative correlation was observed between conversation length and overall satisfaction. Furthermore, the focus of conversations tended to shift from creative/design work in early turns towards error correction in later turns, indicating potential issues with LLM consistency and context management over long dialogues.

Conclusions and Implications: This study moves beyond synthetic benchmarks to provide grounded, empirical insights into real human-LLM coding collaboration. The identified patterns, challenges, and satisfaction drivers offer actionable recommendations for improving LLM-assisted development. Key implications include designing adaptive dialogue systems that recognize interaction patterns and tailor support (e.g., encouraging exploration in tree patterns, strengthening context in bug fixing), enhancing LLM capabilities for particularly challenging task types (e.g., via specialized fine-tuning or tool integration), and developing better strategies for managing long-term interaction context to sustain user satisfaction. Ultimately, this work contributes to a deeper understanding of human-LLM synergy, paving the way for more effective and satisfying AI-assisted software development.