How AI Impacts Skill Formation

AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation – particularly in safety-critical domains.

💡 Research Summary

The paper investigates how AI coding assistants affect skill formation in novice software developers, focusing on a newly introduced asynchronous Python library (Trio). The authors conduct a randomized controlled experiment where participants are assigned two programming tasks that require learning the library’s concepts (structured concurrency, error handling, channels). The control group works without AI assistance, while the treatment group uses a GPT‑4o‑based chat assistant that can generate full code, answer conceptual questions, and provide explanations.

Performance is measured along three dimensions: (1) task completion time, (2) a post‑task knowledge quiz covering conceptual understanding, code reading, and debugging, and (3) interaction logs that are later classified into six distinct AI usage patterns. The main findings are:

1. Productivity: AI assistance does not yield a statistically significant reduction in completion time. Although some participants finish slightly faster, many spend a substantial portion of the allotted time (up to 30 % of the session) formulating prompts and reviewing AI output, offsetting any speed gains.

2. Skill Acquisition: Participants who used AI score on average 17 % lower on the knowledge quiz (Cohen’s d = 0.738, p = 0.010). The decline is most pronounced in code‑reading and debugging sub‑scores, suggesting that receiving ready‑made code bypasses the critical error‑identification phase that underlies deep learning.

3. Interaction Patterns: Six patterns emerge from the log analysis:

   • AI Delegation: Users request the assistant to write the entire solution. This yields modest productivity improvements but the lowest learning scores.
   • Progressive Reliance: Users start with conceptual questions, then gradually request more concrete code snippets.
   • Hybrid Code‑Explanation: Users ask for code together with detailed explanations of each component.
   • Iterative Debugging, Quiz‑Style Queries, and Simple Search are the remaining patterns.

   The latter three patterns (Progressive Reliance, Hybrid Code‑Explanation, and Iterative Debugging) involve higher cognitive engagement; participants in these groups achieve quiz scores comparable to or slightly above the control group. This demonstrates that AI can be a learning aid when it prompts the user to think, explain, and verify rather than merely copy.

The authors frame these results within the concepts of cognitive offloading and overreliance. When AI supplies answers without requiring the user to process underlying logic, the user’s supervisory competence erodes—a risk especially acute in safety‑critical domains such as medical software, autonomous vehicles, or aerospace systems where human oversight must catch AI errors.

Limitations include the focus on a single library, short‑term assessment (no longitudinal retention test), and reliance on a specific model (GPT‑4o). Nonetheless, the study provides robust causal evidence that AI assistance can impair skill formation unless interaction is deliberately structured to maintain active reasoning.

Implications: Organizations should treat AI assistants as partners rather than shortcuts. Recommended practices include: (a) designing prompts that require explanations, (b) mandating a verification step where developers manually inspect and test AI‑generated code, (c) monitoring interaction logs to ensure users are not falling into pure delegation, and (d) integrating AI usage guidelines into onboarding and training curricula. By doing so, firms can reap productivity benefits while preserving the essential human ability to supervise, debug, and improve AI‑augmented systems.