Achieving Productivity Gains with AI-based IDE features: A Journey at Google

We discuss Google’s journey in developing and refining two internal AI-based IDE features: code completion and natural-language-driven code transformation (Transform Code). We address challenges in latency, user experience and suggestion quality, all backed by rigorous experimentation. The article serves as an example of how to refine AI developer tools across the user interface, backend, and model layers, to deliver tangible productivity improvements in an enterprise setting.

💡 Research Summary

Google’s paper chronicles the end‑to‑end development, deployment, and evaluation of two internal AI‑driven IDE features—code completion and Transform Code (a natural‑language‑driven code‑edit tool). The authors emphasize that delivering measurable productivity gains at scale requires more than a high‑quality language model; it demands careful engineering across the UI, backend, and model layers, coupled with rigorous online experimentation.

Code Completion
The feature predicts the code that follows the cursor using a fill‑in‑the‑middle (FIM) formulation. Training data consist of keystroke‑level edit histories from Google engineers, ensuring an in‑distribution signal. Scaling up from earlier 0.5 B‑parameter encoder‑decoder models to larger LLMs introduced latency and cost challenges. Google mitigated these by (1) implementing adaptive caching (“streaks”) that reuses recent predictions, (2) employing speculative decoding to keep the serving model small while preserving quality, and (3) limiting input to 8 K tokens and output to 128 tokens with a compact serialization format.

The adaptive cache works by checking whether an incoming request can be satisfied by a previously generated “streak” response (e.g., reusing a suggestion for “Bu()” after having generated one for “B()”). If not, the request is queued, and the system either cancels it when the user types ahead or adapts a pending model response to pending requests. This approach yielded a 35 % cache‑hit rate, reduced median latency (p50) by 9 % and p90 latency by 2 %, and increased suggestion acceptance by 17 %. Because more suggestions were shown and a higher proportion were accepted, the fraction of code written by ML (FCML) rose by 41 %.

Contextual prompting was treated as a packing problem: relevant snippets from recent edits and opened files were selected, ranked, and rendered with surrounding scopes (class headers, function signatures) to stay within a token budget while preserving semantic context. In a 2‑week A/B test with >2,500 users per arm, this yielded a 5 % lift in acceptance and an 11 % lift in FCML, at the cost of a 46 % increase in median latency and a 5 % reduction in the number of displayed suggestions.

Transform Code
Transform Code enables developers to select a code region, type a natural‑language prompt (e.g., “replace abc with 1‑3”), and receive a concise edit. The model is a Gemini LLM fine‑tuned on internal code and trained to emit edits in a compact Unified Diff format anchored by three unchanged lines.

Key challenges were discoverability, review ergonomics for multi‑line edits, and a distribution gap between high‑quality reviewed code (training data) and the noisy, in‑progress code encountered in the IDE. To improve discoverability, Google added a floating button that appears next to a selection, supplemented by a keyboard shortcut and menu entry. In a month‑long A/B test with >10 k users per group, the floating button increased total prompts by 40 % and the proportion of users issuing at least one prompt by 64 %; shortcut usage rose by 19 %.

For review ergonomics, diff rendering was refined to highlight only truly edited lines, differentiate moved lines from added/removed ones, and reduce visual clutter. This reduced average review time by 7 % and lifted acceptance by 2.2 %; for larger diffs (>133 characters) acceptance rose by 4.5 %.

To bridge the distribution gap, Google collected “user rewrites”: after rejecting a Transform Code suggestion, engineers could record the manual edit they performed. After manual curation to remove unrelated changes, ~100 high‑quality rewrites were added to supervised fine‑tuning, and 200 conversational edit examples were created to teach multi‑turn behavior. Offline evaluation on a held‑out rewrite set showed ChrF improve from 84.5 to 88.8 and unrelated‑file edits drop from 13 % to 0 %. In‑product A/B tests, models trained with these rewrites increased acceptance from 55 % to 63 %.

Productivity Measurement
Google distinguishes proxy metrics (FCML, acceptance rate) from true productivity signals. The latter are derived from log‑based measures such as Change List throughput (CLs per month), active coding time per CL, and mean duration of investigation sessions (time spent searching external resources). Causal inference combines online A/B experiments with offline observational analyses. The paper reports that after rolling out the refined features, CL submission rates and active coding time showed statistically significant gains, confirming that the AI tools translate into real developer productivity improvements.

Conclusion
The study demonstrates that a disciplined, data‑driven loop—model scaling, prompt engineering, UI refinements, and continuous user‑feedback collection—can turn experimental LLM capabilities into enterprise‑scale productivity gains. The authors anticipate that future, more ambitious AI‑assisted coding tasks will follow the same iterative, measurement‑focused methodology.