AI-Assisted Development: What the Evidence Says

TL;DR: The evidence on AI-assisted development is genuinely split, and the split is the useful part. Controlled studies show dramatic speedups on bounded, well-specified tasks and measured slowdowns for experts on mature codebases. Repo-scale data shows quality eroding at high adoption. Org-level research shows individual gains coexisting with delivery losses. Run AI-assisted development as an engineering practice with measurement and it works. Run it as a faith and it invoices you later.

What the evidence on AI-assisted development says

Strip away the vendor decks and the doom threads and the serious research on AI coding tools comes down to a handful of studies that refuse to agree with each other. The disagreement is the most useful thing about them.

On one side: a controlled experiment by Peng, Kalliamvakou, Cihon, and Demirer recruited 95 professional developers, asked them to implement an HTTP server in JavaScript, and gave half of them GitHub Copilot. The treatment group finished 55.8% faster. The 95% confidence interval runs from 21% to 89%, a band wide enough that the point estimate deserves fewer headlines than it gets. Developers with less programming experience benefited the most.

On the other side: METR's randomized controlled trial put 16 experienced open-source developers on 246 real tasks (bug fixes, features, refactors) in mature codebases they'd contributed to for years. With AI allowed, they were 19% slower. Before the study they expected a 24% speedup. Afterward, having lived the slowdown, they still estimated they'd been 20% faster.

Both studies are real, controlled and honest. They measure different work. A bounded greenfield task with a clear spec is the demo condition. A mature codebase full of invariants that live in senior engineers' heads is the Tuesday condition. AI-assisted development has no single effect size. The payoff flips sign with the context you deploy it in, and anyone selling you one number - in either direction - is selling.

Where AI coding tools measurably help

The honest positive case, per the evidence:

• Bounded, well-specified tasks. The 55.8% speedup came from exactly this shape: clear goal, self-contained scope, no hidden context. Boilerplate, scaffolding, one-off scripts, and well-trodden patterns are where generation shines.
• Less-experienced developers on unfamiliar ground. The same study found the largest gains among developers with less experience. As a tireless pair for someone learning a stack, the tools earn their seat.
• Individual experience of work. DORA's 2024 report found AI adoption significantly increases individual productivity, flow, and job satisfaction. That's not nothing: flow and satisfaction are retention currency.

Adoption reflects this. In the Stack Overflow 2024 survey, 76% of developers were using or planning to use AI tools, up from 70% the year before. The tools are here. The open question is whether you'd know what they're doing to your delivery.

Where AI coding tools quietly cost you

The costs show up later than the gains, and in different places.

Expert work on mature systems. METR's result stands as the cleanest warning: on codebases where typing was never the bottleneck, AI access made experts measurably slower - and their perception missed the sign of the effect. The full anatomy of that miscalibration is in the AI productivity paradox.

Code quality at scale. GitClear's 2026 analysis of 623 million code changes measured the AI era's repo-level fingerprint: code-block duplication up 81%, within-commit copy/paste rising from 9.4% to 15.7% of changed lines, refactoring line-moves down about 70%, and two-week churn up 15%. Generation made copy-paste cheaper than refactoring, and teams under pressure took the discount. What the pattern looks like up close, and how to catch it in your own repo, is the subject of AI slop code and AI code churn. What it costs over time is the debt nobody prices.

Downstream defects. Faros AI's 2026 AI Engineering Report measured bugs per developer up 54% at high AI adoption. More code went in; more of it came back.

Developers themselves aren't fooled about the limits: in the same Stack Overflow survey, only 43% trust the accuracy of AI output, and 44.8% of professional developers rate the tools bad or very poor at handling complex tasks. Adoption at 76%, trust at 43%. The 33-point gap is where unreviewed risk lives.

The paradox: individual gains, organizational losses

The most important org-level finding comes from DORA's 2024 report, and it holds both truths at once. Individually, AI adoption improved productivity, flow, and job satisfaction. Organizationally, DORA's estimates associated a 25% increase in AI adoption with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability.

Individuals feel faster while delivery gets slower and shakier. That's not a contradiction, it's a pipeline. Generation is cheap. Everything after it (review, integration, rework, incident response) is not, and those costs land on the organization's metrics instead of the individual's experience.

Survey only your developers and you'll conclude AI is winning. Read your own delivery data and you may conclude something else.

How to run AI-assisted development like an engineer

The evidence doesn't say ban the tools or buy more seats. It says instrument. The playbook, each piece covered in depth in its own guide:

1. Measure ROI on outcomes, not acceptance rates. Cycle time, delivered throughput, and quality costs - before versus after, on your own repo. The full method is in how to measure the ROI of AI coding tools.
2. Track the quality bill explicitly. Short-window churn, duplication trend, and rework rate are where the deferred costs surface first. Price them like the liability they are - see technical debt from AI-generated code.
3. Keep review human where context matters. AI review is a useful pre-filter and a dangerous approver. Where the line sits is the subject of AI code review tools: what they catch and miss.
4. Attribute honestly or not at all. Compare AI-assisted versus non-assisted work only via explicit signals: commit trailers, bot and agent accounts. Guessed attribution produces confident nonsense.
5. Judge trends over quarters, not sprints. Every one of the studies above measured over months. Your evaluation deserves the same patience. A tool that pays off in week one and invoices you in month three will fool any sprint-length assessment.

The teams that come out of this era ahead will be the ones who can answer, with receipts from their own git history, the one question that matters: is this compounding our codebase, or our debt? That's a measurement question, and measurement questions have honest answers.