Three studies, three answers
Ask whether AI coding assistants make engineers faster and you can pick your number. In a 2023 lab experiment, developers with GitHub Copilot finished a small task 55.8 percent faster. In a 2024 trial at Google, the speedup on an enterprise task was about 21 percent. In a 2025 field trial by METR, experienced open-source developers working on their own repositories were 19 percent slower with AI, while believing they had been 20 percent faster.
All three are honest studies. They measured different things. For a team lead or a CTO deciding how to roll out assistants, the useful question is not "faster or slower" but "faster at what, and what does it cost downstream". This article walks through the evidence and ends with a four-week test you can run on your own team.
What each study measured
| Study | Who and what | Result | What it does not tell you |
|---|---|---|---|
| GitHub Copilot experiment, 2023 | 95 developers, one task: write an HTTP server in JavaScript | 55.8% faster with Copilot | Small greenfield task, no review, no existing codebase |
| Google enterprise RCT, 2024 | 96 Google engineers, one realistic task in internal tooling | About 21% faster | One task type; the confidence interval is wide |
| METR field trial, 2025 | 16 experienced maintainers, 246 real issues in their own large repositories | 19% slower; developers predicted 24% faster and felt 20% faster | Early-2025 tools; METR itself says newer tools may differ |
| DORA report, 2025 | Survey of about 5,000 technology professionals | More AI adoption goes with higher throughput and higher instability | Correlation, not a controlled experiment |
Read together, the pattern is consistent. Assistants help most on small, well-defined, unfamiliar tasks. They help least, or hurt, when an expert works in a large codebase they already know by heart. And the time saved writing code reappears later as review, debugging and incidents.
Where the time goes
The METR paper breaks down where the extra time came from. Developers spent less time writing code and searching, and more time prompting, waiting for output and reviewing it. Many suggestions were rejected after reading. On a codebase with years of implicit conventions, the model's draft is often plausible and wrong in a way only the maintainer sees.
DORA describes the same effect at the team level. Throughput rises because more changes are produced. Instability rises because the review, test and release system was sized for the old volume. Their phrase for it: AI amplifies whatever delivery system you already have. Weak tests plus more code equals more incidents.
My reading: the assistant moves effort from typing to judging. Teams that were limited by typing speed get faster. Teams that were limited by review, unclear requirements or flaky tests do not, and some get slower.
What to measure instead of "productivity"
| Metric | Why it matters with AI | Warning sign |
|---|---|---|
| Lead time for changes | The only speed number users feel | Flat while PR count rises |
| Change failure rate | Catches the instability DORA describes | Rising after rollout |
| Review time per PR | Where the saved writing time often goes | Growing, or shrinking suspiciously fast |
| PR size | Assistants make large diffs cheap to produce and expensive to review | Median lines changed doubling |
| Time to restore service | Shows whether the team still understands its code | Incidents take longer to diagnose |
A four-week test on your own team
Week 0: baseline
Pull lead time, change failure rate, review time and PR size for the last 8 weeks. Without a baseline every later number is an opinion.
Weeks 1–4: split the work, not the people
Tag comparable tickets as "AI allowed" or "AI off", alternating. Same engineers on both, which is the design METR used.
Track time on the ticket, not feelings
Developers in the METR trial misjudged their own speed by about 40 percentage points. Use timestamps from the tracker and Git.
Review by ticket type
Split results into new code, changes to old code, tests and boilerplate. Expect different answers for each.
Decide per task type
Encourage assistants where they won, add review rules where failures rose, and repeat the test in a quarter: the tools change faster than most rollout plans.
What this means for hiring
If assistants move work from writing to judging, the skill you hire for moves too. Reading unfamiliar code, spotting a plausible bug and saying no to a large diff matter more than typing speed. Several companies already run an AI-enabled coding round for exactly this reason. If you hire engineers, test review in the loop; if you are an engineer, practise it with a partner on a pair mock with the assistant open.
The short version
The studies disagree because they measure different work. Assistants speed up small, unfamiliar tasks and can slow experts in large codebases. Teams produce more code and, without strong tests and review, more incidents. Measure lead time and failure rate, run a four-week split on your own tickets, and decide per task type.
Related: coding interviews when AI is everywhere, will AI take engineering jobs, structured technical interviews.
Questions readers ask
-
So do AI assistants make developers faster or not?
On small, well-defined tasks the controlled studies say yes, by 20 to 55 percent. On large codebases the developers already know well, one careful trial measured the opposite. Your answer depends on which of those your team mostly does.
Link to this answer -
Why did developers in the METR trial think they were faster?
Waiting for a model feels like progress, and reviewing generated code feels lighter than writing it. The clock disagreed: they forecast 24 percent faster and measured 19 percent slower.
Link to this answer -
Which metric should a team lead watch?
Lead time and change failure rate together. Throughput alone rewards shipping more broken changes; DORA 2025 found exactly that pattern on teams with weak tests.
Link to this answer -
Should we ban assistants until we know?
No. Run a four-week comparison on comparable tickets instead. A ban teaches you nothing and pushes usage onto personal accounts.
Link to this answer -
Does this change how you interview engineers?
Yes. Reviewing generated code becomes part of the job, so it belongs in the loop. Our article on coding interviews with AI covers both formats.
Link to this answer
Sources
- METR (2025): Measuring the impact of early-2025 AI on experienced open-source developer productivity
- Peng et al. (2023): The impact of AI on developer productivity, evidence from GitHub Copilot
- Paradis et al. (2024): How much does AI impact development speed? An enterprise-based randomized controlled trial
- DORA: State of AI-assisted software development 2025
Figures were checked against these pages in October 2026. Pay and dates change; the links show where to recheck.