Should We Buy More Copilot Seats? Check the Repo First

TL;DR: "Should we buy more Copilot seats?" is really "did the seats we already bought change what ships?" Vendor dashboards can't answer that; acceptance rates and active-user counts measure suggestion traffic. The answer lives in your repo: the cycle time, review latency, and rework of AI-attributed merged PRs versus your baseline, per tool. If that cohort ships clean and fast, buy the seats. If it ships at baseline speed with double the rework, you're about to pay twice, at scale.

The question behind the seat question

A seat-expansion request lands on your desk with a vendor quote and a usage report. The usage report says adoption is strong: most licensed developers active weekly, acceptance rates healthy, thousands of suggestions accepted. The quote scales linearly. The missing document is the one that matters: evidence that the existing seats changed delivered outcomes, because that's the thing you're being asked to buy more of.

This is the general shape of AI coding ROI, but the seat decision is its sharpest, most recurring instance: renewal and expansion are the moments a real number is worth actual money, in both directions. Overbuying seats for a tool that isn't landing wastes budget; underbuying one that's printing clean leverage wastes more.

Why acceptance rate and active users can't answer it

Vendor usage metrics measure the top of the funnel: a suggestion appeared, a developer accepted it, a session happened. Between that moment and business value sit review, merge, deployment, and the two-week window where unripe code comes back for rework. Acceptance metrics see none of it. Accepted code that a reviewer rewrites counts as a win. Accepted code that ships and churns out twelve days later counts as a win. A developer who accepts constantly and ships slower than before counts as your most engaged user.

And you can't patch the gap by asking developers whether the tool helps, because perception fails in sign here: in METR's randomized trial, experienced developers believed AI made them about 20% faster while measurement showed 19% slower. Every seat-expansion deck built on survey sentiment plus acceptance rate is two unreliable instruments citing each other.

What the research says the return depends on

The honest reading of the published evidence is that Copilot-class ROI is real, context-dependent, and capable of going negative:

• The best case is genuinely large. In the controlled Copilot experiment by Peng et al., developers with Copilot completed a scoped greenfield task 55.8% faster than control (n=95), with the biggest gains for less-experienced developers.
• The worst case is negative. METR's trial - experienced developers, mature codebases, real tasks - measured 19% slower with AI allowed.
• At the org level, unmanaged adoption dents delivery. DORA 2024 estimated a 25% increase in AI adoption associated with a 1.5% decrease in throughput and 7.2% decrease in delivery stability; GitClear's 623M-change dataset shows where that goes: duplication up 81%, two-week churn up 15%. (GitClear's research is strong enough that we cite it while competing with them honestly.)

Same tool family, results spanning "transformative" to "net negative," depending on task mix, codebase maturity, and the org's absorption capacity. Which means the study you should trust for your seat decision is the one running in your own repo. The feeling-vs-shipping gap is exactly why it has to be measured rather than polled.

The comparison that answers it

Split your merged PRs by attribution and compare cohorts, per tool, over the same window:

What to compare	Why it decides seats
Cycle time p50	Does the tool's cohort actually ship faster than baseline? This is the claim you're buying.
Review latency p50	Is AI volume outrunning your reviewers? A tool that saves authoring time by spending reviewer time may be net negative at current bench depth.
PR size p50	Bigger PRs review worse. A tool whose cohort runs oversized is upstream of your next quality problem.
Rework share	The paying-twice detector: what fraction of the cohort's work gets re-edited shortly after shipping, versus the human baseline.

Three rules keep the comparison honest. Per tool, not blended - Copilot's coding agent, Cursor, Claude Code, and an AI review bot are separate purchase decisions; a blended "AI cohort" lets a strong tool subsidize a weak one, which is precisely the error a seat decision can't afford. Respect the sample floor - a tool with five merged PRs gets "n=5, no verdict," not a verdict. Never per person - the unit of decision is the tool and the team, and the moment cohort data turns into a per-developer usage ranking, people game the attribution and your evidence dies.

Prerequisite, as always: attribution signals must exist; the setup is in how to measure AI adoption. Note that measurement platforms disagree on how to count "AI-assisted" in the first place (telemetry heuristics, classifiers, or signed evidence only), and that choice changes the number your seat decision rests on. We've written cited head-to-heads on exactly this axis against Swarmia, DX, and Faros AI, concessions included.

The Copilot-specific catch

The caveat most measurement pitches skip: Copilot's classic completion mode is invisible to git. Tab-completions leave no trailer, no bot identity, no trace, so a git-based cohort can't isolate completion-mode Copilot, period. What git does see of the Copilot family: the coding agent's commits and PRs (it signs its work), and Copilot's PR review bot (a distinct, measurable fact: the share of your merged PRs it actually reviewed).

For the completion-mode share of the decision, the honest instrument is coarser: treat the rollout as a natural experiment and compare your delivery and quality trends across the adoption boundary. Cycle time, review latency, rework, PR size, before versus after. Closing the blind spot properly would take vendor usage-API integration; anyone claiming precise per-seat completion ROI from git alone is selling you an estimate dressed as a measurement.

The decision rules

• Cohort faster than baseline, rework at or below baseline, healthy sample → clean leverage. Expand seats; consider whether the readiness factors that made it work are strong everywhere you're expanding into.
• Cohort at baseline speed with elevated rework → paying twice. Don't scale it. First check where the damage concentrates and gate those areas; renegotiate or retarget the seats meanwhile.
• Sample too small to judge → the honest answer is "not yet measurable." Fix attribution and decide next quarter with evidence instead of extending a guess.
• Verdicts differ by tool → spend where the receipts are. The seat budget is an allocation problem, and you finally have the allocation data.

Getting the comparison without building it

You can build this: attribution parsing, per-tool cohort joins, medians, rework windows, sample-floor logic - all deterministic, all maintainable forever as tools multiply.

Or connect your repos to Busfactor's AI impact view and read it off the screen: a per-tool table (commits, PRs, cycle p50, review p50, size p50, rework) with each tool's delta against the unattributed baseline, verdicts withheld below the sample floor and labeled n=X instead, the AI-reviewed share with per-bot receipts, and the headline clean-leverage-or-paying-twice verdict with every number quoted from the engine. Tool patterns are configurable, so new agents join the table the week they show up in your commits.

The limits, stated plainly: attribution sees agentic AI that signs its work; completion-mode usage needs the boundary-trend approach above, and Busfactor doesn't ingest vendor seat-activity APIs today. Rework is a file-grain proxy, read as an AI-to-human ratio. And nothing here scores a developer.

Before the renewal call, run the comparison. Either you'll expand with a number the CFO can't argue with, or you'll catch a tool billing you twice - and both outcomes pay for the effort the same week.