You're measuring AI ROI wrong: the metrics that actually show team impact

You bought your team Copilot or Claude, and three months later the CFO sends a simple question: 'Is it worth the money?' And you realize you don't have an answer. You open the dashboard your vendor gives you and see numbers like '8,200 suggestions accepted' or '340,000 lines of code generated.' It sounds impressive. It means nothing.

The vendor showed me the team accepted 71% of suggestions. I asked: are we shipping faster? Silence. — engineering manager at one of my trainings

I see this problem constantly. Companies measure activity because it's easy to measure, and hope activity means value. It doesn't. Let's line it up — which metrics mislead you, and which actually show whether AI is helping your team.

Why LOC and acceptance rate mislead

Lines of code generated is the worst metric you can pick. More code isn't better code — often it's the opposite. AI will happily generate 200 lines of boilerplate where a senior would write 20 lines of an elegant solution. When you reward volume, you reward codebase bloat, more surface area for bugs, and more code to maintain. Lines of code as a productivity metric was nonsense back in the 90s; with AI it's dangerous.

Acceptance rate — the percentage of suggestions a developer accepts — sounds smarter, but it measures willingness to hit Tab, not the quality of the result. A developer can accept autocomplete all day and then spend an hour fixing what they accepted. A high acceptance rate happily coexists with declining productivity. You're measuring how often people say 'yes' to AI, not whether it helped.

What to measure instead: outcomes, not activity

Good metrics share one property: they measure what happens to the work, not how much of it AI produced. Here are the ones that work in practice.

• Cycle time — from first commit to production deploy. The most direct signal of whether the team ships faster.
• Code review time — from PR open to merge. AI often helps most right here (pre-review, PR descriptions, explaining a diff).
• Time-to-first-PR for new hires — how long it takes a new person to ship a first meaningful PR. A good indicator of whether AI speeds up onboarding.
• Incident and bug rate — guards against speed coming at the cost of quality. Crucial: track it alongside cycle time, not separately.
• Developer-reported friction — a short recurring question to the team: 'Where did AI save you time this sprint, and where did it slow you down?'

Notice that none of these come from the AI tool's dashboard. Cycle time, review time, and bug rate are already in your Git history, Jira, or Linear. You don't need to buy a new tool — you just need to look at data you already collect.

The strongest pair is cycle time + bug rate. When cycle time drops and bug rate stays flat or drops too, you have real evidence AI is helping. When cycle time drops but bugs spike, the team is just shipping low-quality code faster — and that's worse than nothing.

Baseline: without a 'before' number you have nothing

The most common mistake: a company rolls out AI and only then starts measuring. Then it has nothing to compare against. 'Cycle time is 3.1 days' isn't information — it's a number with no context. 'Cycle time dropped from 4.8 to 3.1 days over the quarter' is information you can defend to leadership.

Before you roll out AI broadly, capture 2–4 weeks of baseline. Most of these metrics can be pulled retroactively from Git and your issue tracker, so you can often reconstruct the baseline even after the fact — calculate cycle time and review time for the last month before the team started using AI for real.

# Lightweight baseline from Git (before AI rollout)

## Cycle time (commit -> deploy)
- Median over last 4 weeks: ___ days

## Review time (PR open -> merge)
- Median over last 4 weeks: ___ hours

## Bug rate
- Production bugs / month: ___

## Onboarding
- Time-to-first-PR of last new hire: ___ days

=> Measure the same metrics in a quarter and compare.

Don't ignore the qualitative signals

Not everything fits in a chart. Some of the most valuable signals are qualitative and come from regularly asking the team. At retrospective, ask two questions: 'Where did AI help you most this sprint?' and 'Where did it slow you down or lead you into a dead end?'

The answers tell you things the numbers won't: that AI is great at tests but terrible at your legacy module; that it saves juniors hours while occasionally slowing seniors down; that after introducing AI code review, people burn out less on tedious PRs. These are real impacts cycle time won't capture. Track simple sentiment too — 'I wouldn't want to give it up' is a stronger signal than any percentage.

A lightweight measurement plan for one quarter

You don't need a data team or a new tool. One lead can run this over a quarter alongside their normal work.

• Week 0: Pull the baseline from Git and your issue tracker — cycle time, review time, bug rate, time-to-first-PR. Write the numbers down.
• Weeks 1–2: Roll out AI deliberately (not with a 'we use AI now' email), give people time and training.
• Every sprint: Two AI questions at retrospective. Write down what people say.
• Week 12: Measure the same metrics as the baseline. Compare. Add a qualitative summary from the retros.
• Decide: Which use cases clearly work (scale them), which don't (stop pushing them), where more training is needed.

At the end of the quarter you have one page: before-and-after numbers, plus what the team says. That's an investment case a CFO understands — and at the same time a map of where AI genuinely helps your team and where you're just making noise.