Is our AI assistant actually making the team faster? Here is how to check
To check whether your AI assistant is actually making the team faster, take four readings together. Throughput measured against your own pre-AI quarters; the split between how much activity changed and how much the value of each change changed; cycle time read alongside change size; and where the capacity went by type of work. Any one of those alone will mislead you. Together, all four give you an answer you can defend when someone pushes back.
What faster has to mean before you check your AI assistant
Picture the sprint review where somebody asks the question directly. A staff engineer, three months into the rollout, says typing is faster, thinking is the same, and she isn’t sure which one the company bought.
That’s the measurement problem in one sentence. “Faster” can mean five different things, and the easiest to measure ones matter least.
It can mean more changes merged per week. It can mean each change is worth more. It can mean the gap between starting a ticket and shipping it has shortened. It can mean the share of the quarter going to maintenance dropped. It can also mean developers experience less friction, which is real, worth knowing, and completely separate from delivery.
Pick which one you’re claiming before you open a dashboard. Most disagreements about AI productivity are two people measuring against different definitions.
Does throughput exceed your pre-AI quarters?
This is the first reading, and the only one that answers whether the AI assistant made the team faster, the way a CFO will ask.
Take two quarters that ended before any assistant entered the workflow, score them with the same method you’ll use now, and compare. Same repository set, same contributor rules, bots excluded from both windows. The comparison matters as much as the method’s consistency.
Industry averages can’t do this job. Your review culture, merge policy, and test burden are yours, and the average of six other companies still has none of them.
I spent years as a former CTO answering the AI productivity question with conviction and no instrument, which works until the second follow-up question.
Did activity rise, or did the value of each change rise?
Second reading, and the one that changes how people think about the first.
Activity counts go up when an AI assistant is in the loop, almost mechanically. More commits, more lines touched, more pull requests. That movement is easy to find and easy to over-read.
Navigara’s Q2 2026 study split the change into its two components across 699 engineers and 137,592 qualifying commits.
The two components of the Q2 2026 change
| Component | Year over year | Quarter over quarter |
|---|---|---|
| Commits per engineer | +14.9% | -11.2% |
| Performance per commit | +100.4% | +32.4% |
Open cohort, 699 engineers, 65 public repositories, 78-week window
Grouped columns show that commits per engineer rose 14.9% year over year and fell 11.2% quarter over quarter, while performance per commit rose 100.4% year over year and 32.4% quarter over quarter, highlighting changes in the value of each commit rather than the number of commits.
Commits per engineer moved 14.9% year over year. Performance per commit moved 100.4%. Quarter over quarter, commits per engineer fell by 11.2%, while performance per commit rose by 32.4%.
Run that comparison on a team where someone reports commit volume as an AI assistant productivity signal, and you’ll see the story invert. Over the same period, it reads as a slowdown in activity and a large gain in value.
Two things to keep in mind with those figures. The headline quarter-over-quarter figure for performance per engineer came in at +17.5%, with a 95% confidence interval of -0.7% to +38.8%, and the report is clear about what that means: “The quarter-over-quarter change cannot be distinguished from zero.” The study measures only public repositories and doesn’t attribute the shift to assistant adoption. Any measurement of this kind gives you correlation, including yours.
Cycle time and change size have to be read together
Third reading, and the one most likely to produce a false positive on AI assistant productivity.
Cycle time falls when changes get smaller. AI assistants encourage smaller changes, because a suggestion that fits in one screen is easier to accept than one that spans four files. So cycle time drops, the dashboard turns green, and the quarterly report claims a speedup that mostly reflects how work was cut up.
Read the two series side by side. Cycle time down 30%, median change size down 40%, and throughput flat means the work got chopped, and the review queue absorbed the difference. Cycle time down 30% with change size steady and throughput up means something real happened.
If you want the cycle time and change size series read against each other on your own repositories, connect a repo, and both come back in the first pass alongside the historical window.
Where the capacity went, by type of work
Fourth reading. This is the one that answers the planning question rather than the finance question, and it’s the one engineering managers tend to find most useful about themselves.
Score each merged change by what it was, then look at the shares. In the Q2 2026 window (Q1 2025 to Q2 2026), maintenance lost 10.9 percentage points of share, while tests gained 8.0 and features gained 4.4. Fixes fell 1.3 points, and docs stayed flat. .2. Public repositories only, with visibility into private work, code review depth, incident response, planning, or mentorship.
A team that moved 10 points of its quarter from maintenance into tests and features got faster in a way no velocity chart will show, because velocity counted both categories as points and treated them as equal.
AI assistant acceptance rate does not answer this question
AI assistant vendors report it because it’s the metric they can see. It tells you a developer pressed Tab.
It says nothing about whether the suggestion survived review, whether the resulting change shipped, or whether it introduced the bug that consumed Thursday. A team with 40% acceptance and careful review can be delivering considerably more value than a team with 70% acceptance and a growing revert rate.
AI assistant acceptance rate is a usage signal, and usage signals belong in the rollout conversation. Delivery questions need delivery data.
If you’d like someone to roast your readings before they go upstairs, grab 20 minutes with us.
How to report the answer, including when it is no
Write the answer as three sentences and one interval.
What the readings say, over what window, against which baseline. State the uncertainty as a range rather than implying it with a rounded percentage. And what you’d need to see next quarter to change the conclusion.
Sometimes the honest answer is that throughput rose and you can’t separate the AI assistant from the two senior hires who joined in the same quarter. Say that. A number with a stated limit reads as instrumentation, and the engineering leader who reports one gets asked harder questions and keeps more credibility.
Then keep the method fixed. This measurement compounds across quarters, and it resets to zero the moment someone changes the definition to make a quarter look better. Talk to us if you want the four readings run against your own history.
Frequently asked questions
- How long after adopting an AI assistant can I measure the effect?
- Give it a full quarter of normal work after the first few weeks of experimentation, so roughly four months. Before then, you’re measuring the learning curve, which looks like a slowdown followed by a spike, neither of which reflects the steady state.
- Is cycle time a reliable measure of AI assistant productivity?
- On its own, no. Cycle time falls when changes get smaller, which is exactly what assistants encourage, so it can improve while delivered throughput stays flat. Read it alongside median change size and a value-scored throughput figure.
- Should I use developer survey results to measure AI assistant productivity?
- Use them alongside the delivery readings rather than in place of them. Perceived friction is real information about retention and daily experience, and it moves independently of what makes it to production. Reporting a sentiment figure as a throughput gain is what gets a productivity claim walked back.
- Does this measurement report on individual developers?
- No. All four readings aggregate at the team and repository levels, where delivery commitments sit. A per-person view becomes a ranking, and that ranking changes the behavior it observes over two sprints.
- What if throughput went up but I cannot prove the AI assistant caused it?
- That’s the normal result, and stating it is the stronger position. Navigara’s own Q2 2026 study, across 65 public repositories at six organizations, declines to quantify what share of its level shift is attributable to assistant adoption. Report the change, name the other variables in the quarter, and let the decision account for both.
- Can acceptance rate ever be useful?
- As a rollout signal, yes. Near-zero acceptance in one team usually indicates a configuration issue or a language the model handles poorly, both of which are worth fixing. As evidence of delivery speed, it fails because it stops measuring at the keystroke.
