AI is everywhere right now.
It’s writing our user stories, suggesting our code, summarizing our retrospectives, and helping us move faster than ever before. And naturally, organizations are asking the question:
“Is AI actually making us better?”
Not just faster. Not just more productive.
Better.
As Agile practitioners, that puts us in a familiar position — trying to measure something complex with metrics that weren’t necessarily designed for it.
So let’s talk about it.
How do we measure whether AI is improving:
- the quality of what we deliver,
- the speed at which we deliver it, and
- the value it creates?
And more importantly…
Are we measuring the right things at all?
The Trap: Output Dressed Up as Progress
Let’s start with what I’m seeing across organizations right now.
Companies are investing in AI tools and then immediately asking for proof of impact. That usually translates into questions like:
- Are we delivering faster?
- Are we closing more tickets?
- Is velocity increasing?
On the surface, these seem reasonable. But they’re still rooted in output-based thinking.
We’ve just replaced:
“How many story points did we complete?”
with:
“How many more story points can we complete with AI?”
That’s not transformation. That’s optimization of the same mindset. And we already know where that leads. More output doesn’t always mean more value.
What AI Is Actually Changing
AI is undeniably changing how we work. From what I’ve seen (and experienced firsthand), it’s having the biggest impact in three areas:
1. Speed of Execution
AI accelerates the “mechanical” parts of work:
- writing user stories
- generating test cases
- drafting documentation
- suggesting code
I use ChatGPT myself to create epics and user stories for my Agile coaching team. It saves a ton of time and gives us a strong starting point for refinement.
2. Consistency and Quality Baselines
AI helps standardize outputs:
- clearer acceptance criteria
- more structured backlog items
- fewer gaps in documentation
It raises the floor, even if it doesn’t always raise the ceiling.
3. Decision Support
AI can analyze patterns in data:
- identifying bottlenecks
- highlighting risks
- suggesting priorities
This is where things get really interesting — because it starts to influence how we think, not just how we execute.
So… Do Our Metrics Reflect This?
Here’s the problem: most traditional Agile metrics weren’t designed for this world.
Let’s look at a few common ones:
⏱️ Cycle Time & Lead Time
These are still incredibly valuable. If AI is helping reduce time from “start” to “done,” that’s meaningful.
But here’s the nuance:
- Are we faster because we’re better?
- Or faster because we’re cutting corners?
Without quality and outcome context, these metrics can mislead.
📈 Velocity
Velocity might increase with AI.
But should we celebrate that?
If AI helps us write more tickets and complete more work, velocity goes up — but that doesn’t tell us if we’re solving the right problems.
Velocity is a capacity signal, not a value signal. It always has been.
🚀 Throughput
Same story. More tickets completed doesn’t necessarily equal more impact delivered.
AI can inflate throughput while masking deeper issues like:
- poor prioritization
- lack of customer alignment
- unnecessary work
The Shift: Measuring Outcomes in an AI World
If AI is changing how we work, then our metrics need to evolve too. We need to double down on something I come back to often:
Outcomes over outputs.
In the age of AI, this isn’t just a philosophy — it’s a necessity. Here’s what that looks like in practice:
🎯 1. Value Delivered
Instead of asking:
“How much did we build?”
Ask:
“Did it make a difference?”
Examples:
- Customer satisfaction (CSAT, NPS)
- Feature adoption rates
- Reduction in user pain points
AI might help us deliver faster — but only outcomes tell us if it mattered.
⚙️ 2. Flow Efficiency
From the DORA metrics and flow frameworks, we know that efficiency isn’t about speed alone — it’s about how much time work spends actually being worked on.
AI can reduce waiting time, handoffs, and rework.
So instead of just measuring cycle time, ask:
- Did AI reduce bottlenecks?
- Did it improve flow across the system?
🧪 3. Quality Signals
This is critical. If AI is helping us move faster, we need to ensure quality isn’t degrading.
Look at:
- defect rates
- escaped defects
- rework percentage
- production incidents
According to the DORA State of DevOps Report, high-performing teams balance speed with stability — not one at the expense of the other.
AI should help us do both.
🧠 4. Decision Quality
This is the hardest one to measure — but arguably the most important. AI gives us more data, faster insights, and better visibility.
So ask:
- Are we making better prioritization decisions?
- Are we reducing wasted work?
- Are we learning faster?
This is where Agile maturity really shows up.
The Mindset Shift We Still Need
Here’s the honest truth:
The biggest challenge isn’t measuring AI.
It’s changing how we think about measurement.
Because even with all this technology, many organizations are still asking:
“Are we hitting our dates?”
Leadership wants predictability.
Boards want timelines.
Stakeholders want certainty.
And I get it — those pressures are real. But AI doesn’t solve that tension. If anything, it amplifies it.
Because now we can go faster… which makes people expect us to go faster all the time.
So What Should We Do?
As Agile leaders, our role is to bridge that gap.
We don’t ignore metrics like cycle time or lead time — we evolve them.
We connect them to outcomes.
We tell a better story:
- Yes, we’re faster — and here’s the value it created.
- Yes, AI improved our flow — and here’s how it impacted customers.
- Yes, productivity increased — and here’s what we chose to do with that capacity.
Because that last part matters most.
AI doesn’t just make us faster. It gives us choices. What we choose to do with that time – that’s where real agility lives.
Final Thoughts
AI is changing Agile. There’s no question about that. But it’s not changing the fundamentals.
We still need to:
- focus on outcomes
- prioritize value
- inspect and adapt
- keep things simple
If anything, AI is forcing us to get clearer about what matters. Because when the cost of producing output goes down… the importance of producing the right output goes way up.
So yes — measure cycle time.
Track lead time.
Look at throughput.
But don’t stop there.
Because in the age of AI, success isn’t defined by how much we deliver.
It’s defined by whether it made a difference.
References
- DORA, Accelerate: State of DevOps Report (2024)
- McKinsey & Company, The Economic Potential of Generative AI (2023–2024 updates)
- GitHub, State of AI in Software Development (2024)
Scrum Bubbles 
Leave a comment