The moment the dashboard stopped telling the truth

Source: dev.to · Published: 2026-08-01

I watched my team get faster after we adopted AI-assisted coding, and honestly — I felt good about it. More tickets closed. Shorter cycle times. PR volume up. I remember thinking: this is what leverage looks like.

That was the mistake. Not the tool. The assumption.

Two weeks after a change shipped — passed every test, got through review, deployed cleanly — we found it had been quietly degrading a retry mechanism. It only broke under specific load conditions our test environments didn't replicate. Nobody caught it because it didn't look wrong. It worked. It just wasn't safe.

When I asked the engineer to walk me through it, they could. The code made sense. They understood what each part did. But when I asked what would happen if the downstream service was slow — not down, just slow — there was a pause. Not because they weren't capable. Because they'd never needed to ask that question. The AI wrote the handling code, the tests passed, and the whole thing moved forward before that question ever came up.

The dashboard wasn't lying. Things were shipping faster. It just wasn't showing me the part that mattered.

<h2> What I didn't see coming </h2>

Here's what actually surprised me: AI can make a team look more capable before it makes the team actually more capable.

I used to learn by getting stuck. The 11 PM kind of stuck. Staring at a stack trace for three hours, genuinely questioning whether I understood any of this. That friction built something real — the instinct that says this probably works, but something feels off, and I should figure out what before we ship it. I built that through a migration that silently corrupted data for six hours. Through a caching layer that sailed through staging and failed on a Friday afternoon in production.

That's not nostalgia for unnecessary suffering. The suffering was the mechanism. When the first draft is free, that mechanism stops. Learning used to happen inside the act of writing the code. Now it doesn't. And I'm genuinely not sure what replaces it.

Maybe something does. Maybe I'm missing it. But I haven't seen it happen on its own.

Some things break fast. Some take months. One of them you only find out when there's a crisis.

Review quality is usually the first to slip. When one engineer is generating PR volume that would've taken three people before, reviewers get stretched. The natural move is to read faster and trust that things look right. If the tests pass and nothing is obviously wrong, it's probably fine. Subtle problems slip through — not because anyone is careless, but because nobody has the headspace to really push on every change.

After that, debugging gets shallower. When you can paste an error into a chat interface and get a fix in thirty seconds, you stop asking why it broke. The symptom gets fixed, you move on. But the next time a similar failure shows up — slightly different shape, different context — it takes longer because you never really understood the first one. And by then you might not even connect the two.

Incident response is usually where you actually see how bad it got. I had an engineer who'd shipped well for months. Solid PRs, fast delivery, good feedback all around. Their first real incident was rough. Not panic — they just hadn't been in a situation where things were misbehaving and they had to figure it out without a clear answer. You have to have run through three wrong theories at 2 AM before you land on the right one. That kind of confidence doesn't come from anywhere else.

I'm curious whether others have seen this same shape — fast shipping, then the first real crisis lands differently than expected. Or maybe your experience has been different?

Is this inevitable? I'm honestly not sure. But I also don't think it fixes itself.

So we started trying things — mostly after the incident made it impossible to ignore.

<h2> What we actually tried </h2>

We didn't figure this out cleanly. We stumbled into most of it after noticing something was off — the team was moving fast but not quite understanding what they were shipping.

The first thing we tried was adding more structure to standups — asking people to explain what they shipped that day. That didn't work. It just created a performance. People got good at summarising things they hadn't fully understood.

What actually helped was changing what code review was for. We stopped using it as a gate — does this work? — and started treating it as a place to slow down and think together. Reviewers started asking: "What's the failure mode you're most worried about here?" For anything touching auth, payments, data migrations, or external reliability, we added one expectation: the PR description should have a paragraph on what can go wrong.

It felt slower at first. It wasn't. PRs got better. The questions people asked before even opening a PR got better. It turned out the quality check and the learning were the same thing — we just hadn't been honest about what review was actually for.

We also made room for practice that a normal sprint doesn't naturally create. Debugging drills. Post-mortem walkthroughs where someone new would reason through a timeline out loud. Design reviews where we expected challenge, not just sign-off.

And we started getting juniors into incident shadowing earlier than felt right. I used to think that was something you earned access to. I think I had it backwards. Watching a real incident unfold — seeing what questions get asked, what gets ruled out, where people get stuck — you can't replicate that in a drill.

None of it was "no-AI forever." I'm not sure it's a complete answer either. But it was better than doing nothing and hoping the gaps would close on their own.

<h2> What I still haven't figured out </h2>

I don't have a clean way to measure any of this. Review quality improving, sharper questions in design discussions, faster diagnosis in incidents — these feel real to me, but they're slow and indirect. I don't have a dashboard for it, and I don't fully trust anyone who says they do.

I'm also not making the argument that AI is bad for learning across the board. Engineers who previously wouldn't have attempted certain things now attempt them — and the attempt teaches something. AI can explain concepts in a way that removes some of the shame around not knowing. I don't want to write that off.

But there's a specific kind of understanding that comes from being in a broken system and having to figure it out yourself. The logs are ambiguous. The runbook doesn't cover this exact situation. Nobody has a clean answer. That builds something. I'm not sure how to recreate it outside of the real thing.

Maybe there are approaches I'm not thinking about. I'm curious whether others are hitting this same wall — or finding ways around it.

<h2> The part I keep coming back to </h2>

AI writes the first draft. But I'm not sure it teaches you when the first draft is wrong — and that might be the part that matters most.

A team can get faster and shallower at the same time. The velocity metrics won't tell you. You might not notice for a year.

I'm not arguing to slow down AI adoption — I don't think that's the right call, and honestly I'm not sure what the right call even looks like yet. But I do think this is worth talking about more honestly than most teams are.

A few things I'm genuinely curious about:

<ul> <li>How is your team thinking about this? Are you seeing depth hold up as you move faster, or is there a quiet drift you're noticing too?</li> <li>What's your read on where the judgment gap shows up first — review, debugging, incidents, something else?</li> <li>Has anyone found something that actually works here, beyond "pair more and review harder"?</li> <li>And the uncomfortable one: are we measuring the right things, or are we just measuring what's easy to measure?</li> </ul>

I don't think there's one clean answer to any of these. But I'd rather be asking them out loud than pretending the dashboard tells the whole story.