Why Projects Keep Failing Despite Good Estimates

Good team. Reasonable timeline. Clear requirements document. Three months later: late, over budget, and the thing you shipped doesn't actually solve the problem it was supposed to solve. Sound familiar?

I've watched this happen dozens of times. And I've learned that the usual explanations -- scope creep, bad estimates, unclear requirements -- are symptoms, not causes. They're the retro answers that let everyone go home without confronting what actually went wrong.

The usual suspects are symptoms

Let's run through the standard post-mortem:

• "Scope crept." Why did it creep? Because the team discovered requirements mid-build that should have been discovered before build. Why weren't they discovered? Because nobody deeply understood the problem.
• "The estimate was wrong." Why? Because the estimate was based on a surface understanding of what needed to be built. The complexity lived in the parts nobody asked about.
• "Requirements changed." Why? Because the requirements described what someone thought they wanted, not what they actually needed. Once build started, reality intruded.

Every one of these traces back to the same root: the problem wasn't understood before the solution was designed.

Solving the wrong problem precisely

This is the pattern I see most often in engineering teams: incredible execution on the wrong thing.

Here's a real example (anonymized but structurally true):

The request: "Build a dashboard for the leadership team."

Reasonable, right? The team estimates it, builds it, ships it on time. Nobody uses it.

If you'd excavated the request before building, here's what you'd find:

Why does leadership want a dashboard? Because the CEO keeps asking for status updates in meetings. Why does the CEO ask for status updates? Because she doesn't know what's happening between meetings. Why doesn't she know? Because information doesn't flow upward without being pulled.

The physics: information doesn't flow without being pulled. The dashboard is one possible fix, but it's a symptom treatment. The real problem is an information flow problem. Maybe the fix is a dashboard. Maybe it's a weekly async update. Maybe it's restructuring how teams communicate progress. But you can't choose the right solution if you haven't found the real problem.

The cost of skipping excavation

When you skip problem excavation, you pay for it in three ways:

Rework. You build the thing, discover it doesn't solve the actual problem, and rebuild. This is the most expensive line item in most engineering budgets, and it's almost entirely preventable.

Scope creep. Mid-build, someone realizes the solution doesn't quite fit the actual need. So they add requirements. And add more. The scope isn't creeping -- it's searching for the right problem while building the wrong solution.

Team burnout. Nothing kills morale faster than shipping something you know won't matter. Engineers can tell when they're building the wrong thing. They just don't have the language or the process to push back effectively.

Excavation before estimation

The fix isn't better estimates. It's better problem understanding.

Before your team estimates a single story point, spend 30 minutes excavating the request:

• STATE the problem as you understand it. Write it down. Most teams skip this entirely.
• SURFACE the assumptions embedded in the request. "Build a dashboard" assumes a dashboard is the right solution. Is it?
• DRILL into why this matters. Not "what do we build" but "what problem does this solve, and for whom?"
• PATTERN -- has this kind of request come up before? What happened last time?
• CHALLENGE your conclusion. If you think you know the root cause, try to disprove it.
• PHYSICS -- name the irreducible truth. What's the thing that, if it changed, would make the entire request unnecessary?
• FORK -- given the physics, what are your actual options? The original request might be one of them. It might not.

The 30-minute investment

I'm not suggesting you spend weeks on problem analysis before writing code. I'm suggesting 30 minutes.

Thirty minutes of structured excavation before a three-month project can save you months of rework. It can prevent the scope creep that makes every sprint feel like a death march. It can give your team the clarity to push back on requests that don't make sense.

Projects don't fail because of bad estimates. They fail because they solve the wrong problem. The estimate was fine. The problem definition wasn't.

The most productive thing a technical leader can do is refuse to estimate until the problem is understood. Not the requirements. Not the solution. The problem.

---

Try it yourself

The gallery has real excavation sessions where CTOs worked through problems like this. See the method in action, then start your own excavation.