I’ve watched organizations have AI failures. A model or native tool function makes a bad decision. Without guardrails in place, a recommendation goes wrong. A prediction is inaccurate. Leadership asks: “What happened?” And the answer is always about the model, never about the structure.
“The model wasn’t trained on enough data.” “The limited capability of the feature, function, or tooling didn’t account for this edge case.” “The model’s assumptions were wrong.”
Here’s what I keep noticing: we blame models and tools instead of examining why the function was architected or deployed that way.
I remember during the early ML days, I worked with a company that had a simple pricing model that failed. It recommended prices that were too low, and the company lost money. The post-mortem focused on the model: what data it used, what assumptions it made, what it got wrong.
But nobody asked: Why was the model deployed without sufficient testing? Why didn’t anyone notice the prices were too low before customers saw them? Why wasn’t there a human review for large price changes? Why didn’t the monitoring catch the problem faster?
The model failed because of the model. But the impact was large because of the structure around the model. And the post-mortem only examined the model. New ai tools and capabilities are no different.
Here’s what makes this problematic: if you only examine the model, you only learn how to build better models. You don’t learn how to build better systems for deploying, testing, and optimizing. You don’t learn how to build better oversight. You don’t learn how to build better safeguards.
I’ve seen this pattern across organizations. An AI project fails, the team adjusts, and then they deploy the new function with the same lack of safeguards or technical debt reduction that allowed the first failure to have impact. They fixed the feature but not the context or the system. So the next failure has the same impact as the first one.
The organizations that learn from AI failures don’t just examine the capabilities, functions, or models. They examine the solution, the architecture, the operability, and deployment process. They ask: What safeguards should have caught this? What monitoring should have detected this? What human oversight should have prevented this? What structural changes would make this type of failure less impactful?
They treat AI failures as system failures, not just as model failures.
But here’s what makes this hard: examining structure means examining decisions that leaders made. It means asking why safeguards weren’t built. What context wasn’t in place. What bad inputs compounded or cascaded to cause the issue. Where, perhaps, is the spider who made the web. They hone in only on the web. It means questioning whether the deployment process was adequate. It means admitting that the failure wasn’t just about the model. It means, asking hard questions about the solution, the outcomes, or the underlying questions.
Most organizations aren’t ready for that conversation when it comes to newer technology adoption, cloud modernization efforts, or AI programs. So they blame the AI or the tool. And they don’t learn the structural lessons that would prevent the next failure from having the same impact.
Leave a Reply