Home/ Journal/ Memo

What to do after a failed AI pilot, and why it is a diagnosis, not a verdict.

A failed pilot is almost never a verdict on whether AI can help your business; it is usually a diagnosis that the pilot was scoped, sequenced, or measured wrong. The recovery is a sequence, not a restart from zero. Run an honest post-mortem that separates the technology genuinely could not do this from you aimed it at the wrong thing, had no baseline, no owner, or bad data. Recover the assets that survive, the data mappings, the process understanding, the one thing that worked. Re-scope to a single named workflow with a measurable baseline before you touch a tool again. Decide build-versus-buy and ownership deliberately this time, then restart small with a defined outcome and a real owner. Sometimes the honest answer is to wait, because the process was not stable or frequent enough to be worth automating yet.

Owners who fund a first AI pilot and watch it flop tend to draw one conclusion in the week after: AI is not for a business like this. That conclusion feels responsible, it is usually wrong, and acting on it is how a company quietly talks itself out of the one category of improvement its competitors are installing. A pilot does not test whether AI can help you. It tests one narrow attempt, built one particular way, aimed at one target, and measured against one bar, and any of those four can break while the underlying technology is fine. This memo is the recovery playbook for a gun-shy operator: how to run an honest post-mortem, salvage what survived, re-scope down to a single workflow with a real baseline, decide build-versus-buy and ownership on purpose, and restart small, including the cases where the honest move is to wait.

MemoJuly 2026
Read time10 minutes
AudienceOwner-CEOs, COOs, Operators

Why a flopped pilot is a diagnosis, not a verdict.

The most expensive moment after a failed pilot is not the money already spent; it is the conclusion an owner reaches in the week after. A proof-of-concept was funded, a few months went by, the thing did not do what the pitch promised, and the natural reading is that AI is simply not for a business like this one. That reading feels like the responsible one. It is also wrong most of the time, and acting on it is how a company argues itself out of an entire category of improvement. The pilot did not test whether AI can help you. It tested one narrow attempt, built one particular way, aimed at one particular target, and measured against one particular bar, and any of those four could have been the thing that broke while the technology underneath was never the problem.

It helps to be precise about what a pilot is. A pilot is a small, cheap experiment run to buy information before you commit real budget, and the information it returns is almost never a clean yes or no on the underlying technology. When a drug trial fails, a competent team does not conclude that medicine does not work; they ask whether the dose was wrong, the patient group was wrong, the endpoint was measured wrong, or the compound genuinely does nothing. A failed pilot deserves the same discipline. Most of the time the honest post-mortem finds the technology was never the constraint. The constraint was that the pilot was pointed at a vague goal, ran without a baseline to measure against, had no single owner accountable for the outcome, was fed data too messy to work with, or was declared a win or a loss against a bar nobody had agreed in writing.

None of that is comfortable, because it means the failure was partly a planning failure rather than a technology verdict, and planning failures feel like they reflect on the people who ran the pilot. But that reframe is the good news, not the bad news. A technology that genuinely cannot do the job leaves you nowhere. A pilot that was scoped, sequenced, or measured wrong leaves you with a fixable problem and, usually, a pile of things you learned that a second attempt does not have to relearn. The question that separates a recoverable pilot from a dead end is not whether it worked but whether you now understand why it did not, precisely enough to change one thing and try again.

This memo runs the recovery in order: an honest post-mortem that separates a real technology limit from an aiming problem, a salvage pass that recovers the assets and lessons that survive, a re-scope down to a single named workflow with a measurable baseline before any tool is touched again, and a deliberate decision about build-versus-buy and ownership that most first pilots skipped. It also says plainly, more than once, that sometimes the right move after a pilot is to wait, because the underlying process was not stable or frequent enough to be worth automating yet. The failure pattern all of this guards against is the same one that strands rollouts halfway, covered in why most mid-market AI rollouts stall in month four; a flopped pilot and a stalled rollout usually die of the same untreated cause.

Run the post-mortem: separate cannot from aimed wrong.

The first real work is a post-mortem that answers one question honestly: was this a limit of the technology, or a limit of how you aimed it. Those two findings lead to opposite decisions, and conflating them is how companies both abandon workable ideas and keep flogging dead ones. Start by writing down, in one plain sentence, what the pilot was actually supposed to achieve. If that sentence is hard to write, or comes out as something like explore how AI could improve operations, you have already found a prime suspect, because a pilot aimed at a vague ambition cannot succeed or fail cleanly; it can only trail off while everyone slowly loses interest.

The most common finding is that the pilot had no baseline. Nobody measured how long the task took, what it cost, or how often it went wrong before the tool arrived, so when the pilot ran there was no honest way to tell whether it helped. A tool can be doing real work and still look like a failure when the only evidence is a vague sense that things did not feel different. Without a before number, every after number is unreadable, and the pilot gets judged on impression rather than result. The second failure is a vague success bar. See if it improves things is not a target anyone can hit, and pilots defined that way tend to get graded against whatever the skeptic in the room quietly expected, which is usually perfection.

The third is no owner. A pilot that belongs to everyone belongs to no one; if it was somebody's side project on top of a full-time role, it got the attention a side project gets, and its failure says more about bandwidth than about the model. The fourth is data. If the tool was fed inconsistent, incomplete, or scattered inputs, it produced inconsistent output, and that is a data-readiness result, not a verdict on the technology. The fifth, and the one that actually vindicates the skeptics when it is real, is a genuine technology limit: the task truly did require judgment, context, or reliability the tool could not deliver, and no amount of re-scoping changes that.

The way to tell the fifth apart from the other four is to ask whether a competent person, given the same inputs the tool had, could have produced the result you wanted. If a skilled employee could not do the task with the information available, the tool was set up to fail and the problem is upstream, in the process or the data. If a person could do it easily and the tool could not, you may have found a real limit, though even then the more common truth is that the task was handed over whole instead of narrowed to the part that automates cleanly. Whether the process was ever a fit for custom AI in the first place is worth checking against the signs a business is ready for custom AI, because a pilot that failed on readiness grounds is telling you to fix the readiness, not to abandon the idea.

Write the finding down as a single sentence that names the cause, because the whole rest of the playbook branches on it. The technology cannot do this reliably enough sends you toward waiting or toward a much narrower slice. We had no baseline and no owner sends you toward a re-scope you can run with confidence. Most teams that do this exercise honestly land on the second kind of sentence, which is the encouraging part; the pilot failed for reasons entirely within your control to fix.

Recover what survived, then re-scope to one named workflow.

Before you plan a second attempt, spend a day recovering what the first one already bought you, because a flopped pilot is rarely a total loss and treating it as one means paying twice for the same lessons. The most valuable survivors are usually invisible on a budget line. Somebody mapped where your data actually lives and what state it is in, and that map is worth keeping even if the tool built on top of it failed. Somebody documented how a process really runs, as opposed to how the org chart says it runs, and that understanding is the raw material for any second attempt. And almost every failed pilot has one thing that did work, a small corner where the output was genuinely good, which is the most important clue you own, because it shows you the shape of the problem that fits.

Recover the concrete assets too. Any data cleaning or integration work done for the pilot usually survives a change of tool. Prompt patterns, rules, or logic that produced the good corner can be reused. Even the failure modes are assets, because a documented list of what went wrong is a specification for what the next attempt has to handle. The goal of the salvage pass is to walk into the re-scope carrying everything the first pilot taught you, so the second attempt starts from month three of knowledge rather than from zero.

Then comes the move that decides everything, and it happens before you look at a single tool: re-scope down to one named workflow with a measurable baseline. The most common cause of a failed pilot is over-scope, a tool aimed at a broad ambition rather than a specific repeatable task, and the correction is to get almost uncomfortably narrow. Pick one workflow you can name in a sentence, that happens often enough to matter, and that a person does the same way most of the time. Not improve customer service but draft the first reply to a returns request. Not automate operations but turn a signed order into a scheduled job. The narrower and more repetitive the target, the higher the odds of a clean win, and a clean win on something small is what rebuilds the confidence a failed pilot spent.

Attach a baseline before you build anything. Measure how long the chosen task takes today, what it costs, and how often it goes wrong, and write those numbers down, because they are the only thing that will let you judge the second attempt honestly. This is exactly the work the first pilot skipped, and it is not optional the second time. A baseline turns does it feel better into the task took forty minutes and now takes eight, at this error rate, which is a result you can defend to yourself and to anyone holding the budget. The discipline of choosing one bounded target and measuring it is the same one that separates rollouts that land from rollouts that stall, laid out in why most mid-market AI rollouts stall in month four, and the cost side of that bounded target is worked through in what a mid-market AI engagement actually costs.

Decide build-versus-buy, ownership, and when to simply wait.

With a narrow workflow and a baseline in hand, make the two decisions the first pilot probably made by default: how you get the tool, and who owns the outcome. Build-versus-buy is the first. A failed pilot on an off-the-shelf product does not automatically mean you should build custom, and a failed custom attempt does not automatically mean you should have bought; the honest read is whether the workflow you just narrowed to is a common one that a product already serves well, or a differentiated one specific enough that no product fits it cleanly. Buying is right for the commodity task. Building is right for the workflow that gives your business its edge and that you want to own rather than rent forever. That judgment deserves its own deliberate pass rather than a reflex, and it is worked through in build versus buy for a mid-market company.

The second decision is ownership, in both senses. Someone inside the company has to own the second attempt as a real responsibility, not a side project, because a pilot with no accountable owner is one of the five ways the first one failed, and repeating it guarantees the same result. And if you build, you should own the code and the asset at the end rather than renting your core workflow back from a vendor in perpetuity. Whether the capability lives with an outside partner or an internal hire is a real fork with real tradeoffs, weighed in an AI consultant versus an in-house hire, and choosing the partner well if you go that way is its own skill, covered in how to choose an AI consultant.

Only now, after the post-mortem, the salvage, the re-scope, and the two decisions, do you restart, and you restart small. The second attempt should target the single named workflow, measure against the baseline, run under a named owner, and define a win in a sentence everyone agreed to before the work began. Keep it small enough that a clean success is likely and cheap enough that another miss is survivable, because the job of the second pilot is not to transform the company; it is to produce one honest, measured win that proves the shape of the thing and earns the right to widen. How long even a narrow build should take is worth calibrating against how long a mid-market AI build takes, so the second attempt is given a fair runway rather than judged too early.

And sometimes the honest answer, after all of this, is to wait, which is a legitimate outcome rather than a failure. If the post-mortem shows the process is not yet stable, that people do the task a different way every time, or that it happens too rarely to be worth encoding, then automating it now would be building on sand, and the right move is to stabilize the process by hand first and revisit later. Waiting for the right reason is not the same as the gun-shy retreat this memo opened against. The retreat concludes AI cannot help and stops looking. Waiting concludes this specific process is not ready yet, names what would make it ready, and sets a date to look again. One is a verdict; the other is a plan.

The whole arc from a flopped pilot back to a working one is a sequence of deliberate choices, and the fastest way to get on the right side of it is to name your target workflow and test whether it is ready before you spend another dollar, which is what a structured diagnosis is built to do. A first pilot that failed is not money wasted if it becomes the diagnosis that makes the second one land.

Field-note context

What we watch for when a pilot comes to us broken.

The tell that separates a fixable pilot from a dead one.

The single most useful question in a post-mortem is whether a competent person, handed the same inputs the tool had, could have produced the result you wanted. If a skilled employee could not do the task with the information available, the tool was set up to fail and the fix is upstream, in the process or the data, not in the technology. If a person could have done it easily and the tool could not, you may have found a real limit worth respecting. Most pilots that feel like technology failures fail this test: the tool was handed a job a human could not have done with the same messy, incomplete inputs either. That is a fixable diagnosis wearing the costume of a dead end, and the difference is worth an afternoon to establish before you conclude anything.

Why the second attempt should be embarrassingly small.

Owners who were burned once often want the second pilot to prove more, to justify the first one's cost by finally delivering something big. That instinct is exactly backward. The second attempt should be smaller than the first, narrow enough that a clean success is likely and cheap enough that another miss is survivable, because its job is not to transform the company but to produce one honest measured win. A small win on a named workflow, checked against a real baseline, rebuilds the confidence and the internal credibility that the first pilot spent, and it earns the right to widen on evidence rather than hope. A second big swing that misses does not just cost money; it hardens the belief that AI is not for this business, which is the one conclusion this whole exercise exists to prevent.

How we run a post-mortem on a pilot we did not build.

When an owner comes to us after a flopped pilot, the first session is not a pitch; it is a diagnosis of what actually went wrong, usually on a whiteboard with the person who ran the first attempt in the room. We write the intended outcome in one sentence, check it against the five aiming failures, and separate a real technology limit from a scoping or data problem before anyone talks about building anything. Often the finding is that the idea was sound and the setup was not, and the right next step is a much narrower re-scope the owner can run with or without us. Sometimes the finding is that the process is not ready and the honest recommendation is to wait. We would rather tell an owner to stabilize a process for three months than sell them a second pilot that fails the same way the first one did.

Extended questions

The questions operators ask after a pilot flops.

Does a failed AI pilot mean AI cannot help my business?

No, and treating it that way is the expensive mistake. A pilot tests one narrow attempt, built one particular way, aimed at one particular target, and measured against one particular bar. Any of those four can break without the underlying technology being the problem. Most honest post-mortems find the pilot failed because it was pointed at a vague goal, ran with no baseline, had no accountable owner, was fed messy data, or was judged against a bar nobody agreed in writing. Those are aiming failures, and they are fixable. A genuine technology limit, where the task truly needed judgment or reliability the tool could not deliver, is the rarer finding. The way to tell them apart is to ask whether a skilled person, given the same inputs the tool had, could have produced the result you wanted. If yes, the problem was upstream, not the technology.

How do I run an honest post-mortem on a failed AI pilot?

Answer one question honestly: was this a limit of the technology, or a limit of how you aimed it. Start by writing, in one plain sentence, what the pilot was supposed to achieve; if that sentence is hard to write, you have already found a suspect. Then check the five common aiming failures. No baseline, so nobody could tell whether it helped. A vague success bar that let any skeptic call it a loss. No single owner, so it got side-project attention. Bad or scattered data, which produces bad output regardless of the model. And the rarer real technology limit. To separate that last one from the others, ask whether a competent person with the same inputs could have done the task. If they could and the tool could not, you may have a real limit; if they could not either, the problem is upstream in the process or the data.

What can I salvage from a pilot that did not work?

More than owners expect, which is why treating a failed pilot as a total loss means paying twice. The most valuable survivors are usually invisible on a budget line: a map of where your data actually lives and what state it is in, a documented understanding of how a process really runs rather than how the org chart says it runs, and the one corner of the pilot that genuinely worked, which shows you the shape of the problem that fits. Recover the concrete assets too, any data cleaning or integration work, the prompt patterns or logic that produced the good corner, and even the documented failure modes, which are a specification for what the next attempt must handle. The point of the salvage pass is to walk into the re-scope carrying everything the first pilot taught you, so the second attempt starts from months of knowledge rather than from zero.

How should I re-scope after an AI pilot flopped?

Get almost uncomfortably narrow, because over-scope is the most common cause of a failed pilot. Pick one workflow you can name in a sentence, that happens often enough to matter, and that a person does the same way most of the time. Not improve customer service, but draft the first reply to a returns request. Not automate operations, but turn a signed order into a scheduled job. The narrower and more repetitive the target, the higher the odds of a clean win, and a small clean win is what rebuilds the confidence a failed pilot spent. Then attach a baseline before you build anything: measure how long the task takes today, what it costs, and how often it goes wrong. That baseline is the thing the first pilot skipped, and it is what turns does it feel better into the task took forty minutes and now takes eight, which is a result you can defend.

When is waiting the right answer instead of trying again?

When the post-mortem shows the underlying process is not ready, waiting is the honest answer rather than a failure. If people do the task a different way every time, if the process is still changing month to month, or if it happens too rarely to be worth encoding, then automating it now means building on sand. The right move is to stabilize the process by hand first and revisit on a set date. This is not the same as the gun-shy retreat that concludes AI cannot help and stops looking. Waiting names the specific thing that is not ready, names what would make it ready, and sets a time to look again. One is a verdict on the technology; the other is a plan for the process. An operator who can tell the two apart avoids both mistakes: forcing automation onto an unstable process, and abandoning a good idea because the first attempt was aimed wrong.

How does the AI Maturity Index help me re-scope after a failed pilot?

The AI Maturity Index is built for exactly this moment, because it walks you through the re-scope a failed pilot needs before you spend again. It asks you to name the one workflow worth building first, whether you can attach a baseline cost to it, and whether the process is settled enough to encode, which are the same questions that decide whether your first pilot was aimed right. If you can answer them, you are ready to restart small with a defined outcome and a real owner, and the Index hands you the scope and baseline questions to hold any tool or vendor to. If you cannot, it shows you which piece is not ready yet, which is often the honest signal to stabilize the process or wait rather than run a second pilot into the same wall. In about ten minutes and with no call, it turns a flopped pilot into a clear diagnosis.

Ready to turn a failed pilot into a diagnosis you can act on?

Start with the AI Maturity Index. Ten minutes, no call, and it tells you whether you are ready to restart with a defined workflow and a baseline or should stabilize the process first, along with the scope and ownership questions to hold any tool or vendor to.