Turns out that's exactly what it took to catch the bug that would have wiped our freshest data.
A few weeks ago, we almost torched our own best data. Not because someone was careless. Because the plan was good. That’s the annoying part.
Somebody went digging through the account data to clean up a mess. Fake company names like “Confidential” and “Stealth.” A handful of names the system couldn’t read. One test record that snuck into production. Boring janitorial work. Except while checking the plan against the real system, our engineer found something worse already living there. Nine actual executives, quietly mislabeled at the wrong company, because one of those placeholder names used to point there by accident. Whoops. The plan got smarter because of it. By the time it was finished, it looked airtight. A careful reviewer would have signed off without blinking.
We didn’t ship it. Anthony has a saying for moments like this: “These problems keep coming back like cockroaches.” Translation: something is rotten underneath. Even when the plan on top looks great. So instead of hitting deploy, we brought in eight AI reviewers. We told all of them the exact same thing. A balanced opinion is a failed opinion. Go be difficult.
Each one got a different way to be a jerk about it. One had to pretend the plan had already shipped and blown up, then explain exactly how. One was only allowed to care about the single thing a customer would actually notice. One had to prove the plan would survive five years of reality. One had to find who would be stuck fixing this by hand on a random Tuesday. One went hunting for the ugliest input it could throw at the system. One argued that most of the plan should not exist at all. Eight different flavors of "prove me wrong."
Eleven minutes later, they found what the smart, careful, evidence-backed plan had missed. The big one involved a simple question: who currently works where. The system had quietly stopped answering that correctly for anyone who changed jobs after a certain point. Every executive move since then was one step from vanishing off customer lists the second this shipped. No error. No failed test. Just wrong, silently, for everyone who used it. Cool, cool, cool.
There was more. A comparison rule buried in the plan decided that two totally different companies were the same company. It stripped out anything that wasn’t the Latin alphabet before comparing names. Two companies, both named in Chinese characters, would have merged into one inside our system. Somewhere, a real company was about to get told it doesn’t exist anymore. On a completely unrelated note, a backlog of nearly 800 tickets turned out to be permanently stuck, not just slow. The intake process was rejecting the exact type of ticket that made up most of the pile. A queue that eats its own tail.
The eight reviewers didn’t agree on everything, and honestly, thank goodness. Six real arguments went straight to Anthony, each one framed as a plain choice with the reasoning behind it attached. He ruled on all six the same day. One reviewer’s very confident finding got disproven by a different reviewer, simply by checking it against the live system. Confidence is not evidence. Even the robots need a fact check sometimes.
Here’s the part that made the whole thing worth it. Before the fix had even finished rolling out, one of the new safeguards this process produced caught a version of the same bug again. This time, somewhere completely different in the system, on its very first real run. Nobody went hunting for it. It just got stopped at the door.
We are telling on ourselves on purpose, because this is the actual lesson. A good-looking plan is exactly the moment to get suspicious, especially once AI is involved anywhere in the process. This is not a move for every routine change on a Tuesday afternoon. Save it for decisions where being wrong is permanent, or embarrassing in public, or both. Save it for when your gut keeps telling you the same problem keeps crawling back, wearing a different shape.
When you do call it, keep it simple. Give every reviewer a genuinely different reason to hate the plan, instead of asking the same open question eight times. Feed them real numbers, not vague summaries, so they have something to attack. And keep an actual human making the final call. Do not add a ninth AI to summarize the other eight’s opinions. That defeats the entire point.
Try it yourself before you delete anything from your database. You might be surprised how many errors a panel of AI agents arguing with each other can catch.