Stop wasting money on vanity agents
Product leaders are trying to make autonomous workflows a reality, both inside the products they build and in how they run their own teams. And on paper, a lot of them are getting there. The agent is live, and they can check the “autonomous” box for the C-suite.
But look closer and you’ll often find humans still double-checking 100% of the agent’s output, which erases the whole point. Your team is just as busy as before, but now there’s also a fat token bill. We call these “vanity agents.”
Welcome to Beyond The Click by Balboa. Today we’re walking through how to progress from vanity agents to real, governed, autonomous workflows, including:
- High-volume, routine, and vital ✅
- Why AI feels risky 💣
- Stage gates for autonomy 🗝️
- A
leapbaby step of faith 👟
Let’s dive in.
High-volume, routine, and vital ✅
Consider an example from a regulated industry. Every brochure, email, and social media post a financial advisor sends out has to pass SEC rules first. In this business, there are things you cannot say (e.g. guarantee a result), and getting it wrong carries astronomical fines.
But there’s so many opportunities to mess up. One firm might have over 30,000 advisors, equating to hundreds of thousands of posts each day. Even a team of humans can’t read all of that. This approval process is high-volume and routine, but also vital, which makes it the exact kind of workflow that you should automate.
Why AI feels risky 💣
Compliance teams partially automate this workflow with shorthands, such as a deterministic rules engine that just flags any post with the word “guarantee” in it, no matter how it’s used. It’s the same input, with the same answer, every time, which makes it easy to trust, even though the false positive rate is high.
AI, however, is probabilistic, meaning it reads the whole brochure and makes a judgment call. It can catch edge cases like implied guarantees, and avoid jamming up compliant posts that just got stuck in a trigger-happy filter. Most importantly, it scales this effort across all 100,000+ brochures.
But we all know that if you feed the same prompt to an AI twice, you’ll get slightly different answers. That’s just how probabilistic systems work. But it’s enough variance to keep some teams re-checking everything, and the gap between all-human and fully autonomous might look impossibly wide.
But it is not impossible. You can apply a proven change management technique to this new technology, one that helps you accept risk in bite-sized increments. And it applies to any industry with workflows that are high-volume, routine, and vital, be that support ticket triage, content moderation, or expense reports.
That technique is called stage-gating.
Stage gates for autonomy 🗝️
Before you can let the AI run on its own, you run it for a few weeks alongside your human workers, and you keep score. Split the workflow into the specific problems you’re looking for (a guaranteed return in a brochure, a password reset request in a support queue, a mileage claim on an expense report) and score the AI on accuracy for each one. This scorecard is your first stage gate.
Next, say the AI catches guaranteed returns correctly 97% of the time, and your compliance lead gives it a pass. From this point on, your reviewers stop reading every brochure for that one problem. Instead, they pull a random 10-20% of what went through and confirm the AI is still right. If the sample starts missing again (this is called “drift”), you go back to full review for that problem and figure out what changed (new model? new cohort of users who just write sloppy?).
The whole idea behind stage-gating is that you’re asking your team to trust the AI with one thing it has demonstrated it can do, and you sample outputs as insurance. Then next month you pick the next thing, and the month after that, the next.
A leap baby step of faith 👟
What makes this work is that at no point does anyone have to decide to “just trust the AI.” Your compliance lead, your support manager, or your controller only ever signs off on one specific thing the AI has already proven it can do. And every time that happens means work your team gets to stop doing manually, which is the whole reason you invested in these tools.
So kick out those vanity agents and start building the efficiencies you’ve wanted this whole time.
Want the next one in your inbox? Subscribe to Beyond The Click and get the latest on Pendo and Digital Experience every other Thursday.