Issue 27: I had AI check my work. It caught me five times
Second live test from the blueprint. Here's what I almost missed.
Welcome back to The Customer Continuum. Issue #27.
This is the second live test from my own 135-agent blueprint, and the one I least expected to surprise me.
Issue 26 ended with a teaser. The next agent I’d build would be a Chief of Staff validator, an agent whose job is to read the output of other agents and tell you whether the output is ready to ship. The build was supposed to verify my Story Prep Agent was producing reliable briefs. It did, in a way. The validator also kept finding things I’d shipped a week ago that I hadn’t realized were broken.
This is the build report from that run.
What I built
The Chief of Staff validator reviews any agent’s output against three lenses before the artifact gets used.
Format Compliance asks whether the artifact follows the structure its agent’s spec promised. Operator Scan asks whether a customer marketer running between meetings can actually use the artifact in under sixty seconds. Policy and Sensitivity asks whether the artifact surfaces anything that shouldn’t end up downstream, on a recording, in a deck, or in any external content the customer might eventually see.
Each lens returns a status: pass, flag, or required revision. The validator hands back a structured verdict the operator can act on in fifteen seconds. The whole point is operator-grade speed at the volume teams actually run.
The first six tests
I ran the validator against six artifacts in sequence to see whether it would catch what it should catch and pass what it should pass.
Test 1 was the old v1 brief from Issue 26, the prose-heavy version that failed the operator-scan test publicly. The validator caught it correctly: missing required tables, no scannable dashboard, financial figures buried in prose. Good signal that the mechanism works. Unremarkable result.
Test 2 was the v2 brief I’d shipped as clean in Issue 26. I expected pass with notes. The validator returned revise. It flagged two issues: one I half-expected, and one I didn’t. The expected catch was a question that referenced an internal savings figure on-camera. The unexpected catch was a question that quoted a line Sarah had said in a private CSM sync, which I’d imported as if it were public material. That second catch is the moment the build stopped being routine.
Test 3 ran the v2 brief with a deliberately planted bug, an exact ARR figure inserted into one suggested question. The validator caught the planted bug. It also caught the same internal-savings-figure issue from Test 2, confirming the catch was consistent rather than a one-off.
Tests 4 through 6 walked the validator through progressively cleaner versions of the brief, each one fixing what the prior test had flagged. Test 4 fixed the questions, but the validator caught that the outcomes-to-probe section had the same exposure pattern I’d just disciplined out of the questions. Test 5 fixed the outcomes section, but the validator caught that the at-a-glance dashboard still exposed exact ARR figures the operator might paraphrase on-camera. Test 6 finally returned pass with notes.
The arc across those six tests wasn’t what I’d planned. I’d expected the validator to catch known bugs. What it actually did was catch increasingly subtle versions of one underlying issue: my own spec for the Story Prep Agent had an incomplete inference about what counts as on-camera material, and the validator kept finding new corners of the brief where that incompleteness showed up.
The editorial sits in one sentence. The validator wasn’t catching my bugs so much as catching the gaps in my reasoning about my own work.
The practical test
By Test 6 I had a calibrated validator and a sense of what it could catch. But every test I’d run had been a controlled case, an artifact I’d authored with known issues planted in it. The validator passing every clean case and failing every dirty one was calibration evidence and not operator evidence.
For Test 7 I added a realistic complication to Sarah’s record. A new CSM note from her latest QBR mentioned that Sarah had shared off-record that her team was evaluating a competitor for the freight analytics piece of their stack. She explicitly asked that this stay out of customer-facing materials. Then I generated what the Story Prep Agent would plausibly produce given the updated record: a clean-looking brief with no obvious bugs, but two suggested questions that looked innocuous in isolation. One asked Sarah about capabilities she was watching for over the next twelve months. Another asked what she’d be evaluating fresh if she were starting today.
The validator caught both questions. It also caught something I didn’t predict it would. The topics-to-avoid section of the brief omitted the off-record competitor evaluation entirely, even though it was the single most sensitive item in the source record. The validator flagged the omission, not just the inclusion.
That distinction matters. Catching what’s in the artifact is content review, while catching what should be in the artifact and isn’t is structural review. The second move is harder and it’s the one that generalizes to every operator agent with an exclusions section.
The framing I’d use for what the validator demonstrated in Test 7 is that it reasoned about the downstream artifact’s risk surface, not just the brief’s content. Question 7 doesn’t mention the competitor. It doesn’t mention the evaluation. It’s a perfectly reasonable question in isolation. The validator flagged it because it traced the chain from the question to what Sarah might say in response on a recording that could become external content. That’s the inference the Story Prep Agent made unprompted last week, now confirmed as a repeatable validator behavior.
What this means for your team
For most customer marketers reading this, the Chief of Staff isn’t a tool you implement today. Your team probably isn’t running operator agents yet, which means a validator-on-top-of-an-agent solves a problem you don’t currently have.
But within the next twelve to twenty-four months, customer marketing teams at any size will be running operator agents for case study prep, advocacy outreach, review synthesis, community moderation, campaign operations, and cross-functional handoffs. The shift is already happening. When your team reaches that point, the validator pattern becomes the basis for whether you can trust agent output enough to deploy it.
Here’s the implementation guidance I’d give. Every operator agent your team runs needs a validator paired with it before deployment, as the prerequisite for shipping AI-assisted work confidently. The Chief of Staff role generalizes across operator agents: the same three lenses (format compliance, operator scan, policy and sensitivity) apply whether you’re validating case study prep briefs, reference program candidate lists, review responses, campaign updates, or handoff briefs.
The pattern is straightforward. Build the agent. Build the validator. Then deploy them together as a paired system. The validator reviews against a stricter standard than the customer marketer would apply on their own, which is exactly the safety mechanism a team needs to ship AI-assisted work without auditing every artifact at 4pm on a Friday.
A validator in the pipeline catches more than just bugs. It catches the gaps in your reasoning about your own work, applied more consistently than you’d apply them yourself.
What other operators are doing
Scott Thomas runs the customer team at an AI startup, and he’s already in production with six operator agents covering call readouts, sales-to-CSM handoffs, onboarding implementation knowledge bases, tier-one support deflection, customer voice surveys, and weekly CSM briefs. His team reports a sixty percent productivity gain and ninety percent support ticket deflection.
His operating philosophy is the same one I’ve been writing about: human-first, AI-augmented. Every customer touchpoint retains a human opt-out. He still manually reviews AI output before it ships externally. The point isn’t to remove humans from the loop but to give them their time back so they can spend it on the customer relationships that actually drive retention and expansion.
This one example is an independent validation that the agent-plus-validator pattern works at the volume real B2B SaaS teams operate at today.
What’s next
Issue 28 runs Test 8 in the practical suite. The scenario is one customer marketers see constantly: the named customer champion isn’t actually the decision-maker, and the brief implicitly attributes decisions to the champion that the deeper sponsor actually made. Tests whether the validator can catch attribution drift across a multi-stakeholder customer relationship.
Issue 29 runs Test 9, which is cross-customer pattern bleeding, where an agent unconsciously imports language from prior briefs into the current one. The hardest test in the practical suite because it requires the validator to compare the current artifact against the agent’s prior output history.
By Issue 30, the Chief of Staff validator and its companion testing track become a published part of the kit, with a v3 system prompt incorporating everything the seven tests above taught me.
This week’s free starter: Voice of Customer
The free starter this week is the Voice of Customer pillar of the CLG Agentic Blueprint. Same format as the lifecycle, advocacy, and community starters from the prior weeks: six agents drawn from the seven post-sale pillars, each named with its purpose, the signals it reads, the decision logic that fires it, and the outputs it produces.
This is the pillar where structured feedback collection, signal synthesis, and product roadmap influence live. Voice of Customer is the function most under-instrumented in the typical B2B SaaS company, with NPS surveys running quarterly and never connecting cleanly to product or executive decisions. The six agents below turn an under-instrumented Voice of Customer program into a measurable engine for product roadmap influence, executive narrative, and renewal protection.
The six agents are:
NPS Pulse Agent: runs weekly NPS pulse collection and flags response-rate drops or sentiment shifts before they become trends
Verbatim Synthesis Agent: clusters and themes open-ended survey responses into actionable categories tied to product and CS workflows
Customer Voice Voice Agent: runs scheduled voice-call surveys at month 5 and month 8 of the customer lifecycle, inspired by what Scott Thomas is running at Sales AI today
Signal Routing Agent: routes Voice of Customer findings to Product, Engineering, CS, and exec stakeholders with role-appropriate framing
Theme Detection Agent: spots emerging themes across surveys, support tickets, and community posts before they become escalations
VoC Executive Brief Agent: synthesizes quarterly findings into a CMO and CFO-ready brief tied to retention and expansion KPIs
Each one comes with the same structure as the prior pillar drops: purpose, what it reads, decision logic, what it produces, where humans stay in the loop, why it matters, and hours saved per week. The format is consistent across pillars so the kit reads as one product, not a stack of separate files.
Download the file at the link below and save it to your Claude Copilot pack alongside the lifecycle, advocacy, and community modules.
For paid subscribers
This week’s editorial is the build report. The artifacts that produced it are now in your paid Drive bundle and they’re meaningfully more valuable than last week’s drop.
The Chief of Staff validator system prompt is the new addition to your Copilot. Paste it into a fresh Claude Project’s custom instructions and you have a working validator you can run against any operator agent’s output. It’s the same prompt I ran across seven tests in this issue. If you want to run your own Story Prep brief through it tonight, you can.
The full build log with all seven test entries is also in your folder. Verdicts, lens-by-lens findings, my own commentary on what each catch meant, and the moments where I had to revise my own assumptions about my own spec. It’s the working document this issue was written from. Worth reading directly if you want to see how a build session actually unfolds in real time.
The Test 8 and Test 9 designs are in the practical test suite design doc. Those are the scenarios I’ll run in Issues 28 and 29. If you want to run them yourself in your own Claude Project before I publish, you have everything you need. Early access to the next two issues, effectively.
Two folders in your paid Drive bundle, as always:
Next week, Test 8.
— Kevin
P.S. If this issue made you think differently about how your team will ship AI-assisted work in the next twelve months, forward it to one customer marketing leader who’s wrestling with the same question. That’s how this newsletter grows.