Fixing AI Triage Without Rewriting the Checklist

THE SETUP

The compliance AI checks documents and live data against a detailed checklist, then surfaces anything that doesn’t line up. When a user disagreed with a flag, an automated job would fire off a ticket with a guessed cause pulled from the feedback text. The problem: those guesses missed the actual issue more often than they caught it.

That mattered for two reasons. First, engineers were shipping fixes based on those guesses, which meant some fixes addressed the wrong thing entirely. Second, at the peak of a live testing phase, the team was getting more than 140 pieces of feedback a day. That volume is hard for one person to work through carefully, and it’s just as hard for two. A guess that’s wrong 140 times a day compounds fast, in wasted engineering time, in fixes that don’t fix anything, and eventually in trust.

THE CONSTRAINTS

The checklist itself is a spreadsheet owned by subject-matter experts and developers together, and it gets replaced daily. That ruled out any solution that touched checklist logic directly, nothing could depend on the checklist staying the same shape from one day to the next.

Everything downstream also had to fit inside a 4,000-character prompt cap, which is tight when a single piece of supporting evidence can run 20,000 characters across dozens of documents. And production data was read-only: whatever got built could look at live data, but never write to it or act on it directly.

THE APPROACH

The obvious fix is one agent that reads everything, the finding, the citation, the guidelines, the live data, and renders a verdict. That’s also the wrong fix. A single agent working from one context tends to anchor on its first impression and reinforce its own early read of the ticket, especially on cases that look identical from the outside but have completely different causes.

The design that actually worked splits the job into three specialists, each running in its own context, each looking at one slice of the problem: the finding as recorded in the database, the guideline that governs it, and the live data as it stands today. None of them hand back raw dumps of what they found. Each one returns a verdict, a claim about what’s true, not a pile of evidence for someone else to interpret.

An orchestrator then does something none of the specialists do on their own: it goes and reads the source material itself. It pulls the actual document or data the finding cites and checks whether it really says what the AI claimed. That gap, between what the citation captured and what the source actually contains, turns out to be the single most useful signal in the whole system, and no specialist working in isolation can see it, because none of them holds both sides of the comparison at once.

The second piece closes a failure mode the team didn’t anticipate at first: sometimes the AI’s original flag was correct, and the underlying documents or data were corrected afterward. From the outside, that looks identical to the AI being wrong. So before the orchestrator classifies anything, it checks whether the audit trail shows a change after the review ran. If it does, it doesn’t force a verdict, it re-runs the relevant checklist item against the updated data and returns “stale evidence, no fix,” even when the original ticket insists there’s a bug to chase.

THE TRADEOFFS

The command trades speed on any single ticket for accuracy across all of them. It doesn’t write or apply fixes, it’s read-only by design, so a person still has to review the diagnosis, decide what to do with it, and make the change themselves. Getting real value out of it still depends on the person reading the output having enough guideline knowledge and technical context to act on it well, especially on the larger, messier tickets where several things can be true at once.

That’s a real limit, not a minor caveat. The command replaces the guesswork in gathering evidence, not the judgment required to interpret it. Teams without that judgment on hand would get a more accurate diagnosis and still not know what to do with it.

THE RESULTS

Over roughly the last month and a half, daily ticket volume dropped from 140-plus down to an average of about 20, peaking at no more than 30. Part of that drop is fixes landing correctly the first time instead of chasing the wrong cause. Part of it is the “stale evidence” check quietly closing out tickets that were never real bugs. The fixes that do get shipped now target the actual problem instead of a guess about it.

THE LESSON

Don’t build a command or a skill before you know exactly what a person does in that situation and why. Model the automation on that behavior first, test it against real cases, and only extend it once you can see it getting the right answer for the right reason. And keep a person in the loop on anything sensitive, this system reads and diagnoses, but it never gets to fix things on its own. That boundary is doing as much work as the diagnosis itself.

A QUOTE

“The interesting failures all look identical from the outside. Telling them apart takes five sources agreeing, and sometimes the AI wasn’t even wrong, someone just fixed the document after the fact. Automating the evidence-gathering doesn’t replace instinct built from reading a lot of tickets. It just means you’re finally pointing that instinct at the right facts.”