Fixing a Customer-Matching Order Problem in Our Document Pipeline

The Setup

A customer flagged a wrong number in one field. The real problem turned out to have three layers, each hiding the next. A matching rule was reading plain text inside the document and mismatching it against a list of customer names, one customer’s documents happened to contain another customer’s name as ordinary text, and the match rule found that name first every time. That meant the wrong customer’s entire rule set ran against the document, not just the one field being reported. And this exact category of mistake, a shared rule breaking for one customer without any error message, had already happened once before, a few weeks earlier, for the same customer.

The Constraints

The matching rule serves every customer, not just the one reporting the issue, so a redesign was off the table, a large change there risked breaking correct behavior for everyone else. The fix had to work within the existing list-based matching logic, just reordered.

Each customer’s document format is fixed and had to be worked with as-is. The system also had to keep running normally for every other customer while the fix went in, there was no window to take it offline. And rule files are versioned: old files are never edited, so any fix meant adding a new version and pointing to it, not changing anything in place.

The Approach

Before changing anything, the engineer checked which rule file the system had actually used on the failed document. That confirmed the first layer of the problem directly: the wrong customer’s rules had run.

From there, three things happened. First, the order of names in the matching rule was changed so the correct customer’s name would win the match. A test was added using a sample document containing both customer names, checking that the match rule picked the right one. Second, a new rule was written for the actual data field that had been reported as wrong.

That second piece turned out to be incomplete. Three days later, the quality team caught that the new field rule was still wrong, it read a short summary number instead of the full list of numbers the field was supposed to contain. The document had two similar-looking values, and the rule had picked the wrong one. A second version of the rule was written to read the full list, with a written example built into the rule itself, naming the summary number explicitly and telling the next reader not to use it as the full value. That fix was checked three times against the same real document, with consistent results across every data row and field each time.

One decision worth calling out: the field logic wasn’t fixed first, even though the original report pointed straight at a field. The field rule wasn’t even running for this customer’s documents, the matching rule had already routed them somewhere else. Fixing the field first would have fixed a rule nobody was using.

The Tradeoffs

This wasn’t a single clean fix, it was three fixes over five days, because each one got checked against a real document before being trusted, and the second one turned out to be wrong. That’s a slower path than shipping a fix on the first pass, but it’s what caught the summary-number mistake before a third incident.

The matching rule itself also wasn’t redesigned, just reordered. A list checked in a fixed order can produce this same failure again for some other pair of customers whose documents happen to contain each other’s names. That risk was left in place deliberately, since this task was scoped to one customer and a full redesign could affect many others. If this same category of problem shows up a third time, reordering the list stops being the right fix, the matching logic itself needs to change.

The Results

The fix was run three times against the same real document that originally surfaced the problem. All three runs produced the same result, with every data row and every other field matching expectations each time.

There are no numbers yet for how many support tickets the original bug generated or whether the fix reduced manual checking, those live in a different system and haven’t been pulled. Rather than guess at them, the results here are limited to what was actually verified: the fix is correct, and it’s repeatable.

The Lesson

When a report says one field is wrong, check first which rule set is actually running for that data. In a system where one engine serves many customers, the visible symptom and the real cause often aren’t in the same place, the step that picks a customer’s rules usually runs before any field logic sees the document at all.

The second piece: check your own fix against a real sample before assuming it’s right. The first field fix looked correct because it matched one column. It hadn’t been checked against the full set of numbers the field actually needed to produce.

The Quote

“The report said ‘this field is wrong,’ so it is normal to go straight to the field logic. But the field logic was not even running for this customer’s documents, because the match rule sent them to a different set of rules. A fix to the field first would have fixed a rule that nobody used.”