Fixing Speaker Tagging by Rethinking What “Timestamp” Means

The Setup

One engineer picked up the bug as their first task on this part of the codebase, which meant learning the system and hunting down a real problem at the same time. The bug itself sat in an easy place to overlook: the bot generates a live transcript during the call, then reconciles it against a more accurate, post-processed version afterward. That reconciliation step was supposed to line the two versions up. Instead, it was quietly shifting speaker tags onto the wrong words.

The Constraints 

This one was clean, as far as bugs go. No legacy code to route around, no infrastructure to rework, no team or deadline pressure pushing toward a quick patch. The real constraint was unfamiliarity, this was new territory for the engineer fixing it, so step one was understanding how the existing tagging logic worked before touching any of it.

The Approach

Work started with Claude Code, used to trace through the codebase and figure out how speaker tags were being assigned in the first place. The early passes didn’t turn up an answer. A few different theories got tested and dropped.

Instead of continuing to guess alone, the engineer brought in the person who’d originally built the feature. That conversation did two things: it surfaced gaps in the current logic that weren’t visible just from reading the code, and it pointed the investigation at one specific mechanism worth testing instead of several vague ones.

That mechanism was the timestamp itself. The reconciliation logic was cross-referencing the live transcript against the final transcript using wall-clock time, the moment each segment was recorded. The problem is that wall-clock timing during a live call isn’t perfectly steady. Small drifts between when someone actually spoke and when that speech got timestamped in real time were enough to nudge segments out of alignment. Once alignment slips even slightly, speaker tags start attaching to the wrong words.

The fix was to stop aligning on wall-clock time and instead align both transcript versions using word-level timestamps, timing tied to the audio itself rather than to a clock reading taken mid-call. Word-level timestamps don’t drift the same way, since they’re anchored to the words as spoken. Reconciling against them holds up even when the live-capture timing is off by a beat.

This wasn’t the fix that looks obvious going in. The easier move is to patch the existing comparison, tighten the tolerance, adjust an offset, keep the wall-clock approach and paper over the edge cases. That would have addressed the symptom without touching the reason it kept happening. Switching the alignment key to word-level timestamps went after the actual cause.

The Tradeoffs

What got given up here was the original alignment approach, full stop, rather than patching it. That’s not free: patching would have been the faster thing to try, and there was a real stretch of trial and error before the word-level fix became clear.

The approach also has a dependency baked in. It only works if word-level timestamp data is actually available and reliable from the transcription pipeline. If a future transcription source doesn’t expose timing at the word level, or if that data turns out to be noisy, this exact fix won’t carry over, and alignment would need solving again from a different angle.

The Results

The fix was validated by QA against real meetings, including the team’s own standups, rather than synthetic test cases. It’s live in production now.

Testing against actual calls mattered specifically here: the original bug only showed up under the timing conditions of a live call, so a synthetic test wouldn’t have reproduced the drift that caused the mismatch in the first place.

The Lesson

Asked what advice they’d give another engineer facing something similar, the answer was direct: use whatever tools and people are available to you. That meant using AI coding tools to move quickly through unfamiliar code, and it meant going straight to the person who built the original feature instead of trying to reverse-engineer their reasoning alone.

The Quote

“I did use AI coding tools like Claude Code to investigate the issue. I didn’t find the fix immediately, I tried a bunch of approaches until I landed on something concrete. Use whatever tools and people you have access to, human and AI.”