A regulated lender had to prove its agents said the right things on every sales call. A quality team was listening to a sample. Now every call is checked, and a person reviews what is flagged.
The Problem
Two obligations on every call: say what the regulator requires, and never promise what cannot be promised.
The quality team listened to a small sample of calls and extrapolated. Anything outside that sample was unexamined, so the first time the business learned an agent had been mis-selling for months was usually when the complaint arrived.
Risk discovered too late
A pattern of bad practice surfaced through customer complaints or a regulatory query rather than through internal review, by which point the exposure was already built.
Coaching aimed at the wrong people
Agents were reviewed by whoever happened to be sampled, so feedback landed on the unlucky rather than on those who actually needed it.
No defensible record
When a customer disputed what they had been told, the business had a recording somewhere and no practical way to find the moment in question.
Constraints
The non negotiables that ruled out the obvious approach.
A flag can affect someone's pay and standing. The system had to be tuned for precision over coverage, and no finding reaches an agent's record without a human confirming it.
Calls switch between English and the regional language mid sentence, often mid clause. Transcription had to handle the mixture rather than assume one language per call.
Compressed, eight kilohertz, frequently with a call centre floor audible behind the agent and a customer on a moving vehicle.
A compliance officer will not act on a label. Each finding links to the transcript segment and the exact position in the recording so it can be heard in context.
How It Works
Turn every call into searchable, attributable text, then check it against obligations written by the compliance team rather than encoded by a developer.
Agent and customer are split before anything else. Attribution is the whole game here, because a promise made by a hopeful customer is not a breach and the same words from an agent are.
Handles calls that move between languages mid sentence, keeping timestamps at phrase level so any finding can be played back at the exact moment it occurred.
Each mandated statement is checked for having actually been made and understood, not merely for a keyword appearing somewhere in the transcript.
Assurances about returns, guarantees of approval, and pressure tactics. This is a judgement about meaning rather than a search for banned words, which agents learn to avoid within a week.
Every flag reaches a compliance reviewer with the transcript segment and audio position. Nothing reaches an agent's record automatically.
When a reviewer overturns a finding, that call joins a labelled set used to check the next change does not reintroduce the same mistake.
The Hard Part
The obligation is not to avoid a list of words. An agent may accurately say what a fund returned last year. The same agent may not tell a customer what they will receive. The two sentences are nearly identical in vocabulary and completely different in regulatory meaning.
Keyword rules failed immediately, and failed in the worst direction: they flagged accurate, permitted statements, which destroyed the compliance team's trust in the first week. A tool that cries wolf is switched off no matter how good its recall is.
What worked was treating each obligation as a question asked of the passage in context, with the surrounding turns included, so tense, conditionality and who is speaking all carry weight. The output is a judgement with a confidence and the span of text it relied on.
Calibration mattered more than the model. The compliance team labelled a set of real calls, including the genuinely ambiguous ones, and thresholds were set against that set rather than against intuition. The same set runs as a regression check before any prompt or model change ships, because a quiet drop in precision here is invisible until it has damaged several careers.
Technical Decisions
| Choice | Why | Instead of |
|---|---|---|
| Diarisation before transcription | Attribution decides whether something is a breach at all. Splitting speakers first keeps a customer's words from ever being scored against the agent. | Transcribing the whole call and inferring the speaker afterwards |
| Obligations as data | Compliance writes and edits the checks in their own words, with a version history. Regulatory wording changes and cannot wait for a release. | Each rule implemented as code by a developer |
| Labelled call set as a test suite | Any change to a prompt, model or threshold runs against calls the compliance team has already judged, so a regression in precision is caught before it reaches anyone's record. | Spot checking a few calls after each change |
| Phrase level timestamps | A finding is only useful if it can be heard. Every flag deep links into the recording at the moment it refers to. | Storing a transcript per call with no positional index |
| Confidence band, not a verdict | The system reports how sure it is, and only the confident band is queued for review. The uncertain band is sampled instead, which is where genuinely new agent behaviour shows up. | A single pass or fail label per obligation |
Outcome
What changed for the business.
Coverage moved from a sample to the full set, so a pattern of bad practice is visible while it is still a coaching problem rather than a liability.
The quality team stopped listening to compliant calls to find the rare bad one, and now works from a queue of specific, evidenced findings.
When a customer says they were told something, the exact moment in the call can be found and played rather than argued about.
Patterns are visible per agent and per team, so training addresses what people actually do wrong rather than what is assumed.
In Hindsight
What I would do differently.
I optimised for catching everything in the first version. The compliance team lost confidence in it within days because most of what it raised was defensible. Tuning hard for precision, and accepting that some breaches are missed, is what made it usable at all.
The labelled call set should have been built before any detection was written. It was created after the first round of complaints about accuracy, and every decision before that point was made on intuition.
I underestimated how much of the value was the search. Being able to find the moment a thing was said turned out to be worth nearly as much to the business as the automatic checking, and it was almost a side effect.
Get in Touch