Financial services — consumer lending contact centreSpeech AI & multi-agent quality scoring

Automated quality management that scores every contact-centre call, not 2–3% of them

A consumer lender’s QA team could listen to 2–3% of calls. We built a speech-to-text and multi-agent scoring system that transcribes, analyses and scores 100% of them in two languages — delivered in eleven weeks.

Every recorded call, live or post-call and in both of the contact centre’s languages, enters an audio processing agent that runs speech-to-text, speaker separation and acoustic feature extraction. Two agents work on the transcript: a quality agent for emotion, sentiment and conversation flow, and a compliance agent for script and required-phrase checks. A supervisor agent turns the results into a QA report and coaching. Outputs go to agent coaching per call and per agent, live dashboards of KPIs, trends and anomalies, and the CRM, data warehouse and workforce management systems through APIs and MCP. Underneath, one configurable scorecard set by the QA team applies to every call, with an audit trail on every score. Before, QA heard 2–3% of calls; now 100% are scored, with 80%+ transcription accuracy, 70% less QA processing time and 90% better compliance coverage.
Every recording goes through the same agents and the same scorecard. Reviewers see the calls that need them.

Project snapshot

Client
A consumer lender and retail bank running a large bilingual customer contact centre
Industry
Financial services — consumer lending
Business function
Contact-centre quality assurance and agent coaching
Challenge
QA reviewers listened to a manual sample of 2–3% of calls. Script breaches, compliance gaps and unhappy customers in every other call went unseen, scoring varied by reviewer, and coaching depended on which calls happened to be sampled — in a contact centre working across two languages.
Solution
An automated quality management system: speech-to-text with speaker separation in both languages, a quality agent for emotion, sentiment and conversation flow, a compliance agent that checks script adherence and required phrases against a configurable scorecard, and a supervisor agent that turns scores into QA reports and coaching — integrated with the CRM, data warehouse and workforce management systems through APIs and MCP.
Result
100% of calls scored automatically instead of 2–3%, 80%+ transcription accuracy across both languages, 70% less QA processing time and a 90% improvement in compliance coverage — with automated coaching recommendations, delivered in eleven weeks.

Key outcomes

100%
Of customer calls scored automatically
70%
Less QA processing time
90%
Improvement in compliance coverage
80%+
Speech-to-text accuracy in both languages

The client

A lender whose customers mostly meet it on the phone

A consumer lender and retail bank running a large contact centre in two languages. Loan, repayment and service questions all come through it, so the quality of those calls is the quality of the bank, as most customers experience it.

It is also where regulated conversations happen. A missed disclosure on a loan call is a compliance problem, not just a service one.

The challenge

A QA team that could only hear a sliver of the calls

Quality assurance meant reviewers listening to recordings and scoring them by hand. At contact-centre volumes, that covers a small sample. Everything else — the script breaches, the frustrated customers, the agent who needed help — went unheard.

Before

2–3%
of calls reviewed by QA
2
languages the contact centre serves

Scores also depended on who did the listening, and coaching depended on which calls happened to be picked. Agents could not tell a fair review from an unlucky one.

Constraints

What we had to design around

  • LanguageCustomers call in either of two languages. Transcription had to hold up in both, on real contact-centre audio, or every score built on top of it would be wrong.
  • VolumeScoring every call means scoring at peak hour too. The system had to keep up with a high-load contact centre in real time and after the call, not fall behind and catch up overnight.
  • OwnershipThe QA team already knew what a good call sounds like. Their scorecards, required phrases and criteria had to stay theirs to change — not criteria frozen into a model.
  • RegulationA lender’s calls carry personal and payment data. The system had to meet ISO, GDPR and PCI DSS requirements, keep an audit trail on every score, and run on-premise or in the cloud.
  • FitScores are only useful where people already work. Results had to reach the CRM, the data warehouse and workforce management, not sit in a new tool nobody opens.

Our approach

Transcribe everything, then let agents do the listening

A better sample would still be a sample. The only way to see every compliance gap is to score every call, and the only way to afford that is to stop needing a person to listen first.

So we built from the audio up. A speech engine that works in both languages comes first, because every score sits on its transcript. Separate agents then judge separate things — how the call felt, whether the script was followed — against criteria the QA team sets. A supervisor agent pulls it together into something a team lead can coach from.

The solution

Five layers, from a recording to a coaching note

  1. Speech engine — Speech-to-text models for both languages, live and post-call, at 80%+ accuracy. Speaker separation tells the agent from the customer, and acoustic features are kept alongside the words.
  2. Quality analysis agent — Reads emotion and sentiment through the call and measures conversation flow and dynamics, so a call that went wrong is visible even when the script was followed.
  3. Compliance scoring agent — Checks script adherence and required phrases against the QA team’s own templates and criteria. The same scorecard is applied the same way to every call, which removes reviewer-to-reviewer variation.
  4. Supervisor agent — Built on Agno, an audio-processing controller runs the analysis loop and a supervisor agent turns the results into a QA report with coaching recommendations for the agent and their team lead.
  5. Integration and analytics — REST APIs and MCP servers push scores into the CRM, data warehouse and workforce management systems. Real-time dashboards show KPIs, trends, anomalies and the calls that need a person.
A sequence diagram of one call through the system. A contact-centre agent’s customer call starts; the AQM system controller creates an audio processor agent. Inside an iterative call processing loop, the audio processor agent processes the live audio stream, analyses speech patterns, saves the transcription to a memory store and retrieves call context. It creates a quality analysis agent, which creates a compliance scoring agent. The quality analysis agent runs emotion detection and evaluates sentiment, then returns its analysis results. The compliance scoring agent scores script adherence and evaluates quality metrics, then returns its scoring results. The audio processor synthesises the results and either continues processing or ends the call analysis. The controller then sends the results to a supervisor agent, which identifies coaching opportunities and returns a QA report with recommendations. The results are persisted to the memory store and coaching recommendations are returned to the contact-centre agent.
One call, end to end. Quality and compliance are scored by separate agents; the supervisor turns their results into coaching.
Six epics over eleven weeks. Epic one, two weeks: the speech engine — speech-to-text in both languages at 80%+ accuracy, speaker separation, acoustic features and emotion detection. Epic two, two weeks: the scoring framework — configurable templates, script compliance with phrase detection, conversation flow metrics and automated scoring. Epic three, three weeks: enterprise integration — REST APIs and MCP servers for CRM, data warehouse and workforce management, with streaming, webhooks, authentication and data synchronisation. Epic four, one and a half weeks: analytics and reporting — real-time KPI dashboards, trend and anomaly detection, predictive problem flags and a compliance audit trail. Epic five, one and a half weeks: secure deployment in the cloud or on-premise, with monitoring, alerting, backup and recovery and load balancing. Epic six, one week: training and handover with administrator training, user guides and video tutorials.
The speech engine comes first because every score after it is only as good as the transcript.
11 weeksfrom kick-off to every call scored, across six epics

Responsible by design

Four rules for scoring people’s conversations

  • Configure

    The QA team owns the scorecard.

    Criteria, templates and required phrases are settings, not code. When a script changes, QA changes the scorecard the same day.

  • Evidence

    Every score leaves an audit trail.

    Each score and compliance violation is logged with the transcript behind it, so a reviewer can check the reasoning rather than trust a number.

  • Coach

    Scores exist to improve calls, not to rank people.

    The supervisor agent writes coaching recommendations, not just a number. Team leads get the pattern across an agent’s calls, not one bad afternoon.

  • Protect

    Call data is treated as regulated data.

    ISO, GDPR and PCI DSS controls, an audit trail on every score and violation, and a choice of on-premise or cloud deployment.

The third rule shapes the rest. Scoring every call could easily become surveillance. Built around coaching, it becomes the fairest review an agent has had — every call counts, not the unlucky one.

Results

What changed for the contact centre

  • 100% of calls scored automatically, replacing a manual QA sample of 2–3% — every customer interaction now gets the same scorecard.
  • 70% less QA processing time — reviewers spend their hours on the calls the system flags, not listening for them.
  • 90% improvement in compliance coverage, with script and phrase checks and an audit trail on every call.
  • 80%+ speech-to-text accuracy in both languages, live and post-call, with speaker separation.
  • Automated coaching recommendations for every agent, built from all of their calls — and more consistent service quality for customers as a result.

QA stopped being a lottery of which calls got heard and became a full picture of how the contact centre actually sounds.

Beyond the numbers

What else changes

Every call becomes data the rest of the bank can use. Scores, sentiment and compliance flags land in the data warehouse next to the customer record, so trends — a product that keeps causing complaints, a script line customers trip over — show up in analytics instead of anecdotes.

And because the scorecard is configuration, a new product or a new regulation is a change to criteria, not a new project.

If your QA team hears a fraction of your calls

The first question is what a good call means to you. That is answerable from your current scorecard and a sample of recordings, before anything gets built.