How do you stop a voice agent hallucinating on a regulated call?

Share this article:

How to stop an AI voice agent inventing statements on a regulated customer call, which errors remain once it cannot, and how to prove both are controlled.

The short answer

You cannot stop a large language model from producing false sentences. You can stop those sentences reaching the customer. On the regulated parts of a call, do not let a model write what the customer hears: have the agent choose from wording your compliance team approved before the call, insert account data only from your system of record into fields you permitted, and re-confirm or hand over to a person when it is not sure what it heard. That removes invented statements. It does not remove every error. The agent can still choose the wrong approved response, read out wrong data correctly or mishear the customer, so you test for those, monitor for them and keep a record that shows each step.

What counts as a hallucination on a voice call?

Strictly, a hallucination is a fluent statement with no basis in fact or policy. On a phone call, a customer can hear something false in four ways, and only the first is a hallucination in that strict sense:

A generated false statement. The model composes a sentence that is untrue or outside policy: a settlement discount that does not exist, a payment plan the firm does not offer, the wrong consequence of non-payment.
A misheard answer acted on. Speech recognition turns "I can't pay that" into "I can pay that", or captures the wrong date of birth, and the agent carries on as if it were right.
Wrong data in a correct sentence. The wording is approved, but the balance, date or reference inserted into it is wrong because the source record is wrong or mapped to the wrong field.
A wrong summary after the call. A generated call summary records something the customer never said, and later decisions about a complaint, a dispute or vulnerability rely on it.

The customer and the regulator will not distinguish between them. Where CONC applies, CONC 7.9.3G applies the clear, fair and not misleading rule in CONC 3.3.1R to communications with customers about their credit agreement, and the Consumer Duty requires firms to support customer understanding. A false sentence spoken by an agent on the firm's behalf is the firm's communication.

Why are prompts and guardrails not enough?

Instructions such as "only state facts from the account record", retrieval from approved documents and output filters all reduce how often a model invents something. None of them takes the rate to zero, and the rate shifts when the model, the prompt or the conversation changes. A filter that checks generated text has to recognise every possible wrong sentence, which is far harder than approving a finite list of right ones. For a regulated call the useful question is not how rarely the agent invents a statement, but whether it can. We compare the two designs in deterministic vs generative voice AI for FCA-regulated collections.

Seven controls that stop a voice agent hallucinating

Take generation off regulated routes. Statements about the debt, the options, the consequences, the customer's rights and the required disclosures come from a library of approved responses. Map which parts of the call, if any, still use generated text, and keep them away from anything a customer could rely on.
Use the model to understand, not to speak. Language models are good at working out what a customer meant. Let the model choose between defined intents, always including "none of these", so an unexpected answer leads to a clarifying question or a handover rather than an improvised reply.
Insert data only from the system of record. Balances, dates, arrangements and names come from your systems through a defined field mapping. Validation rules for format, range and missing values stop the sentence being spoken if the data fails. The agent never states a figure it was not given.
Handle uncertainty explicitly. When speech recognition is unsure or the classification is ambiguous, the agent reads back what it heard and asks the customer to confirm. Cap the number of attempts, then hand over to a person with the context so the customer does not have to repeat themselves.
Close the back doors. Check what the agent says on silence, on an off-topic question, on an attempt to talk it into something, and when an integration times out. An agent that is approved on the main path but generates text on a fallback path can still hallucinate.
Test every state before go-live and after every change. A defined journey can be tested state by state: each response, each data field, each exit. Add adversarial calls, such as a customer pressing for a discount that does not exist, plus a range of accents and poor lines. Re-run the full set whenever the speech recognition, the language model or the journey changes.
Monitor live calls and keep the proof. Log, for every turn, what was heard, how it was classified, which rule selected the response and which approved version was spoken. Review a sample of calls, track misclassification and handover rates, and trace any complaint about something the agent said back to the step that caused it.

The FCA's October 2026 publication on AI in financial services lists untested vendor assurances as an area for improvement. Test these controls yourself rather than relying on the vendor's description of them.

What if you already run a generative agent?

You can cut the risk without starting again. Work from the highest-risk statements down:

List what it can say. Inventory every statement the agent makes about amounts, offers, consequences, rights and disclosures, and mark which are scripted and which are generated.
Script the regulated moments first. Move disclosures, payment terms, consequences of non-payment and responses to vulnerability onto approved wording.
Restrict what it can commit to. Make sure the model cannot agree an arrangement, a discount or a hold without a rule-based check against policy.
Measure what is left. Review a sample of recorded calls and count statements not supported by policy or the account record. If the rate is not measured, nobody can tell the board it is controlled.

What should you ask a vendor about hallucination?

Which parts of the call, including fallbacks, use generated text? Show us.
Can we export every response the agent can give and approve it before go-live?
What does the agent do when it is not sure what the customer said?
Where does each figure spoken on the call come from, and what happens when it fails validation?
For one of our test calls, can you show which rule selected each response and which approved version was spoken?
When you change a model, how is our tested behaviour protected, and how much notice do we get?

For the full supplier check, use our voice AI vendor due diligence checklist for FCA-regulated firms.

How does MEGA.AI stop its agents hallucinating?

MEGA.AI is a vendor too, so this section sticks to what the platform does today.

Approved wording only. MEGA.AI's agents are deterministic: they speak only wording the client approved before the call, in a journey the client designed, with live data inserted only into permitted fields. Language models classify what the customer said; they do not write the replies.
Uncertain speech. When speech recognition is unsure, the agent re-confirms what it heard or hands the call to a person.
Defined handover triggers. Uncertain speech recognition, vulnerability indicators, a mention of Breathing Space or a debt adviser, an objection to recording and a request for a person.
A person on request. Customers can ask for a person at any point.
Records. A recording, transcript, step-by-step phase log and flags for every call. See what a full audit trail should contain.

Approved wording stops invented wording. It does not correct wrong source data or a wrong choice of response, so the client's own testing, monitoring and sign-off still matter.

Frequently asked questions

Can you stop a large language model hallucinating altogether?

No. Current models can produce fluent, false statements, and no prompt or setting guarantees they will not. What you can control is whether those statements reach the customer, by not letting the model write what is spoken on regulated parts of the call.

Does retrieval-augmented generation solve the problem?

It reduces unsupported statements by giving the model approved documents to draw on. The model still writes the sentence, so it can misstate, combine or leave out what it retrieved. For regulated statements it is a mitigation, not a control you can evidence call by call.

Is a speech recognition error a hallucination?

Not strictly, but the effect on the customer can be the same: the agent acts on something that was never said. Read-back of key answers, a cap on re-asks and a handover to a person are the controls.

Are AI-generated call summaries safe to rely on?

They save handling time, but they can assert things the transcript does not support. Keep the recording and transcript as the record, label summaries as generated, and check them before decisions on complaints, disputes or vulnerability.

Who is responsible if the agent tells a customer something false?

The regulated firm on whose behalf the agent made the call. The vendor answers to the firm under the contract. See who is accountable when an AI voice agent harms a customer.

About MEGA.AI

MEGA.AI provides deterministic voice AI for inbound and outbound customer calls at regulated enterprises in lending and collections, banking, utilities, telecommunications and insurance. Every response the agent can give is approved by the organisation before deployment, with live data inserted only into permitted fields, and every call leaves a record that traces each step back to the approved journey. MEGA.AI is certified to ISO/IEC 27001, 27701 and 42001, is audited under SOC 2 Type II and holds Cyber Essentials Plus; its PCI DSS assessment is in progress. Current certificates are listed on the Trust & compliance page.

Book a briefing to see one of your own journeys run on approved wording, including what the agent does when it is not sure what it heard.

Sources

This article is general information on controlling AI voice agents on regulated calls, not legal advice. Last reviewed: November 2026.