The short version of how to audit AI customer service quality is that you score a fixed sample of AI replies every week against a small rubric, keep a regression test set the AI must pass before any change to sources or prompts ships, and watch a handful of drift signals in production. Most teams still sample 2 to 5% of conversations by hand. AI-assisted scoring lets you review the other 95%, so the question stops being "did we catch it" and starts being "what do we do about it".
Why audit an AI agent at all?
Because the failure modes are not the ones a human team creates. A human agent gets tone wrong, forgets a step, or is slow. An AI agent invents an answer that sounds correct, cites nothing, and delivers it in a confident voice inside three seconds. A 2024 Stanford HAI study found that ungrounded large language models hallucinate in 15 to 30% of customer service responses. Grounded, retrieval-backed systems can bring that number well below 1%, but only if you keep measuring.
There is also the drift problem. The AI agent that shipped clean in April can become worse in September because a policy page changed, a product SKU was renamed, or a translation was updated. Without an audit loop, no one notices until the customer complaint volume tips.
What "audit" actually means here
Audit means three things running together, not one:
- Backward-looking sample scoring. You score a set of AI conversations from last week against a rubric.
- A regression test set. You keep a fixed list of representative questions and gold answers, and the AI must pass them before any change to sources, prompts, or model rolls out.
- Live drift signals. You watch a handful of numbers on the running system that flag a slide before the sample scoring catches it.
Skipping any one of the three leaves a blind spot.
Step 1: Build a small scoring rubric
Six categories cover most of what matters. Score each one 0, 1, or 2. Keep it short so the rubric can be applied fast.
- Accuracy against the source. Is the answer correct according to your help centre or connected system?
- Groundedness. Is the answer cited from a real source, or invented?
- Tone. Does it match the brand voice you asked for?
- Compliance. Does it refuse when it should refuse and disclose when it should disclose?
- Escalation. Did it hand off to a human at the right moment?
- Task completion. Did the reply resolve the ticket, or did the customer come back?
Twelve points total, six per category boundary. Anything under 10 is a fail for that conversation. Anything over 10 with a zero in accuracy or groundedness is also a fail. The industry average QA score in traditional contact centres sits at 85% per SQM Group, so 10 out of 12 is a defensible bar.
Step 2: Pick a sample size that actually catches problems
Manual sampling in classic contact centres reviews 1 to 5% of interactions. A single analyst can score 8 to 10 conversations in detail per day. That looks fine on paper and fails in practice. At a 2% sample, catching a compliance issue that occurs in 1 in 50 calls is a coin flip in any given month.
Two better options.
AI-assisted scoring for full coverage. Have a second AI review the primary agent's replies against the rubric. This is the shift most teams miss: AI-assisted QA tools have made 100% conversation coverage technically feasible, though the cited data shows most teams still touch fewer than 1 in 20 interactions. Score everything, then human-review the auto-flagged failures and a small random sample for calibration.
Stratified human sampling. If auto-scoring is not available yet, sample by category, not at random. Take 10 from refunds, 10 from where-is-my-order, 10 from product questions, weekly. You catch category-specific drift the average sample buries.
Aim to have one of the two in place. Aim to have both in a year.
Step 3: Build a regression test set
The single most useful artefact a support team can own for an AI agent is a fixed list of maybe 50 to 150 canonical questions with gold answers. The AI must pass this set before any change ships: a source refresh, a prompt tweak, a new integration, a model swap.
Rules that keep the set useful:
- One question per row, in the customer's own words, not a paraphrase.
- The gold answer is stored, not the AI's answer from the day the set was created.
- Retire a question the moment the underlying policy changes; add a new one.
- Include known edge cases: the product with the confusing name, the policy that changed last quarter, the question that got escalated three times last month.
- Include a handful of out-of-scope questions where the correct response is a refusal or handoff.
If the AI passes 148 of 150 and the two failures are on the two questions that matter, ship anyway. If it passes 145 with one failure on a payment question, do not ship.
Step 4: Watch four drift signals in production
Between sample scoring runs, four cheap numbers tell you whether the agent is sliding.
Refusal rate. The share of tickets the AI declines to answer. A sudden drop means it has started guessing. A sudden spike means a source went stale.
Handoff rate. The share of conversations that end with a human takeover. Both directions matter. Too high and the AI is not pulling its weight. Too low and it is answering things it shouldn't.
Reopen rate. Tickets the customer comes back on within seven days after the AI closed them. This is your honest resolution signal.
Citation coverage. The share of AI answers that carry a citation to a real source. Coverage falling means the agent has quietly moved from retrieved to invented.
Look at all four on one screen weekly. Any one moving by more than 20% week over week is a trigger for a deep sample.
The audit cadence that works for a team of five to fifty
| Cadence | Activity | Owner | | --- | --- | --- | | Daily | Read the auto-flagged failures from AI-assisted scoring; fix urgent ones | Duty support lead | | Weekly | Score a random sample of 30 conversations against the rubric; log scores | QA lead | | Weekly | Run the regression test set; log pass/fail | AI lead | | Weekly | Review the four drift signals | Duty support lead | | Monthly | Trend the rubric scores and drift signals; decide on training changes | Head of support | | Quarterly | Retire and add regression questions; recalibrate rubric wording | QA lead + AI lead |
A team of five can run this in about three hours a week once it is set up.
Common audit mistakes we see
Rubric inflation. If the rubric grows past six categories, scoring collapses. Keep it short.
Grading on a curve. If the AI's average score is 11.8 out of 12, the rubric is too easy. Rewrite the borderlines so a real failure lands at 8, not 10.
Auditing only the "resolved" bucket. The conversations that ended in handoff are where the interesting failures live.
Cherry-picking the sample. Random or stratified only. Never pick "interesting" conversations to score; the results are unusable.
Grading the transcript out of context. Read the customer's next reply. If they wrote back "no that's not what I asked", the answer was wrong, no matter how well it read.
How Keloa approaches AI answer audits
Keloa's AI agents answer from your connected sources through the integrations layer and cite the source on every reply, so a rubric row for groundedness stops being a subjective read. The unified inbox keeps every AI conversation reviewable and taggable, so weekly sampling is a filter, not an export job. When source coverage is incomplete, the agent declines rather than guesses, per our note on reducing AI hallucinations, and refusals feed the drift dashboard directly.
We also treat the regression test set as a first-class artefact. Adding a question to it is a routine act of support work, not a special project. For the wider set of measurement questions see the note on CSAT for AI-handled tickets and the deflection-rate glossary entry.
Frequently asked questions
How large should the regression test set be? Between 50 and 150 questions is the useful range for most SMB teams. Under 50 and you miss categories. Over 150 and no one maintains it. Add a question when a real support failure teaches you something. Retire one when its policy changes.
Can AI grade its own answers? Yes, for coverage. No, for calibration. Use a separate AI to score against the rubric on 100% of conversations, then human-review a random 5% of the auto-scored set to make sure the grader is not drifting itself. When the grader disagrees with humans by more than a point on average, retrain it.
What is a healthy hallucination rate for a grounded AI support agent? Well under 1% of answers should contain an unsourced factual claim. Best-in-class grounded models can reach 0.7% on tightly scoped tasks. The number is not zero, so citation coverage and a refusal option are the safety net.
Should we audit AI replies before they send, or after? Both, at different bars. Before-send review works for high-stakes categories such as refunds over a threshold or account changes. After-send auditing works for the routine volume. Trying to human-review every reply before it sends erases the speed benefit that made the AI worth deploying.
What is the fastest signal that the AI has drifted? A sudden shift in the refusal rate or in citation coverage, both visible daily. Reopens follow a week later. CSAT follows two weeks later. Watch the leading indicators; do not wait for CSAT to tell you.
Does an audit programme slow down the AI rollout? It slows down changes to the AI, which is the point. The rollout itself is faster because the regression set gives a clear ship gate. Teams that skip the audit ship faster for three months and then spend six months untangling accumulated errors.
Want a working audit setup? Book a demo and we will walk through the rubric, the regression set, and the four drift signals against your own tickets.