How to Train an AI Agent on Your Help Center

To train an AI agent on your help center, you connect the articles you trust, exclude everything stale or contradictory, rewrite key articles so each answers one question, and test the agent against real customer questions before launch. Modern agents do not memorise your docs. They look up the answer when a customer asks, so your help content decides the quality of every reply.

What does "training" an AI agent on a help center mean?

The word "training" still confuses people, because early chatbots really were retrained every time you changed an answer. Modern AI support agents work differently. They use retrieval-augmented generation, usually shortened to RAG. When a customer asks something, the agent searches your knowledge base for the most relevant passages, reads them, and writes an answer grounded in that text.

Nothing about the underlying model changes. You are not teaching it anything in the machine-learning sense. You are giving it a library and a search index. Training, in practice, means three editorial decisions:

• What goes into the library.
• How each article is written.
• How often stale copies get replaced.

That is good news for small teams. The work is editorial, not technical. You do not need an engineer to do it well.

Why does the help center decide whether the agent works?

The help center is all the agent has to work with. Gartner surveyed 5,728 customers and found that only 14% of customer service issues are fully resolved in self-service. In the same survey, 45% of customers who started in self-service said the company didn't understand what they were trying to do. An AI agent reading those same articles inherits the same gaps.

Gartner is blunt about the consequence. A 2024 Gartner Quick Answer concludes that 100% of generative AI virtual assistant projects without integration to modern knowledge management systems will miss their customer experience and cost-reduction goals. And in a separate survey, 61% of customer service leaders said they have a backlog of articles to edit, and more than a third have no formal process for revising outdated ones. That backlog is what the agent reads on day one.

How to train an AI agent on your help center, step by step

Here is the sequence we recommend. Most small teams can do steps 1 to 3 in a week and the rest in the week after.

1. Export an inventory of everything you have. List every help article, FAQ, policy page, macro, and internal doc. Note the owner and the last-updated date for each. You cannot clean what you cannot see.
2. Pull your top 50 customer questions. Take them from the last 90 days of tickets, in the customers' own words. This is the test the agent has to pass, so write it down before you connect anything.
3. Map each question to exactly one article. If a question has no article, write one. If it has three, pick the best one and archive the others. One question, one current answer.
4. Run the exclusion pass. Go through the checklist in the next section and remove everything that would mislead the agent.
5. Rewrite the top 20 articles for retrieval. Answer in the first sentence, explicit conditions, edge cases as separate sections. Details below.
6. Connect live data for order and account questions. "Where is my order" should be answered from the order, not from a generic shipping article. Connect your store and CRM through integrations.
7. Mark internal-only content. SOPs that explain how to resolve a request are useful for reasoning, but should never be quoted to a customer. Flag them.
8. Set the refusal and handoff rules. The agent answers only when it finds a supporting passage. Otherwise it says so and passes the conversation to a person with context. See our guide to AI-to-human handoff.
9. Test against your 50 questions before launch. Check each answer against its source article. Log every wrong, vague, or unsupported answer and fix the content, not the prompt.
10. Launch on a slice, then review weekly. Start with one channel or one topic group. Read low-confidence and handed-off conversations every week, and feed the gaps back into the help center.

What to exclude: a checklist

Connecting everything feels thorough. It usually makes answers worse, because the agent treats every connected page as a valid source. Before you connect, remove or exclude:

• Ended campaigns, seasonal promotions, and expired discount pages.
• Draft articles, abandoned experiments, and anything marked deprecated.
• Duplicate articles that describe the same policy in different words.
• Contradictory articles. If two pages disagree on a refund window, the agent may quote either or blend them.
• Marketing copy that talks around the answer without giving it.
• Old release notes that describe features you have since changed or removed.
• Pricing pages from previous plans or currencies you no longer sell in.
• Pages with personal data, such as exported ticket threads or customer examples with names.
• Internal notes with opinions or workarounds you would not want read aloud to a customer.
• Legal boilerplate that is not a policy answer (the agent will paraphrase it badly).

A smaller, cleaner library beats a large messy one. We regularly see teams get better answers after removing content than after adding it.

How should help articles be structured for an AI agent?

What makes an article easy for a human to skim also makes it easy to retrieve. The patterns that matter most:

• Titles that match how customers ask. "How do I return a sale item?" beats "Returns policy, special conditions." Retrieval matches the customer's wording against your wording.
• The answer in the first sentence. Context and reasoning come after. Retrieval systems split articles into chunks and may only pass one chunk to the model.
• Short paragraphs, one fact each. Long paragraphs hide several facts, and the wrong one may travel with the chunk.
• Explicit conditions. Write "if the order was placed more than 30 days ago", not "for older orders." Precise text produces precise answers.
• Edge cases as their own sections. Sale items, gift orders, damaged goods. Each becomes its own retrievable answer.
• Say what you do not do. "We do not offer phone support" is a sentence the agent can quote. An absence is invisible.

We go deeper on this in writing help content an AI can answer from and in our guide to building a knowledge base for AI support agents.

How often should you refresh the help center?

Set a cadence and keep it. Because the agent retrieves at the moment a customer asks, an updated article takes effect on the next conversation. A stale article also takes effect on the next conversation. This table is the cadence we recommend for small teams:

| Content type | How often to refresh | Trigger for an immediate update | |---|---|---| | Pricing, plans, discounts | Monthly | Any price or plan change | | Returns, refunds, warranty policy | Monthly | Policy change, new exception agreed by the team | | Shipping windows and carriers | Monthly, weekly in peak season | Carrier change, holiday cut-off dates | | Product specs and how-to articles | Quarterly | New release, feature removed or renamed | | Integrations and setup guides | Quarterly | Partner changes their flow or UI | | Articles flagged by the agent as low-confidence | Weekly | Repeated handoffs on the same topic | | Full library audit (duplicates, deprecated pages) | Quarterly | Rebrand, new market, new language |

One rule overrides the table: if a policy changes, update the article before the policy goes live. Otherwise the agent keeps quoting yesterday's policy to today's customers.

How do you know the agent is reading the help center well?

Track three numbers together, per topic:

• Retrieval rate. How often the agent finds at least one relevant passage.
• Grounded-answer rate. How often the final answer is actually supported by the cited passage.
• CSAT per topic. How customers rate the answers, split by category.

Grounding helps a lot, but it does not make an agent infallible. A Stanford RegLab study of commercial legal research tools found that retrieval-based products still hallucinated between 17% and 33% of the time, even though they did better than general-purpose chatbots. Legal research is harder than a returns question, but the lesson carries over: clean source content and a refusal rule matter as much as the retrieval system. Our piece on why support AI agents hallucinate covers the fixes, and auditing AI agent answers covers the weekly review.

How Keloa approaches help center training

Keloa points its AI agents at your help center, policy docs, and live systems, and grounds each answer in your content. Replies carry a citation to the source so your team can check what the agent relied on. When the agent cannot find a supporting passage, it says so and hands off instead of guessing. That lowers the risk of a wrong answer. It does not remove it, which is why we also surface low-confidence questions back to you each week, so the help center improves where customers are actually asking.

If you want to see how this works on your own content, book a demo and bring your ten hardest questions.

Frequently asked questions

Do I need to retrain the AI agent when I update an article? No. The agent retrieves from your live content when a customer asks. Update the article and the next conversation uses the new text. There is no training cycle to wait for.

How long does it take to train an AI agent on a help center? Connecting a help center on a standard platform usually takes an afternoon. The cleanup takes longer: plan one to two weeks for the inventory, exclusion pass, and rewriting your top 20 articles. That cleanup is where answer quality comes from.

What if our help center only has 30 articles? That is often enough for a focused store or service. A small, current, well-written set beats a large one full of duplicates. Use your top 50 customer questions to find the gaps.

Can the agent answer from PDFs and internal docs? Yes, if you connect them. Treat them like public articles: one topic, a clear answer, no contradictions. Mark internal-only content so the agent can use it for reasoning without quoting it.

How do I stop the agent from inventing answers when the help center is silent? Require a supporting passage for every answer and set a refusal rule for everything else. The agent should say it does not have the information and hand off. Then write the missing article so the next customer gets an answer.

Should I write new articles or fix existing ones first? Fix the existing ones. The gap is usually contradiction, vagueness, or stale text, not missing topics. Customers ask the same questions in different words, so make sure each has one clear, current answer.

Sources

• Gartner: Survey finds only 14% of customer service issues are fully resolved in self-service
• Gartner: Quick Answer on knowledge management and GenAI virtual assistants (2024)
• Gartner: Survey reveals 85% of customer service leaders will explore or pilot customer-facing conversational GenAI in 2025
• Stanford RegLab: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools