Building a Website AI Chatbot That Does Not Make Things Up

A chatbot stops fabricating when it can only answer from retrieved passages of your own content, when refusing is treated as a correct outcome rather than a failure, and when uncertain conversations move to a human with full context attached. Prompt wording helps at the margins. Architecture is what prevents invention.

We run an AI agent on our own site with human takeover, so this is written from operating one rather than from theory. The uncomfortable finding is that most of the value comes from what the bot declines to answer.

Why do chatbots invent answers?

Because a language model generates likely text, not verified facts. Asked something it has no information about, it produces the most plausible continuation. Plausible and true are different things.

This is the default behaviour, not a bug to be patched. A model given no source material and asked about your return policy will write a return policy. It will sound reasonable. It will be fiction.

Three forces make it worse in practice. Business questions often have precise answers, so approximately right is wrong. Customers treat a bot on your domain as speaking for you, which makes a fabricated answer a commercial liability. And a confident tone makes errors harder to spot than a hedged one would.

What does grounding actually require?

Grounding means every answer is assembled from passages retrieved from a source you control, and the model is constrained to use only those passages.

Four components make it work.

A curated knowledge base

Not your whole website. A deliberately chosen set of content: services, pricing structure, process, delivery timelines, contact routes, policies. Each item needs an owner and a review date. Stale source material produces confidently wrong answers that are harder to diagnose than fabrication, because the pipeline is working correctly on bad input.

Retrieval that returns the right passages

Content is chunked, indexed and retrieved by relevance to the question. Chunking matters more than teams expect. Split a pricing table across two chunks and the model sees half a table and completes the pattern. Keep semantically complete units together.

A constrained prompt

The instruction must state that the model answers only from supplied context, cites which source it used, and says it does not know when the context does not contain the answer. That last clause has to be explicit and reinforced, because the model's default is to help.

Verification before output

Check that the claims in the answer are present in the retrieved context. Numbers, dates, prices and names are the high-risk fields. Where the answer contains a figure not in the context, suppress the answer rather than sending it.

How do you make refusal work properly?

Treat refusal as a correct answer with a route attached. A bot that says "I don't know" and stops is worse than useless. A bot that says it cannot confirm that and offers to put you through to someone who can is doing its job.

Define when refusal triggers.

Situation	Behaviour
No relevant passage retrieved	Refuse, offer human handover
Retrieved passages conflict	Refuse, flag for content review
Question concerns pricing commitments	Answer with structure only, route to a person for a quote
Question is legal or contractual	Refuse, route to a person
Customer is complaining	Acknowledge, route immediately
Question is outside the business scope	Decline politely, do not improvise

Write refusal copy that sounds like your business rather than like an error message. "I can't confirm that one. Let me get someone who can" reads as competence. "I'm sorry, I cannot help with that request" reads as a wall.

Log every refusal. The refusal log is the single most useful artefact the system produces, because it is a ranked list of what your customers want to know and your content does not cover. We feed ours directly into content creation priorities.

How should escalation to a human work?

The handover must be instant, contextual and visible. Three requirements.

First, the person taking over sees the full conversation. The customer should never repeat themselves. A handover that starts with "how can I help?" after a six-message exchange undoes any goodwill the bot earned.

Second, the handover is offered, not hidden. Make the route to a person available in every conversation from the first message. Businesses that bury it to reduce support load get worse outcomes, because frustrated customers leave rather than persist.

Third, when no one is available, say so honestly and capture the enquiry. Stating the hours and taking a message beats a bot pretending someone is coming.

Our site agent follows this pattern: it answers grounded questions about our services, it refuses anything about a specific quote, and a member of the team can take over the conversation live. Pricing questions always go to a person, because a quote depends on scope and nothing in a knowledge base can know your scope.

What should you measure?

Four metrics tell you whether it is working. Containment rate alone will mislead you, because a bot that answers everything confidently scores beautifully while damaging your business.

1. Grounded answer rate. The share of answers fully traceable to retrieved content. This should be near total.
2. Refusal rate with successful handover. Refusals are healthy when they route somewhere.
3. Escalation satisfaction. How customers rate the conversation after a human took over.
4. Content gap list. Refusal topics ranked by frequency, reviewed monthly.

Audit transcripts by hand every week in the first month. Automated metrics will not show you tone problems or subtly wrong answers that were technically grounded in the wrong passage.

What does this cost to run properly?

The build is not the expensive part. The upkeep is. Someone must own the knowledge base, review the refusal log, and keep source content current. Budget for that from the start, or the system degrades within a quarter.

It is also worth being honest about scope. A grounded support agent handles repeat questions well. It is not a salesperson, and it will not close deals. We treat it as part of an AI and automation engagement that connects to the site itself through web design and development. The same grounding discipline shapes how we build conversational features into our own products, including MindHarbor. Delivered examples sit in our case studies.

Frequently asked questions

Why do AI chatbots make things up?

Language models generate plausible text rather than retrieving verified facts. Without a grounding source, the model fills gaps with statistically likely answers. Fabrication is the default behaviour of an ungrounded model, not a defect, so the fix is architectural rather than a matter of better prompting.

How do you stop a chatbot inventing answers?

Ground every answer in retrieved passages from your own content, instruct the model to answer only from those passages, and make refusal an acceptable outcome. Pair that with a confidence threshold that routes uncertain questions to a human instead of guessing.

Should a website chatbot hand over to a human?

Yes, for anything involving pricing commitments, contracts, complaints or questions falling outside its grounding content. Handover should pass the full conversation to the person taking over, so the customer never has to repeat themselves. A bot that cannot escalate quickly becomes a dead end.

If you want a site agent that answers accurately and knows when to stop, we build and run them, including ours. Call +971 52 961 4900, email info@oldport.ae, or use the contact page.