A chatbot can answer a shipping question in seconds and still create a support problem if it cites an outdated policy, guesses at a delivery promise, or sounds more certain than your business can be. That is why a chatbot evaluation checklist should test more than whether the widget produces fluent answers. It should test whether the system protects the information, policies, and customer expectations your team is responsible for.
For small teams, the practical question is not whether a chatbot appears intelligent in a product demo. It is whether it can handle repeated website questions accurately, show visitors where an answer came from, and stop when the approved information does not support an answer.
1. Start with the support work you want to reduce
Evaluate a chatbot against real recurring questions, not a generic prompt set. Pull questions from support inboxes, sales conversations, site search terms, and contact forms. The useful test set reflects the work your team actually repeats.
For an ecommerce business, that may include shipping regions, delivery estimates, return windows, exchanges, warranties, and product compatibility. A B2B software company may test plan details, onboarding requirements, documentation, contract questions, and support hours. A service business may focus on locations served, appointment policies, scope of work, and cancellation terms.
Separate these questions into three groups: questions with a clear published answer, questions that require clarification, and questions the chatbot should not answer. The third group matters as much as the first. If a customer asks for a special discount, a binding delivery guarantee, or a policy exception, a polished answer is not necessarily a safe one.
2. Check where every answer comes from
A support chatbot should have a defined knowledge boundary. Ask what information it uses, who can review it, and how updates take effect. If the system can answer from material your team has not approved, you have less control over the promises made on your website.
Look for a workflow that lets your team use existing website content and approved business documents as the knowledge base. The source material should be understandable to nontechnical owners: policy pages, product pages, help articles, FAQs, and operational documents that are appropriate to share with visitors.
Then test source visibility. When the chatbot answers, can a visitor verify the relevant source? References are useful for two reasons. They help customers confirm details themselves, and they help your team investigate an answer when something looks wrong. An answer without a visible basis may still be correct, but it is harder to audit and harder to improve.
RobiFox is designed around this controlled model: answers are grounded in approved website content and supported documents, with source references where available. The operational point is simple: the business remains the authority on its prices, policies, and promises.
3. Test refusal behavior before testing creativity
A chatbot should not fill gaps with likely-sounding language. Test it with questions that are deliberately outside the available knowledge base. Ask about an unavailable product variation, an unlisted delivery country, a discontinued offer, or a policy exception not covered in your published materials.
A good response acknowledges the limit plainly. It may direct the visitor to contact the team or point them to the relevant policy, but it should not invent an answer to avoid sounding unhelpful. An accurate “I don’t know” is often better customer service than a confident answer that later needs to be reversed.
Use this checklist when reviewing those responses:
- Does the chatbot clearly state when the available information is insufficient?
- Does it avoid inventing fees, timelines, eligibility rules, or commitments?
- Does it avoid presenting a general policy as approval for an exception?
- Does it guide the visitor toward an appropriate next step without pretending to resolve the issue?
- Can your team review these unanswered questions afterward?
The last question turns refusal into useful operational feedback. A pattern of unanswered questions can reveal a missing FAQ, unclear policy wording, or an issue visitors cannot find on the site.
4. Review accuracy at the detail level
Broadly correct answers can still fail customers. “Returns are accepted” is not enough if the policy has a specific window, condition, or exclusion. Evaluate whether the chatbot preserves the details that change a visitor’s decision.
For each published-answer test, check four things: factual correctness, completeness, wording, and source alignment. Factual correctness asks whether the answer matches the approved material. Completeness asks whether it includes the conditions a visitor needs. Wording asks whether it adds certainty or interpretation that the source does not support. Source alignment asks whether the cited material actually supports the specific answer.
Pay close attention to dates, quantities, geographic limits, product model names, and conditional language such as “may,” “typically,” or “subject to review.” These are easy details for a visitor to miss and costly details for a business to misstate.
It also helps to include conflicting-source tests. If an old FAQ and a current policy page disagree, how does the system respond? The right operational fix may be updating or removing the stale source, not trying to prompt the chatbot around a content-management problem.
5. Evaluate the customer experience, not just the answer
A reliable answer that is difficult to read or slow to appear still creates friction. Test the widget on desktop and mobile with the questions visitors are likely to ask in their own words. Customers rarely phrase requests like your internal documentation.
Review whether answers are concise enough for a website conversation while retaining necessary conditions. Check whether the assistant asks a clarifying question when a request is ambiguous. “Do you ship there?” may need a country or postal code context, depending on what your published information covers.
Also inspect the handoff experience. A chatbot does not replace human judgment for every situation. Visitors should not be trapped in a loop when they need a person, especially for unusual cases, complaints, or questions that depend on account-specific details the widget cannot verify.
6. Include multilingual questions in the evaluation
Multilingual support is valuable only if the facts remain consistent across languages. Test the same policy question in English and the languages your visitors use most. Compare the answers for policy conditions, numbers, dates, and the level of certainty.
Do not assume a translated response is adequate because it reads naturally. A fluent answer can still omit a restriction or soften a requirement. Have a qualified reviewer check high-stakes policies such as returns, cancellations, delivery terms, and warranties before relying on them for customer communication.
The right standard depends on your audience. A business with occasional international visitors may prioritize clear answers to core questions. A company actively serving several markets may need a more formal review process and regular checks after policy updates.
7. Assess the team workflow behind the widget
The control behind a chatbot is the product. Before adopting one, map who owns the knowledge base, who approves updates, who reviews conversations, and how often those steps happen. A tool that is easy to launch but difficult to maintain can become another unchecked customer-facing channel.
Ask whether your team can identify unanswered questions and recurring points of confusion. Ask whether changes to a shipping page, pricing page, or policy can be reflected in the approved information without a complicated technical project. Finally, assign a clear owner. It may be operations, support, ecommerce, or marketing, but it should not be nobody.
A simple monthly review is often enough for a small team: inspect unanswered questions, spot-check sourced answers, remove stale information, and add content where visitors repeatedly get stuck. Review more frequently during promotions, policy changes, product launches, or seasonal fulfillment periods.
A final decision test
Do not choose a chatbot because it gives the most expansive answer. Choose one that can be useful within the boundaries your business sets. If it can answer from approved information, show the basis for that answer, acknowledge what it cannot verify, and give your team a workable review process, it has a credible role in website support. The goal is not to automate every conversation. It is to make the repeatable ones clearer, faster, and safer for both customers and your team.
