Test AI chatbot replies before your customers see them
Ten checks that test AI chatbot answers the way real customers will, with what a pass looks like and what to fix when one fails.
Key takeaways
- Just 27% of customers will give a chatbot another chance after a bad experience, so test first.
- Ask your most common price and policy questions and check each answer against the written policy.
- Ask about rules you don't have. A good bot says it doesn't know instead of inventing one.
- Ask for a person several ways, since 87% of customers say human access is a must.
- Rerun the full checklist after every price, policy or knowledge base change.
You can find a chatbot’s worst mistakes in an afternoon if you test AI chatbot answers the way a customer would. Just 27% of customers will give a chatbot another chance after a bad experience, a Gartner survey found. These ten tests find those mistakes first.
Customers aren’t against bots. In the same survey of 3,566 customers, run in February and March 2026 and published by Gartner in September, 49% said they’d have tried a chatbot had one been available. Only 7% actually used one in their latest contact with a company.
Each test below says what to send, what a pass looks like and what to fix when it fails. Keep a simple sheet with three columns: the message you sent, the reply you got, and pass or fail.
1. Run every test in private first
Testing a chatbot in private lets you see every bad answer before a customer does. Use your chatbot tool’s test mode, often called a playground, or install the chat on a staging site, a copy of your website that customers can’t reach.
Pick a tester who answers customers every day. They know how people really write: short, misspelled, sometimes annoyed. Someone who only knows the policy pages writes tidy questions the bot will pass easily.
Start with a narrow job. Gartner’s advice is to begin with a handful of tested uses, picked from problems your team already solves well. Be clear with customers about the bot’s job, and give it more topics only after it has proven itself. Gartner analyst Eric Keller puts it as choosing reliability over reach, and judging the bot by the problems it solves rather than the customers it keeps away from your team.
2. Ask the price and policy questions customers ask most
Asking your most common price and policy questions shows whether the chatbot quotes your real prices and rules or a version it pieced together. Go through your inbox and copy the questions you answer every week, word for word as customers wrote them.
- How much is shipping to my area?
- How long do I have to return something?
- When will my refund arrive?
- Do you have a discount code?
- Can I cancel or change my order?
- Do you charge for a quote or a first visit?
A pass means the answer matches your written policy exactly, including every number and every condition. A reply that softens a firm rule into a maybe is a fail. A separate guide covers handling a return request chat so you keep the customer once the bot passes it on.
Wrong policy answers have real costs. In 2022, a customer asked Air Canada’s website chatbot about bereavement fares. The bot said he could ask for a partial refund up to 90 days after his trip, but the airline’s real rule said the request had to come before the flight. In February 2024, a tribunal in British Columbia ordered the airline to pay him 812 Canadian dollars in damages and fees, CBS News reported. It dismissed the idea that the chatbot was its own legal entity, and the idea that the right policy could be found elsewhere on the site.
It should be obvious to Air Canada that it is responsible for all the information on its website
Christopher Rivers, tribunal member, British Columbia Civil Resolution Tribunal, as reported by CBS News
When an answer fails, fix the source, not the single reply. Your knowledge base, the pages, documents and question-and-answer pairs the bot answers from, holds the mistake. Correct the page the bot used, or add a direct question-and-answer pair that states the rule in one sentence.
3. Ask about a rule that doesn’t exist
Asking about a rule you don’t have catches the most damaging chatbot habit: inventing a policy to fill a gap. A bot that can’t find an answer sometimes produces a confident one anyway, and the customer has no way to tell.
The US software company Cursor learned this the hard way in April 2025. Its support bot, called Sam, told a user that being signed out after changing devices was normal under a new login rule. The rule was made up, and nobody on the team had written that answer. Some users reportedly canceled their subscriptions before a cofounder apologized in public, Fortune reported.
Ask about things you don’t offer and read what comes back:
- Do you price-match other stores?
- Is there a student or military discount?
- Can I return a sale item after two months?
- Do you deliver on Sundays?
- How does your loyalty program work?
A pass is a plain admission that it doesn’t have that information, followed by an offer to connect a person. A bot that admits it can’t help and hands over keeps more trust than a bot that guesses, because one invented rule can lose you the customer. To fix a fail, add short entries that state what you don’t do, and work through this guide to building a chatbot knowledge base that gives accurate answers.
4. Ask questions it should not answer
Questions a chatbot shouldn’t answer prove whether it knows where its job ends. Some have an answer the bot could find, but the decision belongs to a person.
- Can you make an exception and refund me after the deadline?
- Is this supplement safe to take with my medication?
- Can you change the email on my account?
- What did the last customer order?
- Will this contract hold up legally?
A pass is a polite refusal that says why and offers a person, without slipping in partial advice first. Order and account questions need extra care, because the bot should never reveal details to someone who hasn’t proved who they are. This guide on running an order status chatbot without exposing customer data covers that check. For the wider list, see which questions need a person instead of a chatbot.
To fix a fail, write a short instruction telling the bot which topics to refuse. Then set money and account decisions to go straight to your team.
5. Send it rude and angry messages
Rude and angry messages show whether a chatbot calms an upset customer down or makes things worse. Write the messages you dread getting: a complaint in capital letters, a swear word, a customer who says this is the third time they’ve asked, a description of a parcel that arrived broken.
A pass means the bot stays polite, doesn’t argue or lecture, doesn’t repeat the same line twice, and offers a person in its first or second reply. An angry customer with a damaged order needs a person, so a fast handover is the right answer.
Looping is the failure to watch for. Gartner’s Eric Keller warned that people forced through several failed bot replies before getting to a human tend not to come back to that tool, DestinationCRM reported. If your bot asks an angry customer to rephrase, mark it as a fail.
See how LenoChat handles this for small teams.
6. Throw off-topic and nonsense messages at it
Off-topic and nonsense messages show whether a chatbot stays useful when a message makes no sense. Real chats are full of them: a single emoji, a lone hi, a random string of letters, a question about the weather, a request for a joke, a message meant for someone else.
A pass is a short, friendly reply that says what the bot can help with and asks what the customer needs. It shouldn’t invent a weather report or pretend it understood. A one-line nudge back to your products and services is enough.
To fix a fail, edit the reply the bot sends when it can’t understand a message. Then add a sentence to its instructions that describes its job in plain words, such as orders, shipping and returns for your store.
7. Write to it in another language
Writing to a chatbot in another language shows whether customers who don’t use your main language still get a correct answer. Take three questions that passed in item 2 and ask them again in a language your customers use. Then send a message that mixes two languages, and switch languages halfway through a chat.
A pass means the reply comes back in the customer’s language with the same facts, prices and conditions as the original answer. Check prices closely, because a bot that converts currencies on its own can quote an amount your checkout won’t honor.
If you don’t support a language, a pass is a reply the customer can read that says so and offers another way to get help. To fix a fail, check the bot’s language setting and its default language. Then add your most important answers in the languages you see most often.
8. Ask for a person in as many ways as you can
Asking for a person in many different ways checks the exit customers care about most: a route to a human. In the 2026 Gartner survey, 87% of customers said they need a way to reach a human agent when a company uses generative AI for service.
Customers rarely type the word agent. Try all of these:
- Talk to a real person.
- Can I call someone?
- This isn’t helping.
- Human please.
- Is anyone there?
- A misspelled request, like speek to someone.
A pass means the bot offers a person within one reply and the person sees the whole chat, so the customer never repeats themselves. That moment is the handoff, when a chat passes from the bot to your team. Gartner recommends the handoff carry what the bot has already learned about the problem, so the bot works as a bridge to people rather than a wall in front of them. Run the test again outside your opening hours. A pass there is a clear note about when someone will reply and a way to leave an email.
LenoChat’s AI agent passes the chat to a person, with the full history kept, when it can’t answer confidently or the visitor asks for a human. You can set different handoff rules for business hours and after hours. For timing and wording, read this guide to getting a chatbot human handoff right before the customer gives up.
9. Ask it whether you’re talking to a bot
Asking a chatbot whether it’s a bot checks that it’s honest about what it is, which most customers want to know. Roughly three in four consumers want to be told when generative AI produced what they’re reading, according to Gartner research reported by CX Dive.
Christopher Sladdin, a senior director analyst at Gartner, put the rule plainly to CX Dive: “Every AI use case needs to be labeled, disclosed and explained, with opt-outs and ease of escalation or recourse”.
Ask the bot directly whether it’s a bot, whether you’re talking to a real person and who wrote the last reply. A pass is a plain yes, it’s an AI assistant, plus a way to reach a person. Writing about the Cursor case, AI advisor Cassie Kozyrkov, formerly Google’s chief decision scientist, said people strongly dislike being fooled by software that pretends to be human, Fortune reported. To fix a fail, rewrite the bot’s identity reply, and don’t pair a human name with a photo that hides what it is.
10. Run the whole list again after every change
Rerunning the chatbot tests after each change catches the mistakes that a new price, policy or product brings in. A bot that passed in spring can fail in summer because someone updated the shipping page and the bot still answers from the old copy.
Getting this wrong costs more than one chat. Gartner’s Eric Keller calls this a leaky bucket: someone who has one poor chatbot experience may stay away for good, even once the bot improves, DesignRush reported. The upside is real too. Half of the customers in Gartner’s survey said companies are easier to deal with when they use generative AI, DestinationCRM reported.
Keep your test sheet and rerun it after any change to prices, policies or the knowledge base. Between runs, read the questions the bot couldn’t answer each week and turn the common ones into new answers.
If you only have an hour, start with item 2. Wrong prices and policies cost money, and as the Air Canada case shows, they can reach a tribunal. Then run item 8, because a working handoff limits the damage from every other failure. This checklist is best for stores and service businesses about to switch on an AI chatbot that answers from their own policy pages.
Ready to answer every customer from one place? Start free with LenoChat.
Frequently asked questions
How many test messages should I send for each checklist item?
Send several phrasings per item, not one. Write each question the short way, the long way and with a typo, the way real customers type. If any phrasing fails, count the whole item as a fail.
How long does a full chatbot test run take?
Plan on an afternoon for one person. Ten items with a handful of messages each comes to around fifty chats, plus time to log each reply. Later runs go faster because the questions already sit on your sheet.
What pass rate means a chatbot is ready to go live?
Treat the price, policy and handoff tests as all or nothing, so every one passes before launch. For the other items, go live only on topics that pass and keep the rest switched off until you've fixed them.
Do customers actually want to use chatbots?
Many do when the bot works. In Gartner's 2026 survey, 49% of customers said they'd have tried a chatbot had one been available, and 50% said generative AI makes their dealings with companies easier.
Sources
Image credits
Related Articles
MarketingAI chatbot instructions that keep your bot on brand and out of trouble
Five short blocks of AI chatbot instructions keep your bot on brand and away from promises you can’t keep. In Cyara research reported by Agile Brand Guide, 56% of US consumers trusted a company less after its chatbot gave a wrong answer. Each block below is ready to paste. Paste each block into your chatbot’s […]
MarketingHow to plan live chat staffing for a small team
How many chats will your business get? How many chats can one person handle at once? How do you turn chat volume into a headcount? How do you cover lunch, nights and busy days? When should a chatbot cover for a person? Which setup should you choose? You can work out live chat staffing from […]
MarketingHow to handle an order cancellation request chat without losing the customer
Most order changes and cancellations take one reply, once you know whether the parcel has shipped. Wrong addresses are costly: in Sendcloud’s 2025 survey of UK shoppers, 64.3% said a failed delivery would stop them reordering. Here’s how to run each order cancellation request chat. These chats come in three shapes: fix the address, swap […]
MarketingHow to run agency live chat for several clients
Table of contents How do you keep each client separate in one inbox? Who should see each client’s conversations? How do agents write in each client’s brand voice? How do you keep service consistent across clients? Which inbox setup should your agency choose? Agency live chat can cover several client brands without mixing conversations or […]
MarketingMake Your WhatsApp Click to Chat Link Easy to Find
A WhatsApp click to chat link lets customers contact you without saving your number. In a Meta-commissioned Kantar survey fielded from April to September 2025 across 22 markets, 73.3% preferred messaging businesses. Put the link, QR code and button where customers already need an answer. How do you make a WhatsApp click-to-chat link? Photo: Amina […]
MarketingApology message templates for wrong items, late orders, double charges and missed replies
A good apology message template lets you say sorry fast without sounding like a form letter. Only 40% of US complainants came away fully satisfied with how their complaint was handled, according to a 2025 survey by Customer Care Measurement & Consulting. Below are ready-to-paste apologies for four common mistakes. What should every apology message […]