Our AI Chatbot Lied to a Customer. What Should We Have Checked Before Launch?

AI features are turning up in just about every service.

Here’s an example.

You open KakaoTalk (a South Korean messaging app, similar to WhatsApp) to catch up on a stock-investing group chat you haven’t checked in a while.
There are about 300 unread messages.

Hmm. You’d like to know what everyone’s been talking about, but that’s a lot to read. What do you do?
These days, you just tap a button.

KakaoTalk’s chat summary gives you a rundown of the conversation you missed.

What else?
Coupang (a South Korean shopping platform, similar to Amazon) is beta-testing a feature that summarizes the pros and cons at the top of its reviews tab.
On secondhand marketplaces such as Karrot (a South Korean local marketplace, similar to Facebook Marketplace), upload a product photo and AI will write the listing title and description for you.

And it doesn’t stop there. AI features keep appearing across all kinds of services.
On the accommodation-booking platform Yeogi Eottae (a South Korean travel-booking app, similar to Expedia), a search for “sea scenery” can find properties listed as having an “ocean view.”

These are major platforms in South Korea, popular enough that you’d be hard-pressed to find someone who doesn’t use any of them.
They have something in common.

None of them sells AI as its main product.
They’re messaging, shopping, secondhand marketplace, and travel apps, each adding AI features to the service people already use.

So it’s hardly surprising when someone working on a product gets this request:

“Should we add something with AI too? I hear you can hook up an API pretty quickly now.”

Hooking it up can be quick.
But is that all it takes?

An AI coding company had to apologize after its support bot invented a policy that didn’t exist.
Source: The Register, April 18, 2025

In South Korea, a government chatbot that cost ₩3.1 billion was still giving advice based on old rules more than two months after the law changed.
Source: NewDaily Economy, October 27, 2026

A company that knows AI inside out and a public agency with a multibillion-won budget both ran into trouble after launch.

What went wrong?
For this article, let’s say we run an online store and are planning to add an AI support chat to the website.
Let’s walk through the problems that could actually come up.

Let’s automate our customer support with AI

The store’s CEO was always worried about customer inquiries.
There was someone handling support, but the questions that needed careful answers had a habit of arriving on weekends.
Can I cancel this order? I want a refund, but what does the policy say about this particular case? That sort of thing.

Still, asking an employee to take a laptop home and “just answer a few support questions” over the weekend didn’t feel right either.

What to do with that weekend backlog was a recurring problem.
Then, at the weekly meeting, the CEO said:

“Let’s build an AI chatbot.
You see them everywhere now, don’t you?”

The request went to the product manager, who wrote this in the feature spec:

“When a customer sends a message in the chat, AI responds.”
“Place the AI support chat icon at the bottom right of the website.”
“Clicking it opens a chat window, where AI answers every question in real time.”

For a conventional feature, that might have been enough to avoid major problems.
Those features worked more like a kiosk.
Press the Americano button, get an Americano.
You just had to describe what the user pressed and what came out.

An AI feature is closer to a new support agent than a kiosk.
It can do the work and sound convincing while it’s at it.
But it may phrase an answer differently each time you ask the same question.
Ask something it doesn’t know, and it may answer as if it does.

From launch day onward, questions started arriving that the one-line spec couldn’t answer.

What happened right after launch

Day one, a customer asks: “Are you a real person?”

On the evening of launch day, this message appeared in the chat:

“You’re replying so fast. Are you a real person? I’d like to talk to a person.”

The bot kept the conversation going politely.
But there was no button anywhere to reach a human.

There are two questions here.
Did we tell the customer they were talking to AI?
And can they reach a person when they want to?

In South Korea, the first is now a legal issue too.
Under the AI Basic Act, which took effect on January 22, 2026, businesses that provide generative AI features directly to users must tell them in advance that AI is being used.

That can apply even if you didn’t build the model yourself but offer an AI feature within your own service.
For a chatbot whose responses stay within the service interface, that notice can be given in the terms of service or at the start of the chat.
There is currently a grace period of at least a year, during which administrative fines are deferred.

Source: Newsis, January 22, 2026

Consumers have already answered the second question.
When the Seoul Metropolitan Government surveyed 1,000 online shoppers, 39.4% complained that AI chatbots gave standardized answers unrelated to their questions.
Their most requested improvement, at 40.4%, was to offer both an AI chatbot and one-to-one support from a staff member.
They weren’t asking for chatbots to disappear. They wanted a way to reach a person when the bot couldn’t help.

Source: Seoul Metropolitan Government survey of online-shopping consumers, February 27, 2026

Even a well-built AI will get things wrong.
You need to decide where customers go when they get stuck, and what they see at night when no support agent is available.

Day three, the support lead says: “The AI said sale items were refundable”

Someone from customer support came over with a screenshot.

“Sale items aren’t refundable, but the AI told this customer everything could be refunded within 30 days.
They believed it and have already sent the item back. What do we do?”

What can you do? Accept the return, then find out which policy the AI thought it was following.

This happens even at companies that build AI.
In April 2025, developers using the AI coding tool Cursor started getting unexpectedly logged out across multiple devices.
When a user asked why, a reply arrived from someone called “Sam.”
It said, in effect, that this was expected behavior under a new policy.

Except there was no such policy.
Sam was an AI support bot, and the policy was something it had made up.
After users began talking about canceling their subscriptions, a Cursor cofounder apologized and confirmed that no such policy existed.
The company also said it would clearly label AI-generated replies in its email support.

Source: The Register, April 18, 2025

“April 2025? Isn’t that a bit old to compare with what AI can do this November?”

Fair point.
But a year and a half later, with much better models available, the problem hasn’t gone away.
At last month’s parliamentary audit in South Korea, it emerged that the Ministry of Employment and Labor’s AI labor-law chatbot, operated at a cost of about ₩3.1 billion, was describing wage-arrears compensation limits as “the final three months, up to ₩21 million.”

The current limits were “the final six months, up to ₩31.5 million.”
It had been giving answers based on the old rules for more than two months after the law changed.
A total of 173 certified labor consultants review nearly 20,000 answers a month after the fact, but users had no way to report an incorrect answer when they found one.

Sources: NewDaily Economy, October 27, 2026; materials from Representative Park Jeong’s office

With questions about refunds, prices, or stock, a wrong answer can have financial consequences.
The moment AI fills in a gap with a plausible guess instead of saying it doesn’t know, you have a problem.
Some questions therefore need a firm boundary: answer only from the policy documents, and if there’s no supporting information, don’t guess. Hand it to a person.

Day five, logistics says: “A return was opened without the customer clicking anything”

Getting enthusiastic about AI automation, the CEO connected a return-request function to the chatbot.
A customer asked only, “How much would return shipping cost?” But the AI went ahead and submitted the return.
The logistics team arranged a pickup. The customer had no idea why.

AI doesn’t just talk anymore.
It can click buttons and submit forms on someone’s behalf.
So you need to draw a line: will it only answer, or will it also take action? If it takes action, how far can it go?

For this store, you might let AI look up order history directly, require the customer to click “Submit return request” before opening a return, and allow only a human support agent to issue a refund.

One week in, the developer asks: “We changed models. Is it actually better?”

A developer suggests trying a new AI model that’s faster and cheaper.

“But how do we tell whether it’s better after we switch?”

The spec said only “AI responds,” so nobody had an answer.

It’s hard to judge the quality of AI answers by feel.
In Stack Overflow’s 2026 Developer Survey, published last month with more than 30,000 respondents from 169 countries, the most common response about trusting AI was “when I can verify the results myself,” at 48%.
Only 7% said they would trust AI with important decisions.

Source: Stack Overflow 2026 Developer Survey, October 6, 2026

If even developers who use AI every day trust it only when they can check the results, you need criteria for checking them.
Just as a new support agent is assessed during probation, AI needs a scoring standard too.

An answer that’s completely wrong is easy to spot. An answer that’s almost right can slip through.
Just as a new support agent is assessed during probation, AI needs a scoring standard too.
Collect common questions and their correct answers in advance, then set a minimum number it must get right before launch.
Run the same questions again whenever you change the model or its instructions, known as the prompt.

One month later, the CEO asks: “Why is this month’s AI bill so high?”

The CEO was looking at the month-end bill.
It turned out that a few customers had gone back and forth with the chatbot dozens of times.
The longer the conversations got, the more they cost.

With conventional features, one more button click didn’t noticeably increase the cost.
A feature using an external AI model is different.

Every response carries a usage charge, and longer conversations cost more.
You need numbers: how many questions can one customer ask per day, and when do you cut off a conversation that keeps going?

If you charge users for an AI feature, deciding how to show them their limit is part of planning it.

The store doesn’t charge customers separately for support, so the company pays the bill.
But it still has to decide what customers see when they reach the limit.
If the bot simply stops answering without explanation, it just looks broken.

What if we rewrote the spec like this?

After those five people asked their questions, the store team’s spec grew from one line into five fields.
The left column names each field. The right shows what the team actually wrote for this chatbot.

Other AI features face the same questions

The chat summaries, review summaries, listing writers, and natural-language search features we saw at the start are no different.
If you’re adding something similar to your service, here are a few questions to ask before launch.

The features differ, but the fields for recording the answers are the same:
acceptance criteria, when to stop, where to turn when something goes wrong, what AI can do on someone’s behalf, and costs and limits.

How to get these questions before launch

Everything the store team went through had one thing in common.
The questions all came after launch.
The customer, support lead, logistics team, developer, and CEO each reopened the same spec from their own point of view.

Manyfast’s Review feature gives you a way to get those questions before launch.
It checks your PRD and feature specifications from development, business, UX, design, QA, and security perspectives, linking each comment to its reasoning and the relevant items.
You decide whether to dismiss or resolve each comment, then work through the changes with Manny in chat.

Once you’ve decided on the answers, there are already places to record them in your plan.

Put the passing standard in the requirement’s Acceptance Criteria
Each requirement has a checklist for the conditions it must meet.
Record a launch threshold such as “at least 48 correct answers out of 50 common questions,” and you’ll know exactly what to check again when you change models.

Use Dev-Readiness Slots for boundaries, handoffs, and permitted actions
In Business Rules, record what the AI must not answer. In Display, put the AI disclosure and the button for contacting a support agent.
In Exceptions, define when to hand the conversation to a person. In Permissions, separate who can submit a return from who can issue a refund.

Put cost limits in Product Policies
Rules that apply across several features, such as a daily question limit, can be grouped into a separate policy.

Of course, Manyfast doesn’t grade your chatbot’s answers for you.
Testing the actual responses to those 50 questions is still work for the development stage.
But when the criteria are recorded in the plan, AI coding tools such as Cursor or Claude Code, connected through MCP, can read and work from the same standard.

A final thought

Chat summaries, review summaries, and AI-written listings are no longer unusual features.
Adding AI will only get easier.

That leaves one choice.
Do you wait for the difficult questions to arrive after launch, or answer them in the spec beforehand?

No company tells a new support agent on day one, “Just handle things however you think best,” and leaves it there.
You explain what a good answer looks like, who to ask when they don’t know, and how much they’re allowed to do.
An AI feature needs that working manual too.

If you want to record the decisions that matter and carry the same criteria through to development, start with Manyfast.