Everyone who sells an AI assistant tells you how many messages it can answer. Very few tell you which messages it should never answer. Yet an assistant's value is measured not in the majority of cases where it goes well, but in how it handles the few moments where it can go badly.
This article is not meant to scare you. It is meant to help you widen the work you hand to AI with confidence. First we draw the boundary, then we explain how to enforce it.
This risk is not theoretical
In February 2024, the Civil Resolution Tribunal in British Columbia, Canada, held Air Canada responsible for wrong information its website chatbot gave a passenger. The assistant had said a bereavement-fare refund could be claimed after the flight; the airline's own page said the opposite. The airline argued the chatbot was effectively a separate legal entity responsible for its own words. The tribunal rejected that: the chatbot is part of the company's website.
In April 2025, the support assistant of a software company (Cursor) told a user there was a one-device-per-subscription rule. There was no such rule; the assistant made it up. Users took it as an official statement, some publicly announced they were cancelling, and the company had to apologise.
The common thread: the assistant was not malicious, it was confident. When an AI does not know that it does not know, it states an invented fact in the same tone as a true one. So the real question is not whether an assistant makes mistakes (it does). It is who pays when it does, and how far a mistake can travel.
Six critical processes that belong to a human
- Complaints, anger and crisis moments. An upset customer first needs to feel heard, not informed; a canned apology makes anger worse. The moment the tone changes (capital letters, stacked exclamation marks, words like “outrageous”), the conversation should move to a person. When you have a widespread problem (a late delivery, an outage), every incoming message is a complaint; that day you need one clear public announcement, not automated replies.
- Refund, compensation and exception decisions. Any decision that pays money or bends a rule belongs to a person. An assistant can explain the policy; it cannot recognise or grant an exception.
- A price, discount or delivery date that is not in your catalogue. “We can do it for you at this price” can be treated as a binding promise, as in the Air Canada case. The assistant should only state information that has a source.
- Questions with legal, medical or financial consequences. An answer to “will this cover me”, “am I entitled” or “is this legal” leads a customer to make a decision. General information is fine; the decision that carries consequences should come from an authorised person.
- ID, payment and sensitive personal data. When a customer starts typing an ID number, card details or health information into the chat, the conversation should stop and the customer should be directed to a secure channel.
- Misunderstanding loops and “I want to talk to a person”. A customer who asks the same thing twice in different words without an answer will not ask a third time; they leave. When someone asks for a human, the handover should be unconditional.
Automated decisions and the right to object
In Turkey, Article 11 of the data protection law (KVKK) gives a person the right to object to a result against them that arises solely from the automated analysis of their data. This is not legal advice and you should consult a lawyer about your own situation; but the practical takeaway is simple: a decision that goes against the customer (rejecting an application, refusing a request) should not rest on the assistant's output alone, and a person should approve it.
How to set handover thresholds
A threshold is a written rule, not a feeling. The clearer the rule, the less often the assistant gets it wrong, and the better you sleep. Five triggers you can adapt to your own business:
- Word and tone triggers: words like “complaint”, “refund”, “lawyer”, “fraud”, “outrageous” and signs of anger. Add the words of your own sector.
- Topic triggers: any conversation that mentions refunds, compensation, contracts, warranties, discounts, or ID and card details.
- A loop counter: if the assistant has said twice that it did not understand the same thing, it does not try a third time; it hands over.
- A value threshold: if an order or quote exceeds a certain amount, a person takes the final step. You set the amount.
- An explicit request: “I want to speak to someone in charge” or similar means handover with no further questions.
The handover itself must be done well
A handover should not mean the customer has to start from the beginning. In a system that hands over with a conversation summary, your team does not start from zero and the customer is not left waiting.
If there is nobody to take over (night, weekend, holiday), say so honestly and give a concrete time: “Someone in charge will write to you after 9:00 tomorrow.” “We will get back to you as soon as possible” gives no information; a time manages expectations.
Saying “I don't know” beats making it up
An assistant's most valuable skill is not knowing everything, it is saying what it does not know. Sentences like these do not lose sales; an invented price does.
- “I don't have reliable information on this. To avoid misleading you, I am passing you to someone in charge; you will hear back after 9:00 tomorrow.”
- “I'm not sure I understood correctly: are you asking about the price or the delivery time?”
- “I cannot confirm this price right now. A quote has to come from someone in charge; may I take your contact details?”
This has one prerequisite: the assistant produces its answers only from the source you give it (catalogue, price list, policy text) and says “I don't know” when the source does not cover it. That rule should not be negotiable.
Before you go live: the hard-question test
Before you open the assistant to customers, write twenty “hard” messages yourself and read every answer. The aim is not to see how clever the assistant is, but to find where it invents and where it fails to hand over.
- An angry complaint, in capital letters but without insults.
- The price of a product that is not in the catalogue at all.
- A haggling message such as “you gave my friend this discount, give it to me too”.
- A refund request, both one that fits the policy and one that does not.
- A customer typing an ID number into the chat.
- A customer asking the same thing three times in three different ways.
- A direct “I want to talk to a human”.
Do not stop the test after launch: in the first month, read twenty random real conversations every week. Real customers find ways to trip an assistant that you would never think of.
Summary
- An assistant's value is measured in how it handles the moments where it can go wrong.
- A company is responsible for everything its assistant says; it cannot be defended as a separate legal entity.
- Complaints, money decisions, promises outside the catalogue, sensitive consequences, sensitive data and loops: these six processes belong to a person.
- Set handover thresholds as written rules, hand over with a summary, and give a time when nobody is available.
- An assistant that can say “I don't know” builds trust; run the hard-question test before launch.