What Happens When AI Gets Something Wrong in Your Business?

What happens when an AI system gets something wrong in your business?

An AI system that gets something wrong does what a new employee does wrong, except faster and more consistently, so the containment gets built rather than hoped for.

Three things decide what the error costs. Whether it reaches a customer, whether anybody notices, and whether the same mistake repeats a thousand times before somebody does. A person who miscodes an invoice miscodes one invoice. A rule that miscodes invoices miscodes every invoice of that shape until it is changed, which sounds worse and is actually easier to find, because systematic errors show up in a report and human errors hide in the variance.

The cases that cause real damage share a shape: the output went straight to somebody outside the company, and nothing in between was checking. A quote sent without review. An answer given to a customer as policy. A filing submitted without anyone reading the citations. In each one the fix is not a better model, it is a checkpoint that should have existed.

The cases that cause no damage share a different shape. The system drafts, a person approves, and the error is caught in the two seconds it takes to read a line that looks wrong. The work still disappears, because reading and approving is a fraction of doing, and the failure mode becomes a slightly annoying draft rather than an incident.

So the question to ask about any automation is not whether it will be wrong. It will. The question is what happens on the day it is, and that answer is designed in, or it is absent.

How often does AI actually get things wrong?

AI gets things wrong often enough to design around, and how often depends almost entirely on whether the system is answering from a document in front of it or from memory.

When a model is given a document and asked to summarise it, the error rates are low and falling. Vectara's hallucination leaderboard, which scores models on factual consistency against source documents with its HHEM-2.3 evaluator and was last updated on 22 September 2026, reports hallucination rates from 1.8% for the best model in its table to 24.2% at the other end of the tested field, across a corpus of more than 7,700 articles.

When a model answers from memory, the numbers change completely. Stanford RegLab's study Large Legal Fictions, published in the Journal of Legal Analysis in 2024, put more than 800,000 verifiable questions about federal court cases to general-purpose models and found hallucination rates between 58% and 88%, from GPT-4 at the low end to Llama 2 at the high one. The same study found the models were poor at predicting their own errors and often accepted a user's false premise rather than correcting it.

Two practical conclusions follow. Build systems that work from your records rather than from the model's general knowledge, because that is the difference between the two ranges above. And treat confident tone as worthless evidence, since a model that is wrong sounds exactly like a model that is right, so the checking has to be structural rather than a person's impression that an answer looked fine.

Error rates also vary by task in ways worth measuring on your own work. Reading a structured invoice is not the same job as interpreting a handwritten note, and the only honest way to know your rate is to run the system against real cases before it runs anything live.

Who is liable when an AI tool gives a customer the wrong information?

Liability for an AI tool's wrong answer sits with the business that put the tool in front of the customer, and a Canadian tribunal has already said so in writing.

In February 2024 the British Columbia Civil Resolution Tribunal decided Moffatt v Air Canada, 2024 BCCRT 149, in which the airline's website chatbot told a passenger he could apply for a bereavement fare retroactively. Air Canada's own policy said otherwise. The tribunal rejected the argument that the chatbot was a separate entity the airline was not responsible for, found negligent misrepresentation, and awarded the passenger $650.88 in damages, as reported by the American Bar Association in Business Law Today.

The sum is small and the principle is not. A tool speaking on your behalf is you, and "the AI said it" is not a defence anybody has succeeded with. That applies to a phone agent quoting a price, a chatbot describing a warranty, and an email assistant confirming a delivery date.

The professional versions carry heavier consequences. Courts have sanctioned filings containing citations that no model could support, and the volume has been large enough that researchers maintain public trackers of the decisions, among them Damien Charlotin's AI hallucination cases database. For regulated work the practical rule has not changed since before AI existed: the person whose name is on the document is accountable for what is in it.

None of that argues against using these systems in customer-facing work. It argues for deciding, in advance and in writing, what the system is allowed to assert. A phone agent that books appointments and takes details is a narrow promise. A phone agent that answers any policy question a caller invents is an open one. We design for the first shape, which is also why the industry boards mark some capabilities as early rather than selling them.

How do you design an AI automation so mistakes get caught?

Design an AI automation around three tiers of permission, decided before anything is built, and most error becomes an inconvenience rather than an incident.

Tier	What the system does	Where it fits
Runs alone	Acts without a person in the loop, with a log and a report afterwards	Reversible work with a cheap failure: sorting, tagging, routing, drafting internal notes, chasing for a reply
Drafts for approval	Prepares the whole output and waits for one click	Anything reaching a customer or the accounts: quotes, invoices coded to the ledger, replies that state policy
Flags and stops	Recognises it is outside what it was built for and hands the item to a named person	Exceptions, anything involving money above a threshold, complaints, legal or clinical content

The third tier is the one people underestimate. A system that knows what it does not know is worth more than a system with a slightly lower error rate, because an escalated item costs a minute and a confidently wrong item can cost a customer. Building that means defining the boundary explicitly, by document type, by value, by whether the record already exists in your system.

Two more things belong in the design. Every automated action leaves a record of what it saw and what it did, so a wrong output can be traced rather than argued about. And every system gets run against real historical cases before it goes live, with the outputs compared against what your people actually did, which is where the missing rules surface.

Then there is the boring half that keeps it working: somebody watching, fixing and adjusting as your business changes. Systems we build stay on our infrastructure and we run them from there, so a supplier changing an invoice layout is our problem rather than something that surfaces as a wrong number three weeks later. The security side of the same question, who can see what and where the data sits, is covered in is AI automation safe with my business data.

Which decisions should still have a person's name on them?

Decisions with legal, clinical, financial or relationship consequences should keep a person's name on them, and the line is easier to draw than most vendors admit.

Four tests decide it. Is the output reversible, and how expensive is reversing it? Does it reach somebody outside the company unread? Does a regulator, an insurer or a professional body expect a named human to have judged it? And would you want to explain the decision in a room, in which case somebody had better have made it.

By those tests, tax positions, clinical advice, legal filings, credit decisions, disciplinary matters and anything that sets a price for a specific customer keep a signature. Routing, drafting, summarising, scheduling, chasing, reconciling and extracting do not. Most of the hours in a small business are in the second list, so capacity goes up without anybody losing a decision they actually wanted to make.

Professional firms have the most to gain and the sharpest boundary, because the work being automated is client work rather than back office. Our legal and accounting board sets out what that looks like in practice, with the parts we think are still early marked as early.

Clinical settings draw the line in a different place again, since the constraint is patient safety and record handling rather than professional liability alone. The dental and medical board covers where the boundary sits there.

Write the boundary into the plan rather than leaving it to the day it matters. A roadmap that does not say what stays human has not been thought through, which is one of the tests in what an AI roadmap should contain.

What should you ask a vendor about what happens when it is wrong?

Ask a vendor six questions about error, and the answers tell you more about the product than any demo.

What is the measured error rate on work like mine, and how was it measured? A vendor who has run their system against a client's historical cases can answer. A vendor quoting a benchmark from a model provider is describing somebody else's product.

What does the system do when it is unsure? The answer should be a named behaviour, escalate to a person, leave the field blank, refuse the item, and not a claim that it is rarely unsure.

Who sees the error first, and how? A daily report, an alert, a queue somebody works through. If the answer is that the customer sees it first, that is the design.

What is logged? You want to know what the system saw and what it did, for every action, retrievable later without a support ticket.

Who fixes it, and inside what time? Operation is where the ongoing relationship earns its keep, and the answer should be in the contract rather than in an email.

What would you refuse to automate here? A vendor with no answer has not looked at your work carefully enough to have an opinion. We publish ours by industry, and the maturity labels on the boards exist so that conversation happens before a build rather than after one.

Frequently asked questions

How much does it cost to add proper checking to an AI automation?

Checking is a design decision rather than a line item, and the cost shows up as scope: defining the boundary, building the escalation path, logging every action and running the system against historical cases before launch. A build with those things in it costs more than one without and considerably less than one incident with a customer. What drives the number is how many exception paths the process really has, which is why the scoping conversation asks what happens when the input is wrong rather than only what happens when it is right.

How do you stop an AI system from making things up?

Give it the source and constrain the job. A system that answers from your documents and your records is measurably more reliable than one answering from general knowledge: Vectara's leaderboard, updated 22 September 2026, puts document-grounded error rates between 1.8% and 24.2% across the models it tests, while Stanford RegLab's 2024 study of models answering legal questions from memory found 58% to 88%. Add a defined behaviour for uncertainty, log everything, and keep a person on the outputs that reach customers.

Is a human review step not just doing the work twice?

No, because reading is a fraction of producing. Approving a drafted quote takes seconds against the minutes or hours of assembling one, and the approval step is usually temporary for low-risk work: it runs for a few weeks while the outputs are compared with what your people would have done, then the tier changes for the categories that have proved out. High-risk categories keep the signature permanently, which is a deliberate choice rather than a failure of the system.

Can my business be held liable for what an AI chatbot tells a customer?

Yes, on the evidence so far. The British Columbia Civil Resolution Tribunal held Air Canada liable in February 2024 for a bereavement fare policy its website chatbot described incorrectly, rejecting the argument that the chatbot was a separate entity, and awarded the customer $650.88 in Moffatt v Air Canada, 2024 BCCRT 149, as reported by the American Bar Association. The practical protection is narrowing what the system is allowed to assert, keeping its answers tied to your published information, and routing anything outside that to a person.

What happens if an automation breaks and nobody notices?

That is the failure mode worth designing against, and it is why a report matters more than a dashboard. Every system we build reports what it handled, what it escalated and what it produced, so a silent stop shows up as an absence in a number somebody already reads. Systems live on our infrastructure and we operate them, which means monitoring is ours rather than a task that depends on somebody in your office noticing that the Tuesday emails stopped.