What Happens When AI Gets Something Wrong in Your Business?

What happens when an AI system gets something wrong in your business?

An AI system that gets something wrong does what a new employee does wrong, except faster and more consistently, so the containment gets built rather than hoped for.

Three things decide what the error costs. Whether it reaches a customer, whether anybody notices, and whether the same mistake repeats a thousand times before somebody does. A person who miscodes an invoice miscodes one invoice. A rule that miscodes invoices miscodes every invoice of that shape until it is changed, which sounds worse and is actually easier to find, because systematic errors show up in a report and human errors hide in the variance.

The cases that cause real damage share a shape: the output went straight to somebody outside the company, and nothing in between was checking. A quote sent without review. An answer given to a customer as policy. A filing submitted without anyone reading the citations. In each one the fix is not a better model, it is a checkpoint that should have existed.

The cases that cause no damage share a different shape. The system drafts, a person approves, and the error is caught in the two seconds it takes to read a line that looks wrong. The work still disappears, because reading and approving is a fraction of doing, and the failure mode becomes a slightly annoying draft rather than an incident.

So the question to ask about any automation is not whether it will be wrong. It will. The question is what happens on the day it is, and that answer is designed in, or it is absent.

How often does AI actually get things wrong?

AI gets things wrong often enough to design around, and how often depends almost entirely on whether the system is answering from a document in front of it or from memory.

When a model is given a document and asked to summarise it, the error rates are low and falling. Vectara's hallucination leaderboard, which scores models on factual consistency against source documents with its HHEM-2.3 evaluator and was last updated on 22 September 2026, reports hallucination rates from 1.8% for the best model in its table to 24.2% at the other end of the tested field, across a corpus of more than 7,700 articles.

When a model answers from memory, the numbers change completely. Stanford RegLab's study Large Legal Fictions, published in the Journal of Legal Analysis in 2024, put more than 800,000 verifiable questions about federal court cases to general-purpose models and found hallucination rates between 58% and 88%, from GPT-4 at the low end to Llama 2 at the high one. The same study found the models were poor at predicting their own errors and often accepted a user's false premise rather than correcting it.

Two practical conclusions follow. Build systems that work from your records rather than from the model's general knowledge, because that is the difference between the two ranges above. And treat confident tone as worthless evidence, since a model that is wrong sounds exactly like a model that is right, so the checking has to be structural rather than a person's impression that an answer looked fine.

Error rates also vary by task in ways worth measuring on your own work. Reading a structured invoice is not the same job as interpreting a handwritten note, and the only honest way to know your rate is to run the system against real cases before it runs anything live.

How do you find out an AI system got something wrong?

You do not find out from an error message, because an AI system that is wrong usually reports success. The job finished, the record was written, the reply went out, and the software logged a normal result. Error monitoring watches for the opposite: a crash, a failed request, a stack trace. A confidently wrong invoice code produces none of those, so detection has to be designed rather than assumed.

The system will not tell you either, because models are poor judges of their own accuracy. Stanford RegLab's Large Legal Fictions found the models it tested were bad at predicting their own errors and frequently accepted a false premise in the question instead of correcting it. So confidence in the output is not a signal, and neither is the absence of a red light on a dashboard.

What goes undetected is expensive in a way that rarely gets attributed to the system. BetterUp Labs and Stanford's Social Media Lab surveyed 1,150 US full-time employees for Harvard Business Review in September 2025 and found 40% had received AI-generated work in the previous month that looked finished and did not advance the task. Each instance took an average of one hour and 56 minutes to deal with, which the researchers costed at roughly $186 per worker per month. That time gets recorded as somebody being busy, never as an error rate.

Four things actually surface a wrong output, and a built system should have all four.

How the error surfaces	What it catches	What it costs you
A person approving before it leaves	Anything reaching a customer or the accounts, caught in the seconds it takes to read a line that looks wrong	A click per item, and it is the only one that catches an error before it lands
A rule that checks the output against something else	Totals that do not match the purchase order, dates in the past, a supplier that does not exist, a value outside the range this customer has ever had	Build time, once, and it runs on every item after that
A sample read against what a person would have done	Drift: the layout changed, the model changed, the work changed, and the rules no longer fit	Twenty minutes a week on a handful of items
The error summary somebody reads	Patterns no single item reveals, such as one document type failing all month or escalations quietly climbing	A five minute read, and it is the one most often skipped

The last row is the one most often specified badly, and an error summary nobody can act on is the same as no summary. What belongs in one is set out in the next section.

None of it works retroactively. If the system did not record what it saw and what it did at the time, a wrong output three weeks ago cannot be investigated, only argued about. Logging is what makes the rest possible, and it is decided before the build rather than added after the first incident.

What should an AI system's error summary contain?

An AI system's error summary should contain six lines: the volume it handled, what it escalated and why, what a person corrected after the system had marked it finished, which checks fired, the items the system itself recorded as uncertain, and what each of those did since the last summary. Group them by cause rather than listing them by timestamp. A list by timestamp is a log, and a grouping by cause is a decision about what to fix next.

Most monitoring tools report the opposite set of things, which is why a business running an AI workflow often has a dashboard and no idea how often the system is wrong. Uptime, requests served and response times all describe whether the software ran. None of them describes whether it was right, because a confidently wrong output is a successful request.

The line	What it tells you	What to do when it moves
Volume handled	Whether the work is still arriving at all	A drop is usually a broken connection upstream rather than a quiet month. Check the source before anything else
Escalations, with the reason	Where the boundary of the build actually sits, as opposed to where it was drawn	A reason that keeps repeating is a rule waiting to be written. That is the work that raises the share the system carries
Corrected after completion	What got past every check. This is the only line that counts real errors rather than caught ones	A rising count means the checking is aimed at the wrong thing, not that the model got worse
Checks that fired	Which rules are earning their place	A rule that never fires is redundant or broken, and the only way to tell is to test it deliberately
Items flagged uncertain	Whether the system knows its own limits	Uncertain items that turn out to be right mean the threshold is set too cautiously and a person is reading work they did not need to see
Change since last time	Drift, which is the failure nothing else catches	One bad month is weather. Two in the same direction is the layout, the model or the work having changed

Read the escalation line against a published benchmark rather than against your hopes. Invoice processing is the one workflow with one: Ardent Partners' Accounts Payable Metrics That Matter in 2025 puts the invoice exception rate at 14 percent on average and 9.0 percent among best-in-class operations. A system escalating one item in ten on structured, repetitive work is behaving normally. A system escalating nothing at all is not a better system, it is one whose uncertainty threshold has been turned off, and the errors are still there with nobody looking at them.

Keep the whole thing to a five minute read, for a practical reason rather than a presentational one. A check that takes an hour is a check nobody does by the third month, and an unread summary fails in exactly the way the system it monitors fails: quietly, with everything appearing to work. On the systems we build, that report goes out on a schedule, it comes from us rather than from somebody in your office remembering to look, and the month a number moves the wrong way is the month we are already fixing it.

Can a monitoring tool write the AI error summary for you?

A monitoring tool can write part of the error summary, and the part it cannot write is the part that matters to a business. Several of them now generate one with AI, and they do it well. What every one of them summarises is a failure the software already noticed. The errors that hurt a business running an AI workflow are the ones it did not, because a wrong output is a successful request.

Prefect's error summaries, announced on 27 September 2023, read the logs and produce "automated summaries of logs from your failed flow runs", and they are generated when a flow run fails. HyperTest's AI Summaries are triggered from a failed request, explain the root cause, propose a fix and estimate how likely the error is to be a false positive. Both are genuinely useful and both start from the same precondition: something has to have gone wrong in a way the system recorded as going wrong.

An AI workflow breaks the precondition. PostHog's guide to AI error patterns puts it directly: of the failures it lists, including a model ignoring an instruction and a model answering from the wrong document, "every one of those can return 200 with a plausible-looking body". Its worked case is a summariser that had been dropping the last section of every long document for a fortnight. Every generation call succeeded; one section had stopped on a token limit, so, as the guide puts it, "nothing threw, so the only evidence you have is one word in a stop reason". Of conventional error tracking it concludes that these tools "count the failures your code noticed. They say nothing about the ones it didn't."

What it covers	A monitoring tool's AI summary	The summary a business needs
The crash, the timeout, the rotated credential	Covered, and faster than reading the logs	Covered by the same tool. No reason to build this twice
The output that was confidently wrong and shipped	Invisible. The request succeeded	The line that counts what a person corrected after the system marked it finished
The item the system should have escalated and did not	Invisible. No rule fired	Escalations with their reasons, read against the cases that needed one
Slow drift as documents, models or the work change	Invisible until something breaks outright	The weekly sample read against what a person would have done

So buy the tooling for the first row and design the other three. The practical arrangement is to let your infrastructure monitoring report on whether the system ran, and to have the workflow itself record a decision log: what it saw, what it decided, what it was unsure about, and what a person changed afterwards. Those four fields are what turn the six lines in the section above into something a tool can assemble for you. Without them no amount of AI summarising will find an error, because the evidence was never written down.

Who is liable when an AI tool gives a customer the wrong information?

Liability for an AI tool's wrong answer sits with the business that put the tool in front of the customer, and a Canadian tribunal has already said so in writing.

In February 2024 the British Columbia Civil Resolution Tribunal decided Moffatt v Air Canada, 2024 BCCRT 149, in which the airline's website chatbot told a passenger he could apply for a bereavement fare retroactively. Air Canada's own policy said otherwise. The tribunal rejected the argument that the chatbot was a separate entity the airline was not responsible for, found negligent misrepresentation, and awarded the passenger $650.88 in damages, as reported by the American Bar Association in Business Law Today.

The sum is small and the principle is not. A tool speaking on your behalf is you, and "the AI said it" is not a defence anybody has succeeded with. That applies to a phone agent quoting a price, a chatbot describing a warranty, and an email assistant confirming a delivery date.

The professional versions carry heavier consequences. Courts have sanctioned filings containing citations that no model could support, and the volume has been large enough that researchers maintain public trackers of the decisions, among them Damien Charlotin's AI hallucination cases database. For regulated work the practical rule has not changed since before AI existed: the person whose name is on the document is accountable for what is in it.

None of that argues against using these systems in customer-facing work. It argues for deciding, in advance and in writing, what the system is allowed to assert. A phone agent that books appointments and takes details is a narrow promise. A phone agent that answers any policy question a caller invents is an open one. We design for the first shape, which is also why the industry boards mark some capabilities as early rather than selling them.

What happens to a business that overstates what its AI can do?

A business that overstates what its AI can do is exposed on the claim rather than on the error, and in the United States that exposure is already being enforced. The regulator has not needed a new law to do it.

The Federal Trade Commission announced five law enforcement actions on 25 September 2024 under a sweep it called Operation AI Comply. Its then chair, Lina M. Khan, put the position in one line: "there is no AI exemption from the laws on the books." The cases were about what companies said their AI could do rather than about a model making a mistake.

One of the five is the one to read if you are buying an AI service rather than selling one. DoNotPay sold a subscription it marketed as "the world's first robot lawyer". The FTC's complaint alleged that the company did not test whether its chatbot's output was equal to the level of a human lawyer, and that it neither hired nor retained any attorneys. According to the Commission's own announcement, the proposed order settling the charges required $193,000 and a notice to subscribers from 2021 to 2023. That was a proposed order open to public comment rather than a court's finding, and three of the five actions in the sweep were complaints filed in federal court, which are allegations until they are proved.

Read that allegation from the buyer's side and it describes a test nobody ran. Whether a system performs at the level the marketing implies is answerable, by running it against real work, which is why the first question in the vendor section below is the measured error rate and how it was measured. The same exposure transfers to you the day you repeat a vendor's claim to your own customers, because at that point it is your claim about your own service.

So two things belong in writing before a system goes live: what it is allowed to assert to a customer, and what your marketing is allowed to say about it. Keep the second inside what the first was tested on. The cases above all turn on the distance between those two.

How do you design an AI automation so mistakes get caught?

Design an AI automation around three tiers of permission, decided before anything is built, and most error becomes an inconvenience rather than an incident.

Tier	What the system does	Where it fits
Runs alone	Acts without a person in the loop, with a log and a report afterwards	Reversible work with a cheap failure: sorting, tagging, routing, drafting internal notes, chasing for a reply
Drafts for approval	Prepares the whole output and waits for one click	Anything reaching a customer or the accounts: quotes, invoices coded to the ledger, replies that state policy
Flags and stops	Recognises it is outside what it was built for and hands the item to a named person	Exceptions, anything involving money above a threshold, complaints, legal or clinical content

The third tier is the one people underestimate. A system that knows what it does not know is worth more than a system with a slightly lower error rate, because an escalated item costs a minute and a confidently wrong item can cost a customer. Building that means defining the boundary explicitly, by document type, by value, by whether the record already exists in your system.

Two more things belong in the design. Every automated action leaves a record of what it saw and what it did, so a wrong output can be traced rather than argued about. And every system gets run against real historical cases before it goes live, with the outputs compared against what your people actually did, which is where the missing rules surface.

Then there is the boring half that keeps it working: somebody watching, fixing and adjusting as your business changes. Systems we build stay on our infrastructure and we run them from there, so a supplier changing an invoice layout is our problem rather than something that surfaces as a wrong number three weeks later. The security side of the same question, who can see what and where the data sits, is covered in is AI automation safe with my business data.

Which decisions should still have a person's name on them?

Decisions with legal, clinical, financial or relationship consequences should keep a person's name on them, and the line is easier to draw than most vendors admit.

Four tests decide it. Is the output reversible, and how expensive is reversing it? Does it reach somebody outside the company unread? Does a regulator, an insurer or a professional body expect a named human to have judged it? And would you want to explain the decision in a room, in which case somebody had better have made it.

By those tests, tax positions, clinical advice, legal filings, credit decisions, disciplinary matters and anything that sets a price for a specific customer keep a signature. Routing, drafting, summarising, scheduling, chasing, reconciling and extracting do not. Most of the hours in a small business are in the second list, so capacity goes up without anybody losing a decision they actually wanted to make.

Professional firms have the most to gain and the sharpest boundary, because the work being automated is client work rather than back office. Our legal and accounting board sets out what that looks like in practice, with the parts we think are still early marked as early.

Clinical settings draw the line in a different place again, since the constraint is patient safety and record handling rather than professional liability alone. The dental and medical board covers where the boundary sits there.

Write the boundary into the plan rather than leaving it to the day it matters. A roadmap that does not say what stays human has not been thought through, which is one of the tests in what an AI roadmap should contain.

What should you ask a vendor about what happens when it is wrong?

Ask a vendor six questions about error, and the answers tell you more about the product than any demo.

What is the measured error rate on work like mine, and how was it measured? A vendor who has run their system against a client's historical cases can answer. A vendor quoting a benchmark from a model provider is describing somebody else's product.

What does the system do when it is unsure? The answer should be a named behaviour, escalate to a person, leave the field blank, refuse the item, and not a claim that it is rarely unsure.

Who sees the error first, and how? A daily report, an alert, a queue somebody works through. If the answer is that the customer sees it first, that is the design.

What is logged? You want to know what the system saw and what it did, for every action, retrievable later without a support ticket.

Who fixes it, and inside what time? Operation is where the ongoing relationship earns its keep, and the answer should be in the contract rather than in an email.

What would you refuse to automate here? A vendor with no answer has not looked at your work carefully enough to have an opinion. We publish ours by industry, and the maturity labels on the boards exist so that conversation happens before a build rather than after one.

Frequently asked questions

How much does it cost to add proper checking to an AI automation?

Checking is a design decision rather than a line item, and the cost shows up as scope: defining the boundary, building the escalation path, logging every action and running the system against historical cases before launch. A build with those things in it costs more than one without and considerably less than one incident with a customer. What drives the number is how many exception paths the process really has, which is why the scoping conversation asks what happens when the input is wrong rather than only what happens when it is right.

How do you stop an AI system from making things up?

Give it the source and constrain the job. A system that answers from your documents and your records is measurably more reliable than one answering from general knowledge: Vectara's leaderboard, updated 22 September 2026, puts document-grounded error rates between 1.8% and 24.2% across the models it tests, while Stanford RegLab's 2024 study of models answering legal questions from memory found 58% to 88%. Add a defined behaviour for uncertainty, log everything, and keep a person on the outputs that reach customers.

Is a human review step not just doing the work twice?

No, because reading is a fraction of producing. Approving a drafted quote takes seconds against the minutes or hours of assembling one, and the approval step is usually temporary for low-risk work: it runs for a few weeks while the outputs are compared with what your people would have done, then the tier changes for the categories that have proved out. High-risk categories keep the signature permanently, which is a deliberate choice rather than a failure of the system.

Can my business be held liable for what an AI chatbot tells a customer?

Yes, on the evidence so far. The British Columbia Civil Resolution Tribunal held Air Canada liable in February 2024 for a bereavement fare policy its website chatbot described incorrectly, rejecting the argument that the chatbot was a separate entity, and awarded the customer $650.88 in Moffatt v Air Canada, 2024 BCCRT 149, as reported by the American Bar Association. The practical protection is narrowing what the system is allowed to assert, keeping its answers tied to your published information, and routing anything outside that to a person.

How is an AI error summary different from the error log our software already produces?

An error log records what the software reported. An error summary records what the work actually did, which is a different list, because an AI system that is wrong reports success. The log will show a clean run. The summary shows the volume handled, what was escalated and why, what a person had to correct after the system marked it finished, which checks fired, and what each of those did since last time. Grouped by cause rather than by timestamp, so it reads in five minutes and points at what to fix next rather than at what happened on Tuesday.

Can we just buy a tool that writes the error summary for us?

For the failures your software noticed, yes, and the tools are good at it. Prefect generates summaries of the logs from failed runs and HyperTest explains the root cause of a failed request and rates how likely it is to be a false positive. Both start from something having failed in a way the system recorded. A confidently wrong output did not fail, so none of these tools can see it, which is why PostHog's guide to AI error patterns says conventional error tracking counts only the failures your code noticed and says nothing about the rest. Buy the tooling for the crashes and design the other three lines into the workflow, starting with a record of what the system decided and what a person changed afterwards.

What happens if an automation breaks and nobody notices?

That is the failure mode worth designing against, and it is why a report matters more than a dashboard. Every system we build reports what it handled, what it escalated and what it produced, so a silent stop shows up as an absence in a number somebody already reads. Systems live on our infrastructure and we operate them, which means monitoring is ours rather than a task that depends on somebody in your office noticing that the Tuesday emails stopped.