AI sometimes makes things up. Can my business rely on it?

What is an AI hallucination?

It's a wrong answer presented as fact. The model writes a made-up reference with the same confidence as a real one, and nothing in the tone warns you.

The forms you'll see most at work: a source that doesn't exist, a number that was rounded or invented, an imaginary court ruling, a contract clause summarized wrong, a person or client mixed up with someone else.

Why do language models make things up?

A language model doesn't look facts up in a database. It works out, word by word, the most likely continuation. When it knows the answer, the likely continuation is usually right. When it doesn't, the likely continuation just sounds right.

In September 2025, OpenAI researchers published "Why language models hallucinate." Their argument: training, and the benchmarks used to rank models, reward guessing over admitting uncertainty. Like a student on a multiple-choice test, a model that guesses sometimes scores, while one that says "I don't know" never does. Their example: guessing someone's birthday gives a 1-in-365 chance, abstaining gives zero.

Risk goes up when:

• The question comes with no source, so the model answers from memory with nothing to copy from.
• The fact is rare: an obscure ruling, a small company's revenue, a contact's name. The OpenAI researchers point out that isolated facts like these can't be inferred from any pattern.
• Dates and recent events are involved: the model stops at its training cutoff and doesn't know what has changed since.
• Math is involved: a model writing a total without a calculator can get the sum wrong.
• The question assumes something false ("which case held that..."): the model tends to play along.

Three cases where a made-up answer cost real money

Imaginary case law. In 2023, lawyers in Mata v. Avianca filed a brief in New York citing six court decisions that didn't exist. One of them had gotten the cases from ChatGPT, which assured him they were real. On June 22, 2023, federal judge P. Kevin Castel fined the two lawyers and their firm $5,000. French courts have seen it too: according to Le Monde Informatique, a court in Périgueux noted on December 18, 2025 that cited references did not appear to match any published decision, and asked the claimant to check they weren't hallucinations.

A customer assistant that binds the company. On February 14, 2024, a British Columbia tribunal ordered Air Canada to compensate a passenger. The assistant on its website had told him he could claim the bereavement fare after his trip, which the airline's policy didn't allow. Air Canada argued the assistant was a separate entity responsible for its own actions. The tribunal said it was part of the website, and the company answers for what it says.

A consulting report sent back. According to The Register, Deloitte delivered a report to Australia's employment department in July 2025, under a contract worth AU$440,000. A University of Sydney academic found references that didn't exist and an invented quote attributed to a judge. The corrected version dropped more than a dozen references and disclosed the use of a GPT-4o-based tool. Deloitte agreed to refund the final installment of the contract.

What do public benchmarks show?

Vectara's leaderboard, updated October 22, 2026, measures how often a model adds information that isn't there when it summarizes an article it's been given, across more than 7,700 articles. The best model on the board does it in 1.8% of summaries; the weakest go above 23%.

Even with the document right in front of it, no model reaches zero. The gap between models is wide, so the choice of model and setup matters, and it's worth testing on your own cases. The benchmark covers article summaries; it says nothing about your specific use.

How do you reduce the risk in practice?

From the most effective to the easiest to set up:

• Give the AI your documents instead of relying on its memory. An agent that searches your files before answering (this is called RAG) copies what it reads instead of guessing. That's the job of a context layer or an AI search engine connected to your files.
• Require cited, checkable sources: every answer points to the document it came from, one click away.
• Add automatic checks: a number or name missing from the documents the AI read gets flagged or removed before anyone sees it.
• Keep human review on anything that commits the company: a client email, a quote, a contract, a report.
• Keep tasks narrow. An AI agent for business that does one specific job on a known scope makes fewer mistakes than an assistant asked to do everything.
• Allow "I don't know": the tool should be able to say it found nothing, and your teams should prefer that to a made-up answer.
• Test on your own cases before rolling out: twenty questions you already know the answer to, including some that have no answer.

Which tasks can you trust it with?

The risk comes mostly from the job you give the tool. Finding a document, summarizing a file you have the source for, drafting a first reply: a mistake shows up on review and costs little.

Quoting a law, giving a client a price, answering the public on its own, signing off a report: the mistake leaves the company and can bind it, as Air Canada found out. These uses need a source and a review every time. Put it in your AI policy: which tasks, which data, who reviews.

What does EU law say about human oversight?

For uses classed as high-risk, such as screening job applicants, Article 26 of the AI Act requires human oversight by people who are competent, trained and have the authority to step in. Those obligations apply from December 2, 2027. For everything else you use AI for, review is still your best protection. This page isn't legal advice.

What Saména does about made-up answers

Saména builds custom AI agents connected to your documents in SharePoint, OneDrive or Dropbox, read-only, in the folders you choose. The agent answers from what it reads in your files, and every answer cites the documents it came from.

There's an extra check: a sentence in an answer that states a number or a proper name that no document the agent read contains is removed before it is displayed. This check compares numbers and names; it doesn't judge meaning.

Nothing goes out until someone at your company approves it: the agent prepares, a person reviews and signs off before it's sent. The risk of errors goes down without going away, which is why review stays in the loop.

Frequently asked

Why does ChatGPT make up sources?

Because it predicts the most likely text, and a citation has a very predictable shape. With no document to copy from, it produces one that looks real. OpenAI researchers explained in September 2025 that training rewards a guess over admitting uncertainty.

Will hallucinations go away with newer models?

They're declining, not disappearing. Vectara's leaderboard, updated in October 2026, shows that even the best model still adds information that isn't there in about 2% of summaries of a document it was given.

Who is liable when an AI gives a customer wrong information?

The company using it. In February 2024, Air Canada was held liable for a wrong answer from the assistant on its website, which the tribunal treated as part of the site. This isn't legal advice.

Does RAG eliminate hallucinations?

It reduces them, because the model works from your documents instead of its memory. Reading and summarizing errors still happen, so you still need cited sources, automatic checks and human review on anything that commits the company.

Does Saména guarantee error-free answers?

No, and no AI tool can. Agents cite their sources. A sentence in an answer that states a number or a proper name that no document the agent read contains is removed before it is displayed; this check compares numbers and names, it doesn't judge meaning. And nothing goes out until someone at your company approves it.