The most dangerous property of a language model is not that it makes mistakes. Everything makes mistakes. It is that the mistakes arrive in exactly the same confident tone as the correct answers.
A colleague who is unsure sounds unsure. A model that is unsure sounds identical to one that is certain. That is the whole problem, and understanding it is most of what you need to use these tools safely.
Why it happens
Language models generate text by predicting what should come next based on patterns in their training data. They are not looking anything up. There is no internal database being consulted and no fact being retrieved.
This is why a model can produce a citation to a paper that does not exist, a legal article number that is close but wrong, or a statistic that sounds plausible and was never measured. The output is not a lie in any meaningful sense. It is a fluent completion of a pattern, and fluency is not accuracy.
Some products reduce this by retrieving real documents and generating from those, or by connecting to live systems. That helps considerably. It does not eliminate the underlying behaviour.
Where the risk is highest
Not all tasks are equally exposed.
Highest risk: specific numbers, dates, legal article references, citations, quotations, names of people, technical specifications, and anything jurisdiction-specific. Regulations are particularly bad, because training data contains many countries' rules and many outdated versions of each, and a model will confidently give you an American rule for a French question or last year's threshold for this year's.
Medium risk: summaries of documents you have supplied, structured drafting, analysis of data you provided. Errors are less common and usually visible if you know the source.
Lowest risk: rephrasing, tone changes, formatting, brainstorming, structuring something you already know. Here the model is manipulating text rather than asserting facts, and the failure mode is that it is unhelpful rather than wrong.
The practical implication is that the same tool can be used with almost no checking for one task and needs full verification for another.
How to check efficiently
Verifying everything defeats the purpose. Verifying nothing is how people end up filing something wrong.
Verify anything with a number, a date or a reference. These are cheap to check and the most likely to be wrong. A regulatory threshold, a filing deadline, a legal article, a price. Look it up.
Verify anything you will act on or send externally. Internal thinking, low stakes. A client deliverable, a filing, a contract, high stakes.
Ask for sources, then check that they exist. Not just that they are cited, that they are real and say what the model claims. Fabricated sources are usually plausible-looking, so scanning the list is not the same as checking it.
Test on something you know. When evaluating any tool, ask it several questions where you already know the answer, ideally specific to your jurisdiction. You learn its accuracy and, more importantly, what it does when wrong. Good systems hedge. Poor ones assert.
Watch for excessive fluency on hard questions. A confident, well-structured answer to a question with a genuinely complicated answer is a warning sign, not a good sign.
Building checking into the workflow
The reliable pattern is to treat AI output as a draft from a fast, well-read colleague who is sometimes wrong and never says so.
That means the workflow is generate, then review, never generate then send. And the reviewer needs enough domain knowledge to catch errors, which is why these tools accelerate people who already know the subject and mislead people who do not.
For anything regulated, keep a qualified human in the decision. Every serious vendor says this in their own terms. Mirage Cloud's acceptable use policy states plainly that output may contain errors and must be verified before use, and prohibits presenting AI output as professional advice in regulated fields without qualified supervision. That is the correct position, and a vendor whose terms do not say something similar is either not thinking about it or hoping you will not.
There is now a regulatory dimension too. Since 2 August 2026, EU transparency rules require that people are told when they are interacting with an AI rather than a person, and that synthetic content is marked. That does not address accuracy, but it means the "nobody needs to know" approach is no longer available.
What actually reduces errors
Give more context. Most wrong answers come from a model filling gaps you left. State the jurisdiction, the year, the specifics.
Ask for uncertainty explicitly. Telling a model to flag anything it is unsure about produces noticeably more hedging, which is useful signal.
Use specialised tools for specialised questions. A tool configured for one domain, with reference material and a defined scope, is more reliable than a general assistant for that domain. It is not infallible, but the failure rate is meaningfully different.
Break big requests into smaller ones. Long, complex outputs contain more unverified assertions and are harder to check.
The right mental model
Not an oracle. Not a search engine. A very fast writer with broad shallow knowledge, no ability to tell you when it is guessing, and no stake in whether it is right.
Used that way, these tools are genuinely useful for a lot of business work. Used as a source of truth, they will eventually produce something confident and wrong at a moment when it matters.
Comments
Log in or sign up to join the conversation.