🎯 How To Catch AI's Made-Up Facts

The unsettling thing about AI isn’t that it sometimes gets things wrong. Humans get things wrong constantly and we cope. The unsettling thing is how it sounds while getting things wrong: exactly as fluent, exactly as confident, exactly as tidy as when it’s right. There’s no stammer, no “erm”, no shifty look. The made-up statistic arrives in the same warm, assured voice as the correct one.

The tech world calls this “hallucination”, which makes it sound mystical. I prefer confident guessing, because that’s what it is - and once you understand it that way, catching it stops being scary and becomes a habit, like checking your mirrors.

Why it happens (one paragraph, promise)

AI builds its answers by predicting what words are most likely to come next, based on patterns from everything it’s read. Most of the time that produces truth, because true things get written down a lot. But when it hits a gap - a fact it doesn’t solidly have - it doesn’t stop and say “no idea”. It fills the gap with something plausible, because plausible is literally what it’s built to produce. It isn’t lying; it doesn’t know it’s guessing. Think of a keen mate in the pub quiz who’d rather have a confident go than admit they’ve never heard of the question. Brilliant teammate. You still don’t let them fill in the answer sheet unsupervised.

So the skill isn’t “never trust AI” (paralysing) or “always trust AI” (how you end up citing a court case that doesn’t exist). The skill is knowing when to check. Which brings us to…

The Three Check Moments

You don’t need to verify everything - that would cancel out the time AI saves you. You need a reflex for three specific moments. Miss these three and confident guessing can’t really hurt you.

Moment 1: Names, numbers and quotes

What it is: anything specific and checkable. Dates, prices, statistics, opening hours, phone numbers, laws, “as Einstein once said”. These are AI’s classic slip zones, because precise details are exactly where pattern-guessing fails quietly.

How to check: for anything current or local - prices, opening times, deadlines - just ask Claude to use its built-in web search: “search the web and check that”. For quotes and statistics, ask where it’s from before you repeat it.

The non-obvious bit: the more specific a detail sounds, the more suspicious you should be, not less. “Around a third” is probably a safe pattern; “34.7% according to a 2023 study” is precise-sounding decoration until proven otherwise. Confident guessing loves decimal points - they’re the fake moustache of made-up facts.

Moment 2: Anything you’ll repeat in public

What it is: the newsletter, the social post, the flyer, the speech at the retirement do, the “fun fact” for the pub quiz you’re running. The moment words leave your private chat and go where other people can check them, the stakes change.

How to check: before publishing, run the check-yourself prompt below on the draft, then verify whatever it flags. Two minutes, tops. And if the draft quotes anyone by name, find the original or ask the person - a misattributed quote is the fastest way to look daft online.

The non-obvious bit: embarrassment scales with audience, and the internet has a long memory. A wrong date in your notes costs nothing; the same wrong date on the village fete poster is two hundred people at a locked gate. Your private chats can stay breezy - it’s the copy-paste-outward moment that triggers the check.

Moment 3: Anything involving money or rights

What it is: tax, benefits, contracts, refunds, visas, tenancy rules, employment rights, anything where being wrong costs actual pounds or actual options.

How to check: treat AI’s answer as the briefing, not the ruling. Use it to understand the situation and to build a list of sharp questions, then confirm against the official source - gov.uk, your bank, the actual contract - or a qualified human.

The non-obvious bit: AI has read the rules of many countries at once, and they blur. Ask about a refund and you might get an answer that’s perfectly correct… in the wrong country. Adding “UK specifically, and search the web to confirm the current rules” to money questions fixes most of this. Most. Keep reading.

The check-yourself prompt

Here’s the habit that does the heavy lifting: after any answer you’re about to rely on, make Claude audit itself.

Nick this one:

Before I use this, go back over what you just told me and audit it. For each factual claim - every name, number, date, price and quote - rate your confidence: CERTAIN (you'd bet on it), PROBABLY RIGHT (solid but worth a glance), or HONESTLY UNSURE (could be a guess). Group the claims by rating, most doubtful first. Then list anything I should verify before repeating it, and tell me exactly where to check each one. Be strict with yourself - I'd rather you flag too much than too little.

Something interesting happens with this prompt: Claude is often genuinely good at spotting which of its own claims were on shaky ground, and the “honestly unsure” list is usually short and usually right to be there. If you’ve got memory switched on (Settings → Capabilities), you can even say “remember that whenever I ask about numbers, dates or money, offer me a confidence audit” - and it’ll start doing this without being asked. One less thing for you to remember is one more thing that actually happens.

Then there’s the companion habit, the ask-for-sources reflex - for anything that matters, follow up with:

The follow-up:

Search the web and show me sources for the claims I care about here: [LIST THEM]. If you can't find a source for something, say so plainly rather than reasoning around it.

If it searches and comes back empty-handed on a claim… that claim is decoration. Bin it. A fact that can’t be traced isn’t a fact, it’s an aesthetic.

Read this bit properly

Being straight with you: these habits reduce the risk, they don’t eliminate it. There’s something faintly circular about asking a system that can guess confidently to grade its own guessing - and occasionally it will be confidently wrong about being confident. Web search helps enormously with anything current, but sources can be misread, and a rare wrong claim will sail through flagged “certain”.

So keep one human rule on top of everything: anything genuinely important - money you can’t afford to lose, legal or medical decisions, anything with someone else’s name attached - gets checked by a person or an official source, full stop. That’s not AI being useless; that’s the same rule you’d apply to a clever new assistant on their first month. Trust, plus receipts.

The five-minute version of this guide

Make the check-yourself prompt your pre-publish ritual: nothing gets copied out of Claude into public view without a quick self-audit first. One habit, two minutes, and moment 2 - the embarrassing one - is basically covered for life.

Checking what AI tells you is one half of internet self-defence. The other half is checking what people send you, and for that there’s Is This A Scam? Ask AI.