Hallucination — the Mix-Up With Lying and a Software Bug

A language model that gets asked for a supplier's VAT number and confidently returns a nine-digit figure that appears on no invoice anywhere is hallucinating. The word sounds more dramatic than the phenomenon itself — and that gap is exactly where most misunderstandings in small businesses using ChatGPT, Copilot or similar tools begin.

What the term technically means

A language model does not recall facts, it predicts the next plausible word. Asked for a date, a number or a name, it produces an answer that fits the sentence pattern — regardless of whether that exact value ever appeared in its training data. If the model lacks the knowledge, it rarely stops. It fills the gap with something that sounds right. AI research calls this process hallucination: a fabricated statement that is grammatically and stylistically flawless.

That sets the machine fundamentally apart from an employee who honestly answers "I don't know." A language model barely recognises that state unless it has been specifically trained to flag uncertainty.

The mix-up with lying or a software bug

In conversations with clients and staff, the phrase "the AI lied" comes up constantly. It misses the point. Lying requires intent, an awareness of truth, and a will to obscure it. None of that exists in a language model. There is no internal instance weighing true against false and choosing false on purpose.

The second mix-up is subtler: many treat a hallucination like a software bug, fixable with an update or a restart. That is wrong too. A bug is an error in code, reproducible and in principle fixable. A hallucination grows out of how the system works in the first place, out of the probability math behind every generated word. It can be reduced, through better source material or extra verification steps, but not eliminated outright. Expecting otherwise leads to disappointment — and that exact expectation is what makes autonomous AI agents risky in small firms, as an earlier piece on agent autonomy lays out.

Scepticism toward hasty AI anecdotes shows up outside technical circles too. A ZEIT column titled "Erzähl mir bloß nicht, was dein Chatbot so erzählt" mocks exactly this kind of uncritical everyday enthusiasm for AI.

How to spot the difference in practice

Three signals help with sorting cases. First, the tone. A hallucination rarely sounds hesitant; it arrives confident, often padded with details meant to add credibility — an invented paragraph number, a case reference that doesn't exist. Second, repeatability. Ask the same question twice and many models return different specifics, a strong sign that no fixed source sits behind the answer. Third, fabricated citations. Language models occasionally generate links, study titles or statute numbers that look like genuine hits but lead nowhere. That alone is reason enough to treat any unsourced figure with caution.

An example

Suppose a bookkeeping office asks an AI tool to list standard depreciation rates for office equipment. The model names a percentage for one device category that sounds plausible but appears in no official depreciation table. Nobody fed the model that number — it reconstructed it from patterns in similar tables. If the figure gets copied into the books unchecked, the error only surfaces at the next tax audit.

A counterexample

Not every wrong answer is a hallucination. If a business asks an AI model whether e-invoicing rules already apply to all invoice sizes and gets an outdated answer, the cause is usually different: the model's training cutoff. Rules around the German e-invoicing mandate for 2027 have since been refined several times, and a model with an older knowledge base simply repeats what applied at the time it was trained. That is not fabricated content — it is a correctly reproduced but outdated state. In practice that calls for two different reflexes: countering hallucination means verifying the individual claim, countering outdated knowledge means asking for the cutoff date of the information, or picking a tool that pulls from current sources instead of quoting from training.

Anyone reworking a company website right now, perhaps because AI-written copy has crept in, should use the moment to check the technical foundation as well. The free website check from Log-System Development reviews load time, basic SEO factors and a few other metrics within minutes. For businesses that also want a site built to actually generate enquiries, or an app project handled directly with developers instead of through an agency, that free check is a reasonable first stop.

For anyone wanting to go deeper into how language models actually work, KI-Campus offers free courses on the subject — a format that bends to the learner's schedule, not the other way round.

An AI answer without a source stays a suggestion, not information. Making that one distinction before a figure lands in the books or a claim goes to a client takes about a minute.