Module 2 of StartCloud's AI at Work: Foundations learning pathway, in five short units with a knowledge check: why AI tools sound confident even when they are wrong (hallucination), a green-amber-red traffic light for deciding which tasks to trust AI with, five practical verification habits, and the send rule: whatever the tool drafted, the moment you send it, it is yours.
When to Trust AI and When to Double-Check It
Confidently wrong
Module 1 covered how these tools build answers by predicting what comes next, not by looking anything up. This module is about the practical consequence, and it is the single most important thing to know about working with AI: these tools never sound unsure.
A human colleague hedges. They say "I think", or "let me check", or they trail off and go find the file. An AI tool does none of that. A made-up figure arrives in the same polished sentence as a correct one, with no wobble in the voice. There is no tell. The wrong answer is not marked, flagged, or delivered any differently from the right one sitting next to it.
The industry calls this hallucination. A plainer word is confabulation: filling gaps with plausible invention. The tool is not lying, because lying requires knowing the truth and choosing otherwise. It is doing exactly what it was built to do, which is produce text that sounds right. Usually that text is also correct. Sometimes it is not, and the tone gives you nothing.
Two sentences from one answer, produced back to back, in exactly the same confident voice. One checks out. One describes a law that does not exist.
"The ATO generally requires businesses to keep most records for five years."
True. You can verify it on the ATO website in about thirty seconds.
"This is set out in section 47B of the Small Business Records Act 2011."
There is no such act. The citation is fluent, specific, formatted correctly, and completely made up.
Courts in several countries, Australia included, have dealt with lawyers who filed documents citing cases an AI tool invented. The citations looked perfect: real-sounding names, plausible references, confident summaries. The cases did not exist. People have been professionally embarrassed and formally sanctioned over it. If trained lawyers can be caught out by a confident fabrication, the rest of us should assume we can be too. The fix is not being smarter than the tool. It is having a checking habit, which is what the rest of this module is for.