Module 2 of StartCloud's AI at Work: Foundations learning pathway, in five short units with a knowledge check: why AI tools sound confident even when they are wrong (hallucination), a green-amber-red traffic light for deciding which tasks to trust AI with, five practical verification habits, and the send rule: whatever the tool drafted, the moment you send it, it is yours.
When to Trust AI and When to Double-Check It
Confidently wrong
Module 1 covered how these tools build answers by predicting what comes next, not by looking anything up. This module is about the practical consequence, and it is the single most important thing to know about working with AI: these tools never sound unsure.
A human colleague hedges. They say "I think", or "let me check", or they trail off and go find the file. An AI tool does none of that. A made-up figure arrives in the same polished sentence as a correct one, with no wobble in the voice. There is no tell. The wrong answer is not marked, flagged, or delivered any differently from the right one sitting next to it.
The industry calls this hallucination. A plainer word is confabulation: filling gaps with plausible invention. The tool is not lying, because lying requires knowing the truth and choosing otherwise. It is doing exactly what it was built to do, which is produce text that sounds right. Usually that text is also correct. Sometimes it is not, and the tone gives you nothing.
Two sentences from one answer, produced back to back, in exactly the same confident voice. One checks out. One describes a law that does not exist.
"The ATO generally requires businesses to keep most records for five years."
True. You can verify it on the ATO website in about thirty seconds.
"This is set out in section 47B of the Small Business Records Act 2011."
There is no such act. The citation is fluent, specific, formatted correctly, and completely made up.
Courts in several countries, Australia included, have dealt with lawyers who filed documents citing cases an AI tool invented. The citations looked perfect: real-sounding names, plausible references, confident summaries. The cases did not exist. People have been professionally embarrassed and formally sanctioned over it. If trained lawyers can be caught out by a confident fabrication, the rest of us should assume we can be too. The fix is not being smarter than the tool. It is having a checking habit, which is what the rest of this module is for.
"Can I trust it?" is the wrong question, because the honest answer is "it depends", and nobody can work with "it depends" at nine on a Tuesday morning. The better question is "what kind of task is this?" Some tasks are safe to hand over almost entirely. Some need a check before the output goes anywhere. Some should not lean on AI at all.
A traffic light sorts most work in about two seconds. Green means go. Amber means go, then verify. Red means this is not the tool for the job.
- Rewriting your own words: tightening an email, softening a tone, fixing clunky phrasing
- Summarising a document you provided and know well enough to notice a bad summary
- Brainstorming: names, angles, options, first ideas you will judge yourself
- Drafting in an area where you are the expert and will review every line
- Facts, figures, dates, and names of any kind
- Summaries of long documents where a missed nuance would matter
- Anything you are about to forward to someone else under your name
- Legal, tax, medical or financial advice for a real decision
- Recent events: training data has a cutoff, and not every tool searches the web
- Quotes and citations, the things it invents most fluently
- Anything where being wrong costs real money or real trust
A task that feels green but includes a single figure or a client's name is amber. A task that feels amber but touches a contract, a diagnosis, or the BAS is red. Rounding up costs you a few minutes of checking. Rounding down is how a made-up number ends up in a quote with your logo on it.
"Verify the output" is easy advice to nod along to and surprisingly vague to act on. Verify how, exactly? Here are the five habits that do the actual work. None of them needs technical skill, and most take under a minute.
Ask the tool where a claim came from, then actually click through. Invented links, invented papers, and invented case names are common, and they look exactly like real ones until the page fails to load or says something different.
If the tool pulled figures from a document, open the document and find those figures yourself. A transposed digit or a number quietly borrowed from the wrong row survives every read-through that skips this step.
For factual questions, reach for a tool that searches the web and shows its citations, then judge the sources it found. An answer built from live pages you can inspect beats one assembled from memory alone.
Ask the same question again, or put it to a second tool. Matching answers are mildly reassuring. Different answers are a red flag that at least one of them is guessing, and now you know to go to a real source.
When it summarises a document for you, read its take on the section you already understand. If it fumbled the part you know, do not trust it on the parts you do not.
Match the effort to the stakes. A brainstorm needs none of this. An internal draft might get one spot-check. A figure heading into a client proposal gets the source opened and the number traced. The traffic light from the last unit tells you how hard to look.
Everything in this module rolls up into one rule, and it is short enough to stick above a monitor: whatever the tool drafted, the moment you send it, it is yours. Nobody who receives an email cares which sentences a machine suggested. Your name is on it. The client replies to you. The mistake, if there is one, is yours to fix.
That sounds heavier than it is. In practice it means four quick checks against reality before anything leaves your outbox.
- Names: the right people, spelled the way they spell them
- Numbers: every figure traced back to a source you trust
- Dates: deadlines and meeting times checked against the calendar, not the vibe
- Commitments: anything the draft promises is something you actually intend to do
Here is the part worth being cheerful about. If the tool turned a forty-minute drafting job into five minutes, and the checking takes four more, you are still miles ahead. Verification usually costs a fraction of the time the draft saved. The habit is not a tax on using AI. It is the small premium that lets you keep using it with a clear conscience, at speed, on work that matters.
The send rule has a bigger cousin, ownership: what to do when AI helped with a piece of work, how honest to be about it, and the habits that keep your own judgement sharp. That is Module 5, at the end of this pathway.
- Australian Cyber Security Centre: Artificial intelligence
- Australian Cyber Security Centre: Engaging with artificial intelligence
The ACSC's guidance covers the risks of AI-generated content, including inaccurate and fabricated output, and recommends verifying AI responses before relying on them. Details were current at the time of writing (July 2026).
Knowledge check
5 quick questions. Get 4 right and the module is yours.
1. What makes an AI tool's wrong answers so easy to miss?
2. On the traffic light, which of these tasks is squarely in the green zone?
3. The AI backs up a claim with a link to a report. What is the catch?
4. AI hands you a tidy statistic for a client proposal. What happens next?
5. You ask two AI tools the same important question and get two different answers. What does that tell you?