How to check what it tells you

Lesson 1 gave you the problem: the model produces text that sounds right, and most of it is right, and you can't tell which part isn't from the tone. People respond to this in one of two bad ways. Some reread everything, which takes longer than doing it themselves, so they stop using it. Others check nothing, which works until the day it doesn't.

The way out is triage. Not everything needs checking. The things that do are easy to spot.

What to check and what to skim

The rule in one line: check anything specific that you didn't give it. If you pasted the report and it summarised the report, the summary is grounded in your material; skim it for sense. If it added a figure, a name, a date or a source that wasn't in your material, that came from the model's own plausible-text machinery, and that's where you look.

Specifics that look most convincing are the most dangerous: a percentage with a decimal, a study with authors and a year, a quote with a name attached. The more precise the detail, the more likely someone will repeat it, and the more it costs when it's wrong.

Five questions that make it show its working

You don't have to do all the checking yourself. The assistant can point you at the weak spots if you ask. These five follow-ups work with any of the major assistants.

1. "Which parts of this are you least sure about?" It will usually name them, and usually correctly. This single question is the highest-value habit in the course. Ask it after any answer you're going to act on.

2. "What in this answer came from the material I gave you, and what did you add?" Separates grounded content from generated content. Anything in the second list is on your check-list.

3. "Give me the source for that, and tell me if you can't." If the assistant can search the web, it will go and find one, or admit there isn't one. If it can't search, it may produce a plausible-looking reference. Treat any reference you can't open as nonexistent until you've opened it.

4. "Argue the opposite." For judgements and recommendations rather than facts. If the case for the other side is strong and it didn't mention it, you've learned something about the first answer.

5. "What would someone who disagrees with this say?" Same purpose, different angle. Works well on plans, strategy, anything with trade-offs.

Search mode changes the game, a bit

Most assistants can now search the web when you ask, and some do it automatically for questions about recent events. When it searches, the answer comes with links. That's much better than memory, but it's not the end of the job:

• Open the link. Confirm the page says what the summary says. Summaries of sources are themselves generated.
• Notice the source's quality. A forum post and a regulator's website both count as "a source".
• For anything that changes, prices, laws, who holds a job, ask it to search even if it offers an answer from memory. "Search for this before answering" is a legitimate instruction.

When a mistake isn't a mistake

Two things that look like errors and aren't:

It gives a different answer the second time. As lesson 1 said, there's randomness built in. For facts, that's a sign to check. For writing, it's just variety.

It changes its answer when you push back. Assistants are built to be agreeable. If you say "are you sure?", many will fold even when they were right. That means "are you sure?" is a weak test. "Check that against the figures I pasted" is a strong one, because it points at evidence rather than at your mood.

A worked example

You ask for a short briefing on a competitor before a meeting. The reply has: a paragraph on what they do (skim, it's general), their founding year and headcount (check), a quote from their CEO (check hard, quotes are the most fabricated thing), a list of their products (check the names exist), and a suggested angle for your meeting (skim, it's a suggestion). Checking the three specifics takes four minutes. Reading the whole thing critically would take twenty. Not checking would be a gamble you'd lose eventually, in front of a client.

The habit

After any answer you'll act on, two moves:

1. Scan for the five risky types. Check those against a real source.
2. Ask "which parts are you least sure about?" and look at whatever it names.

That's it. Everything else in the answer, read the way you'd read a sharp colleague's first draft: useful, fast, and yours to edit.