What breaks, and how far it reaches

Most writing about AI risk is either abstract enough to be useless or alarming enough to be ignored. This page is neither. It is a plain account of how these tools actually cost people something, arranged by the question that decides how careful you need to be:

If this goes wrong, who does it reach?

That is the useful question because it does not depend on knowing anything technical. You already know whether a task touches your own time, your company’s money, or another person’s life. Find your task on that scale and you know how much checking it deserves.

Level 1 — It only reaches you

Wasted time. A bad draft. An answer that sounded right and sent you down a blind alley for twenty minutes.

This is where most use sits, and it is why so much AI advice is glib: at this level the worst case genuinely is small. Experiment freely here. Try it on the thing you would have done by hand and compare. You cannot break anything that a delete key does not fix.

The only real cost at this level is a subtler one: getting used to not checking. The habits you form on low-stakes work are the habits you will still have when the stakes are higher, and nobody announces the transition.

Level 2 — It reaches your business

A wrong number in a quote. A summary that missed the clause that mattered. A confident paragraph of analysis built on a figure the tool invented.

The failure that matters at this level is not that these tools are wrong sometimes. Everything is wrong sometimes. It is that they are wrong in the same steady, fluent, professional voice they use when they are right. There is no tell. A person who is guessing usually sounds like a person who is guessing; this does not.

So the rule is mechanical rather than intuitive, because intuition is exactly what fails here:

Reshaping something you provided is reliable. Producing something you did not provide is not.

Summarizing your document, rewriting your email, restructuring your notes — the source is in front of it, and errors are visible by comparison. But any figure, date, name, citation, legal or tax claim, or quotation that you did not supply is unverified, no matter how specific it looks. Specificity is not evidence. A fabricated statistic comes with a plausible decimal point.

Check anything that will end up in front of a customer, in a filing, or in a decision.

Pasted material also lives here

What you paste goes to the vendor. Depending on your plan it may be retained, and on some consumer tiers used to improve their models. Business and enterprise plans generally treat this differently — read your own plan’s terms once, then set a rule you can follow without thinking about it.

Most people are fine with: don’t paste anything you would not be comfortable forwarding outside the company. Contracts, anything under an NDA, and unreleased plans deserve a deliberate decision rather than a habit.

Level 3 — It reaches someone else

Now it is not your risk to take.

Other people’s personal data is the clearest case. Customer records, employee information, health or financial details, anything identifying a named individual. Pasting that into a tool moves it somewhere the person never agreed to, and in most jurisdictions that is a legal question rather than a judgement call. If you would need permission to email it to a stranger, you need to think before pasting it.

Anything that speaks as you. A drafted message is fine; a sent message is a different thing. The recipient cannot tell that the tone came from a machine, and they will respond to it as though you meant every word — because as far as they are concerned, you did.

Decisions about people. Hiring, firing, credit, discipline, who gets served and who does not. These tools reproduce whatever patterns sit in their training data, they cannot explain their reasoning in a way that survives a challenge, and in several jurisdictions automated decisions about individuals carry specific legal obligations. Use them to prepare information for a human decision. Do not use them to make the decision.

Level 4 — It cannot be undone

Deleting. Sending. Publishing. Paying. Committing to something on the record.

The distinguishing feature here is not that the failure is likely. It is that there is no version of the day where you get to fix it. Everything above is recoverable with enough embarrassment and effort. This is not.

The example worth naming, because it is advice you will genuinely be given: “use AI to clean out your inbox.” It sounds harmless and it is widely repeated. But an inbox is an archive of things you cannot get back — the one thread with the agreed price, the message you will need in two years, the note from someone who has since died. A tool sorting on what looks unimportant today has no way of knowing which of those it is holding.

The general form:

Before you let anything act rather than draft, ask what happens if it is wrong on the tenth item and you do not notice until the hundredth.

If the answer is “nothing much”, proceed. If the answer is “I would not be able to tell, and I could not get it back”, then you want a human confirming each action, or a reversible version of the task, or not to do it this way at all.

Automating the irreversible is the single most expensive mistake available to you right now. Not because the technology is bad at it, but because the failure is silent: an automation that works on 99 items and quietly mangles the hundredth looks exactly like an automation that works.

What this page is not saying

It is not saying don’t use these tools. Level 1 and most of Level 2 is where the real, unglamorous value sits, and the people getting the most out of this are working there constantly.

It is saying the checking should scale with the reach, and that the reach is something you can judge in a few seconds without understanding anything about how the technology works.

---

Think we’ve got something wrong here, or hit a failure that doesn’t fit any of these levels? That’s genuinely useful to us — tell us.