A human in the loop is not a control.

A review queue that no longer reviews

A customer service team rolls out an assistant that drafts replies to routine requests. The rule is set on day one: nothing goes out without a human reading it first. Nobody argues with the principle, and it appears in the approval memo for the project.

The first weeks go exactly as planned. Volumes are low, drafts get read carefully, one in three is edited. Three months later the assistant handles four hundred requests a day, two people cover the queue, and the drafts look good. Median time on a reply has dropped below ten seconds. The edit rate is approaching zero.

On paper, human control is intact — every outgoing message carries a named approval. In practice, the queue has become a rubber stamp. The issue is not that people stopped caring. They are doing precisely what the system allows them to do.

The mistake is treating review as a guarantee

Put someone in front of a system that is right most of the time, and they will eventually agree with it. This has a name — automation bias — and it is not a lapse in individual attention. It is a predictable outcome of how the work was designed.

The European Union's AI regulation draws practical conclusions from this. Article 14 requires high-risk systems to be built so that people can effectively oversee them, and it spells out what that means: the person must understand the system's capabilities and limits, stay alert to the tendency to over-rely on its output, interpret results correctly, be able to disregard, override or reverse a decision, and be able to stop the system. The same article requires those measures to be proportionate to the risk, the level of autonomy and the context of use.

Most Canadian organizations fall outside that regulation's scope. It still states a design requirement worth borrowing: oversight is not a checkbox, it is a capability you have to build. And a capability comes with four conditions.

• Authority — rejecting a suggestion has to be routine, not an exception the reviewer must justify.
• Time — the expected throughput has to leave room for actual reading, not just a click.
• Information — whatever the decision requires has to be on screen at the moment of the decision, without switching tools.
• Evidence — rejections and edits have to be recorded, or nobody will know whether the control still works.

Someone with neither the time nor the authority to say no is not a control. They are a witness.

Classify the decisions before classifying the model

Oversight is almost always debated at project level: will this assistant be supervised? That is the wrong altitude. A single assistant produces decisions whose consequences have nothing in common — suggesting wording, issuing a credit, denying a claim, editing a customer record.

Three questions settle most of it, and none of them is about the technology or the model's benchmark scores.

Average model accuracy is a poor guide here. A system that is right 97 percent of the time is unacceptable if the remaining three percent wrongly denies someone a service, and perfectly acceptable if it produces awkward phrasing that gets fixed downstream.

Four regimes, not two

The conversation tends to present a binary: full automation or review everything. In practice four regimes cover most situations, and a single process usually combines several of them.

1. Direct execution — reversible, internal, low-consequence decisions. Control comes from aggregate measurement, not from reading each case.
2. Sampled review — a defined share of cases is read, with the share revised upward as soon as a pattern of errors appears. Useful when volume makes full review a fiction.
3. Mandatory review — nothing leaves without an explicit human decision, and staffing is sized accordingly. Reserved for decisions that are hard to reverse or externally visible.
4. Out of scope — contractual exceptions, sensitive situations, incomplete files. The system produces no suggestion at all and routes the case onward.

The fourth regime is the one most often skipped, and it offers the most protection. Naming what the system deliberately does not handle is what keeps approximate answers away from the files that matter most.

Check that the control still works

Oversight degrades quietly. It gets decided at launch, then volume grows, staff turns over, confidence settles in, and nobody revisits the original setting. A few indicators catch that drift, provided someone looks at them on a schedule.

• Edit rate on suggestions — as it approaches zero, review has effectively stopped.
• Median time per case, compared with the time it takes to read the information that matters.
• Share of cases escalated out of the automated path, and how that share moves.
• Downstream failures — complaints, manual corrections, rework — that should have been caught at review.

The NIST AI Risk Management Framework organizes this discipline into four functions — govern, map, measure and manage — and treats the effectiveness of oversight mechanisms as something to be measured rather than assumed. The takeaway is plain: whether a control exists and whether it works are two separate questions, and only the second one protects anyone.

A useful production AI system is not the one that needs the least human involvement. It is the one where you can say exactly which decisions it makes alone, which ones it prepares, which ones it never touches — and how you know.