AI systems make decisions and produce outputs that affect real people and real work. Your role alongside these systems shapes whether those outputs are good and whether problems get caught early. This section builds the knowledge behind that role. You will know how AI should sit alongside human work, the main models of oversight, what good oversight actually looks like, and where it tends to break down.

In this section you will learn:

• the principle that AI should add to human work rather than replace it where it makes sense
• the three main models of how a person sits alongside an AI system
• the difference between oversight that does the job and oversight that only exists on paper
• why AI that takes actions of its own needs a different kind of oversight from AI that only produces text
• the patterns that show up when oversight is going wrong

AI should add to human work, not replace it

What the principle says

The principle is that where it makes sense, AI and automation should be designed to add to human work rather than replace it. The person keeps the judgement, the responsibility, and the connection to the outcome. The AI brings speed, scale, or some specific capability the person does not have.

What “where it makes sense” actually means

The principle is not absolute. Some work genuinely does not need a person in the decision. Setting up an automatic data transfer between two systems at midnight, for example, does not need human judgement. The principle says the human is removed only where there is real reason for it. Not by default. Not by habit.

What this is not saying

Adding to human work rather than replacing it does not mean an AI never decides anything. It means the human role gets chosen on purpose, rather than drifted into. Sometimes the best design is for the person to decide with AI input. Sometimes for the AI to decide and a person to approve. Sometimes for the AI to act and a person to monitor. The point is that the choice gets made carefully. Not that the human is always at the centre.

Seeing It in Action

A customer complaints team in a financial services company has brought in an AI tool to help with reply drafting.

In the first setup, the tool produces a draft reply for each complaint. A team member then opens the draft, checks it against the case, fixes anything that does not fit, and sends it. The team member knows what was sent. They own the reply. Their name is on it.

A second team, in a comparable company, has set things up differently. The tool drafts and sends replies on its own. Team members do sample checks. They look at one in fifty replies after the fact. The volume has gone from two hundred replies a day to six hundred.

Both designs are defensible. Both have real reasons behind them. But they are different choices, and they produce different things in practice.

In the first setup, the principle has been held. A person sits inside the work. They add the judgement the tool cannot bring on its own. They own the outcome.

In the second setup, the principle has not been held. The volume gain is real, but no person owns any individual reply. If a customer challenges a response, nobody in the team can say what was sent or why. The accountability has moved off the team and onto the system.

Neither setup is automatically right or wrong. What matters is that the choice between them is the choice. The principle says to make it on purpose, with full awareness of what is being given up.

The three main models of human oversight

There are three main shapes of how a person sits alongside an AI system. None of them is the safe option by default. Each one fits different situations.

In the Loop

Human in the loop

The person is part of every decision. The AI produces an output. The person looks at it and accepts, rejects, or changes it. Only then does the output take effect. The person sits inside the decision.

An example. An AI tool that suggests replies to customer complaints, with each reply checked and sent by a person.

On the Loop

Human on the loop

The AI runs on its own and takes actions, but a person is watching. The person does not approve every decision. Those happen without them. They are watching the system’s behaviour and they can step in when something looks wrong. The person sits around the decision, not inside it.

An example. An automation that processes supplier invoices and pays them within set limits. A finance team member checks a dashboard of processed invoices each day and steps in when something unusual shows up.

Out the Loop

Human out of the loop

The AI runs on its own and people are not involved in the individual decisions. People may still be involved in designing the system, watching its overall quality, auditing it, and making decisions about whether to keep using it. They are not involved in the individual decisions it makes day to day.

An example. An email filter that sorts incoming messages into folders automatically, with no person checking the individual decisions.

What drives the choice between them

A few things shape which model fits.

The first is how serious it is when the AI gets something wrong. A wrong customer refund, a wrong rejection of a job candidate, a wrong medical decision. These all push the choice toward human in the loop. A filter sorting an email into the wrong folder is at the opposite end of severity. Low severity decisions can sit comfortably with human out of the loop.

The second is whether the decision can be undone. If the AI’s decision can be reversed, less oversight may be acceptable. If it cannot, more is needed.

The third is how much the AI can be trusted for this particular work. A system with a long track record on this kind of decision can take less oversight. A new or untested system cannot.

The fourth is how many decisions the system is making. Hundreds of thousands of low-stakes decisions a day cannot realistically have a person inside each one. The model has to fit the volume.

The fifth is the rules the work sits under. Some decisions have human involvement required by law or policy. This is most often the case for automated decisions that have serious effects on people, and for decisions in regulated areas like finance, healthcare and recruitment.

Seeing It in Action

A team in a retail business is responsible for several kinds of work that an AI system now touches. Three of those tasks are handled under different oversight models, and the contrast shows why no single model is right for all of them.

The first is customer refund decisions. When a customer asks for a refund above a certain value, an AI tool looks at the case and recommends a decision. A team member then looks at the recommendation, the customer’s history, and the reason given. They make the decision and send the response. This is human in the loop. The wrong call here would mean a customer treated unfairly or money paid out wrongly. Both matter, and both are worth a person looking at every case.

The second is supplier invoice processing. The AI processes incoming invoices, matches them against orders, and pays them within set limits. Another team member checks a dashboard each day. They look for anything that does not fit the normal pattern. This includes invoices larger than expected, suppliers the company has not used before, and payments that match a recent flag. They do not approve individual invoices. They watch for patterns. This is human on the loop. The volume is too high for individual approval. The risk in any single invoice is low, and unusual patterns will surface in the dashboard.

The third is internal email categorisation. The AI sorts emails into folders based on what they look like. No one checks individual decisions. If an email gets sorted into the wrong folder, the cost is small. The sender or the recipient just moves it. This is human out of the loop. The volume is huge, the cost of any single error is tiny, and the system has a long track record on this kind of sorting.

What you can see from these three is that the choice of model is the real choice. The same team uses three different models for three different kinds of work. None of them is wrong. None of them is the safest by default. Each one is a fit for the situation it sits in.

Meaningful oversight versus oversight that only exists on paper

The most important thing about oversight is whether it actually does the work. Most real failures are not the absence of oversight. They are oversight that exists in the system on paper but does not protect the outcome in practice.

Signs the oversight is real

Several things show up when oversight is doing what it should.

• The reviewer has time to actually check the decision. The pace is slow enough for the attention each decision needs.
• The reviewer can see what they need to see. They can see the AI’s output, but also enough of the situation behind it to know whether the output fits this case.
• The reviewer has the right knowledge for the decision. A person checking without the skill to judge is in the loop in name only.
• The reviewer can actually do something when they disagree. They can stop the decision, change it, flag it for someone, get something done about it. Disagreement is something that happens, not just something that gets noted.
• Disagreement is genuinely welcomed. People who push back are supported. They are not pressured, overruled without reason, or treated as blocking delivery.

Signs the oversight is only on paper

There are equally clear signs when oversight is only working on paper.

• The review rate is impossible at the volume. A person expected to approve hundreds of decisions an hour is not actually checking them.
• The review screen does not show what the person needs. The output is there. The case context is not. Or the format is one where a real judgement cannot be made in the time available.
• The pattern is rubber-stamping. This means going through the motion of approving without actually checking. Approval rates never change. Disagreements get rarer over time without the system getting better. People describe their job as “clicking approve.”
• The reviewer was chosen for their availability, not their judgement. The person looking at the decision is not the person who would know whether it was right.
• Disagreement gets pushed back on. People who raise concerns find that doing so slows their day, upsets colleagues, or has the decision approved another way. They learn to stop raising things.
• The oversight has drifted. What worked at launch has quietly become routine signing-off. Volume grew. People got used to the system. Other pressures shifted. The check went from a real check to a habit of clicking yes.

Why this matters now

AI is producing very high volumes of decisions that all “have human review” in name. The question is whether the review is real.

Reviewer fatigue is a real failure pattern. With AI-generated content in particular, most outputs look plausible. The reviewer’s job is to catch the small share that are wrong. Plausible outputs at high speed are easy to miss. The fatigue is not laziness. It is a known limit of human attention.

Seeing It in Action

A customer service team has put a setup in place where AI drafts replies to customer emails. Before sending, every reply is reviewed by a person. On paper, this is human in the loop. Every customer email gets a person checking the reply.

The reality is different.

Each reviewer is expected to handle three hundred emails an hour. The screen shows the customer’s email and the AI-drafted reply, side by side, with an Approve button and a Send button. There is no view of the customer’s history. There is no view of any earlier emails in the same case. The reviewer has about twelve seconds per reply before they fall behind the pace.

Approval rates sit at over 99%. Reviewers say things like “if the reply looks reasonable, I just send it.” The few rejections are usually for obvious things. Typos, missing names, replies in the wrong language. The deeper question of whether the reply actually answers what the customer asked is rarely possible to assess in the time available.

When this team’s process is audited, the structure looks fine. There is human review before sending. The team can show that 100% of replies were checked before they went out.

What you can see, watching the team work, is that the review is not doing what it is supposed to do. The structure is right. The protection is not. And nobody in the team meant for the review to stop working. It became that way through volume, through the screen design, and through drift over time.

Why oversight of AI agents is different

Some AI tools only produce text. They write replies, summarise documents, suggest options. The person decides whether to use the output or not.

Other AI tools take actions in the real world. They send emails. They make bookings. They pay invoices. They write code and run it. They change records in a system. These tools are usually called agents.

Agents need a different kind of oversight. The same idea of “a human checks the work” does not work the same way when the AI has already done the thing.

Why agents change the oversight problem

What good oversight of an agent looks like

Several things together make the oversight real.

Scoping says what the agent is allowed to do and not allowed to do.

This includes:

• actions that are out of scope
• limits on how much money it can spend
• limits on who it can contact
• data it is not allowed to access

Checkpoints inside the agent’s flow are moments where the agent pauses and asks for human confirmation before going further. These are most useful at the points where actions become hard to undo or where spending crosses a set amount.

A visible log of what the agent did. What it was asked to do. What actions it took. What the results were. The log has to be in a form a person can actually look through.

A stop control that actually stops the agent. Not just a flag for review. A way to stop the process, even part-way through, when something is going wrong.

Human review of patterns over time. Not every decision. A sample, looked at regularly, that shows whether the agent’s behaviour is drifting.

Why this matters now

Agents are becoming more common. Organisations are putting agents to work on tasks they would not have considered using AI for a year or two ago. This includes sending customer messages, making purchases, processing claims, and writing and running code on real systems.

The oversight question has changed shape. It used to be “is there a human in the loop?” It is now also “is there proper oversight of an agent that runs by itself?”

Seeing It in Action

A small finance team in a manufacturing company has put an agent in place to process supplier invoices. The agent takes each invoice that comes in, matches it against the purchase order, enters it into the finance system, and flags anything that does not look right. For most invoices, this happens without a person involved.

The oversight has been built in carefully.

The agent has a spending limit. Any invoice above £5,000 is paused. The agent does not enter or pay that invoice. It puts the case in a queue for a person to look at. The person makes the call.

Every action the agent takes is logged. The log shows what invoice was processed, what the agent did with it, and what the result was. The team can look back at any invoice and see exactly what happened.

There is a stop control. If anyone on the team sees something going wrong, they can stop the agent immediately. The agent does not finish its current invoice and then check. It stops.

Each Friday, a team member looks at a sample of the week’s processed invoices. They are not approving them. Those have already been processed. They are checking for patterns. Are the same kinds of errors showing up? Has the agent’s behaviour shifted in any way? Are there suppliers the agent is treating differently from before?

What you can see is that the oversight is working, but it is not the same shape as oversight of a system that just produces text. Nobody is checking every invoice the agent processes. They could not. The volume is too high. What protects the work is the design. The spending limit. The log. The stop control. The weekly sample. Take any one of those away and the oversight starts to fail.

Patterns to watch for when oversight goes wrong

There are five patterns worth knowing about. They show up across different kinds of work and different kinds of AI. Knowing the patterns makes them easier to spot when they appear.

Pattern 1: Picking human in the loop because it sounds safest

Human in the loop sounds like the strong choice. A person checks every decision. What could go wrong?

What goes wrong is that the volume planned for the system makes real review impossible. The team picks human in the loop on paper. In practice, the checks are surface-level or skipped because the volume is too high. The structure is there. The protection is not.

The honest answer in this case is to either reduce the volume, change the system to produce fewer outputs that need checking, or pick a different oversight model. Calling something human in the loop when the loop is broken does not make the oversight real.

Pattern 2: Approval clicks treated as review

A person clicks Approve on each decision. Time per decision is a few seconds. They are not actually checking. They are clicking through.

The structure shows human approval. The protection is missing. The person clicking is not the careful reviewer the system was designed to need.

Pattern 3: Choosing human out of the loop on test results

The system performed well in testing. The team picks human out of the loop on the strength of those results. The system goes live.

In real use, the system meets cases the testing did not cover. Patterns shift. New kinds of input appear. Without humans involved, the team finds out about problems only when they have already caused harm.

Testing is not the same as real use. A system that passed tests is not a system you can leave alone. The decision to go human out of the loop should be based on real-world performance over time, not test results.

Pattern 4: Missing the affordability problem

The oversight the system genuinely needs is not the oversight the organisation can afford at the planned volume.

The honest response is to reshape the system or the volume. Either reduce what the system does. Reduce how much it does. Or accept that more reviewers are needed and pay for them.

The common response is weaker oversight than what is needed. Less time per check. Lower-skilled reviewers. Fewer of them. The cost saving is real. So is the gap between the oversight the system needs and the oversight it has.

Pattern 5: Treating oversight as a one-off design choice

The oversight gets designed at launch. It is meaningful at launch.

Then volume grows. Reviewers get used to the system. Other pressures shift. The oversight that worked at launch quietly becomes oversight in name only. Nobody notices because nobody is checking. The original design was right. The current state is not.

Oversight needs to be looked at again as the system runs. Not just at launch. Whether what was designed is still doing the work has to be a question the team keeps asking.

Seeing It in Action

A health insurance company has brought in an AI tool to review claims. The tool looks at each claim, decides whether it should be approved, rejected, or sent for human review, and acts accordingly. Most claims get approved automatically. A small share get rejected automatically. The rest go to a team of reviewers.

This has been running for eighteen months.

The first pattern, confidence based on testing, was there at launch. The system was tested on six months of historical claims. It performed well in testing. The team decided that automatic approvals and rejections were safe.

The fifth pattern is also there. Nobody has gone back to check whether the system is still performing the way it was at launch. The team has not asked the basic questions. What does the rejection rate look like now compared to the start? Are the same kinds of claims being rejected? Are appeals telling them something they should be hearing?

In the past three months, the rate of rejected claims has been climbing. The team has not noticed because nobody is watching the rate. A pattern of false rejections is building up. Customers are appealing more often. The appeals team is overwhelmed. Some appeals are taking weeks to resolve.

When this is finally noticed, neither pattern on its own caused the failure. The system was overconfident at launch. The oversight stopped being checked. Together, the two patterns have produced months of bad decisions that nobody was set up to catch.

What you can see is that the patterns interact. The first pattern set up the system without enough oversight. The fifth pattern meant the lack of oversight was never questioned. Either pattern alone might have caused smaller problems. The two together produced a real failure.

Summary

Working alongside AI well starts with the principle that AI should add to human work rather than replace it. The right oversight model depends on the situation, the volume, and what happens when decisions are wrong. What matters most across all of this is whether the oversight is real, and whether it stays real as the system runs. Oversight is not something you sort out at the design stage and forget. It is a living arrangement, and at this level part of the work is checking that it is still doing what it is supposed to do.