AI and automation systems get things wrong. Some wrong outputs are isolated mistakes. Others repeat because of a continuing fault. And some produce an unfair pattern that affects one group of people more than another. These are different problems, they have different causes, and they need different responses. Telling them apart, and knowing where they come from, is what lets you deal with each one properly.

You work directly with these systems, so a large part of your job is to notice when something is wrong and report it to the right person, before a fault causes harm.

In this section you will learn:

• how to tell the difference between an isolated error, a repeated fault, and a pattern that treats a group of people unfairly
• how the data a system uses can introduce error and bias
• how the way a system is built and set up can introduce error and bias
• how configuring a system, or using it outside what it was designed for, can introduce error and bias
• who is affected when a system goes wrong, and what it costs both the people involved and the organisation

Error and bias are not the same thing

When a system produces a wrong result, the first useful question is what kind of wrong it is. There are three kinds, and telling them apart changes what you do next.

An isolated error

An isolated error is a single wrong output that does not follow a consistent pattern. The system is usually right, and then now and then it is wrong in a way you cannot trace to a repeating cause. This can happen because of a problem in the input itself, such as a blurred scan or a mistyped figure. It can happen because of a temporary fault, such as a dropped connection or a service that was briefly unavailable. It can also happen because some AI models produce slightly different outputs each time they run, so the same input does not always give exactly the same answer.

This matters because a single error still needs handling, and how you handle it depends on what it could cause. A one-off mistake in a low-stakes task may need little more than a note. A single wrong rejection, payment, access decision, or release of someone’s information could need to be recorded, corrected and raised straight away. What you do differently is record and check an isolated error in line with your organisation’s procedures, let its seriousness decide the response, and never ignore it just because it has only happened once. If similar errors keep happening, investigate whether they share a cause or pattern. Repetition may point to a systematic error, but that has to be established from the evidence, not assumed.

A systematic error

A systematic error is a wrong output that comes from a continuing cause in the data, the assumptions, the setup or the method. It does not always produce the exact same wrong output. It may instead push the results in a consistent direction, or fail in a predictable way under particular conditions. A forecasting tool might consistently underestimate demand without ever giving the same figure twice. An automation might calculate VAT twice on every transaction that arrives through one particular integration.

This matters because a systematic error keeps happening until someone finds and fixes the cause. Simply retrying it does not help, because the cause is still there. What you do differently is investigate the cause rather than retry and hope. A systematic error is a reliability problem, and notice that it is not automatically about fairness. A system can be systematically wrong for everyone in the same way.

Algorithmic bias

Algorithmic bias is a systematic tendency that produces distorted or unfair outcomes, usually disadvantaging a particular group of people. It is a kind of systematic problem, but with a particular pattern. The system is not just wrong in a general way. It is wrong in a way that goes against some people more than others, often people who share a characteristic such as their age, sex, disability, or where they live.

This matters because bias causes harm to real people and creates legal and reputational risk for the organisation. What you do differently is learn to look at outcomes broken down by group, not just at the overall result, because bias is often hidden in the overall figure and only shows when you compare one group with another.

Why a consistent system can still be unfair

It is tempting to think that a system which treats every input by the same rules must be fair. Consistency is not the same as fairness. A system can apply the same logic to everyone and still produce worse results for one group, because the logic itself, or the data behind it, already favours one group over another. A system can also be highly accurate on average and still be much less accurate for a smaller group whose cases it rarely saw. When a system is right most of the time, it is easy to miss that it is wrong for the same people every time.

See it in action

A recruitment team uses an AI tool to screen job applications. One week it rejects a strong candidate because their CV was saved in an unusual layout the tool could not read properly. That is an isolated error, a one-off caused by an odd input, and it does not happen again. The team records it and, because a wrong rejection matters, checks that the candidate is put back into the process.

Over the next month, the team notices that the tool subtracts one month from every applicant’s employment history because of an error in its date calculation. That is a systematic error. It affects every application in the same way, and it will continue until the calculation is corrected.

Later, someone checks the shortlist rates by group and finds the tool recommends applicants from one part of the city at half the rate of applicants from another, even for similar experience. That is algorithmic bias. It is systematic, and it consistently disadvantages one specific group. The overall pass rate looked fine, which is exactly why no one had spotted it until they looked group by group.

When the data causes error and bias

A system learns from, or runs on, the data it is given. If the data is wrong or does not fit the situation, the outputs will be too. Two problems in the data are worth knowing well.

Poor-quality data

Data quality is about whether the data is correct and complete enough to rely on. Data can be inaccurate, holding values that are simply wrong. It can be incomplete, missing information the system needs. It can be inconsistent, recording the same thing in different ways in different places. It can be out of date, describing how things were rather than how they are now. And it can be poorly labelled, which matters for any system that learned from examples that a person tagged as correct or incorrect.

This matters because a system will often work from flawed data as if it were correct. Some checks can catch some problems, such as missing values, invalid formats, obvious outliers, or fields that are missing or in the wrong order, but only where those checks have been built in, and no set of checks reliably catches every quality or fairness problem. What you do differently is treat the data as something to check, not something to assume. If the outputs look wrong, the data behind them is one of the first places to look.

Data that does not match where the system is used

A system built on one set of data can fail when it is used in a different situation. This is a mismatch between the population, time period or setting the data came from and the one the system is now working in. A tool trained on customers in one country may misjudge customers in another. A tool built on last year’s patterns may misread this year’s. A tool that learned from one type of case may do badly on a type it rarely saw.

This matters because the mismatch is easy to miss. The system was genuinely accurate where it was built, so its results look trustworthy, and the drop in accuracy only appears once it is used somewhere its data did not represent. What you do differently is ask where a system’s data came from and whether that matches where it is being used now, rather than assuming that accurate-in-one-place means accurate-everywhere.

See it in action

A finance team sets up an automation that flags expense claims as likely errors. It was built using two years of claims data from the head office. When it is introduced at a regional site, it starts flagging a high share of ordinary claims as suspicious. The regional site has different suppliers and different normal costs, none of which were well represented in the head-office data the tool learned from. The tool is not broken. It is being used on a population its data did not cover, and the mismatch is producing a large number of wrong flags.

History, labelling and proxies in the data

Beyond quality and fit, the content of the data can carry bias into a system even when the data is accurate. Three sources are common.

Historical Patterns

Historical patterns in the data

Data recorded from the past carries the decisions people made in the past, including unfair ones. If a system learns from records of who was hired, promoted or approved, it learns the patterns in those decisions. Where those past decisions favoured some groups over others, the system picks that up as the thing to copy.

This matters because the system does not know the history was unfair. It treats the past as the target to reproduce. What you do differently is recognise that “the system just learned from real data” is not a defence of fairness, because real data can record real unfairness.

Labelled

How the data was labelled

Many systems learn from examples that a person marked as one thing or another, such as good or bad, approved or declined, relevant or not. Those labels are judgements, and judgements can be inconsistent or biased. If the people labelling the data applied a stricter standard to some cases than others, the system learns that stricter standard.

This matters because the labels look like neutral facts once they are in the data, even though they were opinions when they were made. What you do differently is remember that labelled data is only as fair as the labelling behind it.

Proxy Features

Proxy features

A proxy is a piece of information that stands in for something else. A proxy feature is a detail the system is allowed to use that is closely tied to a characteristic it is not supposed to use. A postcode can act as a stand-in for ethnicity or income. The name of a school can act as a stand-in for background. Even when the system is never given the sensitive characteristic directly, it can reach the same result through the proxy.

This matters because removing the obvious sensitive field does not remove the bias if a proxy for it is still in the data. What you do differently is look past the fields a system openly uses and ask what those fields might indirectly be standing in for.

See it in action

An organisation builds a tool to predict which customers are worth offering a discount to, based on past sales records. It is not given anyone’s age. It is given the products each customer browses and the time of day they log in. In use, it consistently offers worse deals to older customers. The browsing and timing patterns were acting as proxies for age, so taking the age field out changed nothing. The unfair result still occurred, this time through those proxies.

Choices in how the system is set up

Two systems can use the same data and still behave differently, because of choices made when they were built. These choices together make up the system’s methodology, the way it is built, and each one can change what the system gets right and who it gets wrong.

How the problem is framed and what the system predicts

Every system is built to answer a specific question, and the exact question is a choice. A tool built to predict “who is likely to leave their job” is a different tool from one built to predict “who is struggling and could be supported”, even if they run on the same records. What the system is set to predict, called its target, shapes everything it does.

This matters because a badly chosen question produces a system that is accurate at the wrong thing. What you do differently is ask what a system was actually built to predict, because a system can be accurate at exactly what it was set to predict and still be set to predict the wrong thing.

Which features it uses

The features are the pieces of information the system is allowed to consider. Choosing them is a judgement, and it affects both accuracy and fairness. Leaving out something important makes the system less accurate. Putting in something that acts as a proxy can build in bias.

This matters because the feature list decides what the system can and cannot take into account. What you do differently is treat the question “what is this system allowed to look at” as a fairness question, not only a technical one.

Where the decision threshold is set

Many systems produce a score, and then a threshold decides what counts as a yes or a no. Where that threshold is set is a choice, and moving it changes who passes. A stricter threshold rejects more people, including more of anyone the system already scores low, which is often the group the data disadvantaged.

This matters because the same system can be fairer or less fair depending only on where the threshold is set. What you do differently is recognise that the cut-off is a decision someone made, not a fixed fact of the system.

See it in action

A team sets up a scoring tool that gives each applicant a number, and anything below a set threshold goes to an automatic decline. To cut the number of cases staff have to review, someone raises the threshold. More applicants are now declined automatically. Because many of the group the data already scored lower had scores just above the old cut-off, far more of that group is now declined than before. Nothing in the model changed. Only the threshold moved, and the threshold was a choice someone made.

Choices in the method and how the data is handled

Alongside how the problem is set up, choices about the method and the handling of the data shape the results.

See it in action

A tool is tuned to be as accurate as possible overall, and it reaches a high overall score, which looks like success. Most of its cases come from one large group, so it can score well by doing well on that group, while being much less accurate for a smaller group whose cases barely move the total. Because success was measured as overall accuracy, and not as accuracy for each group, the overall figure did not reveal the poor performance for the smaller group. The choice of what to optimise decided what stayed hidden.

Choices in configuration and use

Much of this kind of work is setting up and connecting existing tools rather than building models from scratch. That work carries its own sources of error and bias.

Prompts, rules, connections and configuration

When you configure an automation or an AI tool, you make choices that shape its outputs. An unclear or leading prompt makes the tool produce a one-sided answer. A rule or a piece of workflow logic that is slightly wrong runs that mistake every time. A connection that maps one system’s fields to another’s incorrectly moves the wrong data into the wrong place. A setting left on the wrong option changes what the tool does without anyone noticing.

This matters because these are repeated, systematic sources of error, and they are caused by the person doing the configuring rather than by a model somewhere else. What you do differently is treat your own configuration as something to test properly, because a fault here repeats on every run.

Using a system outside what it was built for

A tool built for one purpose or one group of people can produce error and bias when it is used for another. The tool may work well within its intended use and fail outside it without warning, because it was never designed or checked for the new situation.

This matters because using a system out of scope is an easy decision to make and a hard failure to see. What you do differently is check what a tool was actually designed for before applying it somewhere new, rather than assuming that a tool which works well in one place will work well everywhere.

See it in action

A team uses a generative AI assistant to answer staff questions about an internal policy. Someone asks a detailed question the policy does not cover. Instead of saying it does not know, the assistant produces a confident, well-written answer that is wrong. This is not a configuration mistake. A generative model produces its answer by predicting likely wording from patterns it has learned, and it does not, in itself, check whether what it produces is true. Even a system that looks information up before answering can still be wrong, because looking something up does not guarantee the answer is correct. So the tool can produce something that sounds right, and is written confidently, while being false.

The team also finds that when they set the tool up, a rule in the workflow sent certain question types to the wrong knowledge source, which produced repeated wrong answers on those topics. One problem came from how generative AI works, the other from how the tool was configured, and both were producing errors at the same time.

Who is affected, and what it costs

When a system produces error or bias, more than one party can be affected. It helps to be clear about who, because the person using the system is often not the person harmed by it.

The different people affected

There is the person operating the system, who runs it and acts on its outputs. There is the person subject to its output, the applicant, customer or member of staff the decision is about, who often never sees the system at all. There are other people affected indirectly, such as the family of someone refused a service, or colleagues who inherit the consequences. And there is the organisation itself, which carries the responsibility and the consequences.

This matters because the operator and the affected person are usually different people, and the operator can be satisfied while the affected person is being harmed. What you do differently is think about the person an output is about, not only the person in front of the screen.

Why an automated tendency can affect so many people

A single person making decisions affects the people they personally deal with, and they vary from day to day. An automated tendency is different, because it can affect a very large number of cases quickly and repeatedly. A set of fixed rules may treat every matching case in exactly the same way. A system that works on probabilities may vary more from one case to the next, but it can still apply the same tendency across very many cases, far faster than a person could. If the tendency is wrong or unfair, the reach of it is what makes it serious.

This matters because it changes how seriously to take a pattern. What you do differently is treat a bias found in a widely used system as a large problem, not a small one, because of how many people it can reach.

The cost to the organisation

The harm is not only to individuals. An organisation can be taken to court or fined where a biased or faulty system breaks the law. It faces reputational damage when the problem becomes public. It faces operational cost when work has to be redone, complaints handled, or a system withdrawn. And it faces a loss of trust, from staff and customers, that takes a long time to recover.

This matters because it shows that getting this right is not only an ethical concern but a practical one for the organisation. What you do differently is raise a suspected error or bias as something with real business consequences, which is more likely to get it acted on.

Bias is more than a data or algorithm problem

It is easy to treat bias as purely technical, something contained only in a dataset or a model. In practice, the way a system is chosen, set up and relied on reflects human decisions and organisational assumptions. A system used without question, or trusted beyond what it can actually do, turns a technical flaw into a real harm. The people and the context around the system are part of where bias comes from.

This matters because a purely technical view can miss the cause. What you do differently is look at how a system is being used and trusted, not only at the data and the model, when you are trying to understand why something is going wrong.

See it in action

A manager uses a tool that assigns shifts. The manager thinks it works well because rotas are produced quickly and with little effort. The people the tool actually decides for are the shift workers. Over time it gives the least convenient shifts to the same few people, because of a pattern in the past rotas it learned from. The manager, who operates the tool, sees a smooth process and is satisfied. The workers, who are subject to its decisions, experience the unfair outcomes, and at first no one operating the tool notices. When it is raised as a grievance, the organisation must handle the complaint, spend the time to put it right, and repair the damage to trust with its staff.

Your part in this

You work closely with these systems, so a large part of your job is to notice error and bias and report it to the right person. It helps to be clear about what that means, because not all of it is yours to decide.

Some things are yours to do directly. When you see a wrong or unfair output, record it in line with your organisation’s procedures and report it, together with what you noticed and any pattern you have seen.

Some things need other people’s knowledge. Judging whether a difference between groups is lawful, whether a data-protection rule applies, whether a system meets an equality duty, or how to change a model, needs technical, legal, data-protection, equality or governance knowledge that usually belongs to other roles.

And some things are not yours to decide on your own. Whether a system is fair enough to keep using, or should be paused or withdrawn, is a decision for someone with the authority to make it. Your part is to give them a clear, honest account of what you found, so that they can decide well.

Summary

Error and bias are not the same problem. An isolated error is a one-off with no repeating pattern, and it still needs recording and checking according to how serious it could be. A systematic error comes from a continuing cause and pushes the results wrong in a consistent way until the cause is fixed. Algorithmic bias is a systematic tendency that treats some people worse than others. A system can be consistent, and accurate on average, and still be unfair to a group.

These problems come from real choices, and knowing where they enter is what lets you spot them. They come from the data, through poor quality, poor fit, history, labelling and proxies. They come from how a system is built, through the question it is set, the features and threshold it uses, the method chosen, what it is optimised for, and how its data is prepared. They come from how a tool is configured and whether it is used within what it was designed for. When they occur, they reach the person subject to the output more than the person operating it, they can affect many people quickly, and they carry real legal, reputational and operational cost for the organisation. Much of this is not yours to fix or decide alone, but it is yours to notice, to record, and to raise.