Most people treat an AI system as a magic black box that produces outputs. At this level you are expected to think differently. An AI system is not a piece of software that arrives fully formed. It is built by people, in stages, against specific decisions. Every one of those decisions is still showing up in what the system does today. Knowing how an AI system came to be is what lets you think about it clearly, instead of treating it as something that just appears.

In this section you will learn:

• how the way an AI system is framed and the data behind it shape what the system can do
• what training and testing mean, and how each of them can go wrong
• what changes when an AI system is deployed and starts running in real work
• how modern AI systems are built from foundation models, with patterns like RAG, fine-tuning and agents layered on top
• what to expect when the system you are using was built by a vendor and most of its history is hidden from you

How an AI system begins

Every AI system starts with someone deciding there is a problem an AI system might help with.

That decision matters more than it sounds. The way the problem is framed shapes everything that follows. A system built to “predict which customers are likely to cancel” is a different system from one built to “find customers who are struggling so the organisation can support them.” The data underneath might be the same. The system you end up with is not.

The framing is the first shaping act. After that point, every later decision is made inside the shape the framing has set.

What good problem framing looks like

Good framing is specific enough that a system can actually be built against it. The purpose is clear. The people who will be affected by the system’s outputs are part of the thinking from the start. The constraints, including legal and ethical ones, are included in the framing itself, not bolted on later.

This sounds straightforward and often is not. Real framing decisions get made under time pressure, with incomplete information, by people who may not have thought through how the system will affect the people it gets used on.

How framing goes wrong

Framing goes wrong in three common ways.

The first is framing a problem the data cannot actually answer. Someone wants the system to “predict employee burnout” but the only data available is sick days and survey scores. The framing assumes the data contains a signal. When you look closely, it does not.

The second is framing for technical convenience rather than for usefulness. The problem is shaped to fit a model that already exists, or to fit data that is already available. The system gets built and works as a piece of technology, but does not really do the job it was supposed to do.

The third is framing that quietly assumes things about the people it will be used on. That they behave in a certain way. That their data is representative. That what happened in the past should keep happening. These assumptions sit inside the framing and often go unchallenged.

What the data does to the system

Once the problem is framed, data is gathered to build the system. The system is built from that data. So the data shapes what the system can do.

Three things about data are worth holding onto.

The first is what the data shows. Data is never a neutral record of the world. It is a record of what was captured, in the situation it was captured in, by the people and systems doing the capturing. A set of past hiring decisions is exactly that. A record of the hiring decisions that were actually made, including any patterns of advantage or disadvantage in those decisions.

The second is what the data leaves out. Every dataset is a selection. The people, situations and outcomes that are not in the data are as important as those that are. A system built on data from one country, one industry or one group of people will often quietly fail when used outside that group.

The third is the quality of the data. Accuracy, completeness, consistency, freshness. All of these shape what the system can do. Poor data produces a poor system, however good the model on top of it.

See it in action

A logistics company wants an AI tool to predict which deliveries will be late.

The team starts by framing the problem. They decide what counts as “late,” which deliveries are in scope, and which are not. They settle on a definition that fits the data they already have, rather than the definition the operations team would actually use. That decision is quiet but it shapes everything afterwards.

Then they gather the data. They use two years of past delivery records. The records cover a period when the company was running unusually smoothly. There were no major weather disruptions, no fuel shortages, no labour issues. The data does not contain the kinds of disruption the tool will need to handle once it starts being used for real work.

What you are watching here is the system being shaped before any model has been trained. The framing has narrowed what “late” means. The data has captured a smooth period and missed the bumpy ones. By the time the modelling team starts building, a lot has already been decided about what the system can do. Most of those decisions happened in two meetings and a query against a database. Neither of those moments looked like AI work.

Training and testing

Once the problem has been framed and the data is in place, the system gets built. Two things happen at this stage. The system is trained, and then it is tested.

How a system gets trained

Training means processing the data through a learning algorithm so the system picks up the patterns in the data. There is a lot inside that sentence. The basic idea is that the system is being fitted to the data it has been given, so its outputs reflect what was in that data.

Most modern AI work does not start from a blank model. It usually starts from a model someone else has already built. There are two main patterns you will meet.

The first is using a foundation model as it comes. A foundation model is a large general-purpose model built by a provider such as OpenAI, Anthropic, Google or Meta. The organisation does not train anything new. They send their input to the model through prompts and use the output. All of the training happened on the provider’s side, before the organisation got involved.

The second is fine-tuning a foundation model. The organisation takes a foundation model and trains it further on their own data, so the model gets better at a specific job. The base model came with general capability.

The fine-tuning makes it better at things like:

• writing in the organisation’s voice
• handling a particular kind of customer query
• recognising a particular type of document

In both patterns, the main work of training has already been done by the provider. The organisation either uses what the provider built as it is, or adds a layer on top of it.

What testing means

Before the system goes anywhere near real work, it is tested. Testing answers three questions. Does it do what it is supposed to do? How well does it do it? What kinds of error does it make?

The principle to hold onto is this. Testing tells you how the system behaved on the cases you tested it against. It does not tell you how it will behave on cases you did not test against. That sounds obvious and is often forgotten.

Good testing reflects the range of real cases the system will face. Easy cases, hard cases, unusual cases, the full mix. Poor testing only checks the system on a narrow slice of cases that look a lot like what it was trained on. That tells you very little about the rest.

Two ways testing goes wrong

Testing goes wrong in two common ways.

The first is testing on data that is too similar to the training data. The test set and the training set are drawn from the same source, in the same period, with the same kinds of input. The system performs well on the test because the test looks like what it has already seen. Once it meets real inputs, performance falls.

The second is testing on a narrow slice of the real situation. The system handles the common case well. The uncommon case is where most of the actual problems will turn up, and those cases were not tested. The system passes its tests and then starts producing strange results once it is being used for real work.

See it in action

A customer service team fine-tunes an AI tool on thousands of past customer conversations. The aim is to help staff draft replies more quickly.

The team runs their tests. The tool performs well. It is producing replies that look sensible and are graded as accurate by the test reviewers.

Look closely at what was tested. The test data was drawn from the same archive of past conversations as the training data. Both sets are full of the same common kinds of enquiry. Refunds, password resets, delivery questions. The system has been trained on those and tested on those, and it handles them well.

What is missing from both the training and the test set is the harder end of the work. Complaints with mixed messages. Customers in genuine distress. Enquiries that mention a fraud concern. Cases where two policies seem to apply at once. None of those are in the test set.

What you are watching here is a system that looks ready to deploy because the test results are good. The test results are good because the test was the wrong test. Once the tool is in front of staff handling real conversations, the gap between what was tested and what is actually arriving will start to show.

Deployment and operation

At some point, the system stops being something that is being built and tested, and becomes something that is being used. That moment is called deployment.

Deployment

Deployment is a specific point in time. Someone, somewhere, has signed off that the system is ready for real work. From that moment, the outputs of the system are doing real things. They go to customers. They flow into other systems. They feed decisions that have real consequences.

Before deployment, mistakes are tolerable. A bad output goes into a test report. After deployment, mistakes affect real people.

How deployment goes well, and how it goes badly

A deployment that goes well is a deliberate decision. Staff have been prepared. The situations the system will face have been thought through. There is a plan for what happens if the system produces something unexpected. Someone is clearly accountable for the system once it is running.

A deployment that goes badly often has no clear sign-off moment. The system was being tested, and somehow people started relying on its outputs while the testing was still going on. No-one has formally agreed it is ready. Staff are not sure whether they are evaluating the system or depending on it. When something goes wrong, it is not clear who owns the response.

The contrast matters. A system that has been deliberately deployed has someone responsible for it. A system that has slipped into use without a clear decision often has no-one.

Operation

Once the system is deployed, it operates. It takes inputs, produces outputs, and affects real work day after day.

In good practice, more than the system itself sits around a running AI tool.

• monitoring of what the system is actually doing, including its outputs, its errors, and the unusual cases that test its limits
• human oversight at the points where it matters
• a feedback loop from the people using the system, telling those running it what is and is not working
• a way to change the system or pause it when needed

In poor practice, very little of that is in place. The system was working at deployment, and then nothing. No-one is checking how it behaves over time. No-one is collecting feedback. No-one has a way to pause it quickly if something goes wrong.

Why running systems can quietly stop working

A running AI system can quietly drift. The world it lives in keeps changing. The inputs change shape. The data the system was trained on becomes less and less like the data it is now seeing. Without monitoring, the drift is not noticed until something visible breaks, by which point the system may have been producing poor outputs for months.

Drift is not a sign that the system was badly built. It is a sign that the world has moved on while the system has stayed where it was. That is true of all AI systems eventually.

See it in action

A mid-sized organisation has a deployed AI tool that produces short customer-facing summaries of policy documents for a self-service portal.

When the tool was first deployed, it ran well. The summaries were accurate. Customers were finding them useful. The team that built the tool moved on to other work.

Six months later, the situation looks different. The product range has changed. New policies have been added. Some old policies have been updated. The tool is still running. It is still producing summaries. The summaries still look right at a glance.

If you check them carefully, the new summaries are quietly missing important details. The tool is generating sensible-looking text based on patterns from the older policies, and the new policies do not always fit those patterns. Customers reading the summaries are getting an account that is mostly accurate and occasionally misleading.

No-one is watching in a way that would catch this. There is no review of what the tool is producing. There is no feedback loop from customers. The team that built it has moved on, and the team that owns the portal does not see the tool as their problem.

What you are watching here is a deployed system that has quietly stopped doing its job, in a setup where there is no easy way to tell. The system itself has not changed. The world around it has, and nothing was set up to notice.

How modern AI systems are actually built

The picture so far covers the basic shape of an AI system over time. There is one more part to add. Most AI systems you meet in present times are not single-purpose tools built from scratch. They are built using patterns that combine a foundation model with extra pieces layered on top.

Three of those patterns matter most. You will meet them often. They are retrieval-augmented generation, fine-tuning, and agent frameworks.

Retrieval-augmented generation

Retrieval-augmented generation, usually shortened to RAG, is a way of connecting an AI model to a set of documents or a knowledge base.

When a question arrives, the system first searches the documents and pulls back the parts that are relevant. Then it uses the model to produce an answer based on what it found. The answer is grounded in specific sources, which means the system can show where the information came from, and the chance of the model inventing something drops sharply.

RAG is how most modern enterprise AI tools handle company-specific knowledge. The model itself stays general. The knowledge is attached to it from outside.

A worked example. A legal firm has an AI tool that can answer questions about its own past cases. The tool searches the firm’s case archive, finds the cases that are relevant to the question, and produces an answer that cites those cases. The model never had to learn the firm’s case history. RAG gave it access to that history at the moment the question was asked.

Fine-tuning

Fine-tuning takes a foundation model and trains it further on specific data so it gets better at a particular job.

Fine-tuning is more expensive and slower than RAG. It is used for different reasons. RAG is the right pattern when you want the model to have access to specific information. Fine-tuning is the right pattern when you want to change how the model behaves. Its style. Its tone. The way it handles a particular kind of input.

A worked example. A healthcare provider fine-tunes a foundation model on thousands of anonymised clinical letters. The aim is not to give the model new information. It is to make the model produce letters in the provider’s house style, which is the way that organisation writes. The model still answers from its general training. It now does so in a way that matches the rest of the provider’s work.

Agent frameworks

An agent framework is the extra software around a model that lets the model take actions, call other tools, and keep working towards a goal across several steps.

When someone says they have built “an AI agent that does X,” what they usually mean is this. There is a model. There is a set of tools the model can call. And there is a framework around them that runs the cycle of work. The model decides what to do next. The framework lets the model do it. The cycle continues until the goal is met or the work is handed back to a person.

Real examples. Claude Code is an agent that can read, write and run code. ChatGPT agent mode lets the model take actions in a browser. Zapier agents sit inside automated workflows and do work the workflow itself was not designed to handle.

Why these three patterns matter together

Most modern AI systems are not just one of these patterns. They are combinations of them, layered on top of a foundation model.

That is the shape of the thing you are usually looking at in 2026. A general-purpose model from a provider sits at the bottom. Several other pieces are layered on top of it.

• RAG, so the system can use the organisation’s own knowledge
• sometimes fine-tuning, so the system produces output in the organisation’s writing style
• an agent framework, so the system can carry out a task across several steps

Understanding the patterns is understanding how the system actually works.

See it in action

A services organisation has built a tool to help its consultants.

The foundation model is a general-purpose one from a provider. The organisation is not training anything new at the model level. They are using what the provider has already built.

The tool uses RAG to connect the model to the organisation’s case archive and to internal documents that describe how the firm runs its work. When a consultant asks about a past project, the tool searches the archive and pulls back the relevant material before producing an answer.

The tool has also been fine-tuned on the firm’s past pieces of work, so its output follows the firm’s style. Reports look like the firm’s reports. Plans look like the firm’s plans.

On top of all that, the tool is built as an agent. A consultant can ask it to draft a project plan for a client engagement similar to the Acme project. The tool will then do several things on its own. It will search the archive for that project. It will look at related ones. It will work out the shape of the plan. It will produce a draft. And it will ask for clarification on the parts where it needs guidance.

What you are watching here is one tool that uses all three patterns at once. The general model gives it broad capability. RAG gives it the firm’s knowledge. Fine-tuning gives it the firm’s writing style. The agent framework lets it take a task with several steps and complete it without being walked through each one. None of those parts on its own would do the job. Together, they do.

That is what most AI tools you meet in your work are made of. Not a single model. A foundation model with patterns layered on.

Retirement, and systems whose history is hidden

The last part of the picture covers two things that often turn up together in real working life.

The first is retirement. Every AI system reaches a point where it stops being useful, and how that ending is handled matters. The second is the case where the system you are dealing with was not built by your organisation, and most of its history is not visible to you.

Retirement

Every AI system reaches a point where it stops being useful. The problem it was built for may stop existing. The data it was built on may have become too out of date. The world may have shifted enough that what the system produces no longer fits the purpose. Retirement is a stage in its own right, not just a quiet ending.

Signs a system should be retired

There are several signs that a system has reached the end of its useful life.

Its outputs start producing decisions that are not as good as the decisions a person would make on the same input. The underlying problem has changed and the system has not kept up. The cost of running it is higher than the value it produces. A newer system has come along that does the job better. Or the law or wider thinking has moved, and continued use of the system is no longer easy to defend.

You do not need to act on these signals at this level. You do need to recognise them when they appear.

What responsible retirement looks like

Retiring a system properly involves four things.

• a clear decision that this is what is happening
• a plan for the work the system was doing, including who or what is going to do it now
• proper handling of the data the system held
• clear communication to the people who relied on the system, so they are not left finding out by accident

A system that just stops being used, without any of that, often leaves traces. Outputs sitting in other systems that pulled from it. Data still in storage. Patterns of work that depended on the system and have not been replaced. Those traces can cause real problems later.

Systems whose history is hidden from you

For most people working with AI, the typical case is not a system the organisation built. Most AI systems used in organisations are vendor-supplied. A vendor is a company that builds and sells the AI tool to other organisations. The model was built by the vendor. The training was done by the vendor. The testing was done by the vendor. The history exists, but it happened inside the vendor, not inside the organisation that is now using the system.

This does not mean the stages are absent. The system still has a problem framing, training data, testing, deployment and operation. It is just that most of those stages happened where you cannot see them.

What you can know, and what stays hidden

Your job at this level is not to reconstruct the hidden history. There are four things to do instead.

• know that the history exists
• ask what can reasonably be asked
• notice what the vendor is and is not telling you
• use the way the system actually behaves to fill in some of the gaps

A system that works well on common cases but falls apart on unusual ones is telling you something about its training data. A system that handles English well and another language poorly is telling you something about its testing.

There is a difference between what is genuinely a black box and what is a question worth asking. The detailed inner workings of a foundation model are a black box for almost everyone, including most of the people building tools on top of it. That is genuinely hidden. The training data the vendor used, the testing they did, the protections they have put in place against known problems. Those are usually questions the vendor can answer. The organisation should usually be asking them.

Knowing the difference between the two is part of what makes you useful in this kind of work.

See it in action

An organisation is using a third-party AI tool to help screen job applications.

The tool was bought from a vendor. The organisation did not build it, train it or test it. The version they are using is the one the vendor sells to many of its other customers.

Look at what the organisation can actually find out about how this tool works.

The vendor will share some things.

• the kinds of data the tool was trained on, in general terms
• the performance testing they have done, including some published results
• the protections they have put in place against discrimination, including which protected groups they have specifically tested for

The vendor is willing to put these in writing.

The vendor will not share other things.

• the exact training data, which is kept confidential for commercial reasons
• the detailed inner workings of the underlying model, which are not theirs to share
• some of what the model does, which even the vendor cannot fully explain, because that is the nature of the model they are using

What you are watching here is what knowing how the system works looks like in practice when most of it is hidden. The organisation is not working in the dark. They are asking the questions a vendor can answer. They are accepting that some questions will not have an answer. They are watching how the tool actually behaves on real applications as another source of information about it.

That is the realistic version of understanding a vendor-supplied AI system. Not full visibility. Not blind trust either. Knowing what you know, knowing what you do not, and knowing which is which.

Summary

An AI system is not a magic black box that produces outputs from nowhere. It is a thing with a history.

It was framed by specific decisions about what problem it was meant to solve. It was built from specific data, with specific things in that data and specific things missing. It was trained, often on top of a foundation model someone else built. It was tested, well or badly. It was put into use at a specific moment. It runs in a specific context, in a world that keeps changing around it. Eventually it will be retired.

When the system was built by a vendor, most of those stages happened where you cannot see them. They still happened. The shape of the system you are using was set by them.

Comparing any AI system you meet to that picture is what allows you to think clearly about it, instead of treating it as something that just appears. That is the picture to carry into the rest of the work.