AI Needs Assurance
As developers we are very focused, to the point of being obsessed about the LLM that steals our living. It is understandable to be worried, but also hopeful. But there is another angle which is as important: it is not about the AI that writes code, but the AI that is called by the traditional code to help with tasks which are either onerous to code with traditional programming languages, or plainly impossible.
So the universe I will describe is the one that uses generative AI (more precisely LLMs) at runtime.
Andrej Karpathy described the change quite nicely, by naming three types of software (Karpathy 2025):
- Software 1.0 is what we’re very used to. It has code, with
if-s, loops, functions, objects, etc., and it is defined by instructions written as code in one of the many programming languages. The code compiles to programs and runs on a computer. You need precision and improve the output by optimising instructions. - Software 2.0 began with neural nets, and it is defined by weights. Developers train a model, it feeds it data, and the improvements happen when weights, not the instructions.
- Software 3.0 is made possible by generative AI, and is defined by prompts. They interpret natural language inputs, and the runtime can be thought of as being the LLM itself (although ultimately it is a GPU and a CPU inside a computer, still). I am forcing the vocabulary here, because a large language model is fundamentally just a file full of numbers, it does not do anything by itself.
Source: (Karpathy 2025)
One interesting feature of the LLMs and associated services, such as Anthropic, OpenAI or DeepSeek is that they are quickly becoming common infrastructure.
And common infrastructure needs assurance to provide safety and trust. The next logical step is obvious: AI needs assurance.
Fortunately there are historical precedents: no one is asking the electricians to be more virtuous when wiring your house. There are certifications, inspections, and standards. Airbus, Boeing, Ryanair or Heathrow Airport have strict standards, but still there are regulatory demands that need to be met before allowing planes to take off. Same with genomics, nuclear, healthcare, and many other industries. There are regulations, audits, and formal verifications.
“In the case of AI, assurance measures, evaluates and communicates the trustworthiness of AI systems.” (‘Introduction to AI Assurance’ 2024)
Generative AI is new, highly complex, and evolving rapidly, but efforts to establish a control structure exist and are quickly coalescing, at least in the UK, into a very coherent structure, with a roadmap and clear deliverables. Details are the subject of a different article, but there is a Minister for AI, an AI Security Institute, there are also an AI Assurance Stakeholder Consortium led by BCS, CETAS of Alan Turing Institute, but also British Standards Institution (BSI), National Physical Laboratory (NPL) and other important British institutions.
There are so many actors because there are so many aspects of AI Assurance: governance, ethics, transparency, legal compliance, cyber security, safety, and today’s main topic, AI Evaluation.
AI Evaluation is one fundamental building block for AI Assurance. AISI defines it as “An evaluation is a structured, controlled process for measuring a property of an AI system. That property might be a capability, a tendency, or a risk.” (‘Evaluation Framework’, n.d.).
So a systematic measurement of the system’s quality, or, plainly put, they are tests that tell if this system is doing what it was supposed to do. This systematic measurement should also output systematic reports that also act as evidence.
Testing non-deterministic software is no easy feat. LLMs and the agentic system they power are powerful, but also unpredictable. As opposed to a traditional software that crashes or returns errors which are easier to catch, these new systems have the habit of failing silently.
There is also the nature of many of these agentic systems: they act in your name, and you are directly accountable for their real-world actions.
Deployed AI systems have promised discounts and cited non-existent legal cases. Medication advice is no safer: in one study of chatbot answers to patients’ drug questions, healthcare experts judged 66% of a preselected subset of inaccurate answers potentially harmful, and 22% potentially severely harmful or deadly (Andrikyan et al. 2025).
Such facts as these are not designed, to give someone great peace of mind.
I will need to deliver a piece of stochastic software in a few months. Its performance will be public, on a site where the logo of UCL and a leading music corporation will be displayed.
What should I, a responsible engineer do? There is an answer in the wisdom of Mark Watney:
“So, in the face of overwhelming odds, I’m left with only one option: I’m going to have to science the shit out of this.” — Andy Weir, The Martian
And this is how I got into AI Evaluations.
All the mishaps I mentioned above could have happened while traditional testing and observability reported nothing unusual: no 404s, uptime 100%, acceptable latency, no ERROR-s in logs or traces. Everything looked like it was operationally in perfect health, while still producing dangerously wrong answers.
Traditional testing tells us if the system is correct against clear specifications. Traditional observability tells us how the system is doing. Non-deterministic systems need some extra layers that tell us if the system does the right thing.
After shipping, in production, chats and agents need to be traced (LLM calls, tool calls, control flow) so that they can be monitored and debugged.
Before shipping a prompt, model, or agent we need to answer an important question: is this system reliable and safe enough to run in production?
This is where AI Evaluation comes in, and it has a lot in common with the kind of testing we’re used to: we run the system against a fixed set of inputs, with as many exceptions and stressful edge cases as we can, and compare the output with what we expect to get.
Yet, the nature of both inputs and outputs is quite different, which renders assert equals useless:
- Inputs are often ambiguous (natural language, images, text) and the context can change everything. We need to think about and collect representative inputs, expected behaviour and challenging edge cases into evaluation datasets
- There could be many acceptable outputs, none “right” or “wrong”, and this output needs to be judged, ideally in an automated way: by a reliable deterministic module, a human, or, other LLMs. This mechanism is called a scorer.
- Even if the same input is given to the same model, the outcome can be different across runs. This means that the evaluation needs to be performed many times against different inputs, models, and prompts.
- Scorers themselves can be unreliable, especially when the judge is another LLM. Humans-in-the-loop are no safe fallback either, because extended use of AI systems degrades the skills that oversight depends on (Mitchell et al. 2026).
- A good score does not mean that the system is behaving well. The cases tested need to be continuously updated with new observations and failure modes.
So a lot of moving parts for a team to manage in a disciplined way. Fortunately AISI, the UK’s AI Security Institute released an open-source framework to help with just that.
It is called Inspect AI and it has composable building blocks – datasets, agents, tools, scorers–, pre-built evaluations, tooling for viewing evaluation results, connectors to most AI labs, support for multi-step evaluations, and a sandboxing system for untrusted models.
It runs the evaluations, computes the scores, and saves the evidence in detailed logs.
To see inspect AI at work, visit github.com/cdinu/ai-evaluation-thermostat-basic