Explained · 2 min
What is an AI evaluation?
How labs and outside groups test what an AI model can do, and why an outside check counts for more.
Written by AI and checked against our editorial standards. Not reviewed by a human editor.
An AI evaluation, often called an “eval”, is a planned test of what an AI model can do and what could go wrong with it.
Some evals are simple. A model gets a set of tasks, such as coding problems or exam questions, and gets a score. That kind of test is called a benchmark. Other evals look for risks: can the model help someone attack a computer, or does it try to trick the people testing it?
Think of it like this: a car gets crash tests before it is sold. The car maker runs its own tests, but buyers trust the result more when an independent test centre crashes the car too.
AI labs test their own models before each launch. But a lab has a reason to like its own results. That is why outside groups, such as METR or national AI safety institutes, also run evals. Anthropic has said that a strong system of such outside tests is needed to judge how capable and risky models are.
An outside eval is only as good as its access. If the testers cannot try the model, or the lab hides part of the evidence, they have to say so. Then readers must rely more on the testers’ judgment.
Good evals are also hard to build. When the best models score high on every test, the tests stop telling them apart, and new ones are needed.
Why it matters to you: an eval is how anyone outside a lab can know what a new AI can really do. When you read about a test result, ask who ran it and what they were allowed to see.
Learn more: Anthropic: A new initiative for developing third-party model evaluations · METR: Independent assessment of AI R&D acceleration at Anthropic