What Are the Goals for AI?

In “Escaping AI’s Measurement Trap” (Issues, Summer 2026), Urs Gasser, Viktor Mayer-Schönberger, and Fabienne Marco make a compelling case that governance of artificial intelligence faces a problem. Once benchmarks become targets, systems and their developers adapt to them, and measured performance may increasingly diverge from behavior in the real world. Their argument leads to a broader question: How can democratic societies not only improve the oversight of increasingly optimized AI systems, but shape what gets optimized in the first place?

The Volkswagen diesel emissions scandal the authors discuss helps illustrate the distinction. Emissions tests assess whether cars comply with limits that were first established through societal discourse and regulatory processes. Engineers can innovate and optimize their products within these constraints. With AI, this sequence is currently reversed. Increasingly powerful systems are built while many of the objectives, boundaries, and trade-offs that should govern their development remain unsettled.

Without societally defined goals and constraints, optimization has no inherent preference for democratic values. Indeed, many principles and practices that democracies value can appear inefficient from an optimization perspective: Deliberation takes time, pluralism complicates decisions, checks and balances slow action, and individual autonomy makes behavior less predictable. These are not shortcomings but mechanisms through which free societies limit power and preserve human agency.

We need to move democratic governance further upstream. It is not enough to conceive of fundamental rights and the public interest merely as guardrails around technological innovation. Guardrails tell us where AI should not go; they do not tell us where we want it to take us. We need a shift toward values by design, with public interest and human empowerment embedded throughout the development process, from university education and research to the objectives, constraints, and design of AI systems.

Public interest provides the collective goal: AI must contribute to objectives that societies democratically determine to be desirable, rather than merely serve private or commercial interests. A human imperative provides the complementary goal: Technological progress must expand people’s capacity to understand, decide, contest, create, and act rather than progressively transfer these capacities to machines. AI should not merely avoid harming human agency; strengthening human agency must itself be an objective.

Gasser and colleagues show why governing AI through measurement must remain adaptive. But there is a step that logically precedes measurement: deciding what we want AI systems to achieve, what values they should preserve, and what boundaries they should respect. These are normative choices, not technical ones. Where appropriate, they can be translated into binding obligations, giving measurement a clearer regulatory reference point and making meaningful accountability possible. After all, Volkswagen ultimately faced billions of dollars in penalties and other costs for fraudulently circumventing emissions standards that were binding and enforceable. But when even the most adaptive measurement reveals that AI systems are harming society or eroding human capabilities, will there be any meaningful consequences without comparable standards?

Urs Gasser, Viktor Mayer-Schönberger, and Fabienne Marco raise a problem that reaches beyond AI benchmarks and evaluation methods. Governance of artificial intelligence increasingly relies on feedback loops: monitoring risks as technology and social conditions change, and continually revising responses. But no matter how quickly those loops operate, if the evaluation processes underlying the feedback fail to capture reality, governance may simply adapt more quickly in the wrong direction.

Since 2020, I have been involved in developing the Japanese government’s framework for “Agile Governance.”  We have emphasized that agile governance does not simply mean changing rules more frequently. The framework proposed two feedback loops: an operational-level loop for assessing and responding to risks from AI systems, and a management-level loop for reviewing and improving the rules, organizational structures, and processes that support those activities.

Gasser and colleagues show that this management-level loop needs to explicitly encompass another critical object of scrutiny: the measurement and evidence infrastructure itself. It is not enough to ask whether a risk assessment has been conducted. We must also ask whether the metrics used in that assessment still capture the reality they are intended to measure. Do improvements on benchmarks translate into improvements in real-world performance? Have metrics lost their meaning because they have themselves become development targets? Are harms or affected stakeholders falling outside the existing evaluation framework? Agile governance must govern its own feedback loops.

From this perspective, the authors’ concept of “governing by learning” can be understood as an important deepening of agile governance. The framework we developed in Japan was intended to subject to continuing scrutiny not only risks and responses, but also rules, organizational arrangements, and even the goals of governance themselves. The measurement trap adds a crucial insight: The instruments that make such scrutiny possible cannot be treated as neutral yardsticks. As AI systems and their developers adapt to evaluation, the reliability of those instruments can itself deteriorate.

The challenge for AI governance, then, is not to keep searching for a perfect benchmark. Institutions should instead draw on different sources of information, including post-deployment evidence, multiple evaluators, incident data, and feedback from affected communities, while continually testing the reliability of the evaluation methods and processes that produce that information. When a metric fails, we should not simply replace it. We should learn from why it failed, use that knowledge to redesign how we measure and evaluate AI, and subject the revised approach to the same continuing scrutiny. The challenge is not merely to build better instruments for observing AI. It is to build institutions capable of recognizing when those instruments have stopped showing us the world as it is.

In the nineteenth century, Lord Kelvin, the Scottish mathematician, physicist, and engineer, famously emphasized the importance of measurement: “When you can measure what you are speaking about, and express it in numbers, you know something about it.” His idea has often been reduced to a simple maxim: “If you cannot measure it, you cannot improve it.”

This principle has shaped modern science, engineering, and management. But artificial intelligence is challenging it in an unexpected way. What happens when the object being measured can adapt to the measurement itself? As AI systems become increasingly capable of recognizing and responding to benchmarks and tests, measuring them becomes more than a technical exercise. Measurement becomes part of the AI ecosystem, raising fundamental questions about how AI should be governed.

These are among the central questions that Urs Gasser, Viktor Mayer-Schönberger, and Fabienne Marco address in their insightful essay. Their discussion points toward a shift from governance based on fixed benchmarks to a system capable of learning, questioning its own measurements, and adapting as AI systems evolve.

The article also opens an important research agenda for understanding institutional frameworks for AI governance. It takes the argument beyond challenges of measurement toward a broader question: How can we build new types of institutions capable of assessing and governing AI under conditions of uncertainty, power asymmetry, and continuous technological change?

The authors’ proposal that regulators and independent auditors should evaluate not only AI systems but also the methods used to evaluate them is compelling. This requires what they call meta-evaluation: continuously asking whether the measures remain reliable. A natural question, then, is what institutions would be needed to carry out this task and how could they be given sufficient authority, resources, expertise, and independence. This is a major challenge because governments and civil society organizations may lack the technical capacity to evaluate rapidly changing AI systems.

The problem is also one of power. Large AI companies control the models, computing resources, training data, evaluation environments, and much of the technical expertise. Regulators, academic researchers, and civil society often have less information and fewer resources. Even sophisticated measurement systems will have limited value if those being evaluated control access to the evidence.

The argument therefore moves beyond the technical problem of benchmarks to a broader question about the design of AI governance. Governance by learning calls for institutions that can continuously evaluate and revise their own methods, bringing together engineers and computer scientists, social and political scientists, legal scholars, and affected communities. Finally, evaluation cannot stop at static benchmarks. AI systems need to be assessed in the real-world environments in which they operate, where technical, institutional, and social conditions are constantly changing. This challenge will become even greater with AI agents, which can act continuously, interact with their environment, and change their behavior over time. The evolution of AI requires new forms of oversight and institutional mechanisms capable of continuously reviewing, learning from, and adapting the metrics, processes, and results used to evaluate it.

Urs Gasser, Viktor Mayer-Schönberger, and Fabienne Marco argue that oversight of artificial intelligence should learn from the failure of its own measurements. I would add a quieter failure mode: Sometimes the measurement does not visibly degrade at all.

Their framework treats benchmark saturation, divergence from real-world outcomes, and other forms of degradation as information. But that assumes the monitoring system receives a signal that something has gone wrong. Correspondence can fail without producing one.

Suppose an evaluation measures proxy P because, under a set of assumptions, P is good evidence for safety property S. The model need not game the evaluation for that relationship to break. The deployment environment can change. A sensor can drift. A data pipeline can alter the meaning of an input. A human workflow can move downstream. The evaluation may continue reporting the same reassuring number even though P no longer warrants a conclusion about S.

Pulse oximetry offers an example. The US Food and Drug Administration has warned that skin pigmentation can affect pulse-oximeter accuracy and that inaccurate readings can result in unrecognized low oxygen saturation. Nothing inside the displayed number necessarily announces that failure. The device can continue producing an ordinary-looking value while the relationship between that value and the patient’s actual condition is weaker than the user assumes. No one has gamed the test.

The authors’ closing example of the Hubble Space Telescope contains the same structure. Hubble’s mirror was tested before launch, and the verification instruments confirmed that its shape was correct. The problem was that those instruments were themselves miscalibrated and lacked an effective independent cross-check. The verification channel did not announce its own failure. The evidence arrived from outside that channel, when Hubble’s images were blurred.

That suggests an additional discipline: every consequential assurance claim should carry a correspondence statement. It should identify the real-world property the metric is intended to represent, the assumptions connecting the two, the independent evidence used to test those assumptions, and the conditions under which the assurance claim becomes invalid even if the metric remains green.

A useful question for auditors would be: What could change underneath this measurement while leaving the measurement unchanged?

If the answer is unknown, monitoring metric decay is not enough. Independence should be a property not only of evaluators and institutions, but of the measurement chain itself. Periodically, consequential claims should be challenged with evidence that does not depend on the same proxy, sensor, calibration path, benchmark family, or intermediate variable that produced them.

Sometimes degradation is data. Sometimes the absence of degradation is the problem.

Urs Gasser, Viktor Mayer-Schönberger, and Fabienne Marco show what happens when artificial intelligence models learn to perform for the test: the measurements pointed at them decay. Their diagnosis has a flip side: The model is no longer only what gets graded, but increasingly the grader. Language models are now adopted across research and official statistics as measurement instruments, classifying occupations and scoring responses at a scale no human panel could attempt. In the very domain the authors worry about, the loop is starting to close: For a growing share of these measures, the entity answering which work AI can do is itself an AI model. The systems taking the test have now started grading it.

The consequence is that measurements inherit the instrument’s variability. Different models can grade the same object differently, and the same model can grade differently after an update. Economics already offers a clean illustration: One of the most widely used tools for gauging which work is exposed to AI swings from under 3% to over 50% of occupations rated “highly exposed” on identical data, depending on which model scores it. These numbers already steer real decisions: Federal retraining dollars, financial-firm macro forecasts of AI’s economic impact, and a growing wave of state AI legislation all lean on them.

None of this argues against the practice. Measurement through models is a genuine expansion of what can be measured, and it should be welcomed. It argues instead for holding model-based measurement to standards the psychometric literature already settled for fallible human judges: stability (whether independent raters agree); replication (whether a repeated measurement returns the same answer); and calibration (whether scores track observed reality). A model-graded measure that cannot demonstrate these three properties is not yet an instrument. It is an opinion with a sample size.

There is nonetheless potentially a more subversive risk: It can compromise the measurement itself. Asked to apply a new measure, a model may instead find a different, correlated measure’s published results online and copy them, lightly adjusted—the student who copies a classmate’s homework and changes a few answers; both get a good grade, but only one understands the assignment. Call it epistemological cheating. Because the two measures tend to move together, the cheating can pass unnoticed and turns dangerous the moment the new measure is validated against the one it copied: The validation is cheated in the same stroke, and the failure goes unnoticed.

The implication is that the regime the authors propose—continuous monitoring, post-deployment validation, institutions built to learn from their instruments’ decay—applies twice over: once to the measurements we point at AI, and again to the AI we now point at everything else. The authors close with the Hubble Space Telescope, whose flaw lay in the instruments that verified the mirror. As language models become the instruments verifying economies and institutions, the first requirement is the one their essay teaches: being able to tell when the ruler has changed length, and also whether it is cheating.