Most AI safety evaluation focuses on what a model can do in isolation — run a benchmark, check the outputs, report a score.

But that misses something fundamental: AI systems don't cause harm in a vacuum. Harm happens when a model meets a user, in a context, within a society.

How does a chatbot giving slightly wrong medical advice become a public health problem when millions of people use it? How does a hiring tool that seems fair on average create systematic disadvantage for specific communities?

These are not questions you can answer with a benchmark.

This session introduces the sociotechnical approach to AI evaluation — a framework that moves beyond testing model capabilities in isolation and adds two critical layers: how humans actually interact with the system, and what systemic effects it produces at scale. It's about evaluating AI not just as a technology, but as something embedded in the real world.

Weidinger argues this is the harder part of making AI safe, but it is also the part that matters most—because systems that seem safe in a lab often fail when real people with varied backgrounds, languages, and levels of understanding start using them.

Part of Module 9: Real-World Evaluation — Societal Impacts of AI.

Laura Weidinger is a Staff Research Scientist at Google DeepMind, where she leads research on AI safety evaluation. Her work focuses on operationalising ethical and societal concerns into methods for evaluating risks from generative AI systems. She developed the sociotechnical safety evaluation framework used at DeepMind and is a leading voice in building what she calls an "evaluation science" for AI.

Previously, Laura worked in cognitive science research and as policy advisor at UK and EU levels. She is a trained philosopher and cognitive neuroscientist, with degrees from the University of Cambridge and Humboldt University of Berlin.