Belief, Uncertainty, and Truth in Language Models

What does it mean for a language model to “know” something—and how should it communicate uncertainty to the people who use it? In this talk, Jacob Andreas, Associate Professor of Electrical Engineering and Computer Science at MIT, explores new approaches to building language models that not only model the world but also model themselves. 

Andreas shows how optimizing for coherence and calibration—beyond accuracy alone—can produce models that are both more factually consistent and more reliable in expressing confidence. These advances raise pressing questions for governance: How should large models present information? What standards should guide their expression of reliability or doubt? 

Moderated by Josh Joseph, the Berkman Klein Center’s Chief AI Scientist, this conversation situates cutting-edge technical work on belief and uncertainty in language models within the wider debates about interpretability, trust, and the responsible use of AI. Navigating computer science and the questions it raises for broader society, the conversation concludes by examining the policy implications raised by these complex issues.

Speakers

Jacob Andreas is an associate professor at MIT in EECS and CSAIL. He did his PhD work at Berkeley, where he was a member of the Berkeley NLP Group and the Berkeley AI Research Lab. His research aims to understand the computational foundations of language learning, and to build general-purpose intelligent systems that can communicate effectively with humans and learn from human guidance.

Josh Joseph is Berkman Klein’s Chief AI Scientist, the first role of its kind at BKC. Josh leads teams to explore measuring and controlling the agency of AI systems, benchmarking these systems beyond measures of“intelligence,” and to help build the infrastructure for in-house AI research. In addition to being BKC’s Chief AI Scientist, Josh currently holds appointments as a Visiting Scientist at MIT and a Lecturer on Law at Harvard Law School.