Who Watches the Watchers Building Trust in AI Governance
20 Feb 2026 18:00h - 19:00h
Who Watches the Watchers Building Trust in AI Governance
Session at a glance
Summary
This discussion focused on the current state of AI governance and safety, examining progress made since the 2023 Bletchley Park AI Safety Summit and exploring new models for oversight and verification. Stephen Clare, co-lead author of the International AI Safety Report, opened by highlighting significant progress in AI safety measures, noting that technical safeguards have improved dramatically, with models becoming much harder to jailbreak and taking hours rather than minutes to circumvent protections. He emphasized that while risks are now materializing in real-world applications with over a billion people using AI globally, the toolkit for managing these risks is expanding, though implementation remains inconsistent across the industry.
Hiroki Hibuka provided a global perspective on AI governance approaches, explaining that all countries employ both hard and soft law approaches rather than purely regulatory or voluntary frameworks. He distinguished between the EU’s holistic approach through the AI Act versus the sector-specific approaches favored by Japan and the US, noting that Japan prioritizes establishing rules in advance while the US relies more on ex-post litigation and court resolution.
Shana Mansbach introduced the concept of independent verification organizations (IVOs) as a solution to the growing trust problem in AI deployment. She argued that traditional command-and-control governance is inadequate for AI’s rapid pace and technical complexity, proposing instead a government-authorized marketplace of independent verifiers that would use outcomes-based approaches rather than procedural compliance. This system would create financial incentives through liability protection, insurance advantages, and competitive market benefits.
The panelists discussed the challenge of creating standardized evaluation methods, noting that current benchmarks are often narrow and quickly outdated as AI capabilities advance. They explored how independent verification could address the information asymmetry between frontier labs and external auditors while maintaining the technical expertise needed for effective oversight. The discussion concluded with recognition that democratic debate is necessary to determine acceptable risk levels and that lessons from other industries like automotive safety regulation could inform AI governance approaches.
Keypoints
Major Discussion Points:
– Current State of AI Safety and Governance: The discussion highlighted significant progress in AI safety measures since the 2023 Bletchley Summit, with improved technical safeguards making models much harder to “jailbreak” and 12 leading companies now having frontier safety frameworks. However, challenges remain in ensuring consistent application across the industry.
– Global Regulatory Approaches: Different countries are taking varied approaches to AI governance – the EU with holistic regulation through the AI Act, Japan with sector-specific soft law approaches emphasizing compliance, and the US with an “ex-post” liability-focused system. All face the challenge of regulating rapidly evolving “black box” technologies.
– Trust Gap and Independent Verification: A central theme was the trust problem affecting all stakeholders – public, deployers, regulators, and developers. The panel discussed the need for independent verification organizations (IVOs) as a marketplace-based solution to provide outcomes-based assessment rather than procedural compliance.
– Stakeholder Responsibilities and Incentives: The conversation explored how responsibilities should be distributed among model developers, deployers, and end users, emphasizing the need for “defense in depth” approaches. Key incentives for adoption of verification systems include liability protection, insurance requirements, and competitive market advantages.
– Technical Challenges in Evaluation: The panel addressed significant gaps in current AI evaluation methods, noting that existing benchmarks are often narrow and quickly outdated. The stochastic nature of AI systems and the complexity of real-world applications make safety testing particularly challenging.
Overall Purpose:
The discussion aimed to assess the current state of AI governance and safety three years after the Bletchley Park AI Summit, examining what progress has been made, what challenges remain, and what new governance models might be needed to address the trust gap between AI capabilities and public confidence in their safety and reliability.
Overall Tone:
The tone was professional and constructive throughout, with participants building on each other’s points collaboratively. While acknowledging significant challenges and gaps in current approaches, the overall sentiment was cautiously optimistic about progress made and pragmatic about solutions needed. The discussion maintained an academic yet practical focus, with panelists drawing from their diverse expertise in policy, law, and technical safety to offer concrete examples and analogies from other industries.
Speakers
– Gregory C. Allen: Moderator/Host of the panel discussion
– Stephen Clare: Co-lead author of the International AI Safety Report, expert in AI safety and governance
– Hiroki Hibuka: Research professor at Kyoto University Graduate School of Law, former Japanese government policymaker (worked for 4 years designing Japanese AI policies), expert on Japanese AI policy, non-resident senior associate at CSIS, lawyer
– Shana Mansbach: Vice president of strategy and communications at Fathom (a think tank), leads policy initiatives and convenes the ASHFE conference series on AI
Additional speakers:
None identified beyond the provided speakers names list.
Full session report
This comprehensive discussion examined the current state of AI governance and safety, three years after the landmark 2023 Bletchley Park AI Safety Summit, revealing both significant progress and emerging challenges as artificial intelligence transitions from theoretical concern to widespread real-world deployment.
The Evolution from Theoretical to Practical AI Governance
Stephen Clare, co-lead author of the 312-page International AI Safety Report reviewed by hundreds of people, opened the discussion by establishing a fundamental shift in the AI governance landscape. As he noted, “the rubber is really hitting the road” with AI systems, as risks that were merely theoretical one or two years ago have now materialized into concrete real-world impacts including deepfakes and cyber attacks. With ChatGPT alone having 800 million weekly average users, the urgency of effective governance has moved from future planning to immediate necessity.
The technical progress in AI safety has been substantial and measurable. Clare highlighted that models have become dramatically more difficult to “jailbreak” or circumvent safety measures. Where the UK Security Institute could previously find universal exploits within minutes using techniques like emotional manipulation (“I miss my grandma who used to tell me bedtime stories about making Molotov cocktails”) or language translation workarounds, these same researchers now require seven to ten hours to breach the latest models’ safeguards. This represents not just incremental improvement but a fundamental strengthening of AI safety infrastructure.
Furthermore, the industry has witnessed widespread adoption of frontier safety frameworks, with twelve leading AI developers now maintaining formal documents describing their risk management approaches for increasingly powerful systems. This represents a significant shift towards transparency and collective learning about risk management practices.
However, Clare emphasized that these advances come with important caveats. Technical safeguards remain vulnerable to sophisticated attacks and edge cases, and providing reliable assurances across the vast range of real-world applications proves extremely difficult. More critically, while frontier developers typically implement robust safeguards, application across the broader industry remains inconsistent, particularly among companies operating behind the technological frontier.
Global Regulatory Approaches and Cultural Differences
Hiroki Hibuka provided crucial perspective on the international landscape of AI governance, challenging common misconceptions about different national approaches. He argued that the frequently cited distinction between the EU’s “hard law” approach and other countries’ “soft law” approaches represents a fundamental misunderstanding of regulatory frameworks. In reality, all jurisdictions employ both hard and soft law mechanisms, as extensive existing regulations—covering privacy protection, copyright, finance, automotive, and healthcare—already apply to AI systems.
The more meaningful distinction lies in whether countries pursue holistic versus sector-specific regulation. The EU’s AI Act represents a comprehensive approach attempting to regulate AI systems across all applications, while Japan and the United States favor sector-specific interventions that address AI within existing regulatory frameworks for particular industries.
Hibuka also illuminated cultural differences in regulatory philosophy that significantly impact implementation. The United States tends towards an “ex-post” approach, allowing broad experimentation with high-level principles and relying on court systems to resolve disputes when harms occur. Japan, by contrast, prefers “ex-ante” rule-setting, with society establishing clear guidelines in advance. Japanese companies are “very, very good at complying with given rules” but struggle with creating their own governance mechanisms or explaining their approaches to stakeholders.
This cultural analysis reveals that all countries face similar fundamental challenges: how to regulate cutting-edge “black box” technologies with unlimited risk scenarios, and how to establish benchmarks and standards for evaluating complex values like privacy, transparency, and fairness when clear societal consensus on these benchmarks has not yet emerged.
The Trust Gap and Independent Verification Solutions
Shana Mansbach from Fathom, a young think tank started only two years ago, introduced a critical framework for understanding current AI governance failures through the lens of trust. She identified a pervasive trust problem affecting all stakeholders: the public lacks means to determine what is actually safe; deployers such as hospitals, banks, and retailers need AI systems but cannot assess their reliability; regulators struggle to confer earned trust rather than mere compliance; and even developers face declining adoption if trust erodes. Deployers also worry about a “populist backlash” if AI systems cause harm.
Traditional command-and-control governance proves inadequate for addressing this trust gap due to two fundamental problems. First, the speed problem: AI capabilities advance so rapidly that even well-intentioned regulations become outdated quickly. Second, the technical capacity problem: expertise for understanding AI systems and their risks remains concentrated primarily within frontier laboratories, creating an information asymmetry that undermines independent oversight.
Mansbach proposed independent verification organizations (IVOs) as a potential solution—a government-authorized and overseen marketplace of independent verifiers tasked with developing testing and tooling to determine whether AI systems meet safety requirements. This represents an outcomes-based approach rather than procedural compliance, where governments specify desired outcomes (children’s safety, data privacy, controllability) while independent verifiers develop and continuously update testing methodologies to ensure these outcomes are met.
The IVO model offers several potential advantages over traditional approaches. Independence ensures that companies are not “grading their own homework.” Democratic accountability maintains government oversight of outcome specification while leveraging private sector technical expertise. Flexibility allows verification organizations to continuously update testing criteria to match technological advancement. Finally, it creates a “race to the top” by incentivizing ever-better testing and tooling through market competition.
Stakeholder Responsibilities and Market Dynamics
The discussion revealed the complexity of distributing responsibilities across the AI ecosystem. Clare advocated for a “layered approach” with “defense in depth,” recognizing that no single intervention or actor can provide complete safety assurance. Model developers implement training techniques to reduce harmful outputs; deployers add monitoring systems and query classifiers; ecosystem monitoring bodies track AI content spread; and society adapts through resilience measures.
Allen, drawing from his experience at a rocket company, provided the analogy of ride-hailing services to illustrate how responsibility distribution might work: automobile manufacturers ensure safe car design and manufacturing; platform companies like Uber maintain vehicles appropriately; and drivers follow traffic laws and operate vehicles safely. Similarly, in AI systems, model developers, business deployers, and end users each bear distinct but interconnected responsibilities.
However, current incentive structures often work against safety adoption. Hibuka noted that independent auditing lacks clear economic incentives except in highly regulated sectors like healthcare, automotive, and finance where trust is essential. Companies may even prefer “willful blindness”—avoiding safety assessments that might reveal problems and increase legal liability.
Allen observed that the insurance industry has emerged as an unexpected but powerful governance mechanism. Major insurers are increasingly refusing to cover AI products due to uncertainty about their contents and risks. This creates de facto market regulation, as companies requiring insurance coverage—which includes most major enterprises—find themselves unable to deploy AI systems without meeting insurer requirements. Allen referenced his experience with AS9100 certification in the aerospace industry, where certification became essential not due to regulatory mandate but because insurers and customers demanded it.
Technical Challenges in Evaluation and Standards Development
A critical gap exists in current AI evaluation methodologies, which Clare described as “already not super informative about real-world risk because they’re too narrow.” Existing evaluations typically consist of question sets related to specific topics like biosecurity or cybersecurity, with models deemed risky if they score above certain thresholds. However, as models become more capable, general, and widely adopted, these narrow evaluations fail to capture the vast range of real-world applications and risks.
Mansbach elaborated on the technical challenges inherent in AI system evaluation. The fundamentally stochastic nature of AI means identical queries can produce different outputs, complicating safety assessment. Additionally, model outputs do not directly correlate with user actions—a model might provide identical harmful suggestions to ten users, with nine dismissing the advice while one acts upon it with serious consequences. The multi-turn, relationship-building nature of AI interactions adds further complexity that simple evaluation frameworks cannot capture.
These technical limitations highlight why current benchmarks, while useful, remain inadequate for comprehensive safety assessment. The rapid pace of capability advancement means evaluation methods quickly become outdated, creating a persistent gap between assessment tools and actual system capabilities.
Democratic Accountability and Risk Tolerance
Hibuka raised fundamental questions about democratic governance of AI risks, noting that technical solutions cannot resolve value-based decisions about acceptable risk levels. Using autonomous vehicles as an example, he highlighted that Japan experiences over 2,000 traffic fatalities annually from human drivers, raising questions about what safety standards should apply to AI-driven vehicles. Should they merely match human performance, exceed it, and if so, by what margin?
These decisions require democratic deliberation rather than technical expertise alone. Different societies may reasonably reach different conclusions about acceptable risk-benefit trade-offs, and governance mechanisms must accommodate this variation while maintaining technical rigor in implementation.
The challenge extends beyond risk tolerance to evaluation methodology design. Testing autonomous vehicles on safe highways produces different results than testing in complex urban environments, but determining appropriate test conditions involves value judgments about representative use cases and acceptable failure modes.
Information Asymmetries and Ongoing Challenges
A persistent challenge throughout the discussion was the concentration of technical expertise within frontier AI laboratories. Clare acknowledged this information asymmetry while noting that external oversight cannot succeed without drawing upon internal knowledge. The International AI Safety Report attempted to address this through partnerships that leverage industry expertise while maintaining transparency and incorporating diverse perspectives from academia, civil society, and government.
However, this approach faces limitations. External actors remain dependent on publications from laboratories, leaving significant gaps in understanding across different risks and applications. The report frequently acknowledged uncertainty due to lack of comprehensive data outside laboratory walls.
Some positive trends suggest this may improve over time. Frontier safety frameworks have become institutionalized in governance mechanisms like the EU AI Act’s codes of practice. Companies have made new commitments to sharing usage data, and broader societal attention to AI impacts creates pressure for greater transparency.
Allen referenced recent incidents like the XAI Grok situation as examples of how real-world deployment continues to reveal unexpected challenges and the need for better oversight mechanisms.
Legal Frameworks and Verification
The discussion touched on how independent verification could address critical gaps in legal frameworks, particularly around standards of care in liability cases. Currently, when AI systems cause harm, courts must retrospectively determine whether defendants met appropriate standards—a challenging task for technically complex systems where even experts disagree on best practices.
Mansbach suggested that verification could potentially confer legal protection for companies that undergo independent assessment, establishing clear expectations before harms occur rather than leaving courts to determine appropriate standards after the fact. Such clarity could reduce legal uncertainty while incentivizing proactive safety measures.
The insurance implications prove particularly significant. Just as other industries require various forms of assessment, AI insurance could require verification, with verified systems potentially receiving coverage at more favorable terms. This creates market-driven incentives for safety adoption without requiring regulatory mandates.
Unresolved Questions and Future Directions
Despite the comprehensive discussion, several fundamental challenges remain unresolved. The stochastic nature of AI systems continues to complicate safety testing, as identical inputs can produce varying outputs. The relationship between model outputs and real-world harms remains poorly understood, particularly given the complex, multi-turn interactions users develop with AI systems.
Information asymmetries between frontier laboratories and external actors persist, though some progress toward greater transparency appears promising. The question of how to maintain technical competence in independent verification organizations as they become separated from cutting-edge development remains open.
Perhaps most fundamentally, the challenge of determining acceptable risk levels across different applications and societies requires ongoing democratic deliberation that technical solutions alone cannot resolve. The governance frameworks discussed provide potential mechanisms for implementing societal choices about risk tolerance, but cannot substitute for the democratic processes needed to make those choices.
Conclusion
The discussion revealed a field in transition from theoretical concern to practical implementation, with significant technical progress in AI safety measures but persistent challenges in governance and oversight. The emergence of independent verification organizations represents a promising but experimental approach to addressing trust gaps while maintaining democratic accountability and technical rigor.
The insurance industry’s growing influence as a de facto regulator highlights how market mechanisms may drive safety standards regardless of formal regulatory approaches. However, success will require careful attention to incentive design, democratic input on risk tolerance, and continued innovation in evaluation methodologies to keep pace with rapidly advancing AI capabilities.
Most significantly, the conversation demonstrated growing recognition among experts that traditional regulatory approaches may prove inadequate for AI governance, necessitating novel frameworks that blend public oversight, private sector expertise, and market-driven incentives. The specific mechanisms may vary across jurisdictions, but the fundamental challenge of building trustworthy AI systems while maintaining innovation and democratic accountability remains universal. As Clare humorously noted about the length of their report, the complexity of these challenges defies simple solutions, requiring sustained attention and experimentation with new governance approaches.
Session transcript
Again, to my immediate right, we have Stephen Clare, who wrote the International AI Safety Report as the co -lead author, if I’m not mistaken. And he earned that applause, because that report is a remarkable document that I do think is the foundation upon which all conversations about AI governance now must rest for the next year. It’s the sort of minimum amount of knowledge that you must have to participate in the conversation, which I think is really a tribute to him. Then we have Hiroki Hibuka, who is currently a research professor at the Kyoto University Graduate School of Law, and was also deeply involved in drafting Japan’s first set of soft law regulations, and is an expert on all things AI, but also especially astute at what’s going on in Japan.
We also have a privilege of collaborating with him at CSIS, where he’s a non -resident senior associate. And I must say, he is probably the best person writing about Japanese AI policy in Japanese, but he is definitely the best person writing about it in English. And so I often tell Hiroki that, like, if he doesn’t write about it, nobody in Washington, D.C. knows about it. So it’s important, his work. And then finally, we have Shana Mansbach, who’s the vice president of strategy and communications at Fathom, which is a young think tank, started only two years ago, but has already succeeded as one of the best conveners of the ASHFE conference series on AI, and also now leading a policy initiative, which I think she’s going to tell us all about.
So without further ado, I’d like to start with you, Stephen. I just said that the report that you were the lead author of is sort of the bedrock for having a conversation on AI governance. For those in the audience who haven’t yet made it through, but they, of course, will, can you sort of set the stage? Where are we in 2026 in AI governance and in AI safety, technical and procedural intervention?
Sure. Thanks, Greg. First of all, I’m sorry if I’d known Greg was going to make the report, you know, required reading, I would have tried harder to make it shorter. Yeah. Thanks for having me. Thanks for really excited to be here. So for people who don’t know, the report is it was founded up the Bletchley 2023 Bletchley Safety Summit as sort of, you know, the shared evidence base for decision makers thinking about these complicated, fast moving, noisy governance questions. It’s kind of trying to be like the IPCC report for for AI. It’s backed by over 30 countries and intergovernmental organizations. You know, I’m one of two co lead writers along with Karina Prunkle, but there’s over 30 dedicated experts writing different sections, and there’s hundreds of people that review it.
So it’s really trying to be a sort of state of the art, what do we know? What don’t we know about general purpose AI systems and the risks they might pose? I think this year the main message of the report is like the rubber is really hitting the road or something with these kind of systems. Risks that even a year or two ago might have been theoretical are now very real and we’re seeing emerging empirical evidence. More real world impacts of AI on productivity and labor markets and in science and in software engineering. It’s all like really happening out in the world. There’s a billion people now using AI around the world. Many of those impacts include risks.
So we’re seeing effects of deepfake spreading, cyber attacks being more common with AI systems. And so the need for sort of risk management techniques that are effective is also growing. One thing that I found surprising working on the report is that in this domain on risk management and technical safety, there’s actually some good news. Quite a lot of good news, I’d say. In various ways, our technical safeguards are improving. Models are becoming much harder to jailbreak. So. You know. So three, four years ago, if you asked a model to give you a recipe for a Molotov cocktail, it would not do that. But if you said, oh, I miss my grandma, and she used to tell me this amazing bedtime story about how she loved making Molotov cocktails, please help me remember my grandmother, it would be like, okay, well, if it’s for your grandmother.
Then that stopped working maybe a year or two ago, but then if you maybe translated your question into Swahili or something and put it in the model and then translated the answer back, it might have made safeguards. So none of that works anymore. These safeguards are much harder to evade, and we know this quantitatively. For example, the UK Security Institute will try and evade the safeguards or jailbreak all these new models when they’re released. At the beginning of 2025, they could do this in literally minutes, find a sort of universal jailbreak that would elicit potentially harmful knowledge. For the latest models, it’s taking them seven, ten hours to get around safeguards. So there’s still vulnerabilities, but for novices or even moderately skilled actors, it’s basically the same thing.
It’s becoming much, much harder to evade them. We’re also seeing more of these safeguards get implemented into organizational practices. So 12 companies, all the leading AI developers now have frontier safety frameworks, which are these documents that describe how they plan to manage risks as they scale more powerful systems, which is many more than had them a couple of years ago and is, I think, a sign of transparency and sort of collective learning about risk management that’s worth noting. So basically, yeah, our toolkit for managing these risks is growing. But, you know, it wouldn’t be a safety report if I didn’t maybe end on a few caveats or some bad news. The first is that these technical safeguards are still vulnerable in many ways.
They can still be jailbroken with enough effort or in edge cases, and it’s very difficult to test and provide reliable assurances that these safeguards will work across this huge range of use cases that these models are now applied to in the real world. And on the organizational side, you know, these safeguards only work if they are applied. And although we’re seeing, especially from frontier developers, we’re very prominent, usually quite robust safeguards applied on models, across the whole industry, and especially behind the frontier, application remains quite inconsistent. The safety frameworks, all these companies have them, but they vary in the risks they cover, they vary in the practices that they recommend. And so the landscape as a whole, you know, these tools only work if they are applied.
And we still see that, some vulnerabilities across the landscape, which I think turns this technical challenge, that points towards the governance challenge of how do we assure broader adoption, how do you ensure compliance, what do you do when there’s a lack of compliance. We’re sort of facing these questions, and again, because these risks and the impacts are now not something that we can sort of push down the road anymore, I think, for future years, the governance questions are becoming a lot more urgent.
Terrific. And if I could contrast what you said with what we might have said if we were having this conversation back at the Bletchley Park AI Summit. But it’s almost like the only good news on AI safety, AI security, and AI governance at Bletchley was, well, at least we’re all here talking about it. And now, three years later, the good news is we’ve done a lot about it. We have techniques that can provide demonstrable increases in safety. We don’t know everything that we need to work, but we know a lot of stuff that does work. And really, a lot of the challenges, I think, as the report says, it’s now in the hands of policymakers to make sure that these safeguards get implemented robustly and diversely.
So with that, I now want to turn to Hiroki, who I hope can give us a state of where we are in the story of AI governance around the world. If the next steps are really in the hands of policymakers, where are we globally?
Thank you, Greg. And again, congratulations. Stephen was the publisher of the great report. And I think, first of all, I feel very glad that now the discussion on AI governance is such advanced compared to three years ago. I’m a lawyer and I’m a former policymaker. I worked for the Japanese government for four years, designing the Japanese AI policies, mainly in terms of regulation and governance. And as a lawyer and policymaker, the question after reading the report is, where is the end? And to what extent stakeholders have to manage the risks? Because in the end, you can’t remove all the risks. AI is black box and the technology advances so fast. And even though there is advance and progress of Godwins, the next day you may find another risk.
So there is no end to the story of how regulators should design the regulations. That is the main question. All countries. Countries are facing and different nations, regions take different approaches. Maybe the most famous regulation is the EU AI Act. And in that context, a lot of people say, hey, EU takes a hard law regulatory approach on AIs while Japan or UK or United States takes a software approach. But I think it’s a completely wrong understanding of the regulatory framework because, as you know, there are already lots of regulations that can be applied to AI systems. Privacy protection laws, copyright laws, or sector -specific laws such as finance, automotive or healthcare. We already have a lot of regulations out there.
So the real question is not whether or not to regulate AIs, but the real question is how to update our existing regulations and whether or not we need additional regulations targeting AI systems. In addition to the existing regulatory framework, so in that sense all countries take the hard law approach and also all countries have soft laws because European Union there are a lot of technical standards to implement the EU AI Act that are now under discussion but anyways all countries have both hard laws and soft laws that is the start of the discussion and then when we compare EU approach and Japan approach the clear difference is whether to regulate AI holistically or not sector -specific and when I compare the Japanese policy and the US policy we are on the same position as to taking a sector -specific regulation the main difference I understand is whether you prioritize the exempt approach or exposed approach the US takes more exposed approach you can do whatever you want to do and the regulation is usually very high level the principle is very high But once you have a problem, if you damage others’ properties or lives, then you go to the court and you fight in the court.
The Japanese society is not like that. In Japan, actually the number of losses is very low. People prefer to set the rules in advance. Japanese companies are very, very good at complying with the given rules. But they are not very good at creating their own governance mechanisms or explaining to stakeholders why you are doing that. And now Japanese stakeholders are starting to realize that it doesn’t work. So we need to have more agile and multi -stakeholder approach. So we are trying to leverage the power of soft laws, negotiating among different stakeholders, and give the standards, guidances. But in the end, again, if you violate the existing hard laws, of course you will be sanctioned. So that’s the main differences in American approach and Japan approaches.
And in the end, all countries are facing difficult questions of how to deal with this cutting -edge technologies that are black box and there are unlimited risk scenarios. And sometimes we don’t know how to evaluate the values such as privacy or transparency or fairness. There has been no clear benchmark standards so far in the society. So how to design those benchmarks and regulation methods are the challenges all countries are facing.
Terrific, Hiroki. And Shaina, I know you have a unique perspective on this because your organization is now proposing sort of additional models of AI governance that are not really reflected in existing law, whether in the United States or Europe or Japan or India. So walk us through what you see as the important work we’re doing now.
Sure. My panelists have set me up very well to say this. So I think as the International AI Safety Report shows, the capabilities around these models are surging. And as the capabilities surge, so too does the uncertainty around the risks, by which I mean, do these systems work safely, securely, and as advertised? That uncertainty creates a trust problem, a trust problem for the public, which doesn’t have a way of figuring out what is actually safe, a trust problem for deployers, by which I mean hospital systems, retail, banks, who want to and indeed need to use these systems, but have no idea what they can actually trust. So there’s a trust problem for the regulators, too.
They don’t know, how do you confer not just trust, but how do you confer earned trust? And I would say there’s a trust problem for the developers also, because if and as trust starts to grow, there’s a trust problem for So if the trust starts to decline, you’re going to see adoption decline as well, so this is something that developers should be focused on too. The current approach is just not the current approach to tech governance is not equipped to handle this trust problem very well. Traditional command and control governance says here are the rules, here are all the things you have to do, here are the procedures, here is what compliance actually here’s what compliance actually looks like.
There are a bunch of problems with this approach in the context of AI, but I’ll focus on two, which is the speed problem. AI moves really, really quickly, and even well -intentioned regulations are going to become outdated very, very quickly, and then there’s the technical capacity problem. Even with the rise of the AI safety institutes, which are doing amazing work, the talents, the expertise for understanding these systems and understanding their risks is largely concentrated in the frontier labs, which of course leads some people to say, well, let’s just go to the frontier labs. They can regulate themselves. I don’t think I have to spend too much time explaining why there are problems with that approach but it’s simple incentives I think all of us know people in the labs who are doing amazing, amazing work they are the people who make sure that I can because of them I sleep better at night but the incentives are just not there there are always going to be trade -offs between investing in safety testing and tooling and investing in development so we’re going to have problems with self -regulation in terms of addressing that trust gap so where does that lead us?
at Fathom, my organization, we’re very focused on coming up with new models that can solve this trust gap so we’re very focused on independent verification specifically the marketplace of independent verification organizations by which I mean a government -authorized and overseen marketplace of independent verifiers which are trying to be charged with creating testing and tooling to determine whether these AI systems are actually safe The difference here is that this is an outcomes -based approach. Instead of, as I said, having procedures, here are the rules, here are all the things you need to do, here are all the boxes you must check to be certified as being good, you have an outcomes -based approach where you have a government saying, here are the things that we care about.
We care about children’s safety. We care about data privacy and protection. We care about controllability and interpretability. And then you have independent verifiers that can actually go out, do the testing, have updated testing constantly to make sure that those outcomes are being met. We think that independent verification solves for a couple of these deficits in the trust context. First, they are independent. The labs are not grading their own homework. Second, democratic accountability. You have governments that are creating outcomes instead of the industry doing it itself. Third, flexibility. Under this system, the IVOs, independent verification organizations, are constantly updating their testing and criteria to make sure that they’re keeping up with the pace of technology and the pace of risks as well.
And I think the fourth thing, which is pretty interesting, is it creates a race to the top here. Right now, the only people working on safety testing and tooling are in the labs. What we’re envisioning is a marketplace that incentivizes ever better testing and tooling here. I could talk about IVOs for days and days, but let me just end on one point. I was talking to Greg about this earlier, and Greg asked, are there analogous systems or industries or sectors that we could talk about? And I said, yeah, sort of. I mean, in America, we have Underwriters Lab. There’s LEED certification. There are some analogies. But the honest answer is there’s not a perfect analogy.
We have had the same regulatory system for the last century. And I think that with the rise of AI, we’re seeing that system is no longer built for purpose. And when we try to use old systems, hard law, soft law, any of these things, we’re really struggling to make it work. So what I’m trying to do, what I’d encourage all of us to do is to say, you know, we do need to think a little bit differently. Because this is what this technology in this time calls for.
Well, that’s great. So there’s a few points I want to pull together there. The first is, you know, as Hiroki pointed out, in the U.S. system, liability law looms extremely large, right? The lawsuits at the end of this story when things go wrong. And when you have, as, for example, ChatGPT does, 800 million weekly average users, something’s going to go wrong every week, right? And the question is… How is that going to intersect with our existing body of regulation? How is that going to intersect with liability law? The second thing is this is going to, because we’re talking about these general purpose technologies, this is going to be adopted in so many different sectors of the economy.
And right now, as Shana pointed out, the number of people who have, you know, Steven’s expertise on what it takes to really make AI systems safe and well -governed and perform reliably as intended across the whole range of potential applications, that’s not a lot of humans on planet Earth who are good at that stuff. And because these AI models are going to be deployed in just about every sector of the economy, we need some level of those capabilities in every sector of the economy. And so the question is, you know, if I am a financier, if I am a finance company, if I am a health care company, you know, how am I going to know and how are my consumers going to know?
that when they use AI -related capabilities, it’s going to work reliably as intended over the full range of acceptable use cases. And so, Stephen, I want to come to you and ask, when it comes to governance, when it comes to oversight and verification, how do you see the balance of responsibilities in terms of what responsibilities need to fall upon the model developers, what responsibilities need to fall upon the users, what responsibilities need to fall on independent third parties, whether that’s the government, whether that’s auditors, whether that’s this marketplace of verification that Shana is talking about. So what do you see as the balance of responsibilities, and how might this go wrong, how might this go right?
In 30 seconds or less.
I mean, I’m sure it’s kind of the boring but true answer. It’s the boring part of it. depends and it’ll vary a lot across use cases and sectors. I think probably it’s not the case that it’s fair or helpful or true to allocate to one actor or another, but instead we need this layered approach of just many different policies and practices at different parts of the stack. Because none of our approaches are foolproof, they all have vulnerabilities, and so we have, instead of safety by design, we have this safety by degree situation where we want defense in depth. So for developers, there will be training techniques that they can implement to make models less likely to elicit dangerous knowledge in the first place.
If there are people building on top of those models and then deploying them, there will be monitoring systems they can put in place and classifiers that identify dangerous queries and stop models from answering them. and then probably for ecosystem monitoring bodies which could be deployers but could also be other institutions in the world there can be tracking how AI content is spreading across borders and around the world and then I think there’s this other aspect of we’re focusing a lot on sort of model or developer safety but as we are moving into this world where many people around the world are having access to powerful, helpful intelligent technologies and we also just need to adapt for that reality and think about resilience at the societal level too of how do we adapt to the beneficial use cases and the various use cases that these models will be used for so thinking about hardening digital systems against increased cyber attacks just sort of admitting the reality of the situation in many ways and adapting to it rather than trying to prevent all harmful uses in the first place I think we need a variety of approaches across all these different actors
Yeah. And just to use an analogy for how broad the group of stakeholders is, if you think about a ride hailing service, a taxi service like Uber, you have the automobile manufacturers who have to make sure that this is a solid car design that was manufactured safely and appropriately to specification. Then you have Uber, where in some countries Uber owns the car, and so they’re responsible for ensuring that it gets maintenance appropriately. And then you have the driver who’s responsible for ensuring that they are actually following the law and driving the car safely. And if you apply that analogy to AI, you have the model developer, then you might have the sort of business use case deployer, which could be a bank, a medical device company.
Who? A financial institution, whoever. And then you finally have the end customer who’s receiving those services and making sure that they’re using them appropriately. And so. If you think about that sort of different body of use cases, as I said before, the capabilities are not symmetric across all of those. But there are sort of obligations. And so, Shana, I want to come back to you and ask this model that you’re proposing, what exactly does it mean for the different stakeholders in the ecosystem? How does their life change if we adopt the system that you’re in favor of?
Yeah, I mean, the overarching answer is we create trust throughout the system, which is the missing piece here. I think there are a couple of pieces that I would pull out. You had mentioned liability earlier, and let me talk about that a little bit. What this system does, it does not assign liability. It doesn’t say, you know, deployers, developer, it’s you, it’s you, it’s you. We’re seeing, at least in America, courts move their way through this. Sister. court cases move their ways through the court system and we’ll see where that is but where that ends up being but what is really missing is a standard of care and this is I think one of the real advantages that this system has so right now at least how it works in our current tort system is that if you’re Waymo kill someone someone can sue and then a judge and a jury has to figure out so again we’re not answering who should be sued but let’s say that the family of someone who got hurt or killed is suing Waymo what happens is that the jury has to decide whether whether the person who was sued did the right thing and if you are not technical that is the hardest thing even if you are technical and maybe even Waymo doesn’t know So what this system would do is confer, if you are verified, it would confer, the verification would confer a rebuttal presumption of having met a heightened standard of care.
So what we’re doing is clarifying and defining up front before an actual harm happens what a deployer or whoever is sued is actually supposed to do instead of having this very, very messy system where someone after the fact has to figure out what went wrong and who’s responsible for that. I can talk about other layers of this back here, but I think the liability piece is really key. I mean, we just see this. I think it’s a reflection of the trust problem here where when you’re a deployer, I mean, God, I think everyone that I talk to, you know, again, hospital systems, retail, banks, anyone who needs to be consumer facing is really worried about this problem.
I mean, when I get sued, what do I do? And maybe there’ll be. a populist backlash and everyone will hate everyone who’s using AI systems. And it’s much better to, ahead of something like that, ahead of that happening, have that standard of care defined up front and have that seal of approval conferred.
And Hiroki, as you think about the different stakeholders in the system and especially the idea of auditors, which now there are a number of organizations being founded, it seems like almost every day, who are proposing to provide external evaluation services that can help companies understand, as Shane has said, this product or this service or this company meets the seal of approval and we vouch for it as an independent entity. What kind of momentum do you see for this independent assessment part of the story across regulatory frameworks?
Independent evaluation. Independent evaluation is essential given that we are all using AI systems for all different situations, starting from language models to healthcare systems to car driving. But it would be not easy to persuade corporate executives to use the independent audit without clear economic incentives. For example, if you get the certification for autonomous driving, then you can sell the car to the big market. Then, of course, you pay for the audit. But if you take this audit for this language model, then you can prove that this language model is relatively safer than the other models. But it doesn’t necessarily make enough incentive for model developers to conduct the audit or evaluation systems, independent evaluation, because there is no clear financial incentives.
Actually, could I? I can ask you to elaborate on that. So where might these financial incentives come from? You mentioned one, which is the regulators force you to do it. That’s one. Maybe insurance is one. another like where where might these incentives come from
I think it should start from the regulated areas such as cars health care systems finance systems or infrastructures because everybody needs a strong requires strong trust on those systems if it doesn’t work well then somebody might have a baby kills that’s a big problem and maybe you could say hey but in the end if you are killed you can be compensated but it’s not the end of the story while if the damage could be compensated by money by the company and stakeholders are okay with that maybe companies like to just run the system go and and compensate to the victims for example if the language model says something discriminated the company can just say hey we’re very sorry we introduce better guardrails and we pay for that if you want compensation
in terms of what is possible, what interventions work, what the risks are. But I want to ask about how we go from that degree of consensus to something that might be more of like a standard around procedural implementation. You know, Shana’s term of art is standard of care, which matters a lot in the American legal system. I’m sure it matters a lot in other legal systems. I’m just ignorant about, you know, how and where. And so I’m curious, you know, what do you see as the gap? If these independent evaluators, these independent auditing organizations are emerging, how do they go from we think we’re good at this to, no, this is the accepted best practice?
You know, we have accepted consensus on the risks and the interventions, but, like, how do you turn that into a procedure? Just to give an example to the folks in the audience, I used to work at a rocket company, and the safety standard in the American aerospace industry is AS9100. And in the history, of our company, there’s kind of like a before AS9100 moment, and then there’s an after AS9100 moment. And everything changed for our company, you know, after we got that third -party audit evaluation. A lot of our customers, you know, just said, we do not sign checks for companies that are not AS9100 certified. So, you know, you are deeply steeped in where we are today on the consensus, but how far are we from converting that into standards and procedures for third -party evaluation?
Yeah. I’ll also say one follow -up to Hiroki’s point, too, about auditing. Not only is there sort of a lack of incentives to conduct audits voluntarily now, but there might even be disincentives where one is it’s costly, and it slows you down, and there’s very intense competitive pressures to release faster. And there’s also potentially… like, information or security risks to sharing. You spent hundreds of millions, maybe billions of dollars developing a model, and then you have to share it with an external party before deployment. Like, serious risks to, or perceived risks, at least, to having that information leak or… So I think, yeah, there’s some serious challenges there. I guess there’s one other potential part of the story, which is sometimes you see companies want to be willfully blind, right?
If they have a report that says my product is not safe, well, now they know they’re going to lose the lawsuit. Whereas if they never commission the report, maybe they’ll win the lawsuit. So, Shana, what do you see as meaningful interventions that can help address this problem, both the cost side that Stephen mentioned and the other parts of the incentive structure?
Yeah, let me make a couple of points. I mean, I think we’re talking about the cost of audits, and I think this… this is a big issue that we think about a lot. This system will not work if everyone, if there’s a flat fee, everyone is paying a ton. I mean, we are really, we think that an unsuccessful, there are many ways that a system looks unsuccessful, and one of those ways is if it is just protecting incumbents. And we’re thinking, we envision the system as something that works for, you could verify a general purpose LLM, you could also have narrow AI, you could have a tiny little tool, a little chatbot that is used in schools.
Those three different products should not be audited, not only at the same cost, but in the same way. I mean, compliance isn’t just the check that you’re writing, it is how much of a pain in the butt is it? How many lawyers do you need? How long will this take? So the great thing about this being a marketplace is that the system is right -sized to risk type, to size of these products. and again instead of having just a one size fits all this is what you have to do to comply because I think that that is a real issue it really quickly I just want to go back to you know the question that you asked Hiroki about incentives I mean you can imagine a system where this is mandatory and maybe in some areas you can imagine that but I think that there are three real real carrots for wanting to get verified we talked a little bit about liability so obviously the liability clarity that this is a big carrot I think the insurance piece the insurance piece is real right now we are seeing the big insurers saying we’re not going to touch this we’re not going to insure any AI products because we have no idea what’s inside of them at least in America the way that life insurance works is if you want insurance you have to have a lot of money and a lot of money and a lot of money and a lot of money you have to jump on a scale and tell someone how healthy you are and what are the things that you do and the insurer decides okay are you worthy of being insured and at what premium I think that’s actually a pretty direct analog for what we’re trying to do here where the books are opened and an insurer can look at whether they don’t have to do the testing themselves, but they can look at whether the system has been verified and say, okay, we will actually insure you or we will insure you at a more affordable premium.
I think the third thing is just straight -up market competitive advantage. If I’m a school superintendent and I am choosing between two learning chatbots to put in my schools, I’m not going to choose the one that has not been verified. I want the one that has been verified, that is safest. Yes, because I’m worried about getting sued, but because I want my kids to be safe. And you can imagine a situation much like Underwriters Lab in the United States where basically all consumer products like light bulbs, toothbrushes, basic things that you buy in a store like Walmart, all have the UL seal of approval, and those are the ones that get sold in stores. They have a huge market advantage.
They pay a little bit, but not very much. And in exchange for doing that, they go to market in a way that, or they compete in a market in a way that the ones that don’t go through verification. do. I’m so sorry, Gary, you asked me an actual question and I just answered everyone else’s question and probably not my own.
It’s okay. You get out of jail free card because you mentioned insurance, which is something I’m deeply interested in right now. I mean, in that space orbital launch vehicle example that I just mentioned, you can’t get insurance for space launches of satellites until you’re AS9100 certified. And that is 10 % of the cost of getting a satellite into space is just the insurance on the rocket. And so basically companies that can’t get insurance can’t compete in the market. And as Shana mentioned, and I think this is a super undercovered story, there are now many of the major insurers in the United States at least are saying, for your enterprise risk policy, AI is not included.
So if you are a major bank and you are doing big, important financial transactions, as soon as you start using AI, you’ve lost all your insurance. And I think the Trump administration in the United States has a very light -touch regulatory approach. And my concern there is that, well, just because the government is not doing anything big and bold on regulation doesn’t mean there will be no regulation. The insurers will step in. And if the insurers exit the market, maybe not in legal terms, but in economic outcome terms, that could be very similar to draconian regulation. So, Shana, you’re mentioning the Underwriters Lab, which is an organization that writes standards that are relied upon by underwriters, the people who are issuing insurance.
This is a huge part of the regulatory and governance ecosystem that I think is really important. And so now I’m hoping, Stephen, that you’re going to tell me, that you’ve been reached out to by a bunch of insurance companies, and they’re all reading your report eagerly and thinking about this. But maybe, maybe not. What’s the case?
Not yet, but it’s a really long report. 312 pages, but it goes like that. Maybe I can come back to the best practices point a little bit, because I think we’re talking about auditing here, and at least I know there’s a lot of steps involved, I’m sure, but at least at the technical level, the main tool we have right now to audit the capabilities of the RISC -MD AI model are evaluations. And although in my opening I sort of talked about, oh, it’s great we have this toolkit that’s emerging and it’s strengthening, and that is true, I think on evaluations in particular, as far as like, okay, let’s say we have auditors that are looking at these companies, looking at models, what are they actually looking at to audit or evaluate the models?
I think we actually have a big gap here, a big evaluation gap in terms of, well, how are we actually assessing? So if we’re moving towards best practices, not only do I think we don’t have a sense of the best practices right now, but if we did, they’d be different in a year, because the capabilities are moving too quickly for these technical tools to be in date, for very long. So for example, you’ll have, you know, these evaluations often look like a set of questions related to a certain topic, and you ask the model, so you have a bunch of questions about biosecurity or a bunch of questions about cybersecurity. And if it’s above, if it scores high enough on the test, you say, whoa, this is a dangerous capability, and we need to implement more safeguards or something.
And as far as what’s best practice or safe risk management for a company, we evaluate in terms of, well, does it seem like the safeguards apply proportionately to the risk that you’ve assessed? But I think in many cases, these evaluations we’re using are already not super informative about real -world risk because they’re too narrow. Because you have to build a set of questions that gives you some information about the vast range of use cases in the real world. And as models have become more capable and general and adopted more widely, this has become much more difficult. And I don’t think there’s very many actors out there that are constantly thinking about new ways to evaluate the capabilities.
And so I think this… This is like an important gap in terms of our toolkit. that is, again, quite urgent because these models have been released and we’re using our current evaluations, which are already, in many cases, out of date and not super informative about real -world risk. Shannon, do you want to jump in here?
Yeah. Stephen, I agree with you so much. I mean, all of us are obsessed with benchmarks because that’s kind of all we have, and they’re just so narrow. I spend a lot of time with organizations that we think will become these IVOs, and testing is so, so hard. I mean, think about this. We have a fundamentally stochastic system, so I can ask something 10 times, system 10 times, and I’m going to get 10 different answers. So what does that mean in a safety context? Another problem that we have, what a model outputs is not the same thing as what someone does with it. So think about in the context of mental health. Maybe the model says to 10 different people different versions of, I think you should kill yourself.
Nine times, maybe for nine of those users, that’s fine, they will laugh it out. But for one of those users, there’s going to be a real problem here. and also the multi -turn nature of AI. I mean, you build relationships with these systems and you ask long queries and the stuff just gets really complicated really quickly as technical minds could explain far better than I could. So what we’re trying to do here is incentivize better testing because right now the only people creating evals or eval organizations or doing God’s work, doing awesome stuff, but what does it mean? You’re the best meter out there. I mean, there’s not an incentive to go from good to the best.
And the other actor working, of course, are the labs. And I think many of the labs are actually attempting to be responsible actors here, but again, there’s an incentive gap. I think the only way you’re going to solve this is to have an ecosystem where all of the actors are competing to have the best services, to have the best evaluations, to have the best feedback, to have the best feedback, to have the best feedback, And we hope one day one of these IVOs says, I’ve developed a new type of testing that figures out this kid safety thing that no one has ever thought about. And then the next day someone says, well, we have to be better because then everyone will want to be verified from that organization.
So you are incentivizing ever better testing. And as Stephen says, I think that just given how quickly and dramatically the capabilities and the risks of these systems are increasing, we need really good testing and tooling that can keep up with that. And the only way to do that is to incentivize
So, Stephen, if I could come to you about what Shana just said. You pointed out how the state of the art in evaluations and assessment is constantly shifting as the capabilities are shifting. I sometimes hear the frontier labs say, yes, and that’s why we’re the only ones who can do the testing, because we’re the ones out there on the frontier. But Shana is making this point about misaligned incentives, which I think we saw. In a conversation you and I had a couple weeks ago in the XAI Grok undressing children kind of example, there’s perverse incentives sometimes at work here in terms of the companies evaluating themselves. So how do you reconcile that gap between the frontier AI labs often do have a unique perspective and a unique understanding, but also it’s really hard to see how we could ever be comfortable with them being the only ones assessing themselves?
Well, I can talk about a bit in the context of the report where we try to work with everybody to get the state of the science across the whole landscape. And there I think it is true that there’s this big information asymmetry between the people in the labs who both have the most technical capacity and also the most access to leading models and all of the information about testing and development and all of the information about the technology that’s being used in the lab. And if you don’t draw on that knowledge, you can’t really do anything about it. you’re not going to be able to understand what’s actually going on in the AI world but then I think we brought in a lot of perspectives from academia and society and government feedback to sort of get a full perspective of the landscape as far as what to do going forward to deal with this I think probably it looks something like this with partnerships that are aiming to draw on that knowledge but then aiming for transparency and information sharing that gives third parties and external actors a better understanding of what’s actually going on because it’s true like even writing the report we were reliant on these papers that labs will occasionally publish and drop with like very useful data on how people are using the models or adoption rates but we’re kind of reliant on these like ad hoc publications and then that leaves a lot of gaps across the landscape and different risks and so we you know constantly had the word uncertainty or unknowns in the report because we lack that data outside of the labs
And do you think that that’s likely to remain the case, or do you think that that could change over time? As we’ve seen, literally, the safety staff of some of these labs quit and start their own auditing companies. So are they likely to have their skills atrophy as they get farther from the development process, or do you think it’s credible that these third -party organizations can build, the word that comes to mind is like economies of scale that are relevant to be able to continue advancing the state -of -the -art of safety and governance, even as the technology keeps evolving?
I’m not sure, but I think what we can do is sort of look at the trend, and the trend is towards, I think, a stronger ecosystem around AI labs. As more people, as these problems of lack of data and lack of independent verification are identified more, there’s more people working on it. And then I think we’ve seen some movement towards greater transparency with AI labs as well. So frontier safety frameworks are now a governance mechanism that’s in the EU. AI are in the code of practice, and it’s become institutionalized. It started as a voluntary, anthropic, just published, a responsible scaling policy. And so you see these movements towards sharing more information in more structured ways.
I think also, yesterday, there were the new commitments from the companies at the summit, which were related to sharing data about usage. So I think as a broader set of actors in society are paying attention to AI, because, again, we’re feeling the effects more clearly. It’s becoming more of an economic priority. We’ll see more demand from outside the labs to share this information, and maybe that will lead to some changes.
Hiroki, you’ve written a ton about AI, but in your capacity as a lawyer, you also have a lot of understanding of many different industries. Are there any lessons learned from other industries here that have solved this sort of technical expertise exists here, but the need for independence exists here? What kind of precedents do you see that we can learn from?
Okay, so before that, let me add one more incentive, which is public procurement. If the government says, we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, LLM or model is safe and then government procures this standard then it will be a big incentive for developers so that is one thing and when I try to answer your questions I think democratic debate is necessary as to what kind of risk level is acceptable and also what kind of test measures are good because there is any single specific answer as to this is acceptable level of perspective.
For example in Japan every year more than 2 ,000 people were killed by a human driven car and the question is what kind of safety would we require for the autonomous vehicles? Is it okay if the kill number is less than 2 ,000 or would we like to require more safety than human drivers? If so, what would be the level? There is no single answer to that kind of question so we need to debate. in a democratic manner as to what is our acceptable goal. And also about the test measures. For example, we can just simply compare the number of rates per kilometers, but if you test in a very safe straight highway, of course it’s easier to get to safety.
While if you try to drive in a pretty complex city, it’s gonna be very difficult. So how to measure how to define the test method is another question. And I don’t go into the details, but the thing that discussion has been done in a lot of industries, car industries, or finance industries, or aerospace industries, we can certainly do a lot of lessons learned from the existing.
Yeah, one analogy that as you were talking, you jogged my memory, is the National Highway Transportation Safety Administration in the United States, which actually industry begged for this organization to be created. They did. in the 60s and 70s because they said, look, all of us are going to claim that we have safe cars, but only some of us are making big investments in becoming safe, and we want to reward the people whose good behavior is making big safety investments. And so they created this new organization which would give cars a safety rating on one to five, five star or one star. And so now the companies can only get a five star rating if they’re actually doing what it takes to be safe.
And consumers, you know, they’re not always qualified to rip open their car’s engine and see what it looks like under the hood, what’s safe, but they can interpret that five star rating. And so my idea was to ask you, Shana, to elaborate on this in the context of your model, but I’m now scared of the beeper, which is quite loud and scary. So please join me in thanking our terrific panel. Thank you.
Gregory C. Allen
Speech speed
167 words per minute
Speech length
2623 words
Speech time
939 seconds
Policymakers must ensure robust implementation of safeguards
Explanation
Gregory stresses that the next critical step for AI safety lies with policymakers, who need to make sure that the technical safeguards described in the report are put into practice widely and consistently across the industry.
Evidence
“it’s now in the hands of policymakers to make sure that these safeguards get implemented robustly and diversely” [30].
Major discussion point
State of AI governance & safety (2026)
Topics
Artificial intelligence | The enabling environment for digital development
Stephen Clare
Speech speed
189 words per minute
Speech length
2162 words
Speech time
685 seconds
Foundational report shows real, widespread risks
Explanation
Stephen notes that risks once thought theoretical are now evident in practice, underscoring the urgency of the International AI Safety Report’s findings.
Evidence
“Risks that even a year or two ago might have been theoretical are now very real and we’re seeing emerging empirical evidence” [16].
Major discussion point
State of AI governance & safety (2026)
Topics
Artificial intelligence | Building confidence and security in the use of ICTs
Technical safeguards are harder to jailbreak but still vulnerable
Explanation
Stephen reports measurable safety gains as models become significantly tougher to jailbreak, yet he cautions that safeguards remain vulnerable in edge cases and are unevenly applied across the sector.
Evidence
“Models are becoming much harder to jailbreak” [43]. “The first is that these technical safeguards are still vulnerable in many ways” [38]. “application remains quite inconsistent” [35].
Major discussion point
Technical safeguards: progress and remaining limits
Topics
Artificial intelligence | Building confidence and security in the use of ICTs
Layered allocation of safety responsibilities
Explanation
Stephen argues for a defense‑in‑depth approach where developers embed safety in training, deployers monitor and filter usage, and ecosystem bodies track AI‑generated content globally.
Evidence
“I think probably it’s not the case that it’s fair or helpful or true to allocate to one actor or another, but instead we need this layered approach of just many different policies and practices at different parts of the stack” [121]. “And then probably for ecosystem monitoring bodies which could be deployers but could also be other institutions… tracking how AI content is spreading across borders” [127].
Major discussion point
Allocation of responsibilities (layered, defense‑in‑depth)
Topics
Artificial intelligence | Capacity development
Evaluation suites are narrow and outdated
Explanation
Stephen points out that current benchmark suites fail to capture real‑world risk, are quickly obsolete, and do not reflect the stochastic, multi‑turn nature of modern models.
Evidence
“But I think in many cases, these evaluations we’re using are already not super informative about real‑world risk because they’re too narrow” [20]. “They can still be jailbroken with enough effort or in edge cases, and it’s very difficult to test and provide reliable assurances that these safeguards will work across this huge range of use cases” [41].
Major discussion point
Evaluation and benchmarking gaps
Topics
Artificial intelligence | Monitoring and measurement
Hiroki Hibuka
Speech speed
149 words per minute
Speech length
1274 words
Speech time
509 seconds
Update existing privacy, copyright and sector laws instead of new AI statutes
Explanation
Hiroki argues that the core regulatory question is not whether to regulate AI, but how to adapt and update the many existing legal frameworks—privacy, copyright, finance, health—to cover AI systems.
Evidence
“So the real question is not whether or not to regulate AIs, but the real question is how to update our existing regulations and whether or not we need additional regulations targeting AI systems” [69].
Major discussion point
Global regulatory approaches (hard law vs. soft law, sector‑specific)
Topics
Artificial intelligence | The enabling environment for digital development
EU adopts hard‑law AI Act; Japan and US favor sector‑specific approaches
Explanation
Hiroki contrasts the EU’s comprehensive, hard‑law AI Act with Japan’s and the United States’ preference for sector‑specific, either “exempt” or “exposed” regulatory styles.
Evidence
“Maybe the most famous regulation is the EU AI Act” [12]. “And in that context, a lot of people say, hey, EU takes a hard law regulatory approach on AIs while Japan or UK or United States takes a software approach” [58]. “the main difference I understand is whether you prioritize the exempt approach or exposed approach the US takes more exposed approach” [62].
Major discussion point
Global regulatory approaches (hard law vs. soft law, sector‑specific)
Topics
Artificial intelligence | The enabling environment for digital development
Japan needs an agile, multi‑stakeholder soft‑law framework
Explanation
Hiroki calls for a more flexible, multi‑stakeholder soft‑law approach in Japan to complement its traditionally compliance‑driven hard‑law culture.
Evidence
“So we need to have more agile and multi‑stakeholder approach” [36].
Major discussion point
Global regulatory approaches (hard law vs. soft law, sector‑specific)
Topics
Artificial intelligence | The enabling environment for digital development
Economic incentives for audits are weak
Explanation
Hiroki notes that without clear financial benefits, corporations are unlikely to adopt independent audits or evaluations of AI systems.
Evidence
“But it would be not easy to persuade corporate executives to use the independent audit without clear economic incentives” [133]. “there is no clear financial incentives” [134].
Major discussion point
Incentives for audits and verification
Topics
Artificial intelligence | Financial mechanisms
Lessons from other industries – standards and third‑party audits create market pressure
Explanation
Hiroki points to established standards such as aerospace’s AS9100 and consumer product certifications (UL) as models for how third‑party audits can drive safety and consumer trust in AI.
Evidence
“the safety standard in the American aerospace industry is AS9100” [159]. “And you can imagine a situation much like Underwriters Lab in the United States where… all consumer products… have the UL seal of approval” [160]. “And I don’t go into the details, but the thing that discussion has been done in a lot of industries, car industries, or finance industries, or aerospace industries, we can certainly do a lot of lessons learned from the existing” [165].
Major discussion point
Lessons from other industries and standards analogies
Topics
Artificial intelligence | The enabling environment for digital development
Shana Mansbach
Speech speed
175 words per minute
Speech length
2464 words
Speech time
843 seconds
Pervasive trust gap across stakeholders
Explanation
Shana describes a widespread lack of trust affecting the public, deployers, regulators, and developers, stemming from uncertainty about AI safety and outcomes.
Evidence
“That uncertainty creates a trust problem, a trust problem for the public, which doesn’t have a way of figuring out what is actually safe, a trust problem for deployers…” [91]. “So there’s a trust problem for the regulators, too” [92]. “I think it’s a reflection of the trust problem here where when you’re a deployer…” [93]. “And I would say there’s a trust problem for the developers also” [95].
Major discussion point
Trust problem and the case for independent verification
Topics
Artificial intelligence | Building confidence and security in the use of ICTs
Marketplace of government‑authorized Independent Verification Organizations (IVOs)
Explanation
Shana proposes an outcomes‑based, government‑overseen marketplace of IVOs that continuously update testing to keep pace with AI advances, providing an external seal of safety.
Evidence
“we’re very focused on independent verification specifically the marketplace of independent verification organizations by which I mean a government‑authorized and overseen marketplace of independent verifiers” [103]. “Under this system, the IVOs, independent verification organizations, are constantly updating their testing and criteria to make sure that they’re keeping up with the pace of technology and the pace of risks as well” [104].
Major discussion point
Trust problem and the case for independent verification
Topics
Artificial intelligence | The enabling environment for digital development
Liability, insurance and market advantage as carrots for verification
Explanation
Shana identifies three strong incentives for organizations to seek verification: clear liability regimes, access to insurance, and a competitive market edge.
Evidence
“the liability clarity that this is a big carrot” [142]. “the insurance piece is real right now we are seeing the big insurers saying we’re not going to touch any AI products” [142]. “I think the third thing is just straight‑up market competitive advantage” [143]. “They have a huge market advantage” [144].
Major discussion point
Incentives for audits and verification
Topics
Artificial intelligence | Financial mechanisms
Testing difficulty due to stochastic, multi‑turn outputs
Explanation
Shana highlights that the unpredictable, multi‑turn nature of modern models makes reliable testing hard, creating a need for better incentives to develop richer benchmarks.
Evidence
“and also the multi -turn nature of AI” [9]. “They can still be jailbroken with enough effort or in edge cases, and it’s very difficult to test and provide reliable assurances” [41]. “we need really good testing and tooling that can keep up with that” [154].
Major discussion point
Evaluation and benchmarking gaps
Topics
Artificial intelligence | Monitoring and measurement
Agreements
Agreement points
AI governance challenges are urgent and real-world impacts are happening now
Speakers
– Stephen Clare
– Hiroki Hibuka
– Shana Mansbach
Arguments
AI risks that were theoretical 1-2 years ago are now real with a billion people using AI globally, creating urgent governance needs
All countries face the challenge of regulating black box technology with unlimited risk scenarios and no clear benchmark standards
Current AI governance creates a trust problem for public, deployers, regulators, and developers due to uncertainty around system safety and reliability
Summary
All speakers agree that AI governance has moved from theoretical concerns to urgent real-world challenges affecting billions of users, requiring immediate attention from policymakers and stakeholders
Topics
Artificial intelligence | Building confidence and security in the use of ICTs
Multi-stakeholder approach is necessary for AI safety and governance
Speakers
– Stephen Clare
– Hiroki Hibuka
– Shana Mansbach
Arguments
AI safety requires layered approach with different responsibilities across developers, deployers, and monitoring bodies rather than single actor accountability
Democratic debate is necessary to determine acceptable risk levels and testing measures, similar to discussions in automotive and aerospace industries
Independent verification through government-authorized marketplace of verifiers offers outcomes-based approach rather than procedural compliance
Summary
All speakers recognize that effective AI governance cannot be achieved by any single actor and requires coordinated efforts across multiple stakeholders including developers, deployers, regulators, and society
Topics
Artificial intelligence | The enabling environment for digital development
Current evaluation and testing methods are inadequate
Speakers
– Stephen Clare
– Shana Mansbach
Arguments
Current evaluation methods have significant gaps and are often outdated, not informative about real-world risks due to AI’s stochastic nature
Traditional command-and-control governance fails due to AI’s speed of development and concentration of technical expertise in frontier labs
Summary
Both speakers acknowledge that existing methods for evaluating AI systems are insufficient to capture real-world risks and cannot keep pace with technological advancement
Topics
Artificial intelligence | Monitoring and measurement
Information asymmetry between labs and external actors creates governance challenges
Speakers
– Stephen Clare
– Shana Mansbach
Arguments
Information asymmetry exists between frontier labs with technical capacity and external actors needing transparency for effective governance
Traditional command-and-control governance fails due to AI’s speed of development and concentration of technical expertise in frontier labs
Summary
Both speakers identify the concentration of technical expertise in private companies as a fundamental challenge for independent oversight and governance
Topics
Artificial intelligence | The enabling environment for digital development
Similar viewpoints
Both speakers recognize that market-based incentives, particularly through insurance and regulated sectors, could drive AI safety standards more effectively than voluntary compliance
Speakers
– Hiroki Hibuka
– Shana Mansbach
Arguments
Financial incentives for independent auditing are lacking except in regulated sectors like healthcare, automotive, and finance where trust is essential
Insurance companies are refusing to cover AI products, creating potential market regulation through insurance requirements similar to aerospace industry standards
Topics
Artificial intelligence | Financial mechanisms
Both speakers acknowledge progress in AI safety measures while recognizing that different jurisdictions are taking varied but legitimate approaches to regulation
Speakers
– Stephen Clare
– Hiroki Hibuka
Arguments
Technical safeguards are improving significantly with models becoming much harder to jailbreak and 12 leading companies now having frontier safety frameworks
Different countries take varying approaches – EU’s holistic regulation vs Japan/US sector-specific approaches, with Japan preferring ex-ante rules while US uses ex-post liability
Topics
Artificial intelligence | The enabling environment for digital development
Both speakers understand that current liability frameworks create perverse incentives that discourage proactive safety measures, requiring new approaches to align incentives with safety goals
Speakers
– Shana Mansbach
– Gregory C. Allen
Arguments
Independent verification could establish rebuttal presumption of meeting heightened standard of care, clarifying liability before harms occur
Companies may prefer willful blindness to avoid liability, while auditing creates costs and competitive disadvantages without clear benefits
Topics
Artificial intelligence | The enabling environment for digital development
Unexpected consensus
Technical progress in AI safety is substantial and measurable
Speakers
– Stephen Clare
– Hiroki Hibuka
Arguments
Technical safeguards are improving significantly with models becoming much harder to jailbreak and 12 leading companies now having frontier safety frameworks
Different countries take varying approaches – EU’s holistic regulation vs Japan/US sector-specific approaches, with Japan preferring ex-ante rules while US uses ex-post liability
Explanation
Despite the focus on challenges and risks, there was unexpected consensus that significant technical progress has been made in AI safety, with measurable improvements in safeguards and widespread adoption of safety frameworks by major developers
Topics
Artificial intelligence | Building confidence and security in the use of ICTs
Insurance markets could become de facto AI regulators
Speakers
– Shana Mansbach
– Gregory C. Allen
Arguments
Insurance companies are refusing to cover AI products, creating potential market regulation through insurance requirements similar to aerospace industry standards
Companies may prefer willful blindness to avoid liability, while auditing creates costs and competitive disadvantages without clear benefits
Explanation
There was unexpected agreement that insurance companies withdrawing from AI coverage could create more effective regulation than government action, essentially forcing safety standards through market mechanisms
Topics
Artificial intelligence | Financial mechanisms
Need for new governance models beyond traditional regulation
Speakers
– Stephen Clare
– Hiroki Hibuka
– Shana Mansbach
Arguments
AI safety requires layered approach with different responsibilities across developers, deployers, and monitoring bodies rather than single actor accountability
All countries face the challenge of regulating black box technology with unlimited risk scenarios and no clear benchmark standards
Independent verification through government-authorized marketplace of verifiers offers outcomes-based approach rather than procedural compliance
Explanation
All speakers converged on the idea that traditional regulatory approaches are insufficient for AI, requiring innovative governance models that blend public and private sector capabilities
Topics
Artificial intelligence | The enabling environment for digital development
Overall assessment
Summary
The speakers demonstrated strong consensus on the urgency of AI governance challenges, the inadequacy of current approaches, and the need for multi-stakeholder solutions. They agreed on both the progress made in technical safeguards and the fundamental limitations of existing regulatory frameworks. There was notable alignment on the role of market mechanisms, particularly insurance, in driving safety standards.
Consensus level
High level of consensus with complementary rather than conflicting perspectives. The speakers built upon each other’s arguments rather than disagreeing, suggesting a mature understanding of AI governance challenges. This consensus implies that there is a clear foundation for developing new governance approaches that combine technical expertise, democratic accountability, and market incentives.
Differences
Different viewpoints
Who should conduct AI safety evaluations and auditing
Speakers
– Stephen Clare
– Shana Mansbach
Arguments
Information asymmetry exists between frontier labs with technical capacity and external actors needing transparency for effective governance
Independent verification through government-authorized marketplace of verifiers offers outcomes-based approach rather than procedural compliance
Summary
Clare acknowledges the information asymmetry problem but suggests partnerships that draw on lab knowledge while ensuring transparency, whereas Mansbach advocates for fully independent verification organizations to avoid the conflict of interest inherent in self-evaluation
Topics
Artificial intelligence | The enabling environment for digital development
Approach to AI regulation – holistic vs sector-specific
Speakers
– Hiroki Hibuka
Arguments
Different countries take varying approaches – EU’s holistic regulation vs Japan/US sector-specific approaches, with Japan preferring ex-ante rules while US uses ex-post liability
Summary
While not a direct disagreement between speakers, Hibuka presents fundamentally different regulatory philosophies across jurisdictions – EU’s comprehensive AI Act versus Japan/US preference for sector-specific regulation, and cultural differences between ex-ante rule-setting versus ex-post liability approaches
Topics
Artificial intelligence | The enabling environment for digital development
Incentive structures for safety compliance
Speakers
– Hiroki Hibuka
– Shana Mansbach
– Gregory C. Allen
Arguments
Financial incentives for independent auditing are lacking except in regulated sectors like healthcare, automotive, and finance where trust is essential
Insurance companies are refusing to cover AI products, creating potential market regulation through insurance requirements similar to aerospace industry standards
Companies may prefer willful blindness to avoid liability, while auditing creates costs and competitive disadvantages without clear benefits
Summary
The speakers identify different primary drivers for safety compliance – Hibuka focuses on regulated sectors and public procurement, Mansbach emphasizes insurance market forces and competitive advantage, while Allen highlights perverse incentives that discourage voluntary auditing
Topics
Artificial intelligence | Financial mechanisms | The enabling environment for digital development
Unexpected differences
Role of democratic processes in AI governance
Speakers
– Hiroki Hibuka
– Shana Mansbach
Arguments
Democratic debate is necessary to determine acceptable risk levels and testing measures, similar to discussions in automotive and aerospace industries
Independent verification through government-authorized marketplace of verifiers offers outcomes-based approach rather than procedural compliance
Explanation
While both speakers support government involvement in AI governance, Hibuka emphasizes the need for democratic debate to determine acceptable risk levels, whereas Mansbach focuses on technocratic solutions through independent verification. This represents an unexpected philosophical divide between democratic deliberation versus expert-driven governance approaches
Topics
Artificial intelligence | Human rights and the ethical dimensions of the information society | The enabling environment for digital development
Overall assessment
Summary
The main areas of disagreement center on governance mechanisms (self-regulation vs independent verification), regulatory approaches (holistic vs sector-specific), and incentive structures (market-driven vs regulatory mandates). While speakers agree on the inadequacy of current systems and the need for multi-stakeholder approaches, they propose fundamentally different solutions.
Disagreement level
Moderate disagreement with significant implications – the speakers share common concerns about AI safety and governance challenges but propose competing frameworks that could lead to very different regulatory outcomes. The disagreements are constructive and focus on implementation approaches rather than fundamental goals, suggesting potential for synthesis of ideas.
Partial agreements
Partial agreements
Both speakers agree that current evaluation and governance methods are inadequate for AI systems, but Clare focuses on improving existing technical evaluation methods while Mansbach proposes replacing the entire governance framework with independent verification organizations
Speakers
– Stephen Clare
– Shana Mansbach
Arguments
Current evaluation methods have significant gaps and are often outdated, not informative about real-world risks due to AI’s stochastic nature
Traditional command-and-control governance fails due to AI’s speed of development and concentration of technical expertise in frontier labs
Topics
Artificial intelligence | Monitoring and measurement
All speakers agree that AI governance requires multi-stakeholder approaches and that current systems are inadequate, but they propose different solutions – Clare advocates for layered responsibilities, Mansbach for independent verification marketplaces, and Hibuka for democratic debate on acceptable risk levels
Speakers
– Stephen Clare
– Shana Mansbach
– Hiroki Hibuka
Arguments
AI safety requires layered approach with different responsibilities across developers, deployers, and monitoring bodies rather than single actor accountability
Current AI governance creates a trust problem for public, deployers, regulators, and developers due to uncertainty around system safety and reliability
All countries face the challenge of regulating black box technology with unlimited risk scenarios and no clear benchmark standards
Topics
Artificial intelligence | Building confidence and security in the use of ICTs | The enabling environment for digital development
Similar viewpoints
Both speakers recognize that market-based incentives, particularly through insurance and regulated sectors, could drive AI safety standards more effectively than voluntary compliance
Speakers
– Hiroki Hibuka
– Shana Mansbach
Arguments
Financial incentives for independent auditing are lacking except in regulated sectors like healthcare, automotive, and finance where trust is essential
Insurance companies are refusing to cover AI products, creating potential market regulation through insurance requirements similar to aerospace industry standards
Topics
Artificial intelligence | Financial mechanisms
Both speakers acknowledge progress in AI safety measures while recognizing that different jurisdictions are taking varied but legitimate approaches to regulation
Speakers
– Stephen Clare
– Hiroki Hibuka
Arguments
Technical safeguards are improving significantly with models becoming much harder to jailbreak and 12 leading companies now having frontier safety frameworks
Different countries take varying approaches – EU’s holistic regulation vs Japan/US sector-specific approaches, with Japan preferring ex-ante rules while US uses ex-post liability
Topics
Artificial intelligence | The enabling environment for digital development
Both speakers understand that current liability frameworks create perverse incentives that discourage proactive safety measures, requiring new approaches to align incentives with safety goals
Speakers
– Shana Mansbach
– Gregory C. Allen
Arguments
Independent verification could establish rebuttal presumption of meeting heightened standard of care, clarifying liability before harms occur
Companies may prefer willful blindness to avoid liability, while auditing creates costs and competitive disadvantages without clear benefits
Topics
Artificial intelligence | The enabling environment for digital development
Takeaways
Key takeaways
AI safety has made significant technical progress with models becoming much harder to jailbreak and 12 leading companies now having frontier safety frameworks, but governance challenges are becoming more urgent as risks move from theoretical to real-world impacts
A fundamental trust problem exists across all stakeholders (public, deployers, regulators, developers) due to uncertainty about AI system safety and reliability, which traditional regulatory approaches cannot adequately address
AI governance requires a layered, multi-stakeholder approach with different responsibilities distributed across developers, deployers, and monitoring bodies rather than relying on any single actor
Independent verification through government-authorized marketplaces of verifiers could provide outcomes-based governance that is more flexible and democratic than traditional command-and-control regulation
Insurance companies are increasingly refusing to cover AI products, potentially creating de facto market regulation similar to aerospace industry standards, regardless of government regulatory approaches
Current AI evaluation methods have significant gaps and quickly become outdated, creating challenges for establishing reliable standards of care and best practices
Different countries are taking varying regulatory approaches (EU holistic vs Japan/US sector-specific), but all face similar challenges with black box technology and unlimited risk scenarios
Resolutions and action items
Need to establish financial incentives for independent auditing, particularly in regulated sectors like healthcare, automotive, and finance
Require democratic debate to determine acceptable risk levels and testing measures for AI systems
Develop marketplace of independent verification organizations (IVOs) that can provide right-sized compliance based on risk type and product size
Create standards of care that provide rebuttal presumption of meeting heightened legal standards for verified AI systems
Increase transparency and information sharing between frontier labs and external actors to address information asymmetries
Leverage public procurement as an incentive mechanism by requiring government purchases of AI systems to be independently verified
Unresolved issues
How to balance the concentration of technical expertise in frontier labs with the need for independent oversight and verification
What constitutes acceptable risk levels for AI systems across different sectors and use cases
How to develop evaluation methods that can keep pace with rapidly evolving AI capabilities and remain informative about real-world risks
How to address the stochastic nature of AI systems in safety testing when the same query can produce different outputs
What the appropriate distribution of liability should be across developers, deployers, and end users when AI systems cause harm
How to create sufficient financial incentives for voluntary adoption of independent auditing without regulatory mandates
How to prevent the verification system from becoming a barrier that only protects incumbents while excluding smaller players
Suggested compromises
Implement a marketplace approach for independent verification that allows for right-sized compliance based on product risk and company size rather than one-size-fits-all requirements
Create partnerships between frontier labs and external actors that draw on internal technical knowledge while ensuring transparency and independent oversight
Develop sector-specific approaches that focus initial independent verification requirements on high-risk areas like healthcare, finance, and automotive before expanding to other sectors
Establish verification systems that provide liability clarity and insurance benefits as market incentives rather than relying solely on regulatory mandates
Use outcomes-based governance that sets safety goals while allowing flexibility in how those outcomes are achieved and measured
Thought provoking comments
The rubber is really hitting the road or something with these kind of systems. Risks that even a year or two ago might have been theoretical are now very real and we’re seeing emerging empirical evidence… There’s a billion people now using AI around the world.
Speaker
Stephen Clare
Reason
This comment fundamentally reframes the AI governance discussion from theoretical future concerns to present-day reality. It establishes that we’ve crossed a threshold where AI risks are no longer hypothetical but are manifesting in real-world impacts with massive scale.
Impact
This set the foundational tone for the entire discussion, shifting focus from ‘what might happen’ to ‘what is happening now.’ It justified the urgency of all subsequent governance discussions and provided the empirical grounding that made other panelists’ policy proposals feel immediately relevant rather than premature.
The real question is not whether or not to regulate AIs, but the real question is how to update our existing regulations and whether or not we need additional regulations targeting AI systems… all countries take the hard law approach and also all countries have soft laws.
Speaker
Hiroki Hibuka
Reason
This comment dismantles a common false dichotomy in AI governance debates between ‘regulation vs. no regulation’ and reveals the more nuanced reality of how different regulatory frameworks actually operate. It challenges oversimplified narratives about EU vs. US vs. Japan approaches.
Impact
This reframing shifted the conversation away from broad comparisons of national approaches toward more specific discussions about implementation mechanisms and stakeholder responsibilities. It elevated the sophistication of the governance discussion by focusing on practical regulatory design rather than ideological positions.
The current approach to tech governance is not equipped to handle this trust problem very well… AI moves really, really quickly, and even well-intentioned regulations are going to become outdated very, very quickly, and then there’s the technical capacity problem.
Speaker
Shana Mansbach
Reason
This comment identifies the fundamental mismatch between traditional regulatory approaches and the unique characteristics of AI technology. It articulates why existing governance models are structurally inadequate, not just temporarily behind.
Impact
This diagnosis of systemic inadequacy justified the need for entirely new governance models rather than incremental reforms. It set up the intellectual foundation for proposing independent verification organizations as a novel solution, making the subsequent detailed discussion of IVOs feel necessary rather than speculative.
Independent evaluation is essential… But it would be not easy to persuade corporate executives to use the independent audit without clear economic incentives… there is no clear financial incentives.
Speaker
Hiroki Hibuka
Reason
This comment cuts through idealistic proposals to identify the core practical barrier to implementation: misaligned economic incentives. It forces the discussion to confront the gap between what’s theoretically desirable and what’s practically achievable.
Impact
This observation redirected the entire conversation toward incentive design, leading to rich discussions about insurance, liability, procurement, and market dynamics. It grounded the theoretical governance models in economic reality and sparked the most practical parts of the discussion.
Right now we are seeing the big insurers saying we’re not going to touch this we’re not going to insure any AI products because we have no idea what’s inside of them… If you are a major bank and you are doing big, important financial transactions, as soon as you start using AI, you’ve lost all your insurance.
Speaker
Shana Mansbach and Gregory Allen
Reason
This reveals a critical but underexplored dimension of AI governance: how insurance markets are creating de facto regulation through risk assessment. It shows how market forces may impose constraints that formal regulation hasn’t yet addressed.
Impact
This insight opened up an entirely new thread about insurance as a governance mechanism, leading to discussions about how market-based incentives could drive safety standards. It demonstrated how economic forces might solve governance problems that political processes struggle with, adding a new dimension to the policy toolkit.
I think in many cases, these evaluations we’re using are already not super informative about real-world risk because they’re too narrow… And as models have become more capable and general and adopted more widely, this has become much more difficult.
Speaker
Stephen Clare
Reason
This comment exposes a fundamental technical limitation in current AI safety approaches: the evaluation methods themselves are inadequate for assessing real-world risks. It reveals that even the technical foundations of governance are shaky.
Impact
This technical reality check sobered the discussion about independent verification, forcing acknowledgment that even well-intentioned third-party auditing faces serious methodological challenges. It added necessary complexity to proposals for verification organizations and highlighted the need for continued innovation in evaluation methods themselves.
Overall assessment
These key comments collectively transformed what could have been a superficial policy discussion into a sophisticated analysis of AI governance challenges. The progression moved from establishing empirical urgency (Clare’s ‘rubber hitting the road’), through dismantling false dichotomies (Hibuka’s regulation reframing), to proposing structural solutions (Mansbach’s verification organizations), then confronting practical barriers (economic incentives), discovering unexpected governance mechanisms (insurance markets), and finally acknowledging technical limitations (evaluation gaps). This created a comprehensive view that balanced optimism about governance solutions with realism about implementation challenges, ultimately producing a more nuanced and actionable understanding of AI governance needs.
Follow-up questions
Where is the end? And to what extent do stakeholders have to manage the risks?
Speaker
Hiroki Hibuka
Explanation
This addresses the fundamental challenge of determining acceptable risk levels in AI governance, as risks can never be completely eliminated and technology advances rapidly
How to design benchmarks and regulation methods for cutting-edge technologies that are black box with unlimited risk scenarios?
Speaker
Hiroki Hibuka
Explanation
This highlights the need for new approaches to evaluate and regulate AI systems where traditional methods may not be sufficient
How do we assure broader adoption of safety frameworks, how do you ensure compliance, what do you do when there’s a lack of compliance?
Speaker
Stephen Clare
Explanation
This addresses the governance challenge of moving from technical solutions to ensuring widespread implementation across the industry
How far are we from converting consensus on risks and interventions into standards and procedures for third-party evaluation?
Speaker
Gregory C. Allen
Explanation
This explores the gap between understanding AI risks and establishing accepted best practices for evaluation and auditing
What kind of safety level would we require for autonomous vehicles compared to human drivers?
Speaker
Hiroki Hibuka
Explanation
This illustrates the broader challenge of determining acceptable risk thresholds for AI systems across different applications
How to measure and define test methods for AI systems given varying complexity of use cases?
Speaker
Hiroki Hibuka
Explanation
This addresses the technical challenge of creating meaningful evaluations that reflect real-world deployment scenarios
How do we reconcile the gap between frontier AI labs having unique expertise but also having misaligned incentives for self-assessment?
Speaker
Gregory C. Allen
Explanation
This explores the tension between technical expertise concentration and the need for independent evaluation
Will third-party organizations be able to build economies of scale to advance safety and governance as technology evolves?
Speaker
Gregory C. Allen
Explanation
This questions whether independent auditing organizations can maintain technical competence as they become separated from frontier development
How to create meaningful evaluations for fundamentally stochastic systems where outputs vary across identical inputs?
Speaker
Shana Mansbach
Explanation
This addresses the technical challenge of testing AI systems that produce different outputs for the same input
How to evaluate AI safety when model outputs don’t directly correlate with user actions and outcomes?
Speaker
Shana Mansbach
Explanation
This highlights the complexity of assessing real-world harm potential from AI system outputs
Disclaimer: This is not an official session record. DiploAI generates these resources from audiovisual recordings, and they are presented as-is, including potential errors. Due to logistical challenges, such as discrepancies in audio/video or transcripts, names may be misspelled. We strive for accuracy to the best of our ability.
Related event
