Who Watches the Watchers Building Trust in AI Governance

Again, to my immediate right, we have Stephen Clare, who wrote the International AI Safety Report as the co -lead author, if I’m not mistaken. And he earned that applause, because that report is a remarkable document that I do think is the foundation upon which all conversations about AI governance now must rest for the next year. It’s the sort of minimum amount of knowledge that you must have to participate in the conversation, which I think is really a tribute to him. Then we have Hiroki Hibuka, who is currently a research professor at the Kyoto University Graduate School of Law, and was also deeply involved in drafting Japan’s first set of soft law regulations, and is an expert on all things AI, but also especially astute at what’s going on in Japan.

We also have a privilege of collaborating with him at CSIS, where he’s a non -resident senior associate. And I must say, he is probably the best person writing about Japanese AI policy in Japanese, but he is definitely the best person writing about it in English. And so I often tell Hiroki that, like, if he doesn’t write about it, nobody in Washington, D .C. knows about it. So it’s important, his work. And then finally, we have Shana Mansbach, who’s the vice president of strategy and communications at Fathom, which is a young think tank, started only two years ago, but has already succeeded as one of the best conveners of the ASHFE conference series on AI, and also now leading a policy initiative, which I think she’s going to tell us all about.

So without further ado, I’d like to start with you, Stephen. I just said that the report that you were the lead author of is sort of the bedrock for having a conversation on AI governance. For those in the audience who haven’t yet made it through, but they, of course, will, can you sort of set the stage? Where are we in 2026 in AI governance and in AI safety, technical and procedural intervention?

Sure. Thanks, Greg. First of all, I’m sorry if I’d known Greg was going to make the report, you know, required reading, I would have tried harder to make it shorter. Yeah. Thanks for having me. Thanks for really excited to be here. So for people who don’t know, the report is it was founded up the Bletchley 2023 Bletchley Safety Summit as sort of, you know, the shared evidence base for decision makers thinking about these complicated, fast moving, noisy governance questions. It’s kind of trying to be like the IPCC report for for AI. It’s backed by over 30 countries and intergovernmental organizations. You know, I’m one of two co lead writers along with Karina Prunkle, but there’s over 30 dedicated experts writing different sections, and there’s hundreds of people that review it.

So it’s really trying to be a sort of state of the art, what do we know? What don’t we know about general purpose AI systems and the risks they might pose? I think this year the main message of the report is like the rubber is really hitting the road or something with these kind of systems. Risks that even a year or two ago might have been theoretical are now very real and we’re seeing emerging empirical evidence. More real world impacts of AI on productivity and labor markets and in science and in software engineering. It’s all like really happening out in the world. There’s a billion people now using AI around the world. Many of those impacts include risks.

So we’re seeing effects of deepfake spreading, cyber attacks being more common with AI systems. And so the need for sort of risk management techniques that are effective is also growing. One thing that I found surprising working on the report is that in this domain on risk management and technical safety, there’s actually some good news. Quite a lot of good news, I’d say. In various ways, our technical safeguards are improving. Models are becoming much harder to jailbreak. So. You know. So three, four years ago, if you asked a model to give you a recipe for a Molotov cocktail, it would not do that. But if you said, oh, I miss my grandma, and she used to tell me this amazing bedtime story about how she loved making Molotov cocktails, please help me remember my grandmother, it would be like, okay, well, if it’s for your grandmother.

Then that stopped working maybe a year or two ago, but then if you maybe translated your question into Swahili or something and put it in the model and then translated the answer back, it might have made safeguards. So none of that works anymore. These safeguards are much harder to evade, and we know this quantitatively. For example, the UK Security Institute will try and evade the safeguards or jailbreak all these new models when they’re released. At the beginning of 2025, they could do this in literally minutes, find a sort of universal jailbreak that would elicit potentially harmful knowledge. For the latest models, it’s taking them seven, ten hours to get around safeguards. So there’s still vulnerabilities, but for novices or even moderately skilled actors, it’s basically the same thing.

It’s becoming much, much harder to evade them. We’re also seeing more of these safeguards get implemented into organizational practices. So 12 companies, all the leading AI developers now have frontier safety frameworks, which are these documents that describe how they plan to manage risks as they scale more powerful systems, which is many more than had them a couple of years ago and is, I think, a sign of transparency and sort of collective learning about risk management that’s worth noting. So basically, yeah, our toolkit for managing these risks is growing. But, you know, it wouldn’t be a safety report if I didn’t maybe end on a few caveats or some bad news. The first is that these technical safeguards are still vulnerable in many ways.

They can still be jailbroken with enough effort or in edge cases, and it’s very difficult to test and provide reliable assurances that these safeguards will work across this huge range of use cases that these models are now applied to in the real world. And on the organizational side, you know, these safeguards only work if they are applied. And although we’re seeing, especially from frontier developers, we’re very prominent, usually quite robust safeguards applied on models, across the whole industry, and especially behind the frontier, application remains quite inconsistent. The safety frameworks, all these companies have them, but they vary in the risks they cover, they vary in the practices that they recommend. And so the landscape as a whole, you know, these tools only work if they are applied.

And we still see that, some vulnerabilities across the landscape, which I think turns this technical challenge, that points towards the governance challenge of how do we assure broader adoption, how do you ensure compliance, what do you do when there’s a lack of compliance. We’re sort of facing these questions, and again, because these risks and the impacts are now not something that we can sort of push down the road anymore, I think, for future years, the governance questions are becoming a lot more urgent.

Terrific. And if I could contrast what you said with what we might have said if we were having this conversation back at the Bletchley Park AI Summit. But it’s almost like the only good news on AI safety, AI security, and AI governance at Bletchley was, well, at least we’re all here talking about it. And now, three years later, the good news is we’ve done a lot about it. We have techniques that can provide demonstrable increases in safety. We don’t know everything that we need to work, but we know a lot of stuff that does work. And really, a lot of the challenges, I think, as the report says, it’s now in the hands of policymakers to make sure that these safeguards get implemented robustly and diversely.

So with that, I now want to turn to Hiroki, who I hope can give us a state of where we are in the story of AI governance around the world. If the next steps are really in the hands of policymakers, where are we globally?

Thank you, Greg. And again, congratulations. Stephen was the publisher of the great report. And I think, first of all, I feel very glad that now the discussion on AI governance is such advanced compared to three years ago. I’m a lawyer and I’m a former policymaker. I worked for the Japanese government for four years, designing the Japanese AI policies, mainly in terms of regulation and governance. And as a lawyer and policymaker, the question after reading the report is, where is the end? And to what extent stakeholders have to manage the risks? Because in the end, you can’t remove all the risks. AI is black box and the technology advances so fast. And even though there is advance and progress of Godwins, the next day you may find another risk.

So there is no end to the story of how regulators should design the regulations. That is the main question. All countries. Countries are facing and different nations, regions take different approaches. Maybe the most famous regulation is the EU AI Act. And in that context, a lot of people say, hey, EU takes a hard law regulatory approach on AIs while Japan or UK or United States takes a software approach. But I think it’s a completely wrong understanding of the regulatory framework because, as you know, there are already lots of regulations that can be applied to AI systems. Privacy protection laws, copyright laws, or sector -specific laws such as finance, automotive or healthcare. We already have a lot of regulations out there.

So the real question is not whether or not to regulate AIs, but the real question is how to update our existing regulations and whether or not we need additional regulations targeting AI systems. In addition to the existing regulatory framework, so in that sense all countries take the hard law approach and also all countries have soft laws because European Union there are a lot of technical standards to implement the EU AI Act that are now under discussion but anyways all countries have both hard laws and soft laws that is the start of the discussion and then when we compare EU approach and Japan approach the clear difference is whether to regulate AI holistically or not sector -specific and when I compare the Japanese policy and the US policy we are on the same position as to taking a sector -specific regulation the main difference I understand is whether you prioritize the exempt approach or exposed approach the US takes more exposed approach you can do whatever you want to do and the regulation is usually very high level the principle is very high But once you have a problem, if you damage others’ properties or lives, then you go to the court and you fight in the court.

The Japanese society is not like that. In Japan, actually the number of losses is very low. People prefer to set the rules in advance. Japanese companies are very, very good at complying with the given rules. But they are not very good at creating their own governance mechanisms or explaining to stakeholders why you are doing that. And now Japanese stakeholders are starting to realize that it doesn’t work. So we need to have more agile and multi -stakeholder approach. So we are trying to leverage the power of soft laws, negotiating among different stakeholders, and give the standards, guidances. But in the end, again, if you violate the existing hard laws, of course you will be sanctioned. So that’s the main differences in American approach and Japan approaches.

And in the end, all countries are facing difficult questions of how to deal with this cutting -edge technologies that are black box and there are unlimited risk scenarios. And sometimes we don’t know how to evaluate the values such as privacy or transparency or fairness. There has been no clear benchmark standards so far in the society. So how to design those benchmarks and regulation methods are the challenges all countries are facing.

Terrific, Hiroki. And Shaina, I know you have a unique perspective on this because your organization is now proposing sort of additional models of AI governance that are not really reflected in existing law, whether in the United States or Europe or Japan or India. So walk us through what you see as the important work we’re doing now.

Sure. My panelists have set me up very well to say this. So I think as the International AI Safety Report shows, the capabilities around these models are surging. And as the capabilities surge, so too does the uncertainty around the risks, by which I mean, do these systems work safely, securely, and as advertised? That uncertainty creates a trust problem, a trust problem for the public, which doesn’t have a way of figuring out what is actually safe, a trust problem for deployers, by which I mean hospital systems, retail, banks, who want to and indeed need to use these systems, but have no idea what they can actually trust. So there’s a trust problem for the regulators, too.

They don’t know, how do you confer not just trust, but how do you confer earned trust? And I would say there’s a trust problem for the developers also, because if and as trust starts to grow, there’s a trust problem for So if the trust starts to decline, you’re going to see adoption decline as well, so this is something that developers should be focused on too. The current approach is just not the current approach to tech governance is not equipped to handle this trust problem very well. Traditional command and control governance says here are the rules, here are all the things you have to do, here are the procedures, here is what compliance actually here’s what compliance actually looks like.

There are a bunch of problems with this approach in the context of AI, but I’ll focus on two, which is the speed problem. AI moves really, really quickly, and even well -intentioned regulations are going to become outdated very, very quickly, and then there’s the technical capacity problem. Even with the rise of the AI safety institutes, which are doing amazing work, the talents, the expertise for understanding these systems and understanding their risks is largely concentrated in the frontier labs, which of course leads some people to say, well, let’s just go to the frontier labs. They can regulate themselves. I don’t think I have to spend too much time explaining why there are problems with that approach but it’s simple incentives I think all of us know people in the labs who are doing amazing, amazing work they are the people who make sure that I can because of them I sleep better at night but the incentives are just not there there are always going to be trade -offs between investing in safety testing and tooling and investing in development so we’re going to have problems with self -regulation in terms of addressing that trust gap so where does that lead us?

at Fathom, my organization, we’re very focused on coming up with new models that can solve this trust gap so we’re very focused on independent verification specifically the marketplace of independent verification organizations by which I mean a government -authorized and overseen marketplace of independent verifiers which are trying to be charged with creating testing and tooling to determine whether these AI systems are actually safe The difference here is that this is an outcomes -based approach. Instead of, as I said, having procedures, here are the rules, here are all the things you need to do, here are all the boxes you must check to be certified as being good, you have an outcomes -based approach where you have a government saying, here are the things that we care about.

We care about children’s safety. We care about data privacy and protection. We care about controllability and interpretability. And then you have independent verifiers that can actually go out, do the testing, have updated testing constantly to make sure that those outcomes are being met. We think that independent verification solves for a couple of these deficits in the trust context. First, they are independent. The labs are not grading their own homework. Second, democratic accountability. You have governments that are creating outcomes instead of the industry doing it itself. Third, flexibility. Under this system, the IVOs, independent verification organizations, are constantly updating their testing and criteria to make sure that they’re keeping up with the pace of technology and the pace of risks as well.

And I think the fourth thing, which is pretty interesting, is it creates a race to the top here. Right now, the only people working on safety testing and tooling are in the labs. What we’re envisioning is a marketplace that incentivizes ever better testing and tooling here. I could talk about IVOs for days and days, but let me just end on one point. I was talking to Greg about this earlier, and Greg asked, are there analogous systems or industries or sectors that we could talk about? And I said, yeah, sort of. I mean, in America, we have Underwriters Lab. There’s LEED certification. There are some analogies. But the honest answer is there’s not a perfect analogy.

We have had the same regulatory system for the last century. And I think that with the rise of AI, we’re seeing that system is no longer built for purpose. And when we try to use old systems, hard law, soft law, any of these things, we’re really struggling to make it work. So what I’m trying to do, what I’d encourage all of us to do is to say, you know, we do need to think a little bit differently. Because this is what this technology in this time calls for.

Well, that’s great. So there’s a few points I want to pull together there. The first is, you know, as Hiroki pointed out, in the U .S. system, liability law looms extremely large, right? The lawsuits at the end of this story when things go wrong. And when you have, as, for example, ChatGPT does, 800 million weekly average users, something’s going to go wrong every week, right? And the question is… How is that going to intersect with our existing body of regulation? How is that going to intersect with liability law? The second thing is this is going to, because we’re talking about these general purpose technologies, this is going to be adopted in so many different sectors of the economy.

And right now, as Shana pointed out, the number of people who have, you know, Steven’s expertise on what it takes to really make AI systems safe and well -governed and perform reliably as intended across the whole range of potential applications, that’s not a lot of humans on planet Earth who are good at that stuff. And because these AI models are going to be deployed in just about every sector of the economy, we need some level of those capabilities in every sector of the economy. And so the question is, you know, if I am a financier, if I am a finance company, if I am a health care company, you know, how am I going to know and how are my consumers going to know?

that when they use AI -related capabilities, it’s going to work reliably as intended over the full range of acceptable use cases. And so, Stephen, I want to come to you and ask, when it comes to governance, when it comes to oversight and verification, how do you see the balance of responsibilities in terms of what responsibilities need to fall upon the model developers, what responsibilities need to fall upon the users, what responsibilities need to fall on independent third parties, whether that’s the government, whether that’s auditors, whether that’s this marketplace of verification that Shana is talking about. So what do you see as the balance of responsibilities, and how might this go wrong, how might this go right?

In 30 seconds or less.

I mean, I’m sure it’s kind of the boring but true answer. It’s the boring part of it. depends and it’ll vary a lot across use cases and sectors. I think probably it’s not the case that it’s fair or helpful or true to allocate to one actor or another, but instead we need this layered approach of just many different policies and practices at different parts of the stack. Because none of our approaches are foolproof, they all have vulnerabilities, and so we have, instead of safety by design, we have this safety by degree situation where we want defense in depth. So for developers, there will be training techniques that they can implement to make models less likely to elicit dangerous knowledge in the first place.

If there are people building on top of those models and then deploying them, there will be monitoring systems they can put in place and classifiers that identify dangerous queries and stop models from answering them. and then probably for ecosystem monitoring bodies which could be deployers but could also be other institutions in the world there can be tracking how AI content is spreading across borders and around the world and then I think there’s this other aspect of we’re focusing a lot on sort of model or developer safety but as we are moving into this world where many people around the world are having access to powerful, helpful intelligent technologies and we also just need to adapt for that reality and think about resilience at the societal level too of how do we adapt to the beneficial use cases and the various use cases that these models will be used for so thinking about hardening digital systems against increased cyber attacks just sort of admitting the reality of the situation in many ways and adapting to it rather than trying to prevent all harmful uses in the first place I think we need a variety of approaches across all these different actors

Yeah. And just to use an analogy for how broad the group of stakeholders is, if you think about a ride hailing service, a taxi service like Uber, you have the automobile manufacturers who have to make sure that this is a solid car design that was manufactured safely and appropriately to specification. Then you have Uber, where in some countries Uber owns the car, and so they’re responsible for ensuring that it gets maintenance appropriately. And then you have the driver who’s responsible for ensuring that they are actually following the law and driving the car safely. And if you apply that analogy to AI, you have the model developer, then you might have the sort of business use case deployer, which could be a bank, a medical device company.

Who? A financial institution, whoever. And then you finally have the end customer who’s receiving those services and making sure that they’re using them appropriately. And so. If you think about that sort of different body of use cases, as I said before, the capabilities are not symmetric across all of those. But there are sort of obligations. And so, Shana, I want to come back to you and ask this model that you’re proposing, what exactly does it mean for the different stakeholders in the ecosystem? How does their life change if we adopt the system that you’re in favor of?

Yeah, I mean, the overarching answer is we create trust throughout the system, which is the missing piece here. I think there are a couple of pieces that I would pull out. You had mentioned liability earlier, and let me talk about that a little bit. What this system does, it does not assign liability. It doesn’t say, you know, deployers, developer, it’s you, it’s you, it’s you. We’re seeing, at least in America, courts move their way through this. Sister. court cases move their ways through the court system and we’ll see where that is but where that ends up being but what is really missing is a standard of care and this is I think one of the real advantages that this system has so right now at least how it works in our current tort system is that if you’re Waymo kill someone someone can sue and then a judge and a jury has to figure out so again we’re not answering who should be sued but let’s say that the family of someone who got hurt or killed is suing Waymo what happens is that the jury has to decide whether whether the person who was sued did the right thing and if you are not technical that is the hardest thing even if you are technical and maybe even Waymo doesn’t know So what this system would do is confer, if you are verified, it would confer, the verification would confer a rebuttal presumption of having met a heightened standard of care.

So what we’re doing is clarifying and defining up front before an actual harm happens what a deployer or whoever is sued is actually supposed to do instead of having this very, very messy system where someone after the fact has to figure out what went wrong and who’s responsible for that. I can talk about other layers of this back here, but I think the liability piece is really key. I mean, we just see this. I think it’s a reflection of the trust problem here where when you’re a deployer, I mean, God, I think everyone that I talk to, you know, again, hospital systems, retail, banks, anyone who needs to be consumer facing is really worried about this problem.

I mean, when I get sued, what do I do? And maybe there’ll be. a populist backlash and everyone will hate everyone who’s using AI systems. And it’s much better to, ahead of something like that, ahead of that happening, have that standard of care defined up front and have that seal of approval conferred.

And Hiroki, as you think about the different stakeholders in the system and especially the idea of auditors, which now there are a number of organizations being founded, it seems like almost every day, who are proposing to provide external evaluation services that can help companies understand, as Shane has said, this product or this service or this company meets the seal of approval and we vouch for it as an independent entity. What kind of momentum do you see for this independent assessment part of the story across regulatory frameworks?

Independent evaluation. Independent evaluation is essential given that we are all using AI systems for all different situations, starting from language models to healthcare systems to car driving. But it would be not easy to persuade corporate executives to use the independent audit without clear economic incentives. For example, if you get the certification for autonomous driving, then you can sell the car to the big market. Then, of course, you pay for the audit. But if you take this audit for this language model, then you can prove that this language model is relatively safer than the other models. But it doesn’t necessarily make enough incentive for model developers to conduct the audit or evaluation systems, independent evaluation, because there is no clear financial incentives.

Actually, could I? I can ask you to elaborate on that. So where might these financial incentives come from? You mentioned one, which is the regulators force you to do it. That’s one. Maybe insurance is one. another like where where might these incentives come from

I think it should start from the regulated areas such as cars health care systems finance systems or infrastructures because everybody needs a strong requires strong trust on those systems if it doesn’t work well then somebody might have a baby kills that’s a big problem and maybe you could say hey but in the end if you are killed you can be compensated but it’s not the end of the story while if the damage could be compensated by money by the company and stakeholders are okay with that maybe companies like to just run the system go and and compensate to the victims for example if the language model says something discriminated the company can just say hey we’re very sorry we introduce better guardrails and we pay for that if you want compensation

in terms of what is possible, what interventions work, what the risks are. But I want to ask about how we go from that degree of consensus to something that might be more of like a standard around procedural implementation. You know, Shana’s term of art is standard of care, which matters a lot in the American legal system. I’m sure it matters a lot in other legal systems. I’m just ignorant about, you know, how and where. And so I’m curious, you know, what do you see as the gap? If these independent evaluators, these independent auditing organizations are emerging, how do they go from we think we’re good at this to, no, this is the accepted best practice?

You know, we have accepted consensus on the risks and the interventions, but, like, how do you turn that into a procedure? Just to give an example to the folks in the audience, I used to work at a rocket company, and the safety standard in the American aerospace industry is AS9100. And in the history, of our company, there’s kind of like a before AS9100 moment, and then there’s an after AS9100 moment. And everything changed for our company, you know, after we got that third -party audit evaluation. A lot of our customers, you know, just said, we do not sign checks for companies that are not AS9100 certified. So, you know, you are deeply steeped in where we are today on the consensus, but how far are we from converting that into standards and procedures for third -party evaluation?

Yeah. I’ll also say one follow -up to Hiroki’s point, too, about auditing. Not only is there sort of a lack of incentives to conduct audits voluntarily now, but there might even be disincentives where one is it’s costly, and it slows you down, and there’s very intense competitive pressures to release faster. And there’s also potentially… like, information or security risks to sharing. You spent hundreds of millions, maybe billions of dollars developing a model, and then you have to share it with an external party before deployment. Like, serious risks to, or perceived risks, at least, to having that information leak or… So I think, yeah, there’s some serious challenges there. I guess there’s one other potential part of the story, which is sometimes you see companies want to be willfully blind, right?

If they have a report that says my product is not safe, well, now they know they’re going to lose the lawsuit. Whereas if they never commission the report, maybe they’ll win the lawsuit. So, Shana, what do you see as meaningful interventions that can help address this problem, both the cost side that Stephen mentioned and the other parts of the incentive structure?

Yeah, let me make a couple of points. I mean, I think we’re talking about the cost of audits, and I think this… this is a big issue that we think about a lot. This system will not work if everyone, if there’s a flat fee, everyone is paying a ton. I mean, we are really, we think that an unsuccessful, there are many ways that a system looks unsuccessful, and one of those ways is if it is just protecting incumbents. And we’re thinking, we envision the system as something that works for, you could verify a general purpose LLM, you could also have narrow AI, you could have a tiny little tool, a little chatbot that is used in schools.

Those three different products should not be audited, not only at the same cost, but in the same way. I mean, compliance isn’t just the check that you’re writing, it is how much of a pain in the butt is it? How many lawyers do you need? How long will this take? So the great thing about this being a marketplace is that the system is right -sized to risk type, to size of these products. and again instead of having just a one size fits all this is what you have to do to comply because I think that that is a real issue it really quickly I just want to go back to you know the question that you asked Hiroki about incentives I mean you can imagine a system where this is mandatory and maybe in some areas you can imagine that but I think that there are three real real carrots for wanting to get verified we talked a little bit about liability so obviously the liability clarity that this is a big carrot I think the insurance piece the insurance piece is real right now we are seeing the big insurers saying we’re not going to touch this we’re not going to insure any AI products because we have no idea what’s inside of them at least in America the way that life insurance works is if you want insurance you have to have a lot of money and a lot of money and a lot of money and a lot of money you have to jump on a scale and tell someone how healthy you are and what are the things that you do and the insurer decides okay are you worthy of being insured and at what premium I think that’s actually a pretty direct analog for what we’re trying to do here where the books are opened and an insurer can look at whether they don’t have to do the testing themselves, but they can look at whether the system has been verified and say, okay, we will actually insure you or we will insure you at a more affordable premium.

I think the third thing is just straight -up market competitive advantage. If I’m a school superintendent and I am choosing between two learning chatbots to put in my schools, I’m not going to choose the one that has not been verified. I want the one that has been verified, that is safest. Yes, because I’m worried about getting sued, but because I want my kids to be safe. And you can imagine a situation much like Underwriters Lab in the United States where basically all consumer products like light bulbs, toothbrushes, basic things that you buy in a store like Walmart, all have the UL seal of approval, and those are the ones that get sold in stores. They have a huge market advantage.

They pay a little bit, but not very much. And in exchange for doing that, they go to market in a way that, or they compete in a market in a way that the ones that don’t go through verification. do. I’m so sorry, Gary, you asked me an actual question and I just answered everyone else’s question and probably not my own.

It’s okay. You get out of jail free card because you mentioned insurance, which is something I’m deeply interested in right now. I mean, in that space orbital launch vehicle example that I just mentioned, you can’t get insurance for space launches of satellites until you’re AS9100 certified. And that is 10 % of the cost of getting a satellite into space is just the insurance on the rocket. And so basically companies that can’t get insurance can’t compete in the market. And as Shana mentioned, and I think this is a super undercovered story, there are now many of the major insurers in the United States at least are saying, for your enterprise risk policy, AI is not included. So if you are a major bank and you are doing big, important financial transactions, as soon as you start using AI, you’ve lost all your insurance.

And I think the Trump administration in the United States has a very light -touch regulatory approach. And my concern there is that, well, just because the government is not doing anything big and bold on regulation doesn’t mean there will be no regulation. The insurers will step in. And if the insurers exit the market, maybe not in legal terms, but in economic outcome terms, that could be very similar to draconian regulation. So, Shana, you’re mentioning the Underwriters Lab, which is an organization that writes standards that are relied upon by underwriters, the people who are issuing insurance. This is a huge part of the regulatory and governance ecosystem that I think is really important. And so now I’m hoping, Stephen, that you’re going to tell me, that you’ve been reached out to by a bunch of insurance companies, and they’re all reading your report eagerly and thinking about this.

But maybe, maybe not. What’s the case?

Not yet, but it’s a really long report. 312 pages, but it goes like that. Maybe I can come back to the best practices point a little bit, because I think we’re talking about auditing here, and at least I know there’s a lot of steps involved, I’m sure, but at least at the technical level, the main tool we have right now to audit the capabilities of the RISC -MD AI model are evaluations. And although in my opening I sort of talked about, oh, it’s great we have this toolkit that’s emerging and it’s strengthening, and that is true, I think on evaluations in particular, as far as like, okay, let’s say we have auditors that are looking at these companies, looking at models, what are they actually looking at to audit or evaluate the models?

I think we actually have a big gap here, a big evaluation gap in terms of, well, how are we actually assessing? So if we’re moving towards best practices, not only do I think we don’t have a sense of the best practices right now, but if we did, they’d be different in a year, because the capabilities are moving too quickly for these technical tools to be in date, for very long. So for example, you’ll have, you know, these evaluations often look like a set of questions related to a certain topic, and you ask the model, so you have a bunch of questions about biosecurity or a bunch of questions about cybersecurity. And if it’s above, if it scores high enough on the test, you say, whoa, this is a dangerous capability, and we need to implement more safeguards or something.

And as far as what’s best practice or safe risk management for a company, we evaluate in terms of, well, does it seem like the safeguards apply proportionately to the risk that you’ve assessed? But I think in many cases, these evaluations we’re using are already not super informative about real -world risk because they’re too narrow. Because you have to build a set of questions that gives you some information about the vast range of use cases in the real world. And as models have become more capable and general and adopted more widely, this has become much more difficult. And I don’t think there’s very many actors out there that are constantly thinking about new ways to evaluate the capabilities.

And so I think this… This is like an important gap in terms of our toolkit. that is, again, quite urgent because these models have been released and we’re using our current evaluations, which are already, in many cases, out of date and not super informative about real -world risk. Shannon, do you want to jump in here?

Yeah. Stephen, I agree with you so much. I mean, all of us are obsessed with benchmarks because that’s kind of all we have, and they’re just so narrow. I spend a lot of time with organizations that we think will become these IVOs, and testing is so, so hard. I mean, think about this. We have a fundamentally stochastic system, so I can ask something 10 times, system 10 times, and I’m going to get 10 different answers. So what does that mean in a safety context? Another problem that we have, what a model outputs is not the same thing as what someone does with it. So think about in the context of mental health. Maybe the model says to 10 different people different versions of, I think you should kill yourself.

Nine times, maybe for nine of those users, that’s fine, they will laugh it out. But for one of those users, there’s going to be a real problem here. and also the multi -turn nature of AI. I mean, you build relationships with these systems and you ask long queries and the stuff just gets really complicated really quickly as technical minds could explain far better than I could. So what we’re trying to do here is incentivize better testing because right now the only people creating evals or eval organizations or doing God’s work, doing awesome stuff, but what does it mean? You’re the best meter out there. I mean, there’s not an incentive to go from good to the best.

And the other actor working, of course, are the labs. And I think many of the labs are actually attempting to be responsible actors here, but again, there’s an incentive gap. I think the only way you’re going to solve this is to have an ecosystem where all of the actors are competing to have the best services, to have the best evaluations, to have the best feedback, to have the best feedback, to have the best feedback, And we hope one day one of these IVOs says, I’ve developed a new type of testing that figures out this kid safety thing that no one has ever thought about. And then the next day someone says, well, we have to be better because then everyone will want to be verified from that organization.

So you are incentivizing ever better testing. And as Stephen says, I think that just given how quickly and dramatically the capabilities and the risks of these systems are increasing, we need really good testing and tooling that can keep up with that. And the only way to do that is to incentivize

So, Stephen, if I could come to you about what Shana just said. You pointed out how the state of the art in evaluations and assessment is constantly shifting as the capabilities are shifting. I sometimes hear the frontier labs say, yes, and that’s why we’re the only ones who can do the testing, because we’re the ones out there on the frontier. But Shana is making this point about misaligned incentives, which I think we saw. In a conversation you and I had a couple weeks ago in the XAI Grok undressing children kind of example, there’s perverse incentives sometimes at work here in terms of the companies evaluating themselves. So how do you reconcile that gap between the frontier AI labs often do have a unique perspective and a unique understanding, but also it’s really hard to see how we could ever be comfortable with them being the only ones assessing themselves?

Well, I can talk about a bit in the context of the report where we try to work with everybody to get the state of the science across the whole landscape. And there I think it is true that there’s this big information asymmetry between the people in the labs who both have the most technical capacity and also the most access to leading models and all of the information about testing and development and all of the information about the technology that’s being used in the lab. And if you don’t draw on that knowledge, you can’t really do anything about it. you’re not going to be able to understand what’s actually going on in the AI world but then I think we brought in a lot of perspectives from academia and society and government feedback to sort of get a full perspective of the landscape as far as what to do going forward to deal with this I think probably it looks something like this with partnerships that are aiming to draw on that knowledge but then aiming for transparency and information sharing that gives third parties and external actors a better understanding of what’s actually going on because it’s true like even writing the report we were reliant on these papers that labs will occasionally publish and drop with like very useful data on how people are using the models or adoption rates but we’re kind of reliant on these like ad hoc publications and then that leaves a lot of gaps across the landscape and different risks and so we you know constantly had the word uncertainty or unknowns in the report because we lack that data outside of the labs

And do you think that that’s likely to remain the case, or do you think that that could change over time? As we’ve seen, literally, the safety staff of some of these labs quit and start their own auditing companies. So are they likely to have their skills atrophy as they get farther from the development process, or do you think it’s credible that these third -party organizations can build, the word that comes to mind is like economies of scale that are relevant to be able to continue advancing the state -of -the -art of safety and governance, even as the technology keeps evolving?

I’m not sure, but I think what we can do is sort of look at the trend, and the trend is towards, I think, a stronger ecosystem around AI labs. As more people, as these problems of lack of data and lack of independent verification are identified more, there’s more people working on it. And then I think we’ve seen some movement towards greater transparency with AI labs as well. So frontier safety frameworks are now a governance mechanism that’s in the EU. AI are in the code of practice, and it’s become institutionalized. It started as a voluntary, anthropic, just published, a responsible scaling policy. And so you see these movements towards sharing more information in more structured ways.

I think also, yesterday, there were the new commitments from the companies at the summit, which were related to sharing data about usage. So I think as a broader set of actors in society are paying attention to AI, because, again, we’re feeling the effects more clearly. It’s becoming more of an economic priority. We’ll see more demand from outside the labs to share this information, and maybe that will lead to some changes.

Hiroki, you’ve written a ton about AI, but in your capacity as a lawyer, you also have a lot of understanding of many different industries. Are there any lessons learned from other industries here that have solved this sort of technical expertise exists here, but the need for independence exists here? What kind of precedents do you see that we can learn from?

Okay, so before that, let me add one more incentive, which is public procurement. If the government says, we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, and we recognize this is a very important issue, LLM or model is safe and then government procures this standard then it will be a big incentive for developers so that is one thing and when I try to answer your questions I think democratic debate is necessary as to what kind of risk level is acceptable and also what kind of test measures are good because there is any single specific answer as to this is acceptable level of perspective.

For example in Japan every year more than 2 ,000 people were killed by a human driven car and the question is what kind of safety would we require for the autonomous vehicles? Is it okay if the kill number is less than 2 ,000 or would we like to require more safety than human drivers? If so, what would be the level? There is no single answer to that kind of question so we need to debate. in a democratic manner as to what is our acceptable goal. And also about the test measures. For example, we can just simply compare the number of rates per kilometers, but if you test in a very safe straight highway, of course it’s easier to get to safety.

While if you try to drive in a pretty complex city, it’s gonna be very difficult. So how to measure how to define the test method is another question. And I don’t go into the details, but the thing that discussion has been done in a lot of industries, car industries, or finance industries, or aerospace industries, we can certainly do a lot of lessons learned from the existing.

Yeah, one analogy that as you were talking, you jogged my memory, is the National Highway Transportation Safety Administration in the United States, which actually industry begged for this organization to be created. They did. in the 60s and 70s because they said, look, all of us are going to claim that we have safe cars, but only some of us are making big investments in becoming safe, and we want to reward the people whose good behavior is making big safety investments. And so they created this new organization which would give cars a safety rating on one to five, five star or one star. And so now the companies can only get a five star rating if they’re actually doing what it takes to be safe.

And consumers, you know, they’re not always qualified to rip open their car’s engine and see what it looks like under the hood, what’s safe, but they can interpret that five star rating. And so my idea was to ask you, Shana, to elaborate on this in the context of your model, but I’m now scared of the beeper, which is quite loud and scary. So please join me in thanking our terrific panel. Thank you.