From principles to practice: Governing advanced AI in action

10 Jul 2025 10:40h - 11:10h

From principles to practice: Governing advanced AI in action

Session at a glance

Summary

This discussion focused on moving from AI governance principles to practical implementation, featuring experts from academia, government, and industry across different global regions. The conversation was framed as “AI governance speed dating” to explore various stakeholder perspectives on what approaches are working and what needs improvement in governing rapidly advancing AI technology.


Brian Tse opened by identifying four key frontier AI risks: misuse by malicious actors, accidents and malfunctions, potential loss of control scenarios, and systemic societal risks including labor market disruption. He emphasized the need for multi-stakeholder collaboration to address these global-scale challenges. Chris Meserole highlighted the promise of Frontier AI frameworks and voluntary commitments made at the Seoul AI Summit, noting that some risks require entirely new risk management instruments due to their potential for rapid, large-scale emergence.


Yaqin Zhang discussed practical implementation challenges, particularly the lag between rapid technological development and regulatory responses, advocating for continued R&D investment in risk identification and enhanced global collaboration. Juha Heikkila explained the EU’s approach through the AI Act, emphasizing trust as fundamental to AI adoption and describing mechanisms like the Code of Practice that allow for flexibility and updates while maintaining compliance frameworks.


Udbhav Tiwari provided a concrete example of Signal’s response to Microsoft’s Recall feature, illustrating how companies must sometimes take protective action independently when existing governance frameworks prove insufficient. He stressed the importance of addressing current real-world harms rather than focusing solely on hypothetical future risks.


The panelists’ recommendations for scaling effective governance included binding regulations for powerful AI systems, standardized frontier AI frameworks, honest conversations about limitations, safety-by-design approaches, and international cooperation on defining unacceptable AI outcomes. The discussion concluded with calls for developer agency, content identification systems, streamlined initiatives, and transparency mechanisms to build the trust necessary for responsible AI adoption and innovation.


Keypoints

## Major Discussion Points:


– **Frontier AI Risk Categories**: Four key types of risks were identified: misuse (lowering barriers for malicious actors in cyber attacks and dangerous pathogen design), accidents and malfunctions (like AI hallucinations in medical diagnosis), potential loss of control scenarios (where AI systems may deceive humans or evade oversight), and systemic societal risks (particularly profound labor market impacts).


– **Governance Approaches and Frameworks**: Discussion of various regulatory models including voluntary commitments (like the Seoul AI Summit frameworks), legislative approaches (EU AI Act), and industry self-regulation. Emphasis on the need for new risk management instruments specifically designed for rapidly emerging AI risks, particularly those that could scale quickly before harms become apparent.


– **Implementation Challenges Across Jurisdictions**: Participants highlighted the tension between rapid technological advancement and regulatory lag, with different regions (China, EU, US) developing divergent approaches. Key challenges include the need for regulatory clarity, threshold definitions, and coordination across different governance models.


– **Trust and Transparency as Core Requirements**: Strong emphasis on building societal trust through transparency, honest conversations about AI limitations, developer accountability, and “privacy by design” approaches. The discussion included concrete examples like Signal’s response to Microsoft’s Recall feature to protect user privacy.


– **International Cooperation and Standardization Needs**: Call for global alignment on defining “red lines” for unacceptable AI outcomes, standardizing risk management frameworks, streamlining proliferating initiatives, and establishing norms for transparency in frontier AI development.


## Overall Purpose:


The discussion aimed to explore practical approaches for translating AI governance principles into actionable practices, bringing together diverse stakeholders from academia, government, and industry to share lessons learned and identify effective governance models that can keep pace with rapid AI advancement.


## Overall Tone:


The conversation maintained a collaborative and constructive tone throughout, characterized by mutual respect among panelists despite representing different sectors and regions. The tone was serious and urgent given the stakes involved, but remained optimistic about the potential for international cooperation. The moderator’s “AI Governance Speed Dating” framing added a light touch while keeping the discussion focused and dynamic. All participants demonstrated a shared commitment to responsible AI development, even while advocating for different approaches.


Speakers

– **Sasha Rubel** – Head of Public Policy for Generative AI at AWS, Moderator


– **Ya Qin Zhang** – Chair Professor at Tsinghua University


– **Juha Heikkila** – Advisor for the International Aspects of Artificial Intelligence at the European Commission


– **Udbhav Tiwari** – Vice President of Strategy and Global Affairs at Signal


– **Brian Tse** – Role/title not specified, but appears to be involved in AI safety research and policy


– **Chris Meserole** – Role/title not specified, but appears to be involved with the Frontier Model Forum and AI governance


Additional speakers:


– **Unnamed moderator/host** – Introduced the panel at the beginning and concluded the session


Full session report

# AI Governance Speed Dating: From Principles to Practice – Discussion Report


## Introduction and Context


This panel discussion, described as “AI governance speed dating,” brought together experts from academia, government, and industry to explore the transition from AI governance principles to practical implementation. The conversation was moderated by Sasha Rubel, Head of Public Policy for Generative AI at AWS, and featured Ya Qin Zhang (Chair Professor at Tsinghua University), Juha Heikkila (Advisor for the International Aspects of Artificial Intelligence at the European Commission), Udbhav Tiwari (Vice President of Strategy and Global Affairs at Signal), and Brian Tse.


## Frontier AI Risks Framework


Ya Qin Zhang established a comprehensive framework for understanding AI risks, identifying four critical categories requiring immediate attention:


**Misuse by Malicious Actors**: AI systems that lower barriers for malicious activities, particularly in cyber attacks and dangerous pathogen design. Zhang emphasized these represent immediate threats with current AI capabilities.


**Accidents and Malfunctions**: AI systems producing incorrect or harmful outputs due to technical failures. Zhang provided a concrete example from Beijing, where medical prescription regulations were implemented after AI systems began making inappropriate medical recommendations.


**Potential Loss of Control Scenarios**: Situations where AI systems may deceive humans or evade oversight mechanisms, representing perhaps the most concerning category of risks.


**Systemic Societal Risks**: Broader societal impacts, particularly profound labor market disruption that could create systemic challenges requiring coordinated responses.


Zhang noted the regulatory gap with a striking analogy: “we have more regulations on the food safety of dumplings than [on AI systems] today.”


## Voluntary Commitments and Standards Development


Brian Tse highlighted developments in frontier AI governance, particularly voluntary commitments from international forums. He noted that some AI risks require entirely new risk management instruments due to their potential for rapid, large-scale emergence before harms become apparent.


A critical challenge Tse identified is the timeline mismatch between AI development and standards creation. Current formal standards processes, such as ISO SC402001, typically require “four to five years to develop”—incompatible with rapid AI advancement. He emphasized the need for accelerating formal standards development while maintaining rigor.


Tse recommended that firms publish their risk management frameworks and provide systematic implementation updates for public transparency, noting that “15 or 16 companies” have signed commitments at recent international summits.


## European Union’s Legislative Approach


Juha Heikkila provided insights into the EU’s approach through the AI Act, emphasizing trust as fundamental to AI adoption. The EU’s framework aims to build trust through clear requirements and accountability mechanisms that can adapt to technological changes.


Heikkila identified a critical gap in current approaches: the lack of systematic follow-up and effectiveness measurement. He emphasized that “for trust, which has been identified here as one of the key issues, it is actually important to know how and if and how, and to what extent, whatever measures and recommendations are there, were put into practice and how effective they are.”


He also addressed the challenge of proliferating AI governance initiatives, noting that multiple overlapping requirements create excessive burdens on developers and advocating for streamlined coordination between different governance frameworks.


## Academic Research Perspective


Ya Qin Zhang emphasized the critical lag between rapid technological development and regulatory responses, creating windows of vulnerability where harmful applications can emerge before appropriate safeguards are in place.


Zhang advocated for a “five-stage process: testing, auditing, verification, monitoring, and mitigation” and emphasized the importance of “ID, ID, ID” for registration and identification systems. She highlighted China’s regulatory approach, including generative AI registration requirements and the Beijing medical prescription regulations as examples of proactive governance.


From a research perspective, Zhang called for increased investment in R&D for identifying critical risks, developing appropriate benchmarks, and creating early warning systems. She argued that “safety must be foundation of product design, not an afterthought, with academic-industry-policy collaboration.”


## Industry Implementation Challenges


Udbhav Tiwari provided concrete examples of real-world implementation challenges through Signal’s experience. His most significant example was Signal’s proactive response to Microsoft’s Recall feature, which takes periodic screenshots of users’ screens. Signal deployed a feature preventing screenshots when the application is active to protect user privacy.


This example illustrated gaps in current governance frameworks, with Tiwari noting that “AI regulation has existed for some time, voluntary efforts have existed for quite some time, but yet Microsoft could deploy a lot of technology” that created privacy risks for users.


Tiwari emphasized the importance of addressing current, real-world harms: “it is just as important to recognize the very real harms that are occurring to real people with very real consequences that are taking place today because of these AI systems.”


He also criticized declining industry transparency, noting that model cards have become less informative over time due to industry hype, occurring precisely when greater openness is needed for effective governance.


## Key Recommendations


The discussion generated several specific recommendations:


**International Coordination**: Developing international standards for frontier AI risk management within accelerated timelines and establishing global norms for risk management transparency.


**Regulatory Mechanisms**: Implementing binding regulations for the most powerful AI systems while maintaining voluntary frameworks for lower-risk applications, and creating adaptive regulatory mechanisms that can be updated more quickly than full legislative revisions.


**Technical Standards**: Scaling registration and identification systems for AI-generated content globally and implementing content provenance and watermarking systems.


**Accountability and Monitoring**: Establishing follow-up mechanisms to measure governance effectiveness and requiring systematic publication of risk management frameworks and implementation updates.


## Ongoing Challenges


Several critical issues remain unresolved:


– Balancing rapid technological advancement with necessary governance frameworks across different regional approaches


– Lack of consensus on what constitutes “intolerable risks” and appropriate risk thresholds globally


– Tension between transparency needs and proprietary commercial development


– Ensuring inclusive governance that gives meaningful voice to underrepresented communities


## Conclusion


The discussion revealed both promise and challenges in translating AI governance principles into practice. While speakers shared common goals around safety, trust, and international cooperation, they demonstrated different approaches to implementation, particularly regarding the balance between voluntary and binding frameworks, and between addressing current harms versus preparing for future risks.


The conversation highlighted the urgent need for governance frameworks that can keep pace with technological development while addressing both immediate and long-term risks. Success will likely require hybrid approaches combining voluntary flexibility with regulatory accountability, maintaining collaborative approaches while navigating significant implementation challenges ahead.


Session transcript

Sasha Rubel: AI. I’m not hearing the round of applause. I’m calling for to welcome our guests here. Thank you. Yajing Zhang, Chair Professor at Tsinghua University. Yuhua Heikila, who’s Advisor for the International Aspects of Artificial Intelligence at the European Commission. And Udvav Tiwari, who’s Vice President of Strategy and Global Affairs at Signal. In a conversation moderated by Sasha Rubel, who’s Head of Public Policy for Generative AI at AWS. We wish you all a very fruitful conversation. Thank you. Thank you everybody for the warm welcome. Over the next half hour, we are going to deep dive into questions related to how to move from principles to practice in governing AI. That is the official title. The unofficial title could have been called AI Governance Speed Dating, where over the next 30 minutes or so, we will hear different perspectives from different stakeholders and regions of the world in terms of their approaches of what they are seeing working, what they are seeing not working, what they think needs to be scaled. And if you feel like swiping right on all of the above, you can. So I want to make sure that we allocate as much time as possible to the meat of these conversations and lessons learned that all of you are going to share. And I want to kick it off in terms of framing the conversation with you, Brian. Can you share what you’re seeing in terms of key frontier AI risks and what we should be paying attention to


Brian Tse: right now? First of all, it’s a great honor to be on this panel today. To ensure that AI could be used as a force for good, we have to start by identifying and managing some of the global scale challenges that we face as humanity. Drawing from the International AI Safety Report and some of the latest research, I think there are at least four types of frontier AI risk. Number one is misuse. As frontier AI systems achieve human expert level performance on areas like scientific reasoning, coding, and even philology, there’s also the potential for lowering the barrier for malicious actors to launch cyber attacks or even design dangerous pathogens. For example, recent studies have shown that teams of AI agents are able to exploit zero-day vulnerabilities in real-world settings in open-source softwares. So one measure could be providing early and differential access for developing countermeasures to make sure that we have defensive capabilities on the upper hand over the attackers. Number two is accidents and malfunctions. While hallucinations from chatbots could be quite amusing, it could be dangerous in use cases like medical diagnosis. So recently the Beijing government has implemented regulations prohibiting AI from writing medical prescriptions automatically. So classifying some applications as high-risk and making sure that we have robust public safety regulations is critical. Number three is potential loss of control scenarios. As Professor Geoffrey Hinton has mentioned in the summit, we don’t currently have robust mechanisms to ensure that digital superintelligence could be remained under human control. In fact, there is a growing body of evidence showing that frontier AI systems have the tendency to deceive humans or even try to evade human oversight under particular conditions. Now of course, frontier AI systems still struggle to bypass know-your-customer checks and will struggle to automatically replicate in the wild. But I think for catastrophic risk, even if they are low probability or of unknown probability, we should take precautionary measures. The fourth one is systemic risk at the societal level. One particular example is that as general purpose AI outperform humans on many of the economically valuable tasks, the impact on the labor market will be profound. And these gradual and diffuse impacts cannot be managed by any single organization. So we need a multi-signal approach, both at the national and the global level, to ensure that no one will be left behind. And I think that’s what makes the ITU platform so valuable. Thank you.


Sasha Rubel: Brian, thanks so much for that introductory overview. And I think your emphasis on the need for multi-stakeholder collaboration and dialogue and alignment is one of the reasons why this summit is so important, but also why it’s absolutely crucial to get these different groups in the room to agree on what does risk look like, what are we talking about, and what kind of governance is most effective. It’s a challenge, and all of you know this firsthand in the work that you’re doing, to think through governance models that address risks in a technology field where in reality advancements are moving super quickly. And Chris, you’ve done a lot of work at the Frontier Model Forum trying to think through very concretely what kind of governance models or approaches, whether that’s regulatory, institutional, or voluntary, can actually address the fast pace of the development of this technology. What do you believe is showing the most promise in terms of these different approaches that you’re seeing, particularly as it


Chris Meserole: concerns possible global alignment? Well, first of all, it’s great to be here and just, you know, a wonderful chance to speak to such a great multi-stakeholder audience for a really important conversation that hopefully we can make a lot of progress together with. You know, as far as kind of which approaches might have most promise, one thing that I would really flag would be the Frontier AI frameworks and the kind of voluntary commitments that a set of companies made in Seoul at the Seoul AI Summit in May of last year, where they committed to, you know, again, there were about 15 or 16 companies, I believe, globally that signed these commitments, and the commitment was to identify what they considered intolerable risks and, like, their plan for managing those risks. And, you know, over the last year and a half, most of the companies that made that commitment have spent a lot of time trying to flesh out what are now called Frontier AI frameworks or what others might call a safety and security framework. And the goal of this is to try and make sure that we have a risk management framework that is fit for purpose for the specific kinds of risks that were just alluded to. You know, Brian, I think, did a great job of outlining the full array of risks that might come with Frontier AI. Some of those risks, I think, can be managed by adapting existing risk management instruments, but there’s other risks that we might actually need an entirely new type of risk management instrument for, and in particular, there’s a class of risk that, if it appears, it may appear very quickly and at, you know, extreme scale very quickly in a way that, like, the harms don’t materialize until it’s too late to do anything about them. In which case, you need a risk management instrument that’s able to identify those issues in advance or as far in advance as possible, and that’s what Frontier AI frameworks are really designed to do, and I think there’s been a tremendous amount of progress over the last year in identifying which risks fall into that category. I think there’s growing consensus that, like, bio risks, advanced cyber risks, autonomous R&D capabilities, those kinds of capabilities really require, I think, a fundamental new type of risk management instrument that we’re starting to see a lot of. I would say right now we’re in the kind of emerging experimentation phase of what those should look like, but hopefully over time we’ll


Sasha Rubel: increasingly institutionalize them going forward. Chris, I love how you underline that a lot of the risks that we’re going to have to pay attention to are probably going to emerge really quickly, and the question of, okay, how do we see what already exists that can be adapted, and then where do we see the need to completely invent a new approach and framework to risk, and I want to turn, Yachin, to you because I’m curious, in this kind of approach of seeing where the risks are, what might emerge, what exists already, and what we might need to develop, what do you see as the practical challenges when implementing AI governance approaches, especially as models become more and more powerful, and how can we actually overcome some of these challenges?


Ya Qin Zhang: For me? All right. Nice to be here. There are a lot of impedances in implementation. I will just list three. First is just the rapid technological development events versus the governance regulatory lag, and if you look just in the last six months or a year, the tremendous AI safety. In the UK, EU has different guidelines, China has different guidelines, and the US has very different guidelines as well. Obviously, China is quite open and very inclusive in terms of how to approach the thing with companies. We’re very close with industries, defining how you collect data, how do you do testing, how do you register, and how do you release the big models. How do we overcome that? Probably time is not enough, but let me just mention one thing. We need to continue to invest in R&D in terms of how we identify those critical risks, catastrophic risks, to come up with the red lines, the benchmarks, the thresholds, and the warning system. Let me also just, I will talk later if I have time, the five-stage process, testing, auditing, verification, monitoring, and also the mitigation. But overall, I think the most important thing is for all of us to come together for enhanced global collaboration in a forum like this.


Sasha Rubel: Yatin, I love your emphasis on needing to invest in R&D, and also the need to have regulatory clarity in light of the pace of the technological advancement, but also the context that you provide in terms of different countries going ahead and developing different approaches to questions around risk, and also to AI governance. I want to turn to you now, at the forefront of a lot of these conversations in the EU as the EU implements the EU AI Act and the secondary legislation and guidelines related to its implementation, what are you seeing as practical challenges when implementing AI governance, again, especially as these models are becoming more and more powerful? There were conversations during the EU AI Act about thresholds, systemic risk, guidelines that are currently underway with the Code of Practice for general purpose AI. What do you see as the challenges and also the opportunities to overcome them?


Juha Heikkila: Thank you. Thank you very much. It’s indeed a great pleasure to be here and to be a member of this panel. So, first of all, the term governance is used in different ways. So, we have different kinds of international initiatives, principles, codes of conduct, recommendations, etc., on the one hand. Then, on the other hand, we have indeed either soft guardrails or hard law within jurisdictions. So, those have, of course, slightly different consequences, different implications in terms of compliance and monitoring compliance and enforcement, and the possibility to do that. So, this is actually one challenge in and of itself, which is not necessarily even dependent on, though it is complicated by, the progress of this technology. It is, of course, a well-known fact, as you said, that the European Union has decided to regulate AI. For us, it’s key to ensure trust, and we felt that for ensuring trust legislation was necessary, because trust is really the sine qua non for the take-up of this technology, which is a sine qua non for the benefits of AI to materialize, and we do believe that there are really very many benefits in this technology. So, in the AI Act, we have introduced a mechanism of enabling updates in a way which can keep up with this technology, so we have the high-risk areas defined in an annex, and that can be updated much more quickly and much more easily than having a full revision of legislation. On the other hand, you mentioned the Code of Practice. This is a process which is producing an instrument, a tool, to help facilitate compliance with the provisions on general-purpose AI models, and this process, it’s about to be published, this Code of Practice. It has been conducted, led by independent chairs and co-chairs, with strong multi-stakeholder support. We’ve had more than 1,000 stakeholders involved in this, and indeed, this highlights one of the aspects that was mentioned by Brian at the very beginning, this multi-stakeholder aspect is very important in these processes. But the Code of Practice is aimed at taking into account best practice, current industry practice, to the extent that it is feasible, and also to make it as future-proof as possible. But there is also some flexibility in implementing it, so those who sign up to this Code of Practice, they can in some cases choose the way, for example, how to put in place mitigation measures to comply with this Code of Practice, and this then obviously would be a presumption of compliance with the relevant provisions. It also refers to the state of art in a number of places, so that can be interpreted flexibly to take into account where we are. And of course, it is possible also for us to review it and revise it as necessary and clarify concepts if it’s required. Thank you.


Sasha Rubel: Thank you, Juha. I again appreciate your emphasis on the question of trust, because one of the things that we’re seeing in conversations with organizations across the world, as a blocker to AI adoption, is actually the question of trust, and how do we build trust both in the technology by having an approach of responsibility by design, but also between stakeholders that need to come together and work together to design these kinds of governance approaches. I want to turn to you, Udbav, on that specific issue, because I would be curious to hear from you at Signal what are you seeing in terms of the practical challenges in implementing governance, but also building trust? Signal has really been at the forefront of making sure that trust is the center of the ways in which you operate as an industry counterpart. How are you implementing AI governance inside of Signal, and what do you see as both the challenges, again, but also opportunities to build that trust that’s necessary?


Udbhav Tiwari: Thank you. Maybe I’ll start with an example. The purpose of the example is to illustrate how Signal, as a player in the market, is being both impacted and affected by AI, but is in turn also shaping how AI is deployed in practice, and maybe use that to talk about governance and what’s working and what’s not. Quite recently, Signal rolled out a feature on its desktop installations, where if you run Windows 11 and you try to take a screenshot on Windows 11 of the Signal application, all you will get is a blank screen. The reason we deployed that feature was because Microsoft had deployed Recall, a feature that takes periodic screenshots of your screen, stores them, runs an on-device model that analyzes those screenshots and answers questions that you might have about, say, when did I visit this website, or when did I go to this particular page, or when did I last look up this person’s profile? The reason we had to do that was because people use Signal because of the fact that they trust it with their most private information, and not everyone who uses Signal would necessarily understand. I’m here to talk about the summer of 2019. We understand the fact that the recall feature would be taking screenshots of the screen every three seconds and storing them in perpetuity to be able to answer questions that they might have. If you look at signal as a player in this market, AI regulation has existed for some time, voluntary efforts have existed for quite some time, but yet Microsoft could deploy a lot of technology to prevent that from happening, and that’s why we had to take action in our own hands, because we had to protect the privacy of our users against the kinds of technologies and context in which they are deployed, which are quite different from the context of many other AI systems, because a lot of the people who use signal are particularly high-risk individuals. Which is why connecting that to governance, I actually think that the solution to a lot of this governance conversation is all of these things need to happen in parallel. I don’t think signal is under pressure. I don’t think that a law will suddenly come into place by early next year that will prevent things like recall from being deployed without users understanding what it really means. I don’t think it means that only voluntary efforts are enough, because I think we’ve seen a lot of precedence from social media and a lot of other places in the industry that self-regulatory efforts have not really worked in practice, and I think ultimately acknowledging that as much as we talk about advanced AI, it is just as important to recognize the very real harms that are occurring to real people with very real consequences that are taking place today because of these AI systems. And unless we choose to address them rather than wondering whether AI will control Skynet three years from today, I think we’re going to end up in a scenario where some very real harms will occur and the trust that the society has on these technologies, which is already falling quite a bit, because it’s not exactly a very popular technology, if you talk to an artist and talk to a lot of other individuals, will keep decreasing. So if we want to earn society’s trust, I think we all need to both work together but also ensure we prevent today’s harms from occurring rather than focusing too much on what might


Sasha Rubel: be possible tomorrow. I love that call to action, Udbhav. Yeah. I love that call to action. It’s clear that many of you in the room also love that call to action in terms of making sure that we’re thinking through and starting with and working backwards from the impact that this has on individuals and users and communities, but I also think your emphasis on privacy by design and working in parallel are equally important. How do we embed in the approach from the beginning and the ways in which we develop this technology in parallel to efforts of governance and regulation and voluntary commitments privacy and responsibility by design in order to build that trust to encourage responsible adoption. We’ve heard a lot of different kinds of models and lessons learned from the five of you, and I want to shift a little bit to think through concretely if there was one element of governance that you would suggest be widely adopted or better coordinated across jurisdictions and stakeholder groups, what would it be and why? Some of you talked about R&D, voluntary commitments like in Seoul, legislative initiatives like the EU AI Act, internal governance approaches like what’s happening at Signal. What is the model of AI governance that you would want to push forward and make sure that there’s more cooperation on a global level that we can develop, as you said, very elegantly in parallel at the same time as the development of this technology? Brian, I want to start off with you. Thank you. Given the profound impact from front AI systems in the coming years, it’s quite striking that we have more regulations on the


Brian Tse: food safety of dumplings than they are today, and so I think we need more binding regulations, especially for the most powerful AI systems. Just to share two regulatory measures from my region, so the first is pre-deployment registration and licensing. Currently in China, all generative AI would have to be registered with the government before they are deployed to the public, going through a set of safety and security assessments, and this type of registration and licensing could adapt to different regional needs as well as to adapt to emerging challenges as our understanding of the threat landscape deepens. The second would be post deployment transparency. For example, having content provenance and watermarking on AI-generated content to make sure that users and society are able to differentiate the realities between those generated by AI versus humans. I think that’s critical, and that’s something that China has


Sasha Rubel: also implemented through a national standard requiring both developers as well as platforms to have explicit labeling as well as implicit metadata. Thank you.


Chris Meserole: Chris, let’s turn to you. What would be your recommendation in terms of something that’s working to scale and how? I think I would build on the comments I was making earlier in the sense that I think one of the things that hopefully will happen over the next 12 months is an increasing push to start standardizing frontier AI frameworks or risk management systems. There’s a lot of voluntary activity around them. That means there’s a lot of different types of frameworks that have been published, but I think coordinating globally around what a robust frontier AI risk management instrument should look like will be incredibly important. In doing that, though, I think we need to identify how we’re going to create formal standards on a very quick timeline, because I think the things like SC402001 at ISO, which is a great AI risk management standard, I think it took four or five years to develop, and I think we need to really be able to provide something with that kind of rigor, but in a much


Sasha Rubel: shorter timeline. I love that emphasis on the question of standards in terms of their importance in providing clarity, particularly as we think through and hear from organizations that want to scale, but in a safe way. Udbhav, over to you.


Udbhav Tiwari: I would say that two main things. The first is there is, I think, a recognition that the rate at which some of these technological items are being deployed is far faster than the rate at which they’ve really been understood, even sometimes by the organizations that develop and deploy them, and without that understanding, I think it’s very, very hard for us to come out, to come to the conclusion that if we follow process X or standard Y, all of these risks in these systems will be addressed, because in many ways it’s those risks themselves that are not very clearly understood, and the reasons they aren’t really understood is because of the fact that they often happen in incredibly proprietary and closed settings with very little interaction with the outside world, and the second is that even when it does happen, the consequences so much, like every time there’s a big model release, like everybody, that’s all everyone talks about for that day, like, on the internet, like this new model has come out, that people tend to be a lot more risk-averse on talking about the limitations and challenges of the kind of models that have been developed and things that they should not be used for. So if you just look at the model cards for all the prominent model providers over the last three to four years and just analyze the amount of information that used to be present in them and the amount of information that is present in them today for when they come out, we’ve actually seen a regression, and the reason we are seeing that regression is because there is a greater consequence to that information because of all the hype that has really built up around the industry. So I really think that we need to make sure that there’s space and opportunity for people to be able to be clear about faults and drawbacks just as much as there is approaches to what the benefits and progress that this technology is making, because unless we have an honest conversation about that, both from a deployment but also from an addressing problems perspective, I think, independent of whether it’s a self-regulatory body or an actual legislation, we will not be able to have an honest conversation, and that’s what’s really important.


Sasha Rubel: I love that emphasis on the need to have honest conversations. I think in parallel to the question around honest conversations, you also implicitly underlined the need to be asking the right questions. What kind of questions do we need to be asking? And a lot of times very hard questions with regards to both misuse but also missed use and beneficial use of this technology. Yaxin, over to you.


Ya Qin Zhang: I thought the National AI Safety Institute and a lot of the NGOs have played a very constructive and positive role in the last couple of years. you know, the assess model, you know, the work with the industries to monitor and publish the results. So that was great. Let me also mention one thing, and, you know, I’m glad to see top academics and thought leaders like Yashua Bengio and a lot of the AI pioneers are working with industry leaders and policy makers, and to make this work, you know, we have to do this in the very beginning, right? By design and the safety is a foundation of the product.


Sasha Rubel: It’s not an afterthought. I love that. Safety is the foundation and not an afterthought. It’s again one of the reflections that comes up more and more in conversations is how do we move from an approach of responsibility and safety that from bolted on to built in to by design? How do we ensure approaches that are by design safe and trustworthy? And I think Juha, you’ve spent a lot of time thinking about that as well in the work that you’re doing on the EU AI Act. What’s a kind of key lesson that you’ve taken away in terms of conversations on AI governance in the EU? Well, I think there are a number of lessons that could be drawn from that and could be related, but I think, you know, these are complex issues and they don’t actually, complex issues rarely lend themselves to simple solutions as much as we’d like to have simple solutions.


Juha Heikkila: But I think overall one thing that maybe is worth highlighting is the need for follow-up and update mechanisms. I mean, update we have already talked about, but follow-up as well, because for trust, which has been identified here as one of the key issues, it is actually important to know how and if and how, and to what extent, whatever measures and recommendations are there, were put into practice and how effective they are. So I think that this is one aspect that is important, of course, in the context of legislation that requires this kind of a mechanism that is kind of inbuilt, but it would be required also then for other sort of softer approaches.


Sasha Rubel: I want to build on your last statement, Juha, and shift to a kind of closing call to action and reflections in the couple of minutes that we have left, and ask all of you, looking ahead over the next 12 months, and 12 months in the time of AI is probably in dog years, 12 years, what’s one concrete step, is it a norm, a protocol, collaboration that’s needed, a standard you’d like to see move forward to make AI governance more effective in practice? A lot of you have talked about the need for international collaboration and alignment, the need for gaps analysis, the need for multi-stakeholder conversations. What’s one thing that you would like to see on the international agenda prioritized in terms of an approach that you’re seeing work that will move the needle on AI governance and particularly trustworthy and safe AI? Chris, I want to start with you.


Chris Meserole: I would just pick up on Juha’s point about trust and say that one of the things that really needs to happen globally is a norm that’s developed where firms that are developing frontier AI models, like genuinely kind of the newest, most powerful models, that they publish a risk management framework about how they’re going to handle the most extreme risks associated with their technology. And then that they also publish, this is a point that was also made, that they publish kind of systematic updates on how they’re implementing it so that the public can be aware of the activity that they’re doing and can kind of judge for themselves exactly how trustworthy these products are. Great call to action. Brian, over to you.


Brian Tse: I think a concrete step I would like to see is an international effort to define what we view as red lines or unacceptable outcomes from advanced AI systems. And based on those red lines, we can also better define a set of risk threshold and early warning indicators, as Professor Zhang Aqin has mentioned. And what is acceptable and what is safe should not be left to the AI industry alone. And as hopefully we have learned from past financial crises and environmental disasters, some of the unchecked risk-taking by a few elites could actually lead to systemic risk on a global level, including for the Global South and unprotected communities worldwide. So I think this is an area where the international community and civil society have to come together


Sasha Rubel: and have a strong and decisive voice. I love that call to action of coming together. Juha, what would you like to see over the next 12 months?


Juha Heikkila: Well, 12 months is a long time in AI, but it can also be quite a short time for some things. And I’m thinking of one thing in particular, which is the fact that we had kind of a proliferation of initiatives, which also means that there’s been a proliferation of monitoring, reporting, et cetera, requirements, and this can put burden on developers. And I think that maybe some streamlining of initiatives would be quite useful. And for that, however, 12 months is quite a short time. So it really depends on what you look at, whether that’s a long or a short time perspective. Great call to action on streamlining.


Sasha Rubel: Yachin?


Udbhav Tiwari: For the last three years, I’ve been a strong advocate for registration, identification of AI-generated contents, models, and particular agents. I’m glad to see some of this is happening in China, and I hope that can scale to the rest of the world. It’s ID, ID, ID. ID, ID, ID. Udbav, over to you. For me, I’d say it would definitely be developer agency, very similar to the recall example that I mentioned earlier. I think it’s vital that application providers be able to make decisions for their users in order to protect them from risks that AI systems could pose. Privacy is one of them. It’s okay for a user to be able to override those protections, but right now I think there’s been a pretty symbiotic relationship between operating system providers and applications that run on them. I think we’re going to see that radically change with the deployment of AI systems, and in order to make sure that if something is sensitive, the user doesn’t necessarily need to be able to menu dive deep into settings in order to be able to protect their settings, but an application should be able to protect the privacy of the user on their behalf, like we did with recall. It should become much more of a norm, because without that, we’re not going to have 7 billion people being able to protect themselves to the level that they deserve.


Sasha Rubel: One of the key takeaways that I take from this conversation and the great insights of all five of you is really this need to invest not only in cooperation and research as AI advances in terms of what’s possible with regards to safety, but also making sure that that cooperation is across regions and jurisdictions and across stakeholder groups where transparency and responsibility is prioritized, particularly, and many of you have underlined this, as we become very, very aware in the importance of centralizing responsibility, because at the same time, responsibility drives the trust that’s missing, which will drive adoption, which will ultimately drive innovation and beneficial use for all. This has been the most enjoyable AI governance speed dating ever with the five of you. Thank you very much for your insights and for sharing your learnings about what’s working and what’s not. Chris, Yuhua, Udbav, Yachin, and Brian, I hope, as many of you underlined the importance of conversations and asking the right questions, that this is the beginning of a conversation and not the end going forward, and hope that all of you can interact with some of the audience in the room about how to take those call to actions forward with a great group at AI for good. Thank you very much, all, and have a great rest of your morning. Many thanks. Many thanks to you, Sacha, and many thanks to all your guests. I agree with you. This was a very engaging, very dynamic conversation around how we can move to practice, of course. Many thanks to all of you for your insights this morning. Thank you. And we swiftly move on to our…


B

Brian Tse

Speech speed

156 words per minute

Speech length

691 words

Speech time

264 seconds

Four types of frontier AI risks: misuse, accidents/malfunctions, loss of control scenarios, and systemic societal risks

Explanation

Brian identifies four categories of frontier AI risks that humanity faces. These include misuse by malicious actors, accidents in critical applications, potential loss of human control over AI systems, and broad societal impacts like labor market disruption.


Evidence

Teams of AI agents exploiting zero-day vulnerabilities in real-world settings; Beijing government prohibiting AI from writing medical prescriptions automatically; Professor Geoffrey Hinton’s warnings about digital superintelligence control; AI outperforming humans on economically valuable tasks


Major discussion point

Frontier AI Risks and Challenges


Topics

Cybersecurity | Economic | Legal and regulatory


Disagreed with

– Udbhav Tiwari
– Chris Meserole

Disagreed on

Focus on current vs future AI risks


China’s approach includes pre-deployment registration/licensing and post-deployment transparency requirements

Explanation

Brian describes China’s regulatory framework requiring all generative AI to be registered with the government before public deployment, including safety assessments. Additionally, China requires explicit labeling and metadata for AI-generated content to help users distinguish it from human-created content.


Evidence

China’s national standard requiring developers and platforms to have explicit labeling and implicit metadata for AI-generated content


Major discussion point

Governance Approaches and Implementation


Topics

Legal and regulatory | Human rights


Disagreed with

– Chris Meserole

Disagreed on

Regulatory approach – binding vs voluntary governance


Content provenance and watermarking essential for differentiating AI-generated from human-generated content

Explanation

Brian argues that having clear markers and metadata on AI-generated content is critical for transparency. This allows users and society to understand what content is created by AI versus humans, which is essential for maintaining trust and informed decision-making.


Evidence

China’s implementation of national standards requiring both explicit labeling and implicit metadata


Major discussion point

Trust and Transparency Requirements


Topics

Human rights | Legal and regulatory


Agreed with

– Ya Qin Zhang

Agreed on

Need for transparency and identification of AI-generated content


International effort required to define red lines and unacceptable outcomes from advanced AI systems

Explanation

Brian advocates for global collaboration to establish clear boundaries of what constitutes unacceptable AI system behavior. He emphasizes that these decisions should not be left solely to the AI industry, drawing parallels to past financial and environmental crises caused by unchecked risk-taking.


Evidence

Lessons from past financial crises and environmental disasters where unchecked risk-taking by elites led to systemic global risks


Major discussion point

International Coordination and Standards


Topics

Legal and regulatory | Development


C

Chris Meserole

Speech speed

185 words per minute

Speech length

735 words

Speech time

238 seconds

Frontier AI frameworks and voluntary commitments from Seoul AI Summit showing promise for identifying intolerable risks

Explanation

Chris highlights the voluntary commitments made by 15-16 companies at the Seoul AI Summit to identify intolerable risks and develop management plans. These frameworks represent a new type of risk management instrument designed to identify extreme-scale risks before harms materialize, particularly for bio risks, cyber risks, and autonomous R&D capabilities.


Evidence

15-16 companies globally signed commitments at Seoul AI Summit; growing consensus on bio risks, advanced cyber risks, and autonomous R&D capabilities requiring new risk management instruments


Major discussion point

Governance Approaches and Implementation


Topics

Cybersecurity | Legal and regulatory


Disagreed with

– Brian Tse

Disagreed on

Regulatory approach – binding vs voluntary governance


Need for new risk management instruments for risks that appear quickly at extreme scale before harms materialize

Explanation

Chris argues that some AI risks require entirely new risk management approaches because they can appear very quickly at extreme scale, with harms not materializing until it’s too late to respond. Traditional risk management instruments may be insufficient for these unprecedented challenges.


Evidence

Identification of bio risks, advanced cyber risks, and autonomous R&D capabilities as requiring fundamental new risk management instruments


Major discussion point

Frontier AI Risks and Challenges


Topics

Cybersecurity | Legal and regulatory


Disagreed with

– Udbhav Tiwari
– Brian Tse

Disagreed on

Focus on current vs future AI risks


Need for standardizing frontier AI frameworks globally with formal standards on quick timelines

Explanation

Chris emphasizes the importance of coordinating globally to standardize frontier AI risk management frameworks. He notes that while there’s significant voluntary activity, the diversity of approaches needs coordination, and formal standards must be developed much faster than traditional timelines.


Evidence

ISO SC402001 AI risk management standard took 4-5 years to develop, but new standards need to be developed with similar rigor in much shorter timeframes


Major discussion point

International Coordination and Standards


Topics

Legal and regulatory | Infrastructure


Firms should publish risk management frameworks and systematic implementation updates for public transparency

Explanation

Chris advocates for a global norm where companies developing frontier AI models publicly share their risk management frameworks and provide regular updates on implementation. This transparency would allow the public to assess the trustworthiness of AI products and hold companies accountable.


Major discussion point

Practical Implementation Recommendations


Topics

Human rights | Legal and regulatory


Y

Ya Qin Zhang

Speech speed

133 words per minute

Speech length

328 words

Speech time

147 seconds

Rapid technological development creates governance regulatory lag across different regions with varying guidelines

Explanation

Ya Qin identifies the challenge of keeping governance frameworks current with rapidly advancing AI technology. Different regions like the UK, EU, China, and US have developed different approaches and guidelines, creating a fragmented regulatory landscape that struggles to keep pace with technological development.


Evidence

China’s inclusive approach working closely with industries on data collection, testing, registration, and big model release processes


Major discussion point

Frontier AI Risks and Challenges


Topics

Legal and regulatory | Development


Safety must be foundation of product design, not an afterthought, with academic-industry-policy collaboration

Explanation

Ya Qin emphasizes that AI safety should be built into products from the beginning rather than added later. She highlights the positive role of academics, thought leaders like Yoshua Bengio, working with industry leaders and policymakers to ensure safety is foundational to AI development.


Evidence

National AI Safety Institute and NGOs playing constructive roles; collaboration between AI pioneers, industry leaders, and policymakers


Major discussion point

Trust and Transparency Requirements


Topics

Legal and regulatory | Infrastructure


Agreed with

– Sasha Rubel

Agreed on

Safety and responsibility must be built into AI systems from the beginning, not added later


Enhanced global collaboration through forums and multi-stakeholder approaches essential

Explanation

Ya Qin stresses the importance of bringing together different stakeholders in international forums to address AI governance challenges. She advocates for continued investment in R&D to identify critical risks and develop appropriate benchmarks, thresholds, and warning systems.


Major discussion point

International Coordination and Standards


Topics

Legal and regulatory | Development


Agreed with

– Brian Tse
– Juha Heikkila
– Sasha Rubel

Agreed on

Multi-stakeholder collaboration is essential for effective AI governance


Registration and identification systems for AI-generated content, models, and agents should scale globally

Explanation

Ya Qin advocates for expanding registration and identification requirements for AI-generated content and models beyond current implementations. She sees this as a critical step that has shown success in China and should be adopted more widely internationally.


Evidence

China’s implementation of registration and identification systems


Major discussion point

Practical Implementation Recommendations


Topics

Legal and regulatory | Human rights


Agreed with

– Brian Tse

Agreed on

Need for transparency and identification of AI-generated content


J

Juha Heikkila

Speech speed

148 words per minute

Speech length

711 words

Speech time

287 seconds

EU AI Act implements trust-based legislation with update mechanisms and Code of Practice for multi-stakeholder compliance

Explanation

Juha explains that the EU chose legislation to ensure trust in AI technology, viewing trust as essential for adoption and benefit realization. The AI Act includes mechanisms for updates and a Code of Practice developed through multi-stakeholder processes involving over 1,000 stakeholders, with flexibility in implementation approaches.


Evidence

High-risk areas defined in annexes that can be updated quickly; Code of Practice developed with independent chairs and over 1,000 stakeholder participants; flexibility allowing different implementation approaches while maintaining compliance presumption


Major discussion point

Governance Approaches and Implementation


Topics

Legal and regulatory | Human rights


Agreed with

– Brian Tse
– Ya Qin Zhang
– Sasha Rubel

Agreed on

Multi-stakeholder collaboration is essential for effective AI governance


Trust is fundamental for AI technology adoption and benefit realization, requiring legislative frameworks

Explanation

Juha argues that trust is the sine qua non for AI technology uptake, which in turn is necessary for AI benefits to materialize. The EU determined that legislation was necessary to establish this trust, as voluntary measures alone were insufficient to build the required confidence.


Major discussion point

Trust and Transparency Requirements


Topics

Legal and regulatory | Human rights


Agreed with

– Udbhav Tiwari
– Sasha Rubel

Agreed on

Trust is fundamental for AI adoption and requires transparency and responsibility


Streamlining proliferating initiatives to reduce burden on developers while maintaining oversight

Explanation

Juha identifies the challenge of multiple overlapping initiatives creating excessive monitoring and reporting requirements for developers. He advocates for streamlining these various initiatives to reduce administrative burden while maintaining necessary oversight and governance.


Evidence

Proliferation of initiatives has led to proliferation of monitoring and reporting requirements that burden developers


Major discussion point

International Coordination and Standards


Topics

Legal and regulatory | Economic


Follow-up and update mechanisms necessary for measuring effectiveness of governance measures

Explanation

Juha emphasizes that complex AI governance issues require ongoing monitoring and evaluation to determine if measures are being implemented effectively. He argues that follow-up mechanisms are essential for building trust by demonstrating that governance measures are actually working in practice.


Major discussion point

Practical Implementation Recommendations


Topics

Legal and regulatory


U

Udbhav Tiwari

Speech speed

197 words per minute

Speech length

1234 words

Speech time

375 seconds

Real harms occurring today with AI systems affecting high-risk individuals, as demonstrated by Microsoft Recall feature

Explanation

Udbhav uses Signal’s response to Microsoft’s Recall feature as an example of how AI systems can create immediate privacy risks for vulnerable users. Signal had to implement screenshot blocking because Recall would capture and store sensitive conversations, demonstrating that current governance approaches are insufficient to prevent real-world harms.


Evidence

Signal implemented screenshot blocking on Windows 11 to protect against Microsoft Recall feature that takes screenshots every 3 seconds; Signal users include particularly high-risk individuals who need privacy protection


Major discussion point

Frontier AI Risks and Challenges


Topics

Human rights | Cybersecurity


Disagreed with

– Brian Tse
– Chris Meserole

Disagreed on

Focus on current vs future AI risks


Signal’s proactive privacy protection measures demonstrate need for developer agency to protect users

Explanation

Udbhav describes how Signal took independent action to protect user privacy against AI systems deployed by other companies. This illustrates the need for application developers to have the agency and capability to protect their users from AI-related risks, even when those risks come from other parts of the technology stack.


Evidence

Signal’s implementation of screenshot blocking to protect against Microsoft Recall feature


Major discussion point

Governance Approaches and Implementation


Topics

Human rights | Cybersecurity


Need for honest conversations about AI limitations and drawbacks, not just benefits and progress

Explanation

Udbhav argues that the AI industry has become less transparent about system limitations due to increased hype and competitive pressures. He notes that model cards have actually become less informative over time, and there’s insufficient discussion of what AI systems should not be used for, hindering effective governance.


Evidence

Analysis of model cards from prominent providers over 3-4 years shows regression in information provided; less discussion of limitations due to consequences of hype in the industry


Major discussion point

Trust and Transparency Requirements


Topics

Human rights | Legal and regulatory


Agreed with

– Juha Heikkila
– Sasha Rubel

Agreed on

Trust is fundamental for AI adoption and requires transparency and responsibility


Developer agency crucial for applications to protect users from AI system risks without requiring user technical expertise

Explanation

Udbhav advocates for application providers to have the ability to make protective decisions on behalf of users, particularly for sensitive applications. He argues that users shouldn’t need to navigate complex settings to protect themselves from AI-related risks, and applications should be able to implement protections proactively.


Evidence

Signal’s recall protection as example of application protecting users without requiring them to understand technical details or navigate settings


Major discussion point

Practical Implementation Recommendations


Topics

Human rights | Cybersecurity


S

Sasha Rubel

Speech speed

177 words per minute

Speech length

1922 words

Speech time

649 seconds

Multi-stakeholder collaboration and dialogue essential for AI governance alignment on risk identification and effective governance

Explanation

Sasha emphasizes that bringing different groups together is crucial for agreeing on what risks look like and determining the most effective governance approaches. She highlights the importance of getting diverse stakeholders in the room to have meaningful dialogue about AI governance challenges.


Evidence

The importance of this summit and getting different groups in the room to agree on risk definitions and governance effectiveness


Major discussion point

International Coordination and Standards


Topics

Legal and regulatory | Development


Agreed with

– Brian Tse
– Ya Qin Zhang
– Juha Heikkila

Agreed on

Multi-stakeholder collaboration is essential for effective AI governance


Need to move from responsibility and safety being ‘bolted on’ to ‘built in’ to ‘by design’ in AI development

Explanation

Sasha advocates for embedding responsibility and safety considerations from the very beginning of AI development rather than adding them as afterthoughts. She emphasizes the importance of privacy by design and working in parallel with governance efforts during technology development.


Evidence

Examples of privacy by design and parallel development approaches mentioned in the discussion


Major discussion point

Trust and Transparency Requirements


Topics

Legal and regulatory | Human rights


Agreed with

– Ya Qin Zhang

Agreed on

Safety and responsibility must be built into AI systems from the beginning, not added later


Trust is fundamental blocker to AI adoption and requires responsibility by design and stakeholder cooperation

Explanation

Sasha identifies trust as a key barrier preventing AI adoption across organizations globally. She argues that building trust requires both technical approaches like responsibility by design and collaborative approaches between stakeholders working together on governance frameworks.


Evidence

Conversations with organizations across the world showing trust as a blocker to AI adoption


Major discussion point

Trust and Transparency Requirements


Topics

Human rights | Legal and regulatory


Agreed with

– Juha Heikkila
– Udbhav Tiwari

Agreed on

Trust is fundamental for AI adoption and requires transparency and responsibility


Governance models must address risks in rapidly advancing technology field through various approaches including regulatory, institutional, and voluntary measures

Explanation

Sasha highlights the challenge of developing governance frameworks that can keep pace with rapidly advancing AI technology. She emphasizes the need for multiple types of governance approaches working together, including regulatory frameworks, institutional measures, and voluntary commitments.


Evidence

Discussion of different governance approaches being implemented across regions and stakeholder groups


Major discussion point

Governance Approaches and Implementation


Topics

Legal and regulatory | Infrastructure


Responsibility drives trust, which drives adoption, which ultimately drives innovation and beneficial use for all

Explanation

Sasha presents a causal chain showing how prioritizing responsibility in AI development leads to increased trust, which enables broader adoption and ultimately results in more innovation and beneficial applications. She argues this creates a positive cycle that benefits society as a whole.


Evidence

Observations from the panel discussion about the relationship between responsibility, trust, adoption, and innovation


Major discussion point

Trust and Transparency Requirements


Topics

Human rights | Economic | Development


Agreements

Agreement points

Multi-stakeholder collaboration is essential for effective AI governance

Speakers

– Brian Tse
– Ya Qin Zhang
– Juha Heikkila
– Sasha Rubel

Arguments

Multi-signal approach, both at the national and the global level, to ensure that no one will be left behind


Enhanced global collaboration through forums and multi-stakeholder approaches essential


EU AI Act implements trust-based legislation with update mechanisms and Code of Practice for multi-stakeholder compliance


Multi-stakeholder collaboration and dialogue essential for AI governance alignment on risk identification and effective governance


Summary

All speakers agree that addressing AI governance challenges requires bringing together diverse stakeholders from different regions, sectors, and expertise areas to work collaboratively on solutions


Topics

Legal and regulatory | Development


Trust is fundamental for AI adoption and requires transparency and responsibility

Speakers

– Juha Heikkila
– Udbhav Tiwari
– Sasha Rubel

Arguments

Trust is fundamental for AI technology adoption and benefit realization, requiring legislative frameworks


Need for honest conversations about AI limitations and drawbacks, not just benefits and progress


Trust is fundamental blocker to AI adoption and requires responsibility by design and stakeholder cooperation


Summary

Speakers consistently emphasize that trust is the cornerstone of successful AI deployment, requiring transparency about limitations and proactive responsibility measures


Topics

Human rights | Legal and regulatory


Safety and responsibility must be built into AI systems from the beginning, not added later

Speakers

– Ya Qin Zhang
– Sasha Rubel

Arguments

Safety must be foundation of product design, not an afterthought, with academic-industry-policy collaboration


Need to move from responsibility and safety being ‘bolted on’ to ‘built in’ to ‘by design’ in AI development


Summary

Both speakers advocate for embedding safety and responsibility considerations from the initial design phase rather than treating them as afterthoughts


Topics

Legal and regulatory | Human rights


Need for transparency and identification of AI-generated content

Speakers

– Brian Tse
– Ya Qin Zhang

Arguments

Content provenance and watermarking essential for differentiating AI-generated from human-generated content


Registration and identification systems for AI-generated content, models, and agents should scale globally


Summary

Both speakers emphasize the importance of clearly marking and identifying AI-generated content to maintain transparency and help users distinguish between AI and human-created content


Topics

Human rights | Legal and regulatory


Similar viewpoints

Both speakers recognize that frontier AI systems pose unprecedented risks that require entirely new approaches to risk management, particularly for risks that can emerge rapidly at scale

Speakers

– Brian Tse
– Chris Meserole

Arguments

Four types of frontier AI risks: misuse, accidents/malfunctions, loss of control scenarios, and systemic societal risks


Need for new risk management instruments for risks that appear quickly at extreme scale before harms materialize


Topics

Cybersecurity | Legal and regulatory


Both advocate for international coordination to establish common standards and boundaries for AI development, emphasizing the need for global alignment on what constitutes acceptable AI behavior

Speakers

– Chris Meserole
– Brian Tse

Arguments

Need for standardizing frontier AI frameworks globally with formal standards on quick timelines


International effort required to define red lines and unacceptable outcomes from advanced AI systems


Topics

Legal and regulatory | Infrastructure


Both speakers emphasize the importance of addressing current, real-world AI harms rather than focusing solely on hypothetical future risks, and the need for ongoing monitoring and adaptation of governance measures

Speakers

– Udbhav Tiwari
– Juha Heikkila

Arguments

Real harms occurring today with AI systems affecting high-risk individuals, as demonstrated by Microsoft Recall feature


Follow-up and update mechanisms necessary for measuring effectiveness of governance measures


Topics

Human rights | Legal and regulatory


Unexpected consensus

Need for rapid development of formal standards despite traditional slow timelines

Speakers

– Chris Meserole
– Juha Heikkila

Arguments

Need for standardizing frontier AI frameworks globally with formal standards on quick timelines


Streamlining proliferating initiatives to reduce burden on developers while maintaining oversight


Explanation

Despite coming from different regulatory contexts (voluntary frameworks vs. EU legislation), both speakers recognize the tension between the need for rigorous standards and the urgency of AI development timelines, advocating for accelerated but coordinated approaches


Topics

Legal and regulatory | Infrastructure


Importance of developer agency and proactive protection measures

Speakers

– Udbhav Tiwari
– Ya Qin Zhang

Arguments

Developer agency crucial for applications to protect users from AI system risks without requiring user technical expertise


Safety must be foundation of product design, not an afterthought, with academic-industry-policy collaboration


Explanation

Unexpected alignment between a privacy-focused application developer and an academic policy expert on the need for proactive, built-in protections rather than reactive measures or user-dependent safeguards


Topics

Human rights | Cybersecurity


Overall assessment

Summary

Strong consensus on fundamental principles including multi-stakeholder collaboration, trust as prerequisite for adoption, safety by design, and transparency requirements. Speakers from different regions and sectors align on the need for international coordination while recognizing current governance gaps.


Consensus level

High level of consensus on core principles with complementary rather than conflicting approaches. This suggests a mature understanding of AI governance challenges and potential for coordinated global action, though implementation details may vary by jurisdiction and stakeholder type.


Differences

Different viewpoints

Regulatory approach – binding vs voluntary governance

Speakers

– Brian Tse
– Chris Meserole

Arguments

China’s approach includes pre-deployment registration/licensing and post-deployment transparency requirements


Frontier AI frameworks and voluntary commitments from Seoul AI Summit showing promise for identifying intolerable risks


Summary

Brian advocates for binding regulations with mandatory pre-deployment registration and licensing, citing China’s regulatory framework, while Chris emphasizes voluntary commitments and frameworks as showing the most promise for governance


Topics

Legal and regulatory


Focus on current vs future AI risks

Speakers

– Udbhav Tiwari
– Brian Tse
– Chris Meserole

Arguments

Real harms occurring today with AI systems affecting high-risk individuals, as demonstrated by Microsoft Recall feature


Four types of frontier AI risks: misuse, accidents/malfunctions, loss of control scenarios, and systemic societal risks


Need for new risk management instruments for risks that appear quickly at extreme scale before harms materialize


Summary

Udbhav emphasizes addressing immediate, real harms happening today rather than focusing on speculative future risks, while Brian and Chris focus more on frontier AI risks and potential future catastrophic scenarios


Topics

Human rights | Cybersecurity | Legal and regulatory


Unexpected differences

Transparency in AI model information

Speakers

– Udbhav Tiwari
– Chris Meserole

Arguments

Need for honest conversations about AI limitations and drawbacks, not just benefits and progress


Firms should publish risk management frameworks and systematic implementation updates for public transparency


Explanation

While both advocate for transparency, Udbhav criticizes the AI industry for becoming less transparent over time (noting regression in model card information), while Chris advocates for more systematic transparency through published frameworks. This suggests disagreement on whether current industry transparency efforts are sufficient or moving in the right direction


Topics

Human rights | Legal and regulatory


Overall assessment

Summary

The speakers show broad agreement on fundamental goals (safety, trust, international cooperation) but significant disagreement on implementation approaches, particularly regarding regulatory mechanisms (binding vs voluntary), risk priorities (current vs future), and transparency methods


Disagreement level

Moderate disagreement with high implications – while speakers share common objectives, their different approaches to governance (regulatory vs voluntary, immediate vs future-focused) could lead to fragmented global AI governance if not reconciled. The disagreements reflect deeper philosophical differences about the role of government regulation, industry self-governance, and risk prioritization in AI development


Partial agreements

Partial agreements

Similar viewpoints

Both speakers recognize that frontier AI systems pose unprecedented risks that require entirely new approaches to risk management, particularly for risks that can emerge rapidly at scale

Speakers

– Brian Tse
– Chris Meserole

Arguments

Four types of frontier AI risks: misuse, accidents/malfunctions, loss of control scenarios, and systemic societal risks


Need for new risk management instruments for risks that appear quickly at extreme scale before harms materialize


Topics

Cybersecurity | Legal and regulatory


Both advocate for international coordination to establish common standards and boundaries for AI development, emphasizing the need for global alignment on what constitutes acceptable AI behavior

Speakers

– Chris Meserole
– Brian Tse

Arguments

Need for standardizing frontier AI frameworks globally with formal standards on quick timelines


International effort required to define red lines and unacceptable outcomes from advanced AI systems


Topics

Legal and regulatory | Infrastructure


Both speakers emphasize the importance of addressing current, real-world AI harms rather than focusing solely on hypothetical future risks, and the need for ongoing monitoring and adaptation of governance measures

Speakers

– Udbhav Tiwari
– Juha Heikkila

Arguments

Real harms occurring today with AI systems affecting high-risk individuals, as demonstrated by Microsoft Recall feature


Follow-up and update mechanisms necessary for measuring effectiveness of governance measures


Topics

Human rights | Legal and regulatory


Takeaways

Key takeaways

AI governance requires parallel development alongside technology advancement, not sequential implementation


Trust is fundamental for AI adoption and must be built through transparency, responsibility by design, and legislative frameworks


Four critical frontier AI risks need addressing: misuse, accidents/malfunctions, loss of control scenarios, and systemic societal risks


Multi-stakeholder collaboration across regions, jurisdictions, and stakeholder groups is essential for effective AI governance


Current governance approaches show promise but need standardization and coordination to avoid fragmentation


Safety must be foundational to AI product design rather than an afterthought


Real harms from AI systems are occurring today and require immediate attention alongside future risk prevention


Developer agency is crucial for protecting users from AI risks without requiring technical expertise from end users


Resolutions and action items

Develop international standards for frontier AI risk management frameworks within 12 months


Create global norms requiring firms to publish risk management frameworks and systematic implementation updates


Establish international effort to define red lines and unacceptable outcomes from advanced AI systems


Scale registration and identification systems for AI-generated content, models, and agents globally


Streamline proliferating AI governance initiatives to reduce developer burden while maintaining oversight


Implement follow-up and update mechanisms to measure effectiveness of governance measures


Invest in R&D for identifying critical risks, benchmarks, thresholds, and warning systems


Enable developer agency for applications to protect users from AI system risks proactively


Unresolved issues

How to balance rapid technological advancement with necessary governance regulatory frameworks across different regions


Lack of consensus on what constitutes ‘intolerable risks’ and appropriate risk thresholds globally


Insufficient understanding of AI system risks due to proprietary and closed development settings


Gap between voluntary commitments and binding regulations for the most powerful AI systems


How to ensure Global South and unprotected communities have voice in defining AI safety standards


Coordination challenges between different regional approaches (EU, China, US) with varying guidelines


How to maintain honest conversations about AI limitations amid industry hype and competitive pressures


Suggested compromises

Flexible implementation approaches within governance frameworks allowing companies to choose specific mitigation methods while meeting compliance standards


Adaptive regulatory mechanisms that can be updated more quickly than full legislative revisions


Combination of voluntary industry commitments with binding regulations for highest-risk AI systems


Multi-stakeholder processes that balance industry expertise with civil society and academic input


Presumption of compliance for companies that sign up to codes of practice while maintaining oversight mechanisms


State-of-the-art references in governance frameworks that can be interpreted flexibly as technology evolves


Thought provoking comments

Drawing from the International AI Safety Report and some of the latest research, I think there are at least four types of frontier AI risk. Number one is misuse… Number two is accidents and malfunctions… Number three is potential loss of control scenarios… The fourth one is systemic risk at the societal level.

Speaker

Brian Tse


Reason

This comment provided a comprehensive taxonomic framework for understanding AI risks, moving beyond vague concerns to specific, actionable categories. It was particularly insightful because it connected theoretical risks to real-world examples (like Beijing’s medical prescription regulations) and emphasized the precautionary principle for catastrophic risks even when probabilities are unknown.


Impact

This foundational comment set the analytical framework for the entire discussion. It established a common vocabulary and structure that other panelists referenced throughout, with Chris Meserole specifically building on these risk categories when discussing frontier AI frameworks, and other speakers addressing different aspects of these four risk types.


Quite recently, Signal rolled out a feature on its desktop installations, where if you run Windows 11 and you try to take a screenshot… The reason we deployed that feature was because Microsoft had deployed Recall, a feature that takes periodic screenshots of your screen… If you look at signal as a player in this market, AI regulation has existed for some time, voluntary efforts have existed for quite some time, but yet Microsoft could deploy a lot of technology to prevent that from happening.

Speaker

Udbhav Tiwari


Reason

This concrete example was exceptionally thought-provoking because it demonstrated the real-world inadequacy of existing governance approaches. It showed how even well-intentioned AI features can create privacy risks that require defensive action by other companies, highlighting the gap between regulatory frameworks and practical implementation.


Impact

This comment fundamentally shifted the discussion from theoretical governance models to practical, immediate challenges. It introduced the concept of ‘developer agency’ as a necessary governance mechanism and emphasized focusing on current harms rather than speculative future risks. Sasha explicitly praised this as a ‘call to action’ that resonated with the audience.


I think ultimately acknowledging that as much as we talk about advanced AI, it is just as important to recognize the very real harms that are occurring to real people with very real consequences that are taking place today because of these AI systems. And unless we choose to address them rather than wondering whether AI will control Skynet three years from today, I think we’re going to end up in a scenario where some very real harms will occur.

Speaker

Udbhav Tiwari


Reason

This comment was particularly insightful because it challenged the panel’s focus on future frontier AI risks by emphasizing present-day harms. It introduced a critical tension between preparing for hypothetical catastrophic risks versus addressing immediate, tangible problems affecting real users today.


Impact

This comment created a notable shift in the discussion’s temporal focus. It influenced subsequent speakers to balance future-oriented governance with present-day concerns, and Sasha’s enthusiastic response (‘I love that call to action’) indicated it resonated strongly with the audience and moderator.


Given the profound impact from front AI systems in the coming years, it’s quite striking that we have more regulations on the food safety of dumplings than they are today.

Speaker

Brian Tse


Reason

This analogy was brilliantly thought-provoking because it used a mundane comparison to highlight the regulatory gap in AI governance. The contrast between extensive food safety regulations and minimal AI oversight effectively illustrated the urgency of the governance challenge.


Impact

This vivid comparison provided a memorable frame of reference that underscored the need for binding regulations. It supported Brian’s subsequent recommendations for pre-deployment registration and post-deployment transparency, making abstract regulatory concepts more relatable and urgent.


For trust, which has been identified here as one of the key issues, it is actually important to know how and if and how, and to what extent, whatever measures and recommendations are there, were put into practice and how effective they are. So I think that this is one aspect that is important… it would be required also then for other sort of softer approaches.

Speaker

Juha Heikkila


Reason

This comment was insightful because it identified a critical gap in AI governance: the lack of systematic follow-up and effectiveness measurement. It highlighted that having governance frameworks is insufficient without mechanisms to verify their implementation and impact.


Impact

This observation influenced the discussion’s focus on accountability and transparency. It connected to Chris Meserole’s later emphasis on publishing systematic updates and influenced the final recommendations around monitoring and reporting mechanisms.


Overall assessment

These key comments fundamentally shaped the discussion by establishing both the theoretical framework and practical urgency of AI governance challenges. Brian’s risk taxonomy provided the analytical foundation, while Udbhav’s Signal example grounded the conversation in immediate, tangible concerns. The interplay between these perspectives created a productive tension between preparing for future risks and addressing present harms. Juha’s emphasis on follow-up mechanisms added a crucial accountability dimension that influenced the final recommendations. Together, these comments moved the discussion from abstract principles to concrete, actionable governance approaches, with a notable shift toward emphasizing transparency, developer responsibility, and the need for mechanisms that can keep pace with technological development while addressing real-world impacts on users and communities.


Follow-up questions

How can we develop robust mechanisms to ensure that digital superintelligence remains under human control?

Speaker

Brian Tse


Explanation

This addresses the critical challenge of maintaining human oversight over advanced AI systems that may have tendencies to deceive humans or evade oversight, which is essential for preventing loss of control scenarios


How can we create formal standards for frontier AI risk management frameworks on a very quick timeline?

Speaker

Chris Meserole


Explanation

Current standards like ISO SC402001 take 4-5 years to develop, but the rapid pace of AI development requires standards with similar rigor but much shorter development timelines


How can we better understand AI systems and their risks when development happens in proprietary and closed settings?

Speaker

Udbhav Tiwari


Explanation

The lack of transparency in AI development makes it difficult to understand risks and consequences, which is essential for effective governance and risk management


How can we streamline the proliferation of AI governance initiatives to reduce burden on developers?

Speaker

Juha Heikkila


Explanation

Multiple overlapping initiatives create excessive monitoring and reporting requirements that burden developers, requiring coordination to make governance more efficient


How can we develop international red lines and unacceptable outcomes for advanced AI systems?

Speaker

Brian Tse


Explanation

Defining global standards for what constitutes unacceptable AI outcomes is crucial for preventing systemic risks that could affect communities worldwide


How can we invest in R&D to identify critical and catastrophic risks and develop appropriate benchmarks, thresholds, and warning systems?

Speaker

Ya Qin Zhang


Explanation

Better risk identification and early warning systems are needed to stay ahead of rapidly advancing AI technology and implement effective governance measures


How can we establish norms for developer agency to protect users from AI system risks?

Speaker

Udbhav Tiwari


Explanation

Application providers need the ability to make protective decisions for users without requiring users to navigate complex settings, especially as AI systems change the relationship between operating systems and applications


How can we scale AI content identification and registration systems globally?

Speaker

Ya Qin Zhang


Explanation

Registration and identification of AI-generated content, models, and agents is important for transparency and accountability, and successful implementations need to be scaled internationally


How can we create space for honest conversations about AI limitations and drawbacks alongside benefits?

Speaker

Udbhav Tiwari


Explanation

The current environment discourages discussion of AI limitations due to industry hype, but honest assessment of both benefits and risks is essential for effective governance


Disclaimer: This is not an official session record. DiploAI generates these resources from audiovisual recordings, and they are presented as-is, including potential errors. Due to logistical challenges, such as discrepancies in audio/video or transcripts, names may be misspelled. We strive for accuracy to the best of our ability.