Advancing Scientific AI with Safety Ethics and Responsibility
20 Feb 2026 11:00h - 12:00h
Advancing Scientific AI with Safety Ethics and Responsibility
Summary
The panel explored how the rapid emergence of AI-enabled biodesign tools is shifting biosecurity risk from traditional laboratory containment to the upstream design phase, creating a new governance challenge that demands attention to data governance, model evaluation and red-team activities [6-13]. Participants argued that India’s heterogeneous scientific ecosystem cannot rely on a single central authority; instead, oversight must be decentralized to empower biosafety officers, information-security units and other institutional actors, establishing multiple, coordinated checks and balances [24-27][28-31].
To reconcile open-science benefits with high-risk capabilities, a tiered-access model combined with contextual norms and pre-deployment assessments using structured rubrics was recommended, drawing on RAND Europe’s risk index and a “know-your-customer” style credentialing [41-49][50-57]. The speakers emphasized that assessment results should be shared through a credentialed network with tiered confidentiality rather than kept proprietary, and that a six-monthly independent monitoring ritual-potentially housed in an AI-safety institute linked to governments and international bodies-would provide continuous risk oversight [92-99][100-119]. Recognizing limited AI readiness in many Global South institutions, they called for socio-cultural evaluation, deployment of small-model solutions, self-regulation commitments and capacity-building programmes to make safeguards proportionate and functional [62-71][75-79]. A unified yet adaptable framework was proposed, integrating participatory stakeholder involvement, accountability mechanisms for developers to document testing, and self-regulation endorsements [72-77][78-80].
Cross-border challenges were highlighted, with fragmented data standards and divergent legal regimes hampering biosurveillance; the panel urged harmonised federated standards (e.g., HL7-FHIR-style), pre-negotiated legal safe-harbours, and shared evaluation criteria embedded in national systems [226-233][234-241]. To close incident-reporting gaps, a new AI incident taxonomy covering physical, psychological, cyber, algorithmic, socio-economic and environmental harms was described, alongside toolkits for assessing user perceptions and building AI literacy in healthcare settings [270-276]. Emerging powers such as India are creating sandboxes, a Global-South trustworthy-AI network and an AI-safety commons to enable low-resource countries to adopt tailored governance while learning from each other’s experiences [161-169][170-180].
The discussion concluded that effective governance must move beyond model-centric audits to systemic, socio-technical assessments that consider capability uplift, incentive structures and the cross-border diffusion of risk, integrate AI evaluation into grant reviews and biosafety panels, and incorporate tech-sovereignty measures for AI security [189-197][198-202][147-149][155-156]. Overall, a decentralized, collaborative, and context-aware architecture-supported by regular independent evaluation, capacity building and interoperable standards-is essential to safely harness AI-driven scientific innovation [24-27][41-57][122-130][226-233][189-202].
Keypoints
Major discussion points
– AI is moving bio-risk upstream from physical labs to the design stage, creating a new governance challenge.
The panel highlighted that traditional bio-security relied on “physical infrastructure and lab facilities” [7-8] but AI-driven biodesign tools now let researchers “engineer proteins, optimise DNA sequences…” without those constraints [10-12]. This shift means risk must be managed earlier in the design pipeline [12-13] and calls for “more adaptive oversight mechanisms” [23-24].
– Oversight must be decentralized, capacity-building focused, and tailored to heterogeneous ecosystems (especially in India and the Global South).
Speakers argued that a single authority “in Delhi…won’t work” [25-27] and advocated for empowering “information security or biosecurity offices” [28-31] and creating “cross-trained AI biosafety review panels” [147-149]. They also stressed the wide variation in “governance capacity, compliance culture and technical expertise” across institutions [126-129] and the need for “proportionate, capability-aware safeguards” [138-144].
– Open-science benefits must be preserved through tiered, contextual access and pre-deployment assessments rather than blanket restrictions.
The discussion proposed “tiered access and contextual norms” [41-42] and praised the RAND Europe “pre-deployment assessment with structured rubrics” [44-48]. It was emphasized that “differentiated governance at capability level is always better than blanket restriction at access level” [57-58] and that open-source tools remain essential, especially for low-resource settings [53-56].
– Institutionalising independent evaluation (red-teamings) and continuous monitoring is essential, but requires new structures and investment.
A six-monthly “monitoring and assessment of risk” ritual was recommended [105-106] and the creation of an “AI safety or security institute” with formal government links was suggested [113-118]. The need for “non-interactive methodology” and broader integration into institutions was also noted [107-110].
– Cross-border data-standard harmonisation and legal safe-harbors are critical for AI-enabled biosurveillance and pandemic preparedness.
Participants pointed out fragmented standards across Southeast Asia [212-216] and advocated for federated frameworks like HL7-FHIR adapted for public-health [227-230]. They called for pre-negotiated “legal safe harbors” for data sharing [230-234] and shared evaluation criteria embedded in national systems [235-241].
Overall purpose / goal of the discussion
The panel convened to explore how emerging AI-driven biodesign and biosurveillance tools reshape bio-security risk, and to identify governance, policy, and capacity-building measures-especially for the Global South-that can ensure safety while retaining the scientific and societal benefits of open AI research.
Tone of the discussion
– Opening (0:00-5:00): Cautious and exploratory; speakers acknowledge uncertainty (“not an AI safety expert…take it with a pinch of salt” [3-4]) and the novelty of the risk landscape.
– Middle (5:00-22:00): Constructive and solution-oriented; ideas about decentralized oversight, tiered access, and institutional mechanisms are presented with optimism.
– Later (22:00-38:00): Collaborative and forward-looking; emphasis on building networks, commons, and cross-border standards, with a tone of partnership and urgency.
– Closing (38:00-end): Summative and hopeful; participants reiterate key actions, express confidence in emerging frameworks, and thank each other, ending on a cooperative note.
Overall, the conversation moves from identifying a novel problem to proposing concrete, multi-level solutions, maintaining a collegial and proactive tone throughout.
Speakers
– Speaker 1
– Area of expertise: Biosecurity, AI-enabled biodesign, AI safety in life-sciences.
– Role / Title: (not specified in the transcript) – presented as a biosecurity expert discussing institutional readiness and safety measures.
– Citation: [S13]
– Speaker 2
– Area of expertise: AI governance, open-science policy, risk assessment for AI-enabled biological tools.
– Role / Title: (not specified in the transcript) – referenced as a contributor to RAND Europe studies and a proponent of pre-deployment assessments.
– Speaker 3
– Area of expertise: AI policy, socio-technical assessment, AI readiness for emerging economies, governance frameworks.
– Role / Title: (not specified in the transcript) – identified as “Geetha” who works on institutional gaps and AI-trustworthiness initiatives.
– Moderator
– Name: Shyam
– Area of expertise: Session facilitation / AI impact discussions.
– Role / Title: Moderator of the panel.
– Citation: [S16]
– Audience Member 1
– Area of expertise: Psychological harms of AI, AI safety research.
– Role / Title: Researcher in AI safety at the University of York.
– Audience Member 2
– Area of expertise: Model monitoring, data-drift and temporal robustness.
– Role / Title: Audience participant (no further affiliation provided).
– Audience Member 3
– Area of expertise: Biosecurity incident response, cross-border prevention frameworks.
– Role / Title: Audience participant (no further affiliation provided).
Additional speakers:
– None beyond those listed above.
The moderator opened the session by asking whether the emerging challenges should be framed as a data-governance issue, a model-design problem, or a compliance-verification matter [1].
Bio-security perspective (Speaker 1).
Speaker 1, whose expertise lies in bio-security rather than AI safety, framed his remarks in terms of life-science risk governance [2-4]. He noted that traditional bio-security has relied on physical infrastructure, inspections and material-transfer controls [7-8], but the rapid proliferation of AI-enabled biodesign tools-over 1 500 according to a RAND study-has begun to decouple risk from those physical safeguards [9-10][9-13]. These AI-driven capabilities now allow researchers to engineer proteins, optimise DNA sequences and model pathogen-host interactions without laboratory containment [10-12]. Consequently, the risk landscape is shifting upstream to the design phase of biological work [12-13], demanding new, more adaptive oversight mechanisms [23-24]. While data governance, model evaluation and red-team activities remain essential [13-15], the panel argued they must be re-oriented to address this upstream threat.
Open-science discussion (Speaker 2).
Speaker 2 advocated a tiered-access and contextual-norms approach [41-42], supported by pre-deployment assessments using structured rubrics such as RAND Europe’s risk index [44-48]. He emphasized that open-source tools are crucial for low-resource settings and should not be conflated with danger [53-56]; instead, differentiated governance at the capability level should replace blanket restrictions [57-58]. Building on this, he proposed a systematic pre-deployment assessment regime akin to a “know-your-customer” (KYC) approach, where developers of high-risk biodesign tools undergo credentialed scrutiny before release [49-52]. The results of these assessments would be shared across a credentialed network with tiered confidentiality [115-119], helping to prevent the “danger…once released” from spreading unchecked [45-48].
Institutional-gap analysis (Speaker 3).
Speaker 3 highlighted that India’s high global ranking masks substantial intra-regional disparities, with countries such as Indonesia lagging far behind [63-64]. He pointed out that large language models trained predominantly on Western data fail 20-30 % of biological-safety benchmarks relevant to Southeast Asia [66-67][68-70], underscoring the need for socio-cultural evaluations and participatory approaches that involve end-users from the outset [71-73]. He also called for the development of small, edge-deployed language models for low-resource settings [71-73] and stressed the importance of building AI literacy and ensuring privacy protections for marginalized communities [270-274]. Finally, he reiterated India’s self-regulation commitments and argued that a unified yet adaptable framework can be tailored to diverse deployment settings [78-80].
Independent-evaluation / red-team proposal (Speaker 2).
Speaker 2 recommended institutionalising a six-monthly “monitoring and assessment of risk” ritual carried out by an AI-safety institute that is technically credentialed, independent, and formally linked to governments [105-108][111-118]. He cited the recent SECURE-Bio study in which a frontier language model outperformed expert virologists on wet-lab protocol troubleshooting [101-104], underscoring the urgency of continuous, non-interactive risk monitoring [107-110].
Ecosystem-specific safety measures (Speaker 1).
Speaker 1 suggested embedding AI evaluation modules into grant-review procedures and establishing cross-trained AI biosafety review panels at the institutional level [147-149]. He called for investment in domestic evaluation capacity, such as the AI safety institute at IIT Madras [148-149], and for leveraging tech-sovereignty measures to control data flows [155-156].
Emerging Global-South powers (Speaker 3).
Speaker 3 described India’s creation of sandboxes for health-care and ideological AI systems [162-163] and announced the launch of a Global-South network for trustworthy AI together with an AI-safety commons that will provide shared evaluation resources within the next one to two years [164-166]. He also noted the development of an incident-reporting framework customised for Indian contexts, capturing harms across physical, psychological, cyber-incident, algorithmic, socio-economic and environmental dimensions [270-274].
Model-vs-socio-technical focus (Speaker 1).
Speaker 1 warned that even with perfect digital safeguards, a physical infrastructure is still required to synthesise or modify viruses, highlighting the “digital-to-physical barrier” that limits immediate creation of dangerous pathogens [246-251]. He argued that AI can also aid safety, for example by using agentic AI to detect jailbreak attempts in vaccine-development platforms, but that governance must balance model-centric controls with broader socio-technical considerations.
Biosurveillance integration (Speaker 2).
Speaker 2 observed that fragmented data standards and divergent legal regimes in Southeast Asia have led to data hoarding that cost lives during COVID-19 [212-219]. He proposed adopting a federated, HL7-FHIR-style interoperability framework for public-health surveillance [227-230], establishing pre-negotiated legal safe-harbours for emergency data sharing [231-234], and embedding shared evaluation criteria within national surveillance systems [235-241]. He warned that the AI-governance community often treats biosurveillance as a niche, while biosecurity experts see AI merely as a tool, creating a dangerous communication gap [237-240].
Audience Q&A.
An audience member from the University of York raised the issue of psychological impacts, prompting Speaker 3 to present a taxonomy of harms-including physical, psychological, cyber-incident, algorithmic, socio-economic and environmental dimensions-and to share a toolkit for assessing healthcare workers’ perceptions of AI tools [265-276]. The discussion also covered temporal data drift, with Speaker 3 explaining that model-monitoring pipelines must detect distributional shifts over time-a key safety criterion [286-288].
Coordinated incident-response framework.
Speaker 1 advocated empowering biosafety officers at the lab level and providing them with clear reporting channels to central leadership, creating a “decentralised but integrated” system [295-299]. Speaker 2 illustrated Singapore’s multi-agency model (NEA, MOH, Communicable Disease Agency, etc.) as an exemplar of clear role allocation during crises [300-309]. Both agreed that prevention and preparedness, underpinned by robust governance, are essential.
Closing remarks.
The moderator summarised the key points: the upstream shift of bio-risk, the necessity of decentralised yet coordinated oversight, the preservation of open-science through tiered access, the importance of capacity-building in the Global South, the need for harmonised data standards and legal safe-harbours, and the value of a systematic, socio-technical approach to AI safety [255-263]. Speaker 1 added that AI can aid safety-e.g., agentic AI detecting jailbreak attempts-while reiterating the digital-to-physical barrier [246-251]. The panel concluded on a hopeful note, emphasizing collaborative networks, shared safety commons, and adaptive governance as the path forward [252-254].
Key area should we think about it as a governance data governance problem, problem in model design or should it be more on a verification or compliance angle.
Thanks thank you very much Shyam for having me and good morning to everyone and welcome to this session. So I think okay let me maybe just start with saying that I’m not an AI or AI safety expert so whatever I say take it with a pinch of salt. My work is in biosecurity and that’s the angle I’ll come from. I think all of those things whether it’s a model evaluation and other things those are there and those are very very important factors and that those are the things that we need to keep in mind. But on top of that there is also a very important deep structural change that is happening. For example in the field of life sciences historically whatever risk and risk governance things that we had were very much linked to the physical infrastructure and lab facilities and facility inspection and material transfer control and things like that.
But that seems to have changed and seems to be changing very rapidly now with the kind of AI biodesign tools as well as LLMs that are emerging. So I think Rand also did a study on this, but there are more than probably 1 ,500 biodesign tools that are out there, and those are totally transforming how life sciences, but in general, science is done. Now, what kind of change that we are seeing is with these capabilities, now it’s much easier to engineer proteins, optimize DNA sequences to do things that we want, have better pathogen host modeling, interaction modeling, and things like that. Now, these capabilities are… because of AI becoming partly decoupled from the physical containment measures which were usually used in the life sciences.
So we have a lot of this risk landscape shifting a little bit more upstream to the design side when it comes to at least biological side of things. So yes, data governance, things matter. Model evaluation and red teamings are essential and we should be doing that. But also it is very important that especially for a country like India where we have a very vibrant scientific ecosystem but that is also very uneven. How we can use this AI -enabled science which is rapidly evolving into the existing mechanisms to some extent but also at the same time develop those capabilities, have more people with the core capabilities and more people with the core capabilities and more people with the core capabilities chemical security, AI nuclear security, and things like that.
So we need to train more people on those things. So integrating, again, going back to the life sciences, so integrating AI evaluation into biosafety system, strengthening the institutional readiness. Some places there are information, some labs and some institutions have information security labs or information security offices. How we can get them better prepared for these new emerging risks that are coming due to AI. Some places they have biosafety officers or biosecurity officers. How we can enable them better to address the AI risk is what the direction that we need to move towards. And have more adaptive oversight mechanism that is not only based on the, limited to this once in a while inspection that happens, but that goes more with the rapidly evolving things that we are seeing with the AI models coming up.
And I think, I think, So, just in terms of paradigm change that we are seeing and that you mentioned, is that there need to be more decentralized checks and balances and oversight mechanisms. If there is one authority sitting somewhere in Delhi and trying to do everything, that’s not going to work. So that is one of the things that we have to collectively think about. How do we decentralize these kind of oversight systems to some extent? For example, as I was saying, how we can empower the information security or biosecurity offices and create what in the field of disarmament where I have worked on called way of prevention. One measure is not enough. It’s not sufficient.
You need to have a number of measures in place which collectively can help prevent something bad from happening. Thank you.
Thank you. That’s very insightful. And I think we’ve already touched on some areas that, you know, that would be follow -up questions. P .T., focusing a bit more on… open science where high risk domains, especially in biological data and AI capabilities, as Surya was mentioning. How do we preserve the benefits of open science while preventing the destabilizing diffusion of capabilities that we were just discussing about?
Thank you. Thank you for having me today. So I guess like I would love to be able to give like a binary yes or no answer. Right. I think we all want to have that. But unfortunately, that’s not quite the case. So we need to find a way to balance the openness and also the restrictions as well. So I guess my answer here would be sort of like a tiered access and contextual norms. I think those are really important. And I think RAN Europe has done a really great job at establishing the global risk index on AI enabled biological tools. And also just generally looking into AI safety in general, where they do this thing where they call the pre -deployment assessment.
with structured rubrics. And I’m a huge fan of that because I think that when you release very frontier models and frontier tools, the danger is already out there once released. It’s really hard to withdraw the danger. But however, prevention, right? There’s this window before you release where you can do a pre -deployment assessment. So I think I’m a really huge fan of that and also the same way that I’m a big fan of KYC, know your customers. And I guess this principle also pretty much applies whereas in the case of biosecurity, where we differentially allow the development of medical countermeasures and also the defensive measures that is necessary for the research, but also don’t limit the researchers from actually innovating either.
And I guess my point here is that we’re not going to be able to do that. The non -safeguarded access, like private access to credential researchers where necessary for like defensive research is absolutely necessary. And then, you know, like open source tools, it’s necessary. Like we can’t turn away from being open source. Like any governance structure that conflates open source with danger makes a huge mistake because that also is a very critical development point, especially for lower resource settings. So we cannot afford to conflate that altogether. So I guess a very long way to answer this and then to summarize my answer is that differentiated governance at capability level is always better than blanket restriction at access level.
Yeah.
I think that’s a very structured answer and I think, you know, there’s a start of a very valid framework level conversation that’s already happening there. Geetha, turning to you, thinking more about institutional gaps in enabling some of the solutions that we are discussing, potential solutions, what are the most immediate gaps that you see in evaluating systems, technical capability, regulatory and coordination, largely from the policy angle that you work in?
Thank you, Shyam. Good morning, everyone. So on the technical capabilities, right, the most fundamental thing I see is the AI readiness aspect of deployment. So in general, when we see India stands or ranks third globally, and when we see the Southeast Asian countries, I think Indonesia is around 49, and so there we see the gap, right? So whatever we do from the Western context or in the Indian context can never be catered to the AI readiness aspect of deployment. So I think it’s important to the needs, the unique needs of the Southeast. Asian countries and moreover what there is the end user perception where we see that we have to build lot of capacity for creating awareness among the end users who are actually going to use the products and from the policy perspective I would like to give you certain aspects where we think about the socio -cultural aspects that is relevant to the deployment environments.
So in general the large language models are usually trained on the western data and the very recent research work maybe I will cover a bit of both tech and policy here. So there is a Southeast Asia related benchmark, safety benchmark which says that all these leading large language models have failed when they evaluated for more than 20 to 30 percent of the risk. So in the biological settings so which means that we did not have enough safeguards which will protect people from encountering all these risks. And moreover, so this lets us know that we have to build in more sociocultural evaluations and assessments which will cater to the harms that is more particular to that particular deployment environment rather than just having a high level evaluation strategies.
And this cannot come just from the policy side, right? So we need to bring in all the participatory approach which will bring in the end users, the different stakeholders involved in using all these AI systems, be it model, right from the requirements definition, right? So when we assess whether we need an AI system or not, generally now there is a perception saying that for whatever we are going to build or the problem that we are going to solve, by default we assume. We assume that we need a large language model which will not care. which is not even possible to have it deployed in a low resource setting, right? So we need to think about small language models which will enable edge deployments at the low resource settings and also consider all the multicultural and socio -economic diversity that exist in these regions so that your model doesn’t hallucinate, is still fair and also establish some governance and accountability frameworks which will make the developers more accountable and also because having the developers more accountable will enable them considering more safeguards, right?
And also create more awareness about the main fundamental thing is that they will be expected to document whatever testing that has been gone through. And on the policy side, there is one more aspect which is the Indian government also endorses, right? The self -regulation. voluntary commitments on managing and mitigating risk that comes out of all these AI models. So I think we have to have a unified framework which can still be adaptable to different deployment settings.
I think we are already getting a diversity of perspectives here and it is very useful to hear. Moving ahead and thinking about institutionalizing these kind of capabilities in scientific AI context, PT turning to you. Should independent evaluation and red teaming of AI systems from a technical kind of solution perspective for this problem that generate biological outputs, especially thinking biosecurity, given your perspective on this, should it become a norm and part of the global scientific specialist infrastructure? And if so, how would we go about that?
I think we have to have a clear understanding of the role of the AI system and how it can And I think that is a key point. And I think that is a key point. And I think that is a key point. And I think that is a key point. And I think that is a key point. And I think that is a key point. And I think that is a key point. And I think that is a key point. So I guess a good example to use here is probably we’re thinking of nuclear weapons, right? Which falls under this organization called the International Atomic Energy Agency, the IAEA. Now, from my perspective, I think fissile materials, correct me if I’m wrong, they’re very scarce.
And they are, to a certain degree, technically trackable. And they are also, more than anything else, highly regulated. Whereas biology, on the other hand, is everything but that. It’s diffused, it’s dual -use by nature, and it’s also nearly impossible to trace. And also, most importantly, commercially available, right? And so in the recent study, actually, this was done by this organization called SECURE. Bio, where they actually tested frontier large language models against expert virologists. And it turns out that ChachiPT -03 actually outperformed expert virologists by 94 % at troubleshooting wet lab protocols. So that’s a very shocking number, right? And then, I mean, obviously you mentioned earlier that there’s a very concentrated effort that is happening between the US, UK, and China, like the global superpowers, basically.
And I guess there’s, we, in the recommendation from the RAND Europe that I was, you know, helping out with is that we recommended that governments and also independent researchers do this six -monthly ritual of monitoring and also assessment of risk on a continuous basis. And we also suggested, obviously, like using AI as an automation tool to increase the efficiency of this risk monitoring system. But I think, to your point, I think stuff like this, stuff like that is non -interactive methodology that doesn’t require, you know, researchers to actually query directly with the danger systems is actually already in and of itself a very meaningful, you know, safeguard. But that is not enough. You know, we need something that is much larger than that.
That is the integration into, like, you know, institutionalizing it. And I would argue that, like, a six -monthly, you know, ritual, that refresh cadence, for it to be delivered, it’s going to require a very significant investment from the government at multilateral level, right? And so we can’t go without any investment at all. So my suggestion would be to actually implement this AI safety or security institute model that we’ve been applying where largely… It is technically credentialed. It’s independent, but also has a very… formal relationship with the government. And something that I would caveat from the bio side is that for the institution to have some kind of anchoring around biological weapons convention or the WHO.
Because right now that relationship is not quite there yet. And I think, you know, back to my point of like pre -deployment assessment, I think that is definitely needed and then the result has to be shared then across the credential network with tiered confidentiality that rather than being kept, you know, as a proprietary to the different state. I think it’s kind of a
That’s an interesting position, PT. Suryesh, thinking more about safety measures at large, how can we make sure that they remain rigorous and feasible within research ecosystems that you’re quite familiar with, you know, from a biosecurity angle, if you will, but largely also in the larger scientific ecosystem.
Thanks Shyam. I think first, yeah first thing that we need to understand is how that ecosystem is and then see if certain measures will work there or not, right. One of the hallmarks of let’s say Indian scientific ecosystem is there is a lot of heterogeneity. There are some places which are really extremely well performing and there are other places who are not well resourced or have other all kind of challenges. So, understanding how the ecosystem is, what kind of regulation within the institutes that are there, what kind of administrative measures that are there, what kind of safety teams these kind of institutes might have, all of those things are extremely important, right. The governance capacity, compliance culture and technical expertise varies widely in Indian institutions.
And I believe this is true for many other countries in the global south as well. So it’s not something very unique. Particularly to India, we have challenges related to different kind of resources. And even when the resources are there, sometimes it’s also problematic to use them efficiently enough. Now, given that context, if we just import safety frameworks that are developed in a well -resourced place in a Western country or any developed countries, I don’t know if those would be a very good fit for the kind of system that we have here. So those might become more performative than functional to some extent. Another challenge that also P .T. mentioned to some extent is that the speed and scale of AI is huge, right?
And we need these traditional review mechanisms that institutes have for safety audits and all of those things are not going to work. We need something which is far more adaptive and quick. And also what we had traditionally is this periodic paper -based facility -centric kind of measures. And those are very much outdated in the era of AI that we live in. Now, so what… Now the question becomes, how do we design proportionate capability -aware safeguards that would be better matched for the challenges that we have? One of the major challenges, as I think a lot of us realize, is that there is limited awareness about AI safety when it comes to scientific issues, even among the scientists.
So a huge number, a large majority of scientists just don’t know what they are putting, let’s say in chat GPT might be harmful or what they are getting out of biodesign tools could be harmful to some extent. So there is some understanding about the privacy -related issues, but safety and security is still a big gap in understanding of even the scientific experts that are there. Now also regarding AI, I think there needs to be a tiered risk classification. So not everything is highly risky. There are certain biodesign tools, for example, that are trained in… in virus data. Those we’ll put in a higher risk category compared to something which is just working, let’s say, on certain animals which are not dangerous.
Now, also the safety measures, as I was mentioning earlier, as the risk has moved a bit upstream, it has come more on the design side, we should also have more safety measures moving upstream. And as Piti was mentioning that, you know, certain kind of evaluation that are before launching AI tools are necessary, but also integrating AI evaluation modules into grant review processes, creating cross -trained AI biosafety review panels, so panels specifically for AI biosafety at, from the bottom -up side, instead of having them from the top -down approach. Investing more in domestic evaluation capacity, having more AI safety institutes like Geeta’s home institute at IIT Madras. So we need a lot more of that. And lastly, I think what we have in the US and UK are these, a lot of AI safety work is being done there, right?
And as I was mentioning, importing that directly might not work. And we in the global south are largely the users and importer of this technology. So we have to see from the bottom up side, where do we put those safety measures? Do we, like when it comes to import, what kind of, when the data is being transferred, is there certain places where we can put those kind of safeguards? Also, how we can use some tech sovereignty measures in this context, right? That tech sovereignty measures are used for a number of things, but AI security is something, AI safety and security is something where those could also be used to some extent. So, yeah, I would stop here and then we can discuss.
Thank you.
Thank you. And I think a lot of useful thoughts here for us to explore a bit more. I think we’ve… just crossed the mid mark and I’m going to use Geeta to kind of like bridge between the next two topics by combining two of your questions sorry for that so just as Surya just mentioned will the emerging scientific powers you know global south middle powers would they be able to shape governance in this context especially you know enable science or will they continue to inherit the frameworks and if they were to show leadership what would that look like in scientific AI and research ecosystems and you know you’ve already been working on some of this so I’m looking forward to kind of hearing concrete measures that you know are happening
Sure. So in general what I think is definitely the emerging powers right they are putting on all efforts to bring in all the tools and frameworks that are required for governing these AI systems and for example, so India’s strategy towards all these emerging techs is that they are trying to create sandboxes which are highly essential for deploying or evaluating safety aspects for the models, right? So they do it for healthcare systems, they do it for ideology systems and whatever, right? So these type of tools and frameworks come from Indian settings will actually help the other underdeveloped countries to learn from the strategies that we use and then build something of their own or something which cannot go cross border can still happen through learning and collaboration, right?
So for example, we are going to launch a global south network for trustworthy AI and we are going to launch a global south network for trustworthy AI and we are going to launch a global south network for trustworthy AI which will enable all these mechanisms to happen, enable people to… develop and deploy AI systems which will be deployed in the low resource settings. And the other initiative which is going to give a very big leap in evaluating AI safety is coming up with an AI safety commons for the global south. That is part of the safe and trusted AI pillar that is one of the pillars in this impact summit and I think in another one or two years we will have safety commons which will help us evaluate and assess how these AI data models and systems work for different deployment settings.
Another important thing is that as Suresh mentioned about the audit frameworks. So when we come with, when we focus on the kind of risk and audit mechanisms that we have here, we still have it from an organization perspective and not from the end user perspective. So at CRI, we have come up with an incident reporting mechanism and a framework that caters to the Indian settings. So it tells you how to operationalize incident, AI incident reporting in the Indian settings, which is completely different from the Western settings. And here we have to get the harms that the people experience in the marginalized communities, which will never be recorded everywhere, right? So how do we enable all these things?
So since it is all about all these CERN -based systems, right, even those things will have certain impacts to the marginalized communities, which may be an indirect impact. But how do they are knowing about such things are happening to them, right? So those kinds of gaps we should mitigate by building more awareness, creating more AI literacy. And we should also be able to provide more privacy to all these people. The final thoughts about combining all these things is that we have to bring in some kind of collaborative work between the different stakeholders who are involved in developing and deploying these systems. And the governments have already given certain prompt knowledge about how to enable all these things through the techno -legal framework and guidelines that was recently published and the AI governance guidelines.
Which was recently published by METI. So the Southeast Asian countries can learn from the developing countries like India and then have curated a more tailored approach towards their unique needs. So that is what I think. So whoever has an opportunity or a willingness to have more things that will actually help them use or leverage these technologies can learn from whatever. Learn from. the mistakes as well as the experience that the other countries have, which is now openly available through all these summits.
That’s very useful and I’m looking forward to following up on IIT Madras’s work in this front as well. Going to Suresh for kind of the last question in this series really, should, you know, safety measures, evaluations, primarily focus, where should the focus be at the model level? And you talked about upstream quite a bit. Should there be more broader socio -technical readiness measures, misuse considerations? Where do you think it should be?
And also, very importantly, how we have to also see it from the context of, you know, people doing their own thing, DIY kind of science that happens. And also, small -scale commercial activities which are not fully under the oversight mechanism of the government, right? So, considering all of these points, right, the policy evaluation must expand from model -centric assessment to socio -technical assessment. And this would include, you know, evaluating things like how much capability uplift relative to the government capacity that is there. So, government has certain capacity to manage or do oversight, but these AI tools, how are they changing that? Incentive structures, very, very important, that shape the model deployment. Also, the diffusion of risk across borders.
All of these things don’t respect national borders, right? So, how it’s going to spread. If people using VPN or other things, a number of other things that are there. So integration, lastly, the integration with existing biosafety and resource security systems as I had already mentioned. So briefly, like performance evaluation is necessary, but governance -relevant evaluation must be systemic. And otherwise, we risk auditing algorithms while ignoring the institutions that operationalize them. And that is very, very important, how we focus on that institutional level mechanisms. Thank you. Thank you.
Piti, kind of the last structured question before we move into a bit more of an open conversation. AI becomes embedded not just in new capacities, but also existing programs like biosurveillance, public health systems. And so there’s a mix between emerging kind of scientific knowledge with more legacy, let’s call engineering knowledge as well. So. So how do we make sure that safety, evaluation, interoperability, all of that exists in this divide without fragmentation happening across the ecosystem? Because, you know, you can easily imagine everyone’s doing their own AI, you know, safety evaluation and not necessarily talking to each other.
Thank you, Shyam. I think this is a very important question. And it’s also a topic that I’m really passionate about as well, which is biosurveillance. To your point, I think, you know, countries are already deploying AI -enabled biosurveillance systems that are, you know, either syndromic surveillance or it could be, you know, genomic sequencing pipelines or outbreak modeling. The countries are already doing that, but they are not building on… the unified data standards. So they’re basically building on very incompatible data standards with very different legal regimes across the borders. We’ve seen that in Southeast Asia. We’ve seen that even countries like, for example, Singapore to Malaysia, you see different legal regiments on how they monitor the data and also the biosurveillance.
And so the fragmentation risk is actually not a technical risk, I would argue, because it’s not just a technical risk, because we’ve seen COVID. I feel like if anyone is anybody saying, I think we all were a little bit traumatized by COVID. We’ve seen how data hoarding and incompatible reporting actually cost lives. And I saw that especially happening across the region in the lower resource settings. Like countries like Cambodia, for example. AI systems that are trained on non -representative data obviously are going to perform much worse. And guess what happens? When they perform worse, the region that is most affected is the region that needs the help the most. And because of that, and also that region is also the same region with the least data infrastructure.
And so I guess to sort of like answer your question and what I think we need to do, I think there are three things to be addressed here. The first one is obviously the data standards harmonization. Currently, we don’t have that. I think we would need not like a global overhead standard that enforces on every country, but more of a federated interpretability that applies frameworks that applies to different countries. So I can think of like HL7FHIR, which is the federated… healthcare interpretability resources that are attempting to address these very specific issues on clinical data, but this one would be adapted for public health surveillance. And the second point is the legal safe harbors for basically just kind of cross -border sharing of data for public health emergencies that are negotiated beforehand because, and this is important, beforehand, because if you negotiate during an outbreak, people are going to be freaking out.
People are going to be like, I’m not going to share my data to you. What are you going to do with that data? So this needs to be done beforehand. And the last point, and also the most politically challenging point, is actually to have some kind of shared evaluation criteria across the board between different countries that are embedded into the national surveillance systems. And, for example, like Singapore data infrastructure environment might not apply to countries with like different climate data or like different demographic data. So this needs to be applied into, you know, the national surveillance systems. And what I noticed, I guess like the last message is that what I noticed the AI governance framework often thinks of biosurveillance as like an edge, like a niche edge case.
And then people in biosecurity frameworks, like doing biosecurity frameworks, thinks that AI governance is like a tool. And these people don’t talk to each other. And that gap, that gap right there is where the risk happens. So, yeah, we just need to talk to each other more. That’s easier said. Yeah.
So I think I’m just about to close with maybe five minutes or just under that for audience questions. Thank you, Justin. 10 second final thoughts on each of you from the panel. Suresh.
Just wanted to very quickly, we need to also keep in mind that how AI could help solve some of these AI safety challenges. How agentic AI could be used, let’s say, when people are trying to develop vaccines. CEPI has developed this platform where agentic AI is being used to check if there is someone who is trying to jailbreak or someone who is trying to misuse the tool that is there. Second very quick point, also, with all that what I said, there is still a gap to transfer things from digital to physical, what is called digital to physical barrier. So, even if you have everything, you still can’t just develop, modify viruses without having a proper physical infrastructure and there are still some ways to control that.
Thank you.
I think we should move on transforming from issues to intelligence like learning from the risk that happens and feeding it back to the model training and other assessment activities to mitigate the risk in real time so that is where we need to move towards bringing in more people into evaluations and then making it safer for people to use
I guess I’ll make it quick the point that I want to make here is that Shurya should echo his point I think you’re right that we should not shoot ourselves in the foot especially for developing countries, I think it’s really important and so my message for the last message here is just kind of like while we are forging ahead in innovation and while we are innovating ahead in whatever domains of scientific domains that we’re doing we need to be conscious of the impact that we have and I think in the AI Impact Summit is one of the really good places to jumpstart that kind of conversations and break the silos. Thank you.
Thank you everyone. I’m just going to take probably one minute to kind of summarize key points Evaluation, I see largely a systemic question, safety measure systemic question. I especially like the point on incident response not being already there. And a couple of points on the cross -border solutions and problems, we already have that. Discussion on open signs, we talked about how managed access, safeguards, and comparing government capacity to manage that versus letting it out for more DIY -oriented signs, which is a good term, I really like that. That’s a key area. And for emerging scientific powers, of course, collaboration is key. Tailored approach, that’s something that I’m again waiting to see from IT Madras as well, their contribution on this.
And some cross -border work on legal safe harbors, data standard harmonization, PT that you mentioned, really land well from this panel. I’m going to… I’m going to stop my… summary right now and you know more of this would be kind of put together in a blog at some point in the uh nearby future uh perhaps uh we can go for questions uh first uh yes please i think i can give you mine
Thank you so much for your wonderful insights i really enjoyed this session as a researcher in safety of ai at the university of york so i focus on psychological harms of ai and so what i want to ask particularly gita is um when it comes to the definition of harms and traditional safety engineering they’re catering more to physical harms and now we see the whole spectrum of harms expanding beyond that so i would love to know the work being done by karai and you in this area and and in fact enrich my research with it
Yeah, sure uh so when we actually assess harms and impacts right we do we have to do it from the different two different perspectives one is on the functional side where we assess all these algorithmic risk and other stuff. From the human centric perspective, like you said, we can keep doing everything from the psychology perspective and other ethics and other stuff. So, here at CIRI, we do work on assessing bias, determining whether the model is stereotypical or not and how do we generate explanations for the high level scientific models and all. So, from the perspective of the psychological things, there is this cognitive science or cognitive capabilities of AI models which will actually enhance or degrade the capabilities of humans.
So, those things are we are trying to do some assessments from the incident perspective. So, if you go to read the incident reporting framework that we have, we have a taxonomy of risk and harms and also the impacts. So, from the kinds of harms that we have defined, we have categorized it as physical, psychological, cyber incident based harm set. And moreover, we have all the generic kinds of harms like algorithmic harms, socio -economic harms, the environmental harms and all. So, we are trying to come up with a taxonomy that will cater to the different hierarchies that will be applied to these kind of harms and impacts which will again be model specific, use case specific and the domain specific.
So, that is where we are trying to work on. And we also have a healthcare based tool, a toolkit which will enable people to actually assess the perceptions of how they treat these models, how they see whether these AI applications are helpful for them or not and then come up with some capacity building programs for different roles in which they are working on. And this has been done with CMC Wellure Hospital and we have been assessing the perceptions of healthcare workers. and then come up with a training module which will enable them to use AI models or tools more confidently rather than, say, being resistant or not relying on them for so much.
Last, probably last quick question. Maybe keep it short on the responses as well, please. Sorry.
Hi. So my question is about, like, we are discussing all the geographical barriers, right? The modality is geography. When we change the geography, the models tend to perform poorly. Are we concerned about the temporal modality as well? When we go forward in time, the data is going to change eventually, and that is going to affect modeling. And how do we plan on, like, you know, mitigating such a problem if it arises?
Yeah. So this comes under the model monitoring, the system monitoring approach, where we consider the data drifts out of distribution. So we consider the distribution aspects of the data and models. So definitely this is one of the criteria where you assess safety and evaluate the impacts of it.
Yes, I think last question
Thank you so much for the insightful discussion, really appreciated the expertise that you’re bringing to the topic and thanks PT for bringing up COVID because my question is about that. As we learn from COVID biosecurity risk can quickly become a cross border existential threat. So what would a successful web of prevention and incident response framework look like and who are you looking up to in this space? Like who’s doing it well in this space?
I can start maybe PT can add. So I think as I was mentioning, it will have to be more decentralized but at the same time integrated to the leadership. So I think there needs to be more empowerment of people who are like biosafety officers in the lab or who are institutional biosafety committee members, who are people who are working on the ethics and research security side at the institutes. So those are the people who need to be empowered. So there needs to be more capacity building of those people and at the same time there needs to be a mechanism established so they can report those incidents to the very top and there is top leadership sitting in the capitals.
They can in some way get an overview or monitor the situation as it is going on at different institutes level.
Thanks. I can add a little bit to that. So in Singapore we actually have different agencies responsible for this. So we have the National Environmental Agency and then we have the MOH, obviously the Ministry of Health and then we also have different smaller agencies like Communicable Disease Agency and also like Prepare Agency where they are responsible for different tasks. But I want you to envision this as almost like the way that Singapore is trying to establish itself. I think it’s trying to establish itself almost as a firefighter. So when there’s an incident where there’s a crisis, who is actually doing what is very clear but it’s always not always clear across like different countries. For example, in Laos, Vietnam, might be looking very different, but I think having a very coordinated response across the different agencies on who is doing what.
Like, for example, National Environmental Agency is responsible for wastewater surveillance. So monitoring how the sickness is increasing or spiking or not, those are the people, yeah, that you would look up to. And I think that’s the last word, right? It all comes down to prevention and preparedness, even in this much like anything else with biocontext.
Thank you, everyone, for the question, and thank you to my brilliant panelists, Suryesh, Geeta, and P .T. This was a very insightful discussion. On the screen is the work from RAND Europe with CLTR, some of what was referred to by P .T. and other panelists as well, some aspects of what we were discussing about risk typification. You’ll probably get some ideas there as well. And with that, I close. I’m surprised. I’m supposed to hand over these mementos to apparently including me, so let us do that now. Thank you.
“Speaker 1 is a bio‑security expert with experience in disarmament.”
The knowledge base lists Speaker 1 (Suryesh) as a bio-security expert who works in the field of biosecurity and disarmament, confirming the report’s description [S3].
“Open‑source tools are crucial for low‑resource settings and help democratise expertise.”
A source notes that AI democratises expertise previously limited by resources, giving people in underserved areas access to sophisticated diagnostics, which adds nuance to the claim about the importance of open-source tools for low-resource contexts [S105].
“Balancing security concerns with open‑source approaches requires case‑by‑case solutions.”
The knowledge base highlights a discussion on the tension between national security and open-source approaches, emphasizing the need for ongoing dialogue and tailored solutions, providing additional context to the report’s tiered-access/open-science discussion [S110].
“Data governance, model evaluation, and red‑team activities remain essential for responsible AI deployment.”
A source describes the practice of publishing model cards, evaluation benchmarks, and data to make model behavior transparent and to flag risks, which supports and expands on the claim about the continued importance of data governance and model evaluation [S108].
“The panel discussion examined biosecurity challenges within the Biological Weapons Convention (BWC) framework.”
Another source discusses the focus on biosecurity within the BWC, emphasizing non-proliferation of dual-use research, which adds background to the bio-security perspective presented by Speaker 1 [S101].
The panel shows strong convergence on several core themes: the upstream shift of bio‑risk due to AI, the need for decentralised and capacity‑building‑focused oversight, tiered and adaptive governance, and the integration of AI safety into existing institutional processes. Participants from different backgrounds (bio‑security, AI policy, regional capacity building) largely reinforce each other’s proposals rather than contradict them.
High consensus – most speakers align on the structural nature of the problem and on concrete policy levers (decentralised checks, tiered risk regimes, continuous monitoring, capacity building, and collaborative standards). This broad agreement suggests that future work can move quickly toward implementing multi‑layered, region‑specific governance frameworks without needing to resolve major conceptual disputes.
The panel shows strong consensus on the need for enhanced AI‑biosecurity governance, capacity building, and multi‑stakeholder collaboration. The principal disagreements centre on the architecture of oversight (decentralised national networks vs a centralised international institute) and on the cadence of monitoring (continuous adaptive mechanisms vs a fixed six‑monthly ritual). These divergences reflect differing assumptions about feasibility, resource allocation, and legitimacy, but they do not undermine the shared recognition of risk.
Moderate – while all participants agree on the problem and the overarching goal of safer AI‑enabled biology, they diverge on structural and procedural solutions. The implications are that any policy outcome will need to reconcile decentralised national capacities with some form of coordinated, possibly internationally‑anchored, monitoring framework, and must balance the desire for real‑time adaptability with the practicality of periodic reviews.
The discussion evolved from recognizing a fundamental shift in biosafety risk—AI moving threat creation upstream—to debating concrete governance mechanisms that balance openness with security. Early insights about upstream risk and tiered access reframed the conversation, prompting participants to surface regional capacity gaps, propose institutionalized monitoring bodies, and stress the need for interoperable data standards. These pivotal comments redirected the dialogue from abstract concerns to actionable, context‑sensitive solutions, ultimately converging on a shared vision of decentralized yet coordinated oversight that can be adapted by emerging scientific powers in the Global South.
Disclaimer: This is not an official session record. DiploAI generates these resources from audiovisual recordings, and they are presented as-is, including potential errors. Due to logistical challenges, such as discrepancies in audio/video or transcripts, names may be misspelled. We strive for accuracy to the best of our ability.
Related event
