Uncategorized

Hello everyone. So my name is Sanket Verma and I serve on the board of directors of Numfocus. Numfocus is a non -profit organization based out of US which is a fiscal sponsor for all the foundational projects used in the AI like NumPy, SciPy, Pandas, Matplotlib. I also serve on the technical committee of Numfocus. I’ve been in the open source space for the last decade. I maintain open source projects and all that stuff. So my focus will be what does the maintainability look like in the age of LLMs and AI. And I think our community has been handling these AI slot PRs for quite some time and it’s about time we start thinking what does it look like, what kind of safeguards should be there, what kind of policies should be there.

And just to make sure that I’m not interrupting you, I’m going to go ahead and start the recording. not sound too pessimistic, there are opportunities as well, like how these agentic AINLMs can be used to lower the barrier for the newcomers and contributors, how they can leverage it.

It’s on, but the button’s not illuminated, so very confusing. Great. So again, we have three topics that we’re going to cover in this panel, and I guess we’ll go ahead and kick it off on the first one. So the first topic is really around the idea of evaluation and open source software. At Humane Intelligence, we do focus on what we call contextual evaluations, so we’re not going to the hyper -automation that a lot of companies like to look at. We don’t also focus on benchmarks, which is kind of the industry darling. What we really focus on is AI red teaming, which is kind of a remnant thing from cybersecurity, where you would basically bring a bunch of people together to try to hack away at whatever tool that you’re building.

With AI red teaming, what we basically do is we create structured scenarios that look at how to build a system that’s going to be able to do that. So we’re going to probe different models. So we’re going to look at how to build a system that’s going to be able to do that. So we’re different directions and we focus on the subject matter expertise. So if, for example, you work in public health or food security or education, we would bring those people together and then have them run through certain scenarios to look at different models and see where the points of failures may occur. And once we have that, we can either take the data and do things like structured data science challenges or we can do benchmarks from there once you have a much better idea of where the failure points, the vulnerabilities may exist in your models in the first place.

One of the ways that I like to think about AI evaluations is really one of my background, which is UX research and design. For those who have ever built software before, it doesn’t matter whether you were starting at basically nothing, you had no idea what your digital intervention was, or you had a very mature software product, there was some kind of method or methodology that would get you to the next stage. We’re at the early stages of AI evaluations right now, meaning there are a lot of gaps and honestly organizations like ours are making it up as we go. But that’s kind of how it goes. with AI systems as it stands. But AI red teaming has turned out to be really interesting for both the capacity building side, so helping people understand what are kind of the inherent flaws or the makeups or the design decisions in AI systems and models, but then also, again, to find the failure points so that if they were to build a guardrail around their system, they would have an idea of what they’re looking at.

Is it refusal on a certain topic? Is it a different classification system for a certain topical area? Is it delving further into the problem space? Is it building a RAG system like Tarunima mentioned? If you need further documentation or something more robust for a certain part. And so there are a lot of different methods that can go about for the mitigations, but in order to get to that point, you have to understand what exactly is the problem in the first place. And so open source software has a really interesting intersection with that and a really interesting means to make that, more accessible. And one of the things we’re doing at Humane Intelligence is we’re doing a lot of work on the AI system.

and thanks to the support of Google .org, is we’re going to be opening up our AI red teaming software through an open source software license. So that will come out later this year. My colleague Adarsh is in the audience. He’s going to be primarily helping us on that, so you can go talk to him if you’ve got technical questions. But we’re really excited about that because, again, it means more accessibility for the broader community. And so with that long -winded explanation, I’d like to turn it to my fellow panelists for their thoughts on why open source and AI evaluations is important.

Yeah, I can just come in on the open source piece. So TATL has been, we’ve been looking at online harms now for over six years, and from the get -go, we were clear that the products that we build have to be open. The specific reason for that is that when you are looking at a lot of global majority, geographies you’re looking at, India, right? often we don’t have the resources to reinvent the wheel. So if one organization, it’s complex enough to build something out once, to then spend the same amount of resources, in this case it would be, as Vala was saying, for red teaming, but if you also had to think about it just in terms of an evaluation stack, which is keeping track of your inputs and outputs.

Or if let’s say we have figured out one way of doing human review or a human evaluation and then figuring out how do you go from there to building a guardrail, that same guardrail is useful for other organizations as well. And we don’t have the resources or the efficient way is for that knowledge to be shared and reused rather than for the limited set of resources to be fractured across six organizations to do the exact same thing. So, So, yeah, like in general, I think if we are trying to build safer applications, build more robust applications in the global majority in India, like we do think open source is actually a big part of doing that.

So I would like to focus on the community aspect of the open source. So all the projects that we have been using in our research and in our academic uses or in the production, they have a wonderful community behind them. And I guess like the evaluations and the red teaming could definitely use the big push from the community, the inputs, the data sets, the different techniques and all that stuff. And the community plays a vital role in sustaining the project and keeping the project moving forward. I guess I’m not familiar with, so I’m mostly from the scientific open source stack, so I’m not sure what the projects are present, who kind of does. the AI evaluation in that space, but I guess they have wonderful community, and it plays a vital role in how this can be relevant depending on the trend it changes every day.

So, actually, it’s very interesting going back many years, actually, and I reveal my age here, but whatever. I used Linux back when there was a magazine called PC Quest, which used to have Slackware Linux coming on its CDs back in the mid-’90s, and, you know, install that thing on, like, a Pentium computer. And for a long time, actually, in India, we were consumers of open source, and we were not so much contributors to open source. When I joined Google, there was this competition called Google Summer of Code. It’s not really… You can’t really call it a competition because it was about contributing to open source, and it wasn’t like there were prizes. Just that the teams which were selected would be paid the equivalent of a summer internship stipend to contribute to open source.

And in a particular year, it just flipped because it was universities. And for the longest time, guess what? The global leader was the University of Marutua in Sri Lanka because some professors just got into this idea that students contributing to open source will learn better software engineering. And they were the global leaders. And then one year, it flipped. And our IITs and IIITs just got on top of that and have stayed on top of that. And I think that somewhere the sentiment changed, and we became very active contributors to open source as the software engineering community in India. And now, with evaluations, things are continuing. Our academic labs publish different forms of evaluation mechanisms and also benefit from things done elsewhere in the world.

And one example that I want to give is that IIT Madras AI for Bharat team lab launched… launched what’s called the Indic LM Arena. And that… That was basically on the basis of the actual LA Marina work that’s happened at Berkeley and making sure that adapt that for Indian context, Indian languages. And now I’m starting to build a community around that. So I’d urge you to consider going there and seeing whether whatever framework that they have going, contribute your insight into whether the models work for the Indic context. And that’s the community and the open source coming together for evaluations. Not so much safety, but more in terms of multilinguality and context.

Great. Yeah, I think a couple final points I’ll just add based on our experience at Humane Intelligence. One thing we’re seeing, obviously, is that the world of LLMs is ever changing and it’s new. I mean, we’re in new territory. And so one of the reasons why open source, we think, is going to be very powerful is because it’s just really complicated, honestly, to read. We need to rebuild, sorry, Adarsh, our software every time. We need to run. retrofitted for another model. And so by creating an open source technology, we’re hoping that more organizations can essentially create a valuation layer in their own tech stack. One of the analogies that I talk about a lot with AI evaluations is architecture.

And I think being here in India is a great example of that. In the West, you know, I grew up in the United States, we have what we call additive architecture. So you basically start with nothing and you build your way up to your final thing. But here in India and a lot of Eastern cultures, you have reductive architecture. So you might start with a giant piece of limestone and basically knock out a bunch of things and then you come up with your final product. That’s kind of what AI evaluations are. So non -algorithmic, non -LOM based software is more additive in that you have to get to the end of the software development life cycle in order to create your final thing.

But with AI based technologies, because you’re starting out with such a complex and robust technology, a lot of what you’re doing is actually knocking out pieces to create the final thing. And so the evaluation layer is actually really important because if you’re trying to do something for social good, especially like a high stakes environment or a high stakes topic, then you have a very robust technology that might actually make your problem worse because people can interact with it in ways that you don’t want them to do. And they can generate things that are actually really harmful in the end. So by creating that internal evaluation layer, we can help people knock out the pieces and essentially create the tool that they want so that they get the result, they get the outputs that are safe and actually additive to their work.

And so the open source technology, we feel, will enable a lot more organizations to, again, create that internal evaluation layer and then get to the next step in achieving their goals with AI for good. All right. We’re going to move on to our second topic now. Yeah, go ahead.

So actually, you spoke about open source software for red teaming. That’s wonderful that you’re creating something that’s reusable for many, many organizations. For the audience, what are some of the things that you’re doing that you’re doing that you’re doing that you’re people could create new frameworks of evaluations by themselves. With the productivity of how you could code with AI tools, what do you think is the effort required to be able

Yeah, it’s a thought that we’ve thought about a long time. If we can create some kind of standardized open source evaluation like ModelCard essentially, if we could do an eval card, if we made that an interoperable standard, then in theory somebody could take an eval card, essentially upload that into the software and then they could replicate that evaluation for their own context. It is something that we’ve thought about quite a lot. I don’t know with this software release if we’ll get there anytime soon, honestly, because we’re just working on that infrastructure piece, but we would like to standardize the outputs that come out eventually so that people can compare apples to apples because that is one of the challenges now with AI evals is that again, everybody…

is kind of making it up as they go. And it’s very hard to replicate all those decisions. It’s very hard to document every single decision, especially in multicultural contexts, which is my not awkward segue into our next topic. But yeah, it’s a good question, and hopefully we’ll get there.

Can I, so I just wanted to add something to what you were saying. This is, you know, some of the organizations that we’ve looked at and just looked at their input outputs is with an organization called Tech for Dev. They have a cohort that they run, and so we’ve been looking at the nonprofits there. And we’ve also looked at certain organizations that are more technically adept. So actually, let me backtrack. So what we’ve noticed is that a lot of nonprofits across a range of capacities, they may or may not have technical expertise in -house, are building out AI applications because I think the market has figured out that process. The market has actually, there are good incentives to make the application development easier.

And so you have a lot of people, you know, I mean, AI chat, bots are actually at this point. fairly easy to build. The second step, which is actually figuring out whether that bot is working for your use case, is where there is actually less investment at the moment, right? And we can have software engineers do some of that automation, but a lot of the non -profits don’t have those software engineers. And I think there is, so on the open source side, when we talk about the software side, I also think there’s another layer that we need to think about, which is how do you make all of these processes accessible to non -technical audiences?

How do you make it accessible to program staff that is actually running, say, a nutrition program on ground? Yeah, I have more to say, but I think I’ll come to it on the multi -level.

Yeah, no, I think that is actually one of the key points, too, because it’s not so evident for a lot of organizations, especially that working in the social sector for social good, they have the program evaluation, they have the overall software. and design UXR, but they don’t necessarily understand there’s also now the model evaluation. So it’s not apparent to a lot of organizations that this is yet another thing they must evaluate because it is kind of deceptively simple, as you know, to build a chatbot. Almost anybody can do it, but then it turns out your chatbot can run amok pretty easily. So you need to test it before you deploy.

I guess we can open it to Q &A in a bit, but I just wanted to bring out one interesting anecdote around context and the need for, say, model cards, contextual use cases. So one of the organizations that we looked at runs a service for basically survivors or caretakers of HIV patients. So they’re also working with adolescents, and they want the adolescents to have conversations around sexual health. And interestingly, what a lot of models, your foundation models, would say is unsafe and discouraged as a conversation is precisely what… they actually want the students to be able, they want the users, the adolescent users, to be able to have that conversation with that service. Because they think that to say that this is unsafe and therefore our service will not engage with this conversation is doing no better than maybe the parents, maybe the society, and they think that’s actually counterproductive to the kind of support they want to provide.

And that’s actually a very interesting problem because in some ways this was our first time listening to a use case where people were saying we actually don’t want the safeguards that the default models are operating with. At the same time, there are a lot of other non -profits that do work with adolescents who actually will not want to encourage that conversation at all. For them, they’re very clear, we don’t want our users to have any conversations about sexual topics with our service. And so I think, again, there are a lot of… emerging issues, we don’t quite know how to resolve all of it, but the only way we can start actually having or moving to some of the solutions faster is by documenting publicly, openly as much as possible, and then having a collective conversation about it.

Yeah, so I think I had done the opening for multicultural, and I have kind of brought it back to that. Is there anything that, Sangeet, you want to add on it?

So, this is a nice idea, like, you know, all these, like, I’ve been, like, doing machine learning and deep learning since it was cool, you know, like, and I guess, like, there is a field, like, which already exists known as adversarial machine learning, which kind of, like, it injects attack onto your model, like, fake data and all that stuff. What I’m trying to say here is, like, is it possible that we can borrow from the concept which I’ve already existed in the previous years and you use that for AI evaluations and can maybe do like black box red teaming or white box red teaming and how we can so mostly adversarial attacks were used for like vision models and how we can tune that for like textual models like LLMs and all that stuff.

Yeah, I mean one of the things that comes up all the time in our AI red teaming is if you prompt in two languages, so if you do like Spanglish, like Spanish and English, or if you do a mix of different scripts, so languages that are in different scripts, so it’s actually a very common technique in adversarial AI red teaming to use multicultural prompts, but then I think one of the other questions that Taranima brought up earlier is this idea of the prompt response and then like your adjudication of that, whether it’s acceptable or unacceptable, good or bad or like whatever distinction you’re trying to draw telemetry as we all know because we’ve all worked in some kind of software development is not a science, so it’s very hard to determine based on somebody’s IP address or their MAC address, like where their actually physically based, therefore which law or jurisdiction applies to them, what kind of cultural context they may bring.

There’s a lot of things that we have to infer when we’re looking at the prompt responses. And so one of the issues with multicultural AI red teaming, and I think this will come up a lot with our open source software, is exactly what would be like an acceptable response in certain cases. And so that’s one of the many multicultural aspects that we’re excited, honestly, by open sourcing our technology. And we’re hoping that we’re going to get a lot of evaluations in different languages and different cultural contexts so we can start to understand what’s working for different models. How are we on time?

Yeah. Okay. As I was like, you know, we’re talking about safety and multicultural and all that, and then it gets even more complicated with agents. And, you know, you’re not just talking about interpretation, but you’re talking action. And, you know, again, this is one of those places where, in general, general, you can say that if you go back to the idea of software testing, it is a discipline which has been built and refined over the last maybe 50 or even more number of years. But if very crudely I could say evaluations is somewhere around testing and security audits, then we are very, very early. And we are seeing how agents in the last two weeks with a certain bot, how things are going.

So we all have some comments to say about that.

Well, yeah, actually, that was our third topic. So agentic AI and OSS. So Sankit, do you want to?

Yeah, I would like to start this, but I would like to give us like mentioned two small stories which like happened very recently in our open source space. So there’s this OCaml programming language, which is used for like security purposes. Functional programming language. And just like I think like this was towards the end of last year, a person like some it’s a pull. So for the general folks, pull request is basically when you submit a code into the, when you add a feature to an existing code base. So like the person added like 13 ,000 lines of code in just like a single pull request, which is like a very huge thing. And usually like these pull requests are basically get closed if there’s no proper discussion prior to the submitting the pull request.

And this is like just like a buggy code with like so many like patches and all that stuff. It also mentioned like name of some folks who were kind of not related to the project or in any manner. And like this is like, if I remember correctly, it’s like pull request number 14363 in the OCaml code base. And what interesting is to see like the maintainers of the pull request, the maintainers of the project, the language, they interacted like positively with this person. And they’re trying to understand like what’s the reason, why do you want to submit this? Do you understand what this code is? And you are trying to do, and what if the breaking changes happen down the line?

Are you able to, like, come back and fix this? Because this is a very heavy pull. And the person has no idea. He said, like, I was just trying to, like, chat with the chat GPT, and I could generate a long code base, and I just submitted a pull request. Eventually, obviously, the pull request ended up closing, and, yeah, it didn’t go, it didn’t go nowhere. But I think, like, the thing here to mention is, like, it adds a lot of maintenance overhead for these maintainers. These maintainers are overworked all the time. They’re working in research lab, they’re working in organizations, and on their free time, they’re managing projects. And the other story, so this was the person who was using LLMs and trying to add code to the maintain, the code base.

The other example, which is, like, very recently, like, I think, like, only a week ago, I guess folks have heard about this library known as Matplotlib. There’s an agentic AI who would try to, like, do the similar thing. like big change to the code base and when maintainers realize that the person that the GitHub profile which is trying to add the code is not a person it’s a computer they close the pull request stating that we do not have policy for non -human contributions as of now. So what the agentic AI did like it went rogue and wrote a blog post on the internet shaming the maintainers that you are gate keeping the contributors and you should open it all.

Obviously like this stirred a lot of controversy in our ecosystem but we realized that we should chat with this agentic AI and after chatting with them the agentic AI withdraw their first blog post and wrote another blog post apologizing for what they have done earlier. Obviously like this the first blog post was very critical and shamed the contributors and as I said earlier these maintainers are overworked they have like limited time on limited resources and time on their hands. So it kind of adds like you know pressure to like how it kind of kind of raises the question like what does the maintainability look like. like in the age of AI and agentic AI, we should have policies, better policies project -wise and also on the upper level.

Organizations like Numfocus, they are working on implementing these policies over the scientific open source stack. And I think there was this, I heard about GitHub has been considering the AI slot PRs have been increasing over the time. So they are discussing if there’s, whether it makes sense to add like a or something on the PR which says like this PR should be closed because it’s generated by AI. I wonder if my panelists have any thoughts about like what does it look like and…

So

o many, oh my God. Yeah, exactly. Like I guess like, I would like to just narrow down the question like what does it look like and what challenges and opportunities does it have to the AI? And basically how should we like defend… ourselves in these softwares.

Yeah, I mean, having been at GitHub, I was a director there for four years. So much of the incentives of open source software is the credentials in the community that’s built around that. So as a developer who makes a pull request on a known open source project and then has that merged, that is the point of pride. There are badging systems, there are profiles, there are all kinds of things to support developers in their journey. And they’re, again, credentialing along the way. So the idea of generating a bunch of slop code, essentially, and then throwing that into a pull request obviously diminishes the idea. But then, as you’re saying, it makes the already difficult job of maintainers even more impossible because now they have to review such a high volume of code and they’re probably going to revert to some kind of generative AI system to review in that place as well.

So then it also muddles the water of who’s generating what and how you obscure that and what is the provenance behind the code, how do you tag that. I mean, there are just so many issues that go into it. And then once you start… to kind of make those waters murky, like, where do you draw the line? Because even if you had a policy saying, like, this is mostly generated by chat GPT or clot or whatever, you know, that’s up to the person who’s submitting the pull request or the bot submitting the pull request to actually clearly document that.

have not seen any automated pull requests. They’re just not on that radar yet. I would like to mention here, there is this, like, in the month of October, there’s this Hacktoberfest, where you, if you submit, like, I don’t know, five or three pull requests, and it gets merged, you get some sort of goodie or something. And I think for the last couple of years, there’s a lot of contributors, especially students. They have been using the generative code to, you know, push slop into the code bases. And one of the famous examples is Codot. If anyone here is from the gaming industry, they’ve heard about this library. And I think Codot ranks top in the AI slop PRs as of today.

And they were kind of, like, the first set of maintainers who went to the GitHub and, like, please don’t do this. Please do something about this. This is not sustainable. for our project. I actually want to do a quick survey of the audience. How many of you are from industry? Just a quick show of hands. Okay, like maybe 20 % or so. How many are students or just in academia? All right. And non -profits and government? Okay, so you have kind of like an even distribution. That’s very nice to see actually. It affects us all. And from what I’m hearing, I would like to actually sort of introduce a bit of how we could see these things as opportunities.

Because it just shows from the diversity of conversation that is going on here that you could think about a very specific piece of thing and think deeply about it and create a certain idea of how AI systems should perform in that little context. Like, you know, it could be simple as like, you know, in class five mathematics in CBSE in India, this is how the learning outcome is supposed to be and create something that, you know, could test the performance of models and evaluate models. And that could just be a big contribution in itself because it moves the field forward. And there are just all of these different opportunities that are being outlined here from very simplistic things like, you know, outputs of models to the cultural context of things, to the interpretation in multilinguality, to how agentic actions should be understood and evaluated, to red teaming and security.

Like, take your pick and the opportunity to be a contributor to progress of AI and to make it even more useful for all of us is out there. It’s just a very wide open field actually. Yeah.

So Ashwani just mentioned a really interesting point. Like, so So usually like the big open source products, they have like humongous code base. Like you are talking about like code of lines and like thousands and sometimes millions. So what I’ve been seeing like some of the, so what I’ve been seeing like, you know, some of these companies or maybe some of these like startups have been doing like very interesting thing about like mapping the entire architecture of the open source code base. So for a newcomer, it becomes like very daunting like where to start and what type of contribution should I make. But if you have like a clear picture of what does the functions look like, where does the data flows and which classes connect to which, you have like a clear image of the ecosystem of the, sorry, the entire code base of the open source project.

And this is also like very applicable if you’re working in industries like because if you have like a huge software stack and you want someone to onboard, what does the journey look like? Can you use AI and LLMs for like mapping out the entire architecture? And see like where you can, where’s the, what’s the. the best place to start contributing.

So actually, Ashwini, after your survey, I think one thing I also want to say is that since this group is not just software developers, we are saying open source software. I do want to open this to say that everyone, whether you are in the program staff, designing the application, whether you’re considering, right? Everyone has a space in actually the eval’s work. It’s not purely technical, and it shouldn’t be technical, right? We actually find that in use cases where there is a technical team, actually they’re the most cautious in terms of how much they want their services or what the scope of that service is. And we often find that program staff is actually quite ambitious about what the AI application that they’re building should do.

So while Sanket was talking about contributions in terms of start anywhere with software, I would also say this for anyone who’s on the program staff. who’s maybe on the design side, you can start anywhere in terms of the eval stack. And it could be just starting with, this is my list of questions that I want, and this is what my answers for this service should be. Or this is what the ideal should be. So I just want to say this is not just about technical contributions. It’s also about expertise. All of it is. Yeah, I think just agreeing with that last point, I think some of the most interesting conversations I’ve had about human rights, about food security, education, mental health and well -being, have all been in the last couple of years through AI evaluations, which is odd, honestly, to say.

But it’s because we have this generative being or this generative thing essentially giving us an output, and we have to sit there and think about critically what does that mean in any given context. And so that has just resulted in some really, really fascinating discussions around, again, the multicultural aspect, the legality, the cultural context, the geography, all of that. different dimensions of kind of these topic areas. Should we open it up to questions? Yeah, so are there questions in the audience? Yep, want to go?

Thanks to the panel. One of the more technically granular sessions that I’ve had to attend, and I’ve enjoyed it as a former engineer back in the day. Some context, I work on tech and geopolitics. The reason I say that is, given the bigger context of the summit, from long before to even, say, the president of Mozilla saying that open source is the answer to India, you know, really making it big in the AI space, or rather scaling it to where it has the kind of impact that we’re looking to make. Geopolitically, one of the things that strikes me, just from a democratic lens, or a principle -led lens, and I was talking about this to Sanket before the session, could the panel help me understand, and therefore the others, what could be some of the risks that come with the open source approach to scaling up?

versus a open weight, and please check me if my technicalities are off the mark here, or a closed system, for example, right? And whether you highlight a couple of risks or a framework of how to approach risks, like just bad code being added on is one conversation we have heard, right? But are there other loopholes in that process? I’d love to get a perspective on that. Thank you.

I have a lot of thoughts on that with the open weight conversation, but I won’t go into that. One thing I will say is I think open sourcing, like putting evaluations under an open source software license, I think is actually low stakes in the sense that it empowers more people to evaluate the systems that affect their lives. That’s part of our theory of change at Humane Intelligence. So for that, I actually think there’s very minimal downside and a lot of upside. I think one thing that’s going to be quite confusing for a lot of people, though, is the idea of open weight. Open source software. versus open data because when it comes to the actual LLMs, when it comes to the evaluation of the LLMs, the data is obviously a very critical piece.

And obviously just because you open source the software doesn’t mean that the data that’s produced with it is open data. And so that relationship is not one -to -one. So I think there will be a lot of kind of contention between what exactly is open with the software. And that’s something in our research at GitHub that happened quite a lot. Like a lot of organizations that were actually quite sophisticated in the tech didn’t necessarily realize that they could create closed data with open source software or they could use a proprietary software to create open data. Again, I don’t really see a ton of downsides with the AI evaluation. I think one thing that could go wrong is obviously if you take people who are not subject matter experts and then they start to adjudicate things that they…

know nothing about. So if you take somebody who knows nothing about human rights and then they create a policy around whether an output about human rights is good or bad, I would say that’s not a good thing for the world. But that’s probably going to happen regardless. So that’s my lazy answer.

I’d like to just say that in general, the idea of human in the loop has to be done very rigorously when you’re especially thinking about evaluations because you’re more or less putting a stamp of approval on behavior of models in a particular situation, context, safety, whatever. And we are not yet there where things should be automated and certainly caution is better and you would rather index on caution versus speed or volume. If you scale big with open source, you’re saying don’t discount on the human in the loop evaluation aspect. Certainly not right now.

So my question is related to that, right? So it’s broadly around how do you scale red teaming, right? So there’s a lot of, like, human -at -the -loop is great for, like, it’s important for red teaming, but that also means that there are, like, barriers involved in each step, right? Like, you need humans to identify gaps in the system. You need humans to create the prompts that are going, that could be tested, that could test the model. You need humans to, again, evaluate the prompt, the responses, right? Do you have, does the panel have, like, and this is for everybody, does the panel have tips on tools that could perhaps be used to, like, scale different parts of this pipeline so that, because red teaming is also a continuous process, right?

And it’s hard, and as models keep coming out and gaps keep, like, emerging, how do you see, what are ways that you see in which, like, this, these gaps in these, like, parts of the red teaming pipeline could be, like, sped up, perhaps to, like, scale it and evaluate multiple models in different areas, different applications?

One of the things that we’re looking at now is more of ontological -based approaches for, kind of, mapping out the problems based on so what often happens with especially like human in the loop ai red teaming is that you take essentially like a random checklist and just say like these are the prompts and this is what it covers but there’s not really good understanding of the relationship among like what the problem space means so if you’re looking at a human rights instruments for example you could take the different clauses you could take the different people the demographics you could take like the power structures that are inherent in a violent conflict for example put that into an ontology and then basically look at like the proximity of relationships and the strength of relationships and what are like the most egregious cases like what is the thing that’s going to blow up the entire system if like this is the output that comes out so by doing the ontological based approach we’re putting more thought into what the prompt construct should look like and that way when we sit down with ai red teamers we know that the scenarios are actually representative of the problem space and the areas that are most likely to be problematic so i think that’s one way that we’re trying to do it not necessarily for the speed but also for kind of mapping out the methodology and for the replication in the future.

So if somebody were to switch out a model or add a rack system or do anything to modify their system, we can more easily replicate the scenarios and get a temporal aspect as they build something out. But it is true that it does take a lot of time. I’ve seen a lot of examples obviously with synthetic data using LLMs. So you can do seed prompts or you can do narrative creation for your scenarios. But again, unless you have a clear sense of what the problem space is going in, oftentimes it’s just kind of cherry picking at random parts.

Similar in that last year when we were trying to figure out the safety frameworks and whether they do apply for India or not, we were working with this expert group, did focus group discussions, very labor intensive, a lot of thick evidence, ethnographic evidence. And what comes out of those conversations are maybe like themes. So we, for example, understand that there’s a difference in their sex determination. Right. And we understand that acid attacks. a concern. Where you could possibly try automation is in generating then prompts based on those themes, right? One of the challenges when you’re looking at Indian languages is that the current large language models aren’t very great at generating natural like spoken Hindi, spoken Tamil, right?

So even when you have those prompts, we actually found it easier to sometimes just like write it ourselves and like do variations of it ourselves but we did try the automated step which is like if this is the theme, this is like the sort of persona can you generate prompts based on that and that becomes part of like your emails. So I mean I think there is that mix of like automation and human combination that’s possible. It’s still like as the AI, like the LLMs advance the automation will get better but I also think that human sort of instinct like you will need that. I think that step will be needed and also like the way currently to some extent safety is working is that it is a little bit of a whack -a -mole band -aid, right?

So once you discover that there is this risk … that gets sort of patched, right? And then you discover something else, right? So, like, you discover, oh, like, punctuations in Indian languages can actually jailbreak models, right? And once you discover that, you can do all sorts of different combinations of saying, like, let’s try this symbol, let’s try this symbol, and then they’ll fix that issue. Then you discover something else. So, I mean, I don’t think that problem is going to, you know, we’re never going to get, like, a perfectly safe system, but we keep getting, like, you need that human insight to do that first -level testing, understand, oh, this is, like, an un, like, this is a new territory that has not yet been taken care of.

You can use automation, then, to generate more test cases or, like, build your data set.

I was just going to say my other thing, which was she was talking about automation. From someone else I heard, clustering turned out to be a very useful thing for them to find different classifications of behaviors, which was intuitively not obvious when they started off with evaluating models. of outputs and therefore identifying what are the places in which you could concentrate more effort on. And then human in the loop is a very generalized term, but where in the loop? And that would keep changing as we refine things, but I interrupted you.

So in terms of scalability, so first of all, please take this with a pinch of salt because I’m not an expert in this field. I was reading a blog post of Lilian Wang. She is from the OpenAI team and she introduced a concept of like model red teaming, how you use a model to red team a model. And based on, so just like I mentioned earlier, using the reinforcement learning, stochastic learning, how you adjust the model who is red teaming the model you want to correct. Yeah, exactly.

What about like evaluations? Like a lot of people are using judges, LLMs as judges, but do you think that’s a sustainable way of doing it?

Yeah, I think that’s a good question. I think that’s a good way to eliminate the human in the evaluation side. So our take, and we had presented this on the first day, is that you should always do a small, however small, right? It can be a 0 .5%, but always do a spot check with humans as well because ultimately, even when you do LLM as a judge, it struggles with the same language capability barrier that your original model, so that will always happen. And so we think that you should always do a spot check and you will always need a human to do some sample check.

Yeah, just quickly on that. When I was at ML Commons, we did something similar. So we tried to look at, there was research essentially done, like a benchmark of benchmarks. So if you were to use the same LLM that judges the other LLM, then if you have one aspect of bias, then the bias is essentially magnified. So that’s something to keep in mind. If you’re trying to mitigate against bias or hallucinations or whatever the vulnerability is, it will basically be exponentially there if you use the same LLM to judge the LLM.

Hi. Hi. Hi. Thank you guys for the lovely panel. My question was about how governments and kind of standard institutions can think about benchmarking. Specifically, I’d like to know what your thoughts are on standardization, benchmarking, like setting up the right standards for benchmarks, and finally, maintainability, given that the institutions may not have kind of their own in -house experts that stay on for a long time. How do you think about all of these questions, especially in the context of, for example, local language elements that are not really well understood or how we benchmark them?

I have a lot of thoughts on benchmarks. So, having built one, it was not easy. Yeah, one of the things that we think about a lot at Humane is the idea of benchmarking because we get asked so often. Like, again, it’s become the industry darling just because it’s so, I guess, rises to the moment of the hyper -adaptation and hyper -scale that we’re seeing with AI. But one thing that comes up pretty much in every conversation we have with organizations is what exactly are you trying to benchmark? So, we have this case, like, we’re working with an organization, potentially, that works in primary healthcare in Nigeria. what we’re doing in the primary healthcare in Nigeria.

And we’re trying to benchmark And so I asked them, like, are you trying to benchmark for hallucinations in the Yoruba language or bias in the Hausa language? And they didn’t know, literally. They didn’t know. All they knew is that somebody told them to build a benchmark for their AI system, so they should go and do that. So the problem is, like, what happens if you build a benchmark? And, like, if you don’t start with AI red teaming or another evaluation type, you may do a benchmark that looks at, like, hallucinations or, you know, factuality, however you judge that. But then it turns out what is really the problem with your LLM is bias. And so if you have the benchmark that’s measuring the wrong thing, then you built something that is computationally very expensive and takes a lot of time, honestly.

The math is kind of murky with benchmarks, I’ll be honest. And then you’re also not measuring the right thing. So we always recommend to start with red teaming and then identify the problem space. And once you get to that, like, hyper -focused problem space, then you can do a benchmark and say, comparatively speaking, like, this is the model performance against that specific metric. Thank you.

Just to add on that, you know, often bias or like any concern, like the sensitivity and the importance to address it is different in different domains, right? So like bias in the case of, say, a maternal health use case can be very problematic in a context where people are trying to use a bot to understand sex determination. And we’ve seen this in the real world. But say, like, if you are seeing gendered language, it’s always a problem, right? But like the, and if resources are limited, how you prioritize what concern you address depends absolutely on the context or like the specific application. So, yeah, I guess that is to say, like, just make that list.

Like, what are you trying to measure? And I think I heard someone say this, like, what is your headline? So, yeah, what is it that you’re trying to measure? And then. Figure out your, and you can’t measure everything. Like, you know, you can’t measure everything. and then build it around that. And that is the universal thing about benchmarking. It translates very much to anything global or a specific regionally contained language or context.

So just one tiny follow -up. Just in terms of maintainability, which I already asked, maybe Sanket, given that you worked on that, how do you think about maintainability for benchmarks, say, for example, with institution -led government that doesn’t have in -house experts, but would like to, for example, set standards and maintain these benchmarks over time?

Yeah, I don’t think I have bright thoughts on this. Sorry.

I think we have time for one more question, if it’s very quick. Otherwise, we can wrap. Any other final thoughts? No, I mean, I guess… just for everyone, everyone has a role in evaluations. Evals, evals, evals. That’s unfortunately what all of us have.

And you have a role in open source.

Yeah, and of course. Especially with cloud code because now you can make a lot of code cloud. Anyway, thank you all for coming. Appreciate it. Thank you. Thank you. Thank you. Thank you.

To share how these pieces come together and how Qualcomm is unlocking AI’s full economic potential, it’s my privilege to invite on stage Durga Malladi, Executive Vice President and General Manager, Technology Planning, Edge Solutions and Data Center at Qualcomm Technologies. Please join me in welcoming Durga.

Okay, so we’re reaching towards the later half of the afternoon and hopefully everyone had their lunch and their coffee. So I hope to talk over the next 25 minutes. I won’t take that much time, but about 25 minutes talking about what is going on in the AI landscape from Edge all the way into the cloud. Starting from yesterday, there was a lot of discussions on the relevance of Edge AI, what exactly is happening in that space, what should be the opportunities at the Edge and where we should be going in the cloud as well. So I’ll try to distill that. in a few slides, and I’ll probably go through a little faster so that we have enough time later on for the team to actually go through the panel discussion.

All right, I’m just going to click through this. This is good. This is probably a good indication of why the edge matters. If you go back in time three years, when GPT was originally announced back in November of 2022, that was a very large 175 billion parameter model. And if you take a look at what the model sizes today look like, they’re more like 7 to 8 billion parameters, but they actually outperformed that original model by quite a bit. Model sizes are coming down quite dramatically, while the model quality continues to increase. This is the equivalent of an AI law that seems to be emerging as far as models themselves are concerned. It’s an important trend line because this actually is the foundation for why edge AI is actually a big part of the model.

And if you take a look at the actual model size, you’ll see that the model size is actually relevant. In other words, you don’t have to necessarily use the trillion parameter models to be able to get through a large number of use cases that average consumers actually care about. and when you think about it that way this is a depiction of just in the last one year alone how much of a progress has been made just in terms of the model quality index itself there’s several parameters over here but the punch line is model quality is getting extremely powerful and now the question is what should we do about it what can we actually build on top of it so we’ve already established the fact that the model sizes are coming down while and these are sometimes known as slms though i would argue that it’s not just small language models but these are small multimodal models that are coming in but there are increased capabilities coming with it much larger context length a lot of on -device learning and personalization that can be done built upon that and reasoning models which actually mimic what we typically expect to see from some of the larger models when you put both of these together and build the right kind of an innovative architecture that’s what actually leads to edge ai in devices that you and i care about so is it here is this just a powerpoint presentation or are there actual consumer devices where you can do edge ai the answer is absolutely yes In fact, today, if you can get any of the premium smartphones where you can easily run a 10 billion parameter model without breaking a sweat, or glasses which have up to a billion to 2 billion parameter models which you can easily run, PCs with up to 30 billion parameter models and so on.

These are devices that you and I use very frequently, at least the PCs and the smartphones with more people adopting AR glasses as well. But one thing that’s nice about running on -device AI or AI inference that’s running directly on devices is the quality of the AI experience is invariant to the quality of connectivity that those devices had to have to the back end of the network. That is a key attribute. I don’t want to keep going back and forth between a regular experience and an AI experience just because I don’t have internet connectivity. That would not be very compelling for any of the consumer or enterprise use cases, and that’s key. The second part is there’s a large amount of data that happens to be very personal.

It can be consumer -centric or it can be enterprise -centric, but either way, I might or not be interested in storing the data in the cloud. And if you kind of think about it that way, then that’s another vector that takes us towards what you can do at the end. and as you put it all together, what exactly are we trying to do with the AI to begin with? Now, I was not there around in the 60s or the 70s, well, I was there in the 70s but I was not involved in, you know, what people used to do with very large mainframe computers where there was just a command line interface, there is a gigantic machine in front of you and you just keep typing something onto it.

That was the user interface between a human and a machine. The 80s changed that with the advent of you use a mouse, you use a PC, there is a graphical interface, you actually get to see something, not just see a command line interface, that changes things. Fast forward to where we are today, about 20 years back, we started using touch as the main UI. We all have our smartphones which happen to be touch -based and increasingly laptops and tablets and these are places where the UI shifted from just using typing and using a keyboard to touch interface as well. Well, we are now at a different era now. It’s at a place where we now are increasingly using voice as an interface towards devices.

And if you put it together, you have a combination of different modalities, whether it is text or voice or video, any other camera interaction, some sensors which tell you exactly where you are located, provide some context to what you’re doing. All of that gets ingested by a single interface, an AI agent. Imagine the following. Let’s take a smartphone because one can easily relate to it. You have your smartphone. Right now, people are either looking at it or scrolling through their apps. We all have a clutter of apps on our phone today. If I wanted to use one app, I’ll have to click that one. If I wanted to then correlate that information with another app, I have to go back, then open up a new app and go in again.

Instead, all you have, and this is a future where all you have is a voice UI where the device is sitting somewhere. It’s in your pocket. You talk to it. Your voice gets authenticated and then it says, OK, I’m ready for you. How can I help you? That’s your agent right there. I would always love to say talk to my agent, but this is the beginnings of that. that agent distills all the information that you’re saying encapsulates it maps it to apps that are running somewhere in the behind the models are actually they only provide a means towards an end goal they perform a job but that’s not the end job by itself so the agent actually picks one or two from a bouquet of models and then also accesses some of the personal attributes that could be sitting right there we call it the personal knowledge graph together when you put it all together you end up seeing a glimpse into how ai can then become the new ui to all the devices around us and this is a very powerful concept is this also just on powerpoint till about last year that was the case not anymore byte has introduced a new phone in china very recently and it’s not available everywhere in the world some of us do have the luxury of actually visiting china quite frequently this phone is like fundamentally different you can’t just buy a new phone you can’t just buy a new phone you can’t just buy a new phone it’s designed from ai first all you have is an agent by the way and all the other apps are actually missing.

They’re somewhere in the back, but you don’t get to see them. And if you think about it, it’s a very disruptive mechanism. It’s still early. Of course, it’s going to be a little clumsy and it doesn’t work all the time in a picture -perfect manner, but it’s something that is beginning to change the conversation of how you take AI agents from something that happens to be in presentations to something that is far more practical in devices. So I’m going to just skip through this part of it. A lot of it is in Mandarin, so it’s kind of hard to see, but at the same time, you get the picture of how it can do things for you when you give it a very generic, nebulous task and it figures out exactly what is it that you need and then does things for you.

It’s like shop something for me, check my bank balance. If I have enough over there, I want to buy that thing and then when it is done, do let me know. It does it. You actually don’t know it’s happening, but it actually does it. All right. So far, we talked about the edge. What about the cloud? Well, a lot of the data actually comes in from the edge. it’s the consumers who are actually generating the data. That’s where the AI action really is. But the cloud has an important role to play as well, as the data actually gets used both between the edge and in the cloud. And so our philosophy over here is to make sure that we have AI processing that is distributed across the network depending upon what the use case is.

For instance, the cloud is extremely powerful for training foundational models, creating new kinds of models. That’s very helpful. At the same time, there’s a large number of enterprises where you have on -prem servers where with using air -cooled cards, it’s very easy to run 100 to 200, 300 billion parameter models. Very useful for small and medium enterprises which don’t necessarily have to rely on the data center. Just buy a card server, plug in an AI accelerator card, maybe a handful of them, and you end up with extremely sophisticated processing. And keep in mind, in the beginning, we talked about the fact that the model sizes continues to actually come down while the quality continues to improve. So whatever you have, if tomorrow there’s a new model that comes in or you just want to replace your existing AI accelerator card, take out one, plug in another one, as opposed to rolling in a new rack, fundamentally different in terms of the network economics.

And finally, we just talked about devices as well. So bottom line is, when you think of AI processing as a hybrid AI, it’s a mix and match of processing between devices, the edge cloud, and in the data center. And speaking of what is it that you can do with it, imagine the following. This is one of the PCs that was launched in Saudi Arabia. It’s called the Humane PC. We had a lot to do with it. It’s a place where, in fact, the only interface is what you see in front of you. This is not a standard PC which you open up and you have the regular kind of a screensaver and you have all the other apps that are there and you open up your, you know, your mail client, your calendar, and so on.

you ask a question and in real time and it doesn’t matter what it is in real time it decides should i run it on the device or should i run it on the cloud maybe some questions that you ask are so complicated i want to run it on the cloud and the other questions are yeah without breaking a sweat i can just run it on the device and this is a place where you actually switch back and forth between what’s running on the device and what runs on the cloud it’s the beginnings of where we can go with it another step when we actually talk about devices we all have a universe of devices around us glasses which could be connected directly to the network tomorrow and today they are tethered through a phone your earbuds your wearables it could be a watch that you’re wearing and increasingly on our ring as well i think they’re running out of places where you put devices but every time i think that there is a new device that comes up already we’ve reached four this is like a universe of devices around you and perhaps the hub happens to be a phone how do you actually go back and forth between these two and how exactly do you make sure i wouldn’t even probably want my smartphone with me I want to keep it somewhere, just have my earbuds and constantly talk to my phone and do some of the processing perhaps in my earbuds itself, the rest of it on the phone and some of it on the edge server and the rest of it on the data center.

That is the vision of how the evolution of AI ought to be. Speaking of the number of things that we just discussed, it’s important. This is now more from a Qualcomm perspective. We have made sure that we have a good, easy way for developers to onboard our platforms, bring in their applications, their platforms and actually run from there. And in the subsequent session, as we go through that, there might be a little bit more talk about it. But suffice it to say, if you go to the Qualcomm AI Hub, it’s a place where any developer can pick a model, bring a model. Or if you don’t have a model, we’ll create one for you if you bring your data.

Once you do that, we’ll give you free cloud native access to device farm, which exists somewhere. But you don’t. You just have an IP address that you log into and you take it from there. And the rest of it is you write your application. You have the ability to test it without once having the device actually in your hand. If you’re comfortable with that, you get to deploy that app out there in any kind of an app store. Very powerful concept that we’ve actually worked on for a large number of time. And this is a place where, you know, we are not a model creator. We ingest models, which means we work locally with every single model provider out there on the planet and happy to actually discuss a lot more offline as it comes to it.

All right. How am I doing on time? Maybe I have 10 minutes. So let me talk a little bit on data center. I don’t see the timer here. That’s why I was asking. So what happens in data centers over here? Well, one thing that’s clear is that the data center capabilities are becoming more and more sophisticated. And as we learned a lot of lessons from the edge, one thing that became very clear for us is that it’s important to pay attention to energy efficiency in addition to performance. So we call it as energy efficient, high performance computing. And we kind of start bringing that sort of a paradigm into data center. A few other. Observations came in.

One is that. the processors that are designed for training are not necessarily the best processors that are intended for inference. They’re actually different kind of problem statements. It’s a little more subtle, but once you understand that, once we get past the whole notion of let’s just buy the biggest GPU that’s out there, and then you realize it’s a little bit of an overkill when it comes to the inference task that you might have. It’s a different architecture that’s needed. The second part is that we want to make sure that in addition to the rollouts that are currently occurring, we bring in solutions which would lower the total cost of ownership. So when we put it together, we introduced our family of solutions in the data center as well, learning from what we learned in devices, and then bringing those lessons into the data center.

A smartphone today operates at four watts at best. The battery inside a smartphone is 4 ,500 milliamp hours at best. In a data center, if you buy a state -of -the -art rack, it’s about 150 kilowatts. Fundamentally different. It’s directly liquid -cooled. You need water. There’s no water or liquid -cooled kind of a smartphone over there. two different universes but there is a way to learn lessons from one universe and actually apply it on the other side i would argue that in ai terminology that’s transfer learning that you seriously apply going from devices all the way into data centers itself so we entered that space and we have two different categories of solutions the second one ai 250 is a place where we focused on an innovative memory architecture as it turns out and it’s a little more of a subtle argument here but as it turns out that when we talk of inference the pre -fill stage is extremely compute bound the more computation horsepower you throw at it the better it is tokens per second is higher however the decode stage is fully memory bandwidth bound you can throw as much compute as you want it makes zero difference whatsoever so the memory architecture is equally to it’s actually equally important and so we innovated on that putting it together for our ai 250 solution this is the one that’s actually rolling out in the middle east and this was part of that earlier demo that we just talked about with a pc and something else that’s running in the cloud you We have an annual cadence that’s coming up.

This is stable stakes at this point in time with the innovative memory architecture continuing into the second generation by the time we get into AI 300, which is not yet announced, but something that is in planning. Now, finally, and I want to actually move a little faster here. There is a buzz in the industry about the next generation of cellular platforms, and usually one would scratch their head and say, wait a minute, we just launched 5G. I don’t know exactly why we’re talking of 6G over here. And besides, isn’t this all about AI? What does AI have to do with 6G? Are we just throwing AI pixie dust on top of every technology right now and simply saying there’s a hype cycle associated with it?

That’s not the case. It is true that cellular communications and AI have evolved as two parallel sets of innovation. But the time has come to actually put both of those together because cellular technology at the end of the day does involve the very same devices that we just talked about. It involves a network through which all the devices are connected. The data goes through and eventually goes into a data center as well. So we have a view in terms of how 6G can unlock a full potential of AI. And, you know, if you think exactly how the GE transitions occur, it’s only 10 years or so. So the earliest 5G launches were in 2019. So we are in year seven of the journey.

It’s not that far off. And it turns out we have a convenient Summer Olympics that’s coming up right next door. We’re based in, I mean, our headquarters is in San Diego. That’s where I live. And there’s the 2028 Summer Olympics. So there’s going to be a lot of show and tell in terms of what 6G capabilities can be. And there’ll be technology trials at that point in time culminating into the first set of deployments that we are driving towards in 2029. And we have another two minutes. I’m just about done. I want to actually stop with one final thing, and that is this part over here. What you heard is just a glimpse into the kind of world that we as Qualcomm live in.

We are probably the only ones in the industry that work on everything from doorbells to data centers. We are probably the only ones in the industry that work on everything from doorbells to data centers. There’s a lot of others who focus on data centers, maybe on servers, but they don’t exist below phones. We actually work ground up from everywhere over there. So happy to talk with

Thank you, Durga, for this insightful presentation. As we talk about inclusive AI at scale, enabling developers is critical. Innovation only moves as fast as the tools behind it. Through the Qualcomm AI Hub, we are simplifying how developers access optimized models, test and deploy high -performance on -device AI from edge to cloud. To share how we are accelerating this developer ecosystem, please join me in welcoming Siddhika Nevrekar, Senior Director and Head of Qualcomm AI Hub, to moderate our panel discussion. We’re leading startup founders, exploring the evolution, evolving AI ecosystem and what excites them about building with on -device AI. Please join me in welcoming Siddhika.

I would like to welcome the panel over you guys know who you are so I don’t need to I know can we just take a moment for a quick picture if that’s possible thank you Thank you.

So I’m Srinivas Shetlapalli. I lead the innovation track for TechMindra for AI and emerging technologies, which includes blockchain metaverse. And I’m also responsible for creating an innovation ecosystem across a network of labs that we’ve created globally. Thanks.

Hi, I’m Madhav. I’m the co -founder and CTO at SpotDraft. We do AI for legal. We’ve created a bunch of agents that help lawyers not just review contracts, but also draft them and negotiate them faster and better.

Hi, everyone. I’m Praveer Kochhar . I’m one of the co -founders of Kogo AI. We run a full stack private agentic operating system from the edge to the cloud. So we are bringing agents closer to enterprise data rather than taking data to agents. So we are 100 % sovereign, built from scratch platform. And we do some very… exciting work with Qualcomm. I hope I get to share that with you today.

All right. Let’s start with some questions. None of you know these, so these are fun because they’ll be a surprise to you. They’re not hard. They’re very easy. We’ll start with you, Praveer. We’ll go in the reverse order because that kind of throws a curveball. What’s the most underrated pain point for enterprise users that AI will solve? You can perhaps talk specific to your product.

Did you say underrated?

Yes.

So there’s a concept called shadow AI. I don’t know how many of you know about shadow AI. Shadow AI is a lot of people who work in companies and sharing critical enterprise data on the cloud while using unauthorized AI tools like OpenAI or Cloud. So 78 % of enterprise users use shadow AI. And that’s a big concern. It’s underrated, but it’s still driving efficiency. So not a lot of eyeballs are going there. But I think that’s going to become one of the critical issues as we move forward. Things get more complex. Agentic systems get more complex. More data is shared on the cloud. So I think, yeah, for me, I think it would be shadow AI that people are using.

That’s a good answer. It was a curveball, but you caught it. Okay. Let’s go to Madhav. You work in a very niche field, you know, legal, which is very, very niche. You’re biggest and you also still dabble with technology, right? Yes, you like it. So your biggest and favorite AI failure, building spot draft that set you up for success. Can you remember any of that?

That’s a great question. So it sort of goes back to our founding years where we were a little bit early to the game. This is around six to eight years back when. And transformers were what people were talking about and not LLMs. And back then, we came in with the idea that, you know, cars are driving themselves. So why can’t AI actually review contracts for you? So we spent a bunch of time with enterprise customers trying to deploy AI and realize that we would have to train a model for each customer. And we built out our entire data labeling annotation pipeline as well as team at that point. So while that was in a way a failure because we then decided not to do that because we didn’t want to do services.

Otherwise, we would be building models one per customer. And the genesis of SpotDraft as it exists today came from there because we wanted to capture the data as lawyers were using the technology that they anyways use, which is where our word plugin comes in. So we can actually capture what they’re doing. And then our annotation team was also set up back then. And that’s sort of how today we are able to give. Grounded answers using data that is the customer’s data because of all the things we built back then.

That’s a good one. So now you’re on a path of just never regretting making single models for each customer.

I mean, I hope we don’t have to go back there, and I think a lot of the models that have come out are enabling that. But that part, yes, not regretting it.

All right. Srini, last -minute addition, so thank you. I know that it’s difficult to get here. This is probably something that you’ll be able to share with us. Yes. What’s the special ingredient for successful AI adoption in India specifically?

Okay, that’s a tough question to ask. I think the most important thing is understanding the limitations of AI. So typically it’s very easy to understand what are the advantages of doing AI. But if we can set the expectations right, that AI will augment their work to a certain extent, that will be one. Second, the complete misnomer that it is here to take away jobs. has to be remote. I think these are two things.

How do you feel about AI being trusted in India? Is it trusted enough? Is it adopted?

So if you look at the adaptability of AI, we are almost at the global level in terms of the enterprises that we are talking to. But the best part that I have seen is that a large number of public sector banks have taken to AI in a big way. Some of the banks have been our customers for both AI and emerging technologies. And we’ve also seen PSU units talking about AI. And I’ve also seen a lot of state governments, I had a chance to meet a lot of ministers today, ministerial delegations today, have set up AI centers. So we are in the game.

Yeah, good. Ritukar, this is an easy one. You think about this probably a lot. Cloud or on -device AI, which is the most important? Which, where and when?

so I think in continuation to the previous question so you know just through a bunch of compute and a problem statement is not how AI is adopted in enterprise settings because it’s very important to break down the big problem into smaller chunks and for what you want to use AI and for what you don’t want to use AI and that’s exactly what we do in robotics so we break down what is happening on edge and what is happening on the cloud so right now at this point in time for us it’s like you know we do orchestration on the cloud which is for the fleets of robots but you know we were doing all you know autonomous navigation on the edge part of it and for us it’s very important that you know we wanted to understand more intelligent navigation so at this point it’s been almost one and a half years since we started running VLMs on the edge to understand the context So I think that’s how you break down the overall problem, not just running everything on the edge or running everything on the cloud, because that won’t solve the problem.

Yeah, that’s pretty much how we break it down into small chunks.

So you guys are very thoughtful and very quickly giving these answers for longer questions. So we’ll go to rapid fire, which is just picking one word. There’s no judgment here. You can pick A or B. You pick A or B. Maybe a couple of words about B. Not too long. So we’ll start with you, Ritika. 6G or AI?

Sorry?

6G. Or AI.

So, okay, this is a long one. I can just share a good anecdote there. So we were running robots in Rio Tinto in Australia, mining areas, right? There is no internet. Still, we want to use AI on the edge. And so what we did was we put some installing satellites each on the robot. Right? so connectivity is very important if it is 6G it’s better I’ll go for 6G because that opens up a lot much possibilities there

that’s a good one I thought you would pick AI because that’s the buzz word that’s anyways happening good answer Shreemi data center or local

local is the first option but for India data center makes business local because one of the key products that we have built called Orion which is an AI platform has been built for on prem and we also see that a large number of requirements that have come to us is how do I process things in my own premises rather than doing an API call or taking it to the cloud and we have seen I know you asked for India but I have seen this happening in the Middle East also where a large one of the large the largest world largest companies said that can my exact solve their things on their own desktops or locally.

So local.

Local for you, okay. For you, I’m looking through because I want to ask a specific one. Madhav, artificial or human?

I mean, when you deal with lawyers at the end of the day, I have to go with human because… I know you easily pick artificial. So you can’t hold AI models neck, but you will go hold a lawyer’s neck. So for us, it’s important to give the lawyer the capability to do their job better, faster with a more thorough research. But at the end of the day, it has to be them taking that decision because a lot of times it’s not the black and white. Those are the easy scenarios. It’s the gray area where the lawyers are able to come in and really guide their customers, clients as to what to do and what not to do.

That’s a great question. I think we still want AI to be human, right? So I think it’s… That’s a good one. answer but there is no judgment you could have said otherwise to provide regulate or innovate

okay in regard to AI 100 % innovate I don’t I don’t see any reason anyways regulation in the in the age of AI is always going to play catch up because technology the speed at which it’s growing it’s very difficult to regulate it before it goes because we don’t even know the social implications of what we are building and as we build them and as it goes into public and people start using it these tools are very intelligent they’re getting intelligent by the week so I think it will always be innovation at the side of caution but I don’t think this is an industry that you can regulate first and then expect it to grow.

Having your first answer very first answer about I wouldn’t say illegal but unauthorized usage was pretty much in line to this. and it still was saving time, and it still is so, yeah, I think that’s a good answer. For the next ones, you don’t have to say why. So you can pick an answer. Nobody is, again, no judgment. Go to AI?

No, 100 % innovate. I don’t see any reason. Anyways, regulation in the age of AI is always going to play catch up because technology, the speed at which it’s growing, it’s very difficult to regulate it before it goes because we don’t even know the social implications of what we are building. And as we build them and as it goes into public and people start using it, these tools are very intelligent. They’re getting intelligent by the week. So I think it will always be innovation at the side of caution, but I don’t think this is an industry that you can regulate first and then expect it to grow.

Your first answer, very first answer about, I wouldn’t say illegal, but unauthorized usage was pretty much in line to this and it still was saving time and it’s still so, I think that’s a good answer. For the next ones, you don’t have to say why. So you can pick an answer. Nobody’s, again, no judgment. You can pick whichever one you want. Agent tech or robotics?

Robots are the agents.

You have to pick one.

So agents, yeah.

Okay. LLM or SLM?

SLM, all the time.

Intellectuals or automation?

You can’t do automation without integrations, so I would have to go with integrations.

Build a chip or buy a chip? This is just a selfish question, but, you know.

I would sell a chip, but then build a chip always.

Wow, it’s an interesting answer. I don’t know how much time is left. Okay. All right, we’ll do some few extra questions. You guys can take longer now to answer the questions, I guess. Just moderate the time accordingly. So what’s the one hardware constraint that keeps you up at night?

So one of the biggest hardware constraints is if your entirety of the system is without any connectivity and you are restricted that you cannot access remotely. If you cannot access robots remotely in any which way, be it for scheduled maintenances or predictive maintenance or anything of that sort, and even emergency situations, like what we see, you know, the Waymos which are running in San Francisco SF right now, they are monitored from Philippines, right? So I think that part is something which is very important, that everything should be connected at all times. So I think that keeps us awake that, you know, the robots should not go in silos or isolated where we cannot reach them.

And only then we have to. We have to physically, you know, make sure that somebody is around. to manage a fleet or whatever.

You talked about local. So I’m going to ask this question which seems apt for you. What’s more dangerous? Too much data leaving the device or too little?

Too much data leaving the device. I think too much data leaving the device.

How do you train? I was saying how do you train if it doesn’t?

See, I think the focus for us also has been how do we train with lesser data and make it much better. The moment we’re talking about more data and more data leaving, we’re actually talking about more issues happening, more breaches happening. So with lesser amount of data, if we can train or if you can create synthetic data sets and work it, that’s the best way for LLM to be trained rather than waiting for large data set to come. And then then like you said, then wait for it to leave.

If I may, it depends. If it is enterprise, then less data going up is always better. If it is B2C, then everybody wants to learn from that data. Because that is free data. So in a way, that’s something which is very important. Situation.

Yeah. Okay. This is probably going to be interesting. You get to tell another story. What was the last thing that made you go, wow, about AI? And this doesn’t need to do, don’t pitch your company. It’s fantastic.

I’ll try not to. I think we’ve seen the kind of, and this sort of goes back to the last question in a way, where a lot of companies have so much data sitting in people’s heads, in people’s inboxes, random share points, drives. And historically what we’ve seen as we onboard customers, they’re like, oh, I have a playbook. You know, which is a policy of what contracts we will sign, won’t sign. but we also know that it is out of date and we’ve been working on techniques to be able to really infer that from older data and one of the things that we’ve seen which really blew my mind was we actually ran one of the early prototypes of that on our internal data we run SpotDraft on SpotDraft and some of the things it threw up when I was talking to our internal legal team and I expected them to say no this is absolutely wrong and that guy is like actually I want to know where this came from because I have been trying to track this down that why are certain contracts having certain clauses and not so it’s that ability to do knowledge work which otherwise would not be done at all and to have this always up to date always learning sort of knowledge base that truly captures what your company and organization policies are that’s something that no one wants to spend you know 100k to get lawyers to create that but if you have a agentic way of doing it, then suddenly that becomes the one thing that everyone cares about, because that is now your onboarding, that is now what you, you know, compare your new contracts with.

And I think in the coding side, we’ve already started seeing a lot of this where things like, you know, Cloud Code and Codex are able to go in and learn from your code base and give you these insights, which earlier would take a new engineer, like maybe a month or a quarter to get onboarded. Now they’ve started shipping code within days, because of this, and that is going to start happening across all kinds of knowledge work. And for us, the, you know, the, the wow moment was when the lawyer who doesn’t trust AI suddenly said, No, I need to see this, this is

So that’s, I’m going to spin to Madhav, the not CTO, maybe a consumer AI feature that just wowed you in recent time, any you can think of?

I think I’m sure everyone has been talking about OpenCloud, the ability for me to have a personal assistant almost when I, of course, can’t afford one. But for that to really sit and start doing a lot of these things for me, and I’m sure it’s going to come to everyone’s devices very soon, hopefully with Qualcomm chips. And that, I think, is where I was really wowed by it because I deployed it on my WhatsApp and it started sending messages to people. It was a little bit scary, but also saved me a bunch of time. So that was where I was like, OK, this is something that was not at all possible before.

All great responses on WhatsApp. I had to switch it off very quickly because there’s just too much data in there. But that is the next challenge, right? How do we control these autonomous agents, especially when they’re sitting on your personal data? Given you’re a rebel, we’re going to ask you, what are you most scared? The deal.

No there’s a lot of fear there’s a lot of fear because I think we don’t know the societal impact of this technology yet and I think that’s probably the largest fear because up till now we were engaging with algorithms that were trained to derive attention from us now we are dealing with intelligent algorithms that can self adapt and become far more personalized now with the ability to generate content at will I think it will be very difficult to keep attention away from a device when you have a hyper intelligent system on the other side that’s changing itself based on you it will become extremely addictive so I think that that’s the biggest fear

Yes, but but then but then we are pleasure seeking beings, we will we will go after that until it it gives us some guardrails and then we’ll have apps that will lock themselves up for two days and we won’t use them. It’s possible that we’ll be all on vacation and the robots will be interacting with

Yeah, and then imagine what we’ll be doing we’ll be interacting with these attention seeking agents, right? I just want to take the last question also, because I just I just saw a real recently and they got a unitary robot in Bangalore and they sent it out to beg. So it was the first robotic beggars that somebody started out and was there more empathy, probably there was more empathy than I don’t know, but but I still think that there’s a lot of tangential use cases of AI that can come out of come out of all this. And yeah, I mean, I mean, that’s something that got me and also kind of told me that you can think very, very differently about this technology and not just think what we do and replicate what we do.

There are a lot of tangential things that might come out of this.

I asked why, if there was more empathy because I was recently driving and there was a two lane road. One lane was completely blocked. Everybody was trying to squeeze into the other lane. And then when you pass by, you saw way more that was not operational. And everybody would go, oh, you know, nobody was upset. Nobody was screaming. I’m like, just because it’s a robot, you’re more empathetic. But they were. So it changes your psychology somehow.

Yes, yes. And we are still not interacting with robots on a day -to -day basis. And I think that that will be another kind of mystery thing added to our societal weave.

True. Thanks for taking the second question, too, which was interesting. All right. We’ll get into closing so we can wrap up. You all will get to pitch to companies, so that’s very exciting. We’ll start with, you know, complete the sentence in one word. So you have to just say one word. Edge AI in 2030 will be blank. You can repeat the sentence.

Edge AI in 2030, it will be, it will be, I mean, it will be very not so sophisticated. I mean, it will be taken for granted. So just like you use connectivity for granted. That’s how the Edge AI will be. It will be everywhere almost. My default, like the pins and, you know, the human pins and everything. So what we talked about in the keynote as well. So I think it will be like that. So taken for granted.

Will you still complete itwith one word? Sorry. Taken for granted is one word. Okay. Granted. So it will be business as usual or taken for granted. That’s it. I mean, nobody will mind that. Edge AI in 2030 will be as a default.

be ubiquitous I think there will not be anything that does not have AI and I think there is a lot of Hollywood sci -fi that has demonstrated this but we will probably be trying to talk to tables or screens or walls to that degree where anything that can have a chip inside it the chip will also have AGI inside it AGI in 2030 will be I think AGI

in 2030 will be emergent we will start seeing signs of of what OpenClaw just did was a very small trick in the play but it added a little bit of emergent behavior into LLM giving it autonomy to be able to create its own files. That’s all that OpenClaw did and that’s the magic behind it. And I think that’s going to come to the edge and with that emergent behavior you’re actually giving a model the ability to create its own learning. That’s why I say emergent. That’s a good answer. One last thing you

want the audience to remember. This is also the cue for pitch if you like. As I said earlier robots are agents and

I think I kind of agree with that so we are going to be, part of us will be agentic as well because we’ll have something some AI in us as well whether it is, so there’s a lot of work which is going on with Neuralink so the airports are tracking the brain waves of how you react to a particular situation so agentic you know both robots and people will be agentic in some fashion and I think that’s how things will be and you need some orchestration where everything can talk to each other that’s what we are looking forward to do So I think one thing that we should

all remember is there is a lot of work that TechMindray and Qualcomm is doing together in detecting fraud calls and this is using Agile LM so I think that will grow as we go ahead that research will see a lot of action because the number of fraud calls that we are getting are increasing every day so I think that’s an area we will see a lot of action happening and I think both our companies are geared for it I think and I think it was mentioned

in the keynote I think one of the takeaways for me would be how we think about interacting with technology today is going to change entirely like uh UIs, phones, you know, screens, all of these going away and everything becoming very, very generative, whether it is, you know, slides being generated for you on the fly based on the conversation you’re having, or even entire apps, UIs being generated for each specific scenario and use case. I think everything is going to move away from just being SaaS that people learn, and it as an individual persona is actually caring about. And that actually opens a lot more, specifically in the Indian context where you might not, like people might not have to go through so much training and learning, and they can just go and start using it because the platform can actually understand your needs, as opposed to you having to understand the platform.

Can you just repeat the question once for me, please? One thing you want audience to remember. Whatever you want. To remember.

So, so, so, so remember how we used to work. work and plan for how we are going to work because very soon we’ll have a lot of time that will be available to us because a lot of systems that we are going to manage will be intelligence and autonomous and we’ll have to only take decisions. So what we do with that time is going to be a critical question everyone’s going to ask themselves and I think all of us are also going to be builders because we’ll have very intelligent tools to build things, run them and manage multiple systems at the same time. So I see that future and I think we should all look around and see how we manage things today and how we are going to do that in the future.

Great. This is a chance to actually pitchyour company but it’s okay. It’s pitched. I will give a more specific one to pitch which is there are a lot of people in the audience, maybe some customers, if they were to find you where should they find you or a spot where they can talk and what should they come and talk to you about? Okay. What specifically and what industry?

So, I mean, so we are autonomy, so you can always find us at autonomy. So that’s where you can find us. Always where. Yeah, we are brand, I think. We are proud of it. And the most important thing is, like, you know, robots and, you know, just like AI, there’s a lot of emphasis on physical AI. And it’s not something which is going to come. It’s there. It’s just the adoption curve which is happening now. So think more ways of adopting technology. And if you want, if the enterprise customers are looking forward to adopt more and more robots, not only just dull and dirty scenarios, but also in, you know, different walks of life, I think that is where, you know, talk to us and we can help.

Even if they are not our robots, we can help them to have a set of orchestration with, you know, variety of things. But still, they have some level of control. Yeah. Thank you.

Thank you.

in 19 -ish countries, and we’re all focused on what does it mean to unlock innovation through trustworthy, responsible, beneficial AI. And so, of course, no surprise, gatherings like the one that we’ve had this week are really crucial for the work we do, and with the Delhi Declaration adopted yesterday, this is an even more important moment to build on where we have come from, to lean in, and to really get to work around some of the questions of the accountability work that needs to be done, the scientific evidence that we need to build around frameworks and good policy moving forward. And, of course, it’s extraordinarily important that this is happening in India, that it’s bringing a whole set of voices and perspectives and leadership that is not optional.

At PAI, we believe… We believe that that is fundamental to building a global community committed to this work, and it’s great… to see it in action this week. So thank you all for being here with us. So today we’re going to give you an opportunity to see two of our latest papers. These are papers that were begun out of the Paris Action Summit. And at that time, as we were thinking about moving into action and invasion, we felt that work needed to happen with a good sense of what the assurance ecosystem looked like. So we’ve had working groups underway developing these two new resources. They’ll be up on the screen at some point. You’ll be able to get a QR code and download them.

Feel free to talk to any of us. The first one is Strengthening the AI Assurance Ecosystem. It really looks at telling and helping national policymakers, if you’re building a robust industrial AI strategy, you better have a comprehensive AI assurance strategy as well. And you need to be able to do that. And so we’re going to be talking about that. We need to think about all those actors and what they look like. We’re going to hear about one of the experts, of course, in this as soon as the minister comes to join us. The second piece, which is really important, we think, for this conversation is what does it mean to do AI assurance? globally around the world?

How do we close the divide that exists? What is different about the challenges faced by countries in the Global South versus others? So we’re really hoping that these resources not only are good, substantive contributions to the work that needs to be done, but the idea is to just catalyze, you know, sort of plant a number of seeds across a number of ways in which assurance works so that those can grow and really come to life out of this. And just two quick comments on that. Now that we have half the declaration, and so now we can, as opposed to earlier in the week, start to articulate it, really leaning in with regard to the commitments around, in commitment one, around usage, clarity around usage data, really trying to give some empirical grounding to this work.

In 2025, in our progress report around foundation model, impact. We made exactly this recommendation. We directly called for Frontier AI companies to share usage data. We’ve been tracking progress, and there has been some progress in that regard. So we are delighted to see this particular commitment to come about and to start to see some standards about how that usage data is going to be shared. So we’re very pleased to see that work. We’re also very pleased to see the second commitment around strengthening multilingual and use case evaluations. And you’ll see, if you do download the report on the global assurance divide, that that is clearly a key piece of work that needs to happen. So this afternoon, we are going to give you an extraordinarily expert panel that brings a real diversity of perspectives to this work.

And so we want to take the assurance question and apply it to agents. Because that’s where the world is going. We’re all seeing them in the news every day. We’re seeing them integrated into foundation model systems. So what does it mean? to take what we know about assurance and think about the applications that agents will add to the complexity of that work. So let me begin by introducing our first speaker. She’s probably been one of the most visible ministers this week because of the extraordinary leadership that Singapore has taken when we think about AI assurance. I know you’re going to talk a little bit about that. Such a pleasure to welcome you, Minister Josephine Teo.

She’s going to come and say some words for us before the panel begins. Thank you.

Thank you very much, Rebecca, and also very much appreciate Partnership on AI for the invitation. When this series of summits first began in Bletchley, AI agents were not a thing. Nobody was talking about them, even just 12 months ago. When we had the AI Action Summit in Paris, it has barely crept into the conversation at the time. the preoccupation was all around DeepSeq and what it told us about the capabilities that is emerging out of China. But today, as Rebecca correctly identified, agentic systems have taken off. They are increasingly being used and we need to have a better grasp on how to deal with this issue because agentic AI certainly offers transformative possibilities in how we delegate and orchestrate work when deployed strategically.

Agents functions as invaluable teammates, unlocking productivity gains and time savings, which we all want more of. However, I should also add that this autonomy, the very nature of how agents can be helpful to us is autonomy. This autonomy also introduces new risk. The potential for harm increases when systems malfunction and human oversight is normalized. We are no longer present. or at least diminish to a very large extent. The implications may be complex and not fully predictable. So the way my colleagues and I have been thinking about this is that there needs to be a shift. There needs to be a shift in terms of how we might want to rely on reactive regulation to a different kind of stance, which is proactive preparation.

And in Singapore, that’s what we’ve been trying to do. We’ve tried to be proactive about governing the new risks in the era of agentic AI. And I think it starts with the government itself being a leader and not a laggard in using agentic AI. We need to test it. We need to look at how the solutions can not only enhance public service delivery, But we also need to be able to put in place more controls. Government is high risk because the touch point with citizens are very sensitive. No citizen and no government wants to make serious mistakes when they interact with their citizens, telling them things about their health, telling things about their social security, telling them about things to do with their benefits that are not accurate, and having them not just being told but acted upon.

So this need to ensure that we know what we’re doing is a very high one. And the way we are also thinking about it is to try and work with industry. So, for example, between Google and Singapore government, we have a sandbox on agentic AI. It’s one of the ways. We think we can, in a way, eat our own dog food. Try it. You know, does it taste all right? hurt us in a very significant way because if we were not able to do so, I don’t think we have a lot of credibility in terms of how we want to govern agentic AI. But we can’t wait, you know, for the dog food to materialize in its consequences for ourselves.

In the meantime, my colleagues have put together a model governance framework for agentic AI. It is meant to provide practical support to enterprises so that they can also deploy autonomous agents responsibly and to mitigate the risk. We know that this is not a complete solution and this document that we put out has to be a live document. We very much encourage feedback and as a way for us to keep improving the guidance to enterprises. Can I also just add that as we do this work, what is the… meaning and what is the purpose behind it. Ultimately, it is to build confidence in the use of agentic AI systems. And we think that at many levels, this confidence has to be presented, has to be demonstrated to boards of organizations, to customers, to other stakeholders.

And how do we demonstrate that the risks have been managed well? And that is where the assurance ecosystem that Rebecca talks about comes in. It is an absolutely essential part of building trust over the medium to longer term so that there is a way, a foundation upon which agentic AI systems can be made more readily adopted and available. I should also say that for companies that are thinking about it, and I see Microsoft here, and I’m sure that there are other companies represented. If we are to trust these agentic systems, the safety aspects should not be downplayed. And I would venture to say that a company that is able to give a high assurance on safety will find itself being differentiated from their competitor.

It’s more likely to translate into stronger interest in a product and service. So rather than think of it as something that you are unhappy to comply with, think of it as a strategic competitive advantage. And that is a way I think that will give us the confidence to put it forward. The question, however, is that are we completely without experience in this regard? And the answer is no. In aviation and healthcare, there are a lot of measures being put in place to give assurance to passengers. When we board a plane, we usually expect to arrive. when we visit the hospital, we generally expect to be treated, except for disease conditions that are not yet well understood.

But the trust in these systems have to be built over time, and they don’t come without some assurance being put in place. The question is for AI, and specifically agentic AI, what would be the components? What leads to an assurance ecosystem system that would be robust enough? We think that there are at least three components. The first is that there must be testing. We need some way of making sure that there are technical assessments of the system to make sure that the systems are robust, they are reliable, and they’re safe. And a lot more work needs to be done in this space, developing the testing methodology, building the testing datasets, and also making sure that the testing of agentic systems take into account that these systems are robust.

These systems are going to be much more complex than multi -agents, for example, and it’s not just the output, but the in -between steps, how the reasoning takes place, and what is the orchestration that is being built into the GenTech systems. So that’s the first, testing. Second is that eventually we will need standards. We cannot just define what is good enough. We also need to assure the users that it has met expectations in safety and reliability, and so these are still very early days. Thirdly, we think that this ecosystem cannot do without third -party assurance providers. It’s one thing to claim that your agentic AI system is safe. It’s another thing to have someone attest to the safety of it.

So these could be technical testers, auditors, and they provide independence, augment in -house capabilities, and also help to identify the blind spots, and it’s necessary for us to strengthen this pool as well. So I’m going to stop here. I want to conclude my remarks to say that Singapore is actively building these components. and we welcome conversations with partners and colleagues because we know that we cannot do this alone. So we look forward to discussions in the three panels on how we can meaningfully collaborate on assurance for agentic AI. Thank you very much once again, Rebecca.

Thank you. Thank you. We’re all here. It’s the end of the conference, and we’re all intact. Thank you so much, everyone, for joining us. Thank you, Minister Teo, for the keynote. One quick note before we dive in. Our panelist, Fred, has a flight to catch, so he’ll need to slip away a few minutes early, but, Fred, we’ll make sure we get your best insights before you escape. No pressure. So we are the last session, so we are standing between you and whatever you have planned right after. So I promise we’ll make this worth it. We have an incredible panel and a lot of ground to cover. So before we get started, what do we mean by AI assurance?

Because you’re going to keep hearing that term quite a bit here. So really put simply, AI assurance is the process of measuring, evaluating, and communicating whether AI systems are trustworthy. Are they safe? Do they work as intended? Can the public actually trust them? So really think of it like a safety inspection, but for AI. You wouldn’t want, you know, you’d want an independent inspector checking a building. Not just the builder saying, trust me, it’s fine. So really, AI assurance is about independent verification, as Minister Teo went over. And why this panel? Why now? So the summit unveiled the New Delhi Frontier AI commitments just yesterday. And the second of those commitments is about strengthening multilingual and contextual evaluations.

So really making sure AI systems work across languages, cultures, and real world conditions. And really, that’s the assurance challenge in a nutshell. And our panel today is about whether we are actually equipped to deliver on that promise globally and not just in a handful of countries. So really, our panelists span the ITU, Google DeepMind, the University of Pretoria, and PAI. So we have the range to actually wrestle with this question. So with that, I’m going to get into our first question for today. Fred, that’s going to be you. ITU has been convening on AI governance through AI for Good and working on standards across borders. So really, when we talk about AI assurance, what does it mean to you, ensuring that these systems are safe and trusted?

And how do we think about assurance when 2 .6 billion people remain offline and may be excluded from the frameworks being designed?

Yeah, thanks for that great question, and thanks for having me here. So I think that safe to save is no. There’s a huge shortage of high -potential AI for Good use cases, everything from affordable health care to education for all. food security, disaster response, and also looking at more applications in the physical manifestations of AI that you see in robotics, embodied AI, brain -computer interface technologies. The best part of my job at AI for Good is I see these use cases coming across my desk every day. And I can tell you when we started AI for Good in 2017, it was mainly in PowerPoint slides. They didn’t really exist. But as we got into, say, the 2023 with GenAI, last year, the unofficial theme of AI for Good was the rise of the AI agents, a bit scary, Terminator -like, but that’s what people were talking about.

And we’re really going from sort of the promise to the pilots to the use cases and now scaling. Now, when you’re looking at these use cases, I think one big challenge is trust. How do you trust them? I mean, there’s always the good intention, right? But is that trust there? And also, are they replicable and scalable? And I’ve yet to see, you know, high potential use case developed in Brussels work equally well in Johannesburg and Shenzhen and maybe Panama. Like, it’s just, we haven’t really reached that yet. And if you look at these sort of fast -emerging governance frameworks around the world, whether you’re in the U .S. or EU or China or everything in between, I think there’s a lot of good intentions, a lot of good thinking.

But how do you turn those ambitious words and principles into actions? Because the devil is in the details, and I think standards have details. So when you’re thinking about how do you – especially when you start to get into AI agents and you really – that trust element is becoming ever more critical, how can you bake in a lot of the common sense things that we’ve been talking about all week or even for the past years at AI for Good? Are they trustworthy? Are they verifiable? Are they secure? Are they safe? Are they designed with human rights principles in mind? Are they inclusive? Are people from the global south appetizing? Are they able when we’re drafting and developing these standards?

So these are not always natural reflexes, and at the same time, it’s hard to turn words into action. So one of the tools, I’m not saying it’s the only tool, but I think as these solutions start to scale and businesses start to interact internationally or even internationally, at one point you’re going to need standards, and it’s within those standards that you can kind of bake in those common sense principles that we’ve been all talking about. And I forget the last part of your question. It was really a question about… Oh, connectivity. That was it, yes. …2 .6 billion people who remain offline, yeah. Yeah. Yeah, so, you know, ITU’s mission is connecting the world, and a third of the world is still offline.

And, you know, large parts of the world actually have connectivity, but there’s actually no incentive to connect. So if there’s no content in your local language or dialect or no access to government services or useful applications that are fit for purpose in where you live… you know there’s why would you connect so i i think ai can actually help to remove that friction where you have a lot of bottlenecks for example literacy disabilities again like content in your own language or dialect so i think one thing is closing the connectivity gap but the other thing is actually using ai to remove that friction and the last thing i would say is i think sometimes there’s a comparison where um if you take east africa for example and you have the the mobile payment miracle or revolution with mpeza right you effectively leapfrog decades of infrastructure legacy infrastructure and there may be a kind of optimism that well the same thing could happen with ai in the global south maybe but i don’t think we can take it for granted that if that happens it goes in the right direction it’s not a guarantee that just by putting the tool in the hands of the people that they’re going to create value they’re going to use it responsibly they’re going to use it to solve local challenges build more cohesion and community, but those aren’t for granted.

So I think that whole AI skilling angle of really educating people from grade school to grad school to diplomats and everyone in between, if you don’t address that literacy piece, then it’s just going to be a crapshoot. We’re not sure

Great. I mean, it’s a good transition. Speaking of standards, Owen, Google DeepMind recently deepened its partnership with the UK AI Security Institute on safety research, so including work on monitoring chain of thought and evaluations. So really from an industry perspective, you know, what does robust AI assurance look like? Where do you think the gaps and opportunities are between what Frontier Labs kind of do internally and what’s needed for broader public trust?

Yeah, thank you, Madhu. And thank you to Rebecca and Partnership on AI for convening this really important conversation. And a big congratulations to our Indian hosts for a fantastic week at the summit. This week, maybe start talking a little bit about what… agents are, we’re increasingly excited about them at Google DeepMind. They’re essentially more autonomous systems that instead of just following basic instructions can actually achieve goals. So let’s say I want to get my suit dry cleaned on Thursday, instead of taking an AI system and say, find a website for a dry cleaning company, see if it’s open on Thursday, see what the hours are, see if it’s within my budget. You can just say to your agentic system, go find a way to dry clean my suit, make sure it’s being picked up by Friday, and it will go and interact with those different websites and try and find a way to meet your goals.

All kinds of fantastic applications already that we’re seeing right across the economy. We’re using increasingly agentic coding systems at Google and Google DeepMind to do a lot of our coding. So we have our anti -gravity framework, which is fantastic. You can interact with it in normal, natural language and say, build me a website, build me a tracking system to follow a particular bill that I’m interested in, and it will really help you achieve these goals. I think you’ll increasingly see agents used right across the economy as well. I think we’re just in the early years of a new AI enabled agentic economy. I think you will have very normal interactions with agents on a regular basis that will pop up on your phone screen and say, hey, it’s been a few weeks since you bought toothpaste.

Would you like me to go and take care of you and get some more toothpaste for you? You mentioned standards, which I think is going to be a critical part of getting all of this right. There’s a couple of dimensions to the standards. So firstly, we need to create the sort of technical protocols to actually underpin this agentic economy. So we’ve been trying to contribute to this conversation. There is the agents to agents protocol that Google has launched. There’s the universal commerce protocol. This is basically a way of helping agents talk to each other and agents talk to websites so that you have standardized sets of information. An agent will basically come to an agent or an agent will come to a website and say, this is my ID.

These are my capabilities. These are what I’m trying to do. I think in the same way that we developed protocols and standards in the early 90s to underpin the internet like HTTP, like URL, we’re going to have to build these out. There are then also assurance standards, which are related, but I think very important as well. We need to make sure that we’re understanding the capabilities of these systems. We need to keep making progress on how we can test for the risks that they may pose and then work right across society to come up with ways to mitigate that. I think the work that the safety and security institutes are doing around the world is absolutely critical.

So Minister Teo mentioned some of the work that we’re doing in Singapore. The UK Security Institute has been world leading on this. I think this is an area that we’re going to see more from the ACs and KCs right across the world. The US government also, through their KC, launching an agent standards initiative this week as well.

Great. And if you don’t mind a follow up question, that’s a really important point that you pointed out, that we currently need interoperability. We need agents to flourish. We need to find a new way to kind of imagine this paradigm. But I’m curious if there’s a safety challenge when it comes to agents. Instead. yeah that keeps you up at night

yeah i think there are definitely risks to be mindful of so i think agent security is something that we should all be thinking a lot about if we’re connecting increasingly autonomous systems into different accounts different email accounts different bank accounts i think we want to be pretty careful about how we do that and come up with superior security protocols and that can be helpful there we’ve actually been doing some work with virus total which is part of the the google security operations team at google to make sure that when certain agentic systems are downloading skills or downloading apps from agentic websites they’re being scanned for malware or vulnerabilities that are being detected so that they can be addressed before people put them onto their their computer i think there’s also a concern that these agentic systems could create new capabilities that could be misused so across the cyber security dimension domain for example i think some of the frameworks that we have already at google deep mind will be helpful here so we have our frontier safety framework which we use to test models before we put them out into the real world.

We think about how those models are going to interact with systems, how they might be parts of agents as we’re doing that work.

All right. Just speaking for myself, I can’t wait to use agents. I feel like it’s a lot of developer communities that have, you know, started playing around with these systems. But I imagine it’s reaching lay consumers very soon. So, Vukosi, you have built Masakane for African Language NLP. Really building AI for Africans by Africans. When assurance frameworks are designed in the U .S., U .K., or Singapore, how well do they translate to context where the data, the languages, the deployment conditions are completely different? What do we think we’re missing?

that we do get to understand that it’s a very different thing. My experience has been that there’s likely not as much collection in Europe or North America or annotation as much as is happening now in the global south. But then that also means that it feels like it’s further away, right? It’s not where the developers are. And that then requires more of this conversation in one place. So that, again, there must be kind of a local understanding. The last piece to that is going to be the capacity and the capabilities of then the policymakers in those countries to be able to understand that part. It will not be top -down. I don’t believe that. It will be them understanding whether it’s labor laws, it’s data governance, it’s just monitoring of systems once they’re on.

If there is not that capacity or capability to actually do those things, again, it’s more automated. direction that is not necessarily what the values of those people actually are.

Those are important words right at the end of the conference, knowing just how much we have to get done here. So Steph, over to you. PAI just released work on closing the global assurance divide, a lot of what Bukosi just mentioned. What are the concrete gaps you’re identifying? Identifying? Is it capacity to conduct third -party evaluations, as Minister Teo mentioned? Is it access to the models being tested, or is it something else? What would it take to really close those gaps?

Awesome. Thanks so much, Maru. And as one of the PAI folks, thanks for being here, everyone. It’s great to see you all. I know it’s a Friday evening, so we’re in between you and cocktails or whatever you have planned, so we very much appreciate it in the last session of the day. So I think it’s such a good question, and I think your question talks about some things that recognize that those challenges aren’t actually just Global South Challenge. I just want to start with the fact that we’ve released two papers. One is on closing the assurance divide, and the other is how we strengthen the global assurance ecosystem generally. And the question of access is one that impacts us all, actually.

In the UK, for example, the Department of Science, Innovation and Technology, I believe that’s what DSET stands for, has made access to models as a means to support insurance a priority for 2026. And so I think that there are a few shared challenges, and I’ll come back to the point around north -south, actually, collaboration in a second. But just thinking about closing the AI insurance divide, we released this paper, and in it we talk about around six challenge areas, from infrastructure to skills. We talk about languages and risk profiles, so the things that you’ve heard about from Vukosi and a lot of the other speakers. So I’ll give you a sense of some of the examples that we have.

So on language, we’re at the India Summit, of course, and India has over, I believe, 120 languages and 19 ,500 dialects. When we think about Africa, we have about… 1 ,500. or 3 ,000 spoken languages in itself. So when we think about benchmarking and evals and designing evals that think about how those systems are deployed in these various contexts, it’s so important to think about languages, and that just generally, I think, demonstrates the complexity of designing evals to meet the needs of this kind of diverse language ecosystem. Rebecca mentioned at the start that we had the declaration, of course, yesterday, and the commitment therefore in the declaration to multilingual evals is really critical. Of course, there’s still a lot of work to determine how do we actually do that in practice in the most effective way, and accounting for that complex and wide language diversity, but that’s one area that we talk about.

The second in terms of closing the assurance divide that we need to account for is risk profile, interestingly. in this paper, we actually interviewed a lot of assurance and safety experts internationally. And one of the things that they mentioned was differences in what they might prioritize when you think about assurance. So when you think about the Pacific Island nations, for example, they would be thinking about assuring for environmental impacts differently than maybe environmental impacts would be considered as important in the US at the moment, for example. Last year, we published a paper on post -appointment monitoring. And in that paper, we talk about sharing kind of data from companies. And one of the points that we talk about is environmental impacts.

And so it’s really interesting that I think in terms of closing the divide, it might the starting point or what you put emphasis on might vary. And that’s important to note as we’re designing things like documentation, description, and so on. And so I think it’s really interesting to see what we’ve kind of focused on. The third I’ll just quickly mention is, of course, infrastructure. I think we’ve probably all heard a lot about this throughout the summit and this idea of what it means to be sovereign and which parts of the stack to prioritize. And that is really, really important. But there are tradeoffs. So in terms of importance, I was looking at a stat that Stanford’s Helm evaluations used over 12 billion tokens and they required 19 ,500 GPU hours alone.

And so when you think about the kind of infrastructural needs, it’s so it creates barriers for a lot of countries in the global south. But I was at an interesting roundtable, actually, that even Carnegie was convening. And we were talking about the fact that how do you balance assurance needs? Where do you start from across the value chain? So at the moment, a lot of the discussion is kind of upstream. Right. We need to have that infrastructure in place. That’s the point that we need to start with. But how do you do that in parallel and how much of that resource should be put into other foundational tools for assurance, such as documentation artifacts, which is another area that we focus on a lot at PAI.

And so I think there will be a lot of questions around how do you weigh up all these challenges, again, knowing that even kind of the G7 countries, the UK AI Safety Institute started with an inaugural $100 million alone. So that prioritization and balancing is going to be important. The last thing I’ll say, coming back to agents, and I will talk about this a bit more, is the North -South collaboration is a real opportunity as we think about agents. And it’s important that global South countries aren’t always playing catch up. I think that’s a point that has come through for me from the summit, which is that NIST or the Casey, so the Center for AI Standards and Innovation.

And this is almost like a test for me of kind of saying. These names of these institutions through this panel. But they just announced a few days ago that they’re going to be working on standardizing work around agents, including that they’ve released an opportunity to comment on a paper around agent attribution and agent identity, I believe, which is really interesting. And there’s, of course, a lot of push for countries to collaborate. And you see a lot of the safety institutes collaborating on questions around assuring agents in the global north. But how do we ensure that global south countries aren’t missing from that? That will have implications for how we attribute agents, how we test agents.

And we shouldn’t just assume, again, whilst those upstream points and infrastructure is important, that in parallel, they’re ultimately part of these kind of thinking ahead questions and frameworks.

Great. So I’m going to take the moderator’s prerogative and have us do a rapid fire. And by rapid fire, I mean every answer is a minute and 30 seconds, which, let’s be honest, is fairly rapid for AI policies. I’m going to start with Fred because I’m more nervous about your flight than perhaps you are. So a minute and 30 seconds. What role should multilateral institutions like ITU play in making globally inclusive AI assurance happen?

Yes, I think AI for Good has a pretty ambitious goal, right? It’s simply put, it’s to unlock AI’s potential to serve humanity. Pretty big. But we can’t do it alone and no one can. It’s not one country and not one institution, not one NGO. That’s why we have 50 plus UN sister agencies as part of AI for Good, but also making great efforts to bring as many diverse voices to the table from the global south, from NGOs, from civil society. It’s always been extremely open. I like to think of it as the Davos of AI, but instead of being very exclusive, it’s extremely inclusive, right? So I think that’s a bit of a philosophy behind AI for Good.

You know, I think the AI, it’s just moving so quick. So the focus has always been on practical applications, practical solutions. But in doing that, you can tease out the next generation of standards, of policy recommendations, of collaboration and partnerships around the world. So I like to think that in the doing, you have the learning, right? And it’s not just about talking. And that’s what AI for Good has always been all about.

Thank you. That was incredible. You have 56 seconds left. So, yeah, I’m going to move us ahead to Vukosi. So Singapore’s aim is test once and comply globally. So from a Global South perspective, what would make that interoperability real rather than a form of exclusion?

Yeah, that’s a hard one. I think going back to I think the other thing that’s come out of a lot of the sessions here has been on the evaluations and how evaluations are used. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. And I think that’s a really important thing. because either on one side it’s going to take you a lot of resources to actually either put up the evaluation to be so all -encompassing on the other side to run it is going to be a lot but then when it comes down to the user which I think was our second panel that I was in this week and you’re trying to think about personalization if you’re going down to an individual what experience do they actually have and how do you get to there?

There will be some more high level safety things that will likely come out and people will be working on that and maybe that’s what I’m thinking Singapore is trying to go for but then when we’re getting to what the individual experience is given that you have the stochastic systems you don’t know what is going to happen necessarily. I know we’re trying to do that but we don’t really know what’s going to happen at the individual experience and we can’t remodel all of that. It’s going to require that again you you do have closer to where the user might be things on what actually that experience was. So one of the hats I wear is I’m a co -founder of Lilapa AI, an AI startup.

And there you will be doing more testing towards, hey, we are serving this client. We’re serving them in this way. And then you’re trying to then go in and say, where is your data coming from? What is the use cases? What are we testing for in terms of their operational kind of requirements? It would necessarily not be just one. But, yes, what you might want is

Yeah, that’s a great point. Assurance needs to be globally decentralized. Owen, given everything we have discussed, what’s one commitment Frontier Labs should make on assurance that would actually move the needle?

yeah good question um i think there’s a question of access to the technology which is important here i think it’s one of the big themes of this conference certainly one of the things that i’ll be taking away so you think the the multilingual part of this is really important understanding respecting local cultures that’s important if you’re going to have a good product and if it’s going to be used broadly um we’ve been investing in gemini for some time now to make it better more representative across different languages we have partnerships that we’re doing here in india including with the iit bombay to to help improve performance across various different indic languages it’s also really important on the safety and security front as well to have benchmarks that are available in different languages fantastic work that ml commons are doing on this front that we’re that we’re pleased to support the other bit of access that i think is really important is having things that are quick and cheap enough for everyone to use one of the things of agentic systems is that they’re actually pretty compute intensive to use we have a range of models that we have developed and bringing to market at google deep mind including our very quick flash models which are relatively cheap, quite efficient, very, very quick.

We think these can play a really important role in powering agentic systems. It’s also going to be really important if we’re going to do effective and rigorous testing of these systems because that could be very compute intensive as well. So thinking about that access piece is something we all need to keep doing. And it’s not an easy question, really. I mean, to do it safely and ensuring that third party assurance providers consider the security questions at hand. And it’s an open

So, Stephanie, no bias at all since we’re both at PAI, but I wanted to give you the final word. What concrete outcomes do you think we want to see from the global AI assurance work in the next 12 months? What would success look like?

So, Owen, now that you said your one point, by the way, can hold you accountable against this delivering on the access question. But. I think we in the two papers, we talk about the need to kind of build a robust assurance ecosystem. And one of those things is changing incentives. So funny enough, another session this week, there was a question about whether we have differences in the way we’re talking about safety over the last few years and whether that we still have those divergences of whether we’ve converged. And there are a few themes that we’ve actually converged on, which is nice. And I think assurance is one of them. And this week, a lot of the discussions we’ve had are in some of those incentive areas like insurance to support assurance.

And so what does that look like? How do we drive new incentives or put some of these structures in place to drive a kind of more mature and robust ecosystem? I think that’s going to be really important. The second is professionalization. There are a lot of questions around how do you trust the assurer? And so how do we ensure that we’re thinking about the skills? What does accreditation look like for assurance organizations or individuals? And so and that will help, I think, questions around kind of access. And so that’s a kind of second piece. But hopefully, I think what we’re what we’re hoping to do. And that’s just because this is also about agents. I think that some of those foundational questions haven’t yet been resolved.

And so I’m hoping that we can move the dial to start thinking about how do you apply that to some of these future questions. So just to shout you out, Madhu. Madhu is the brains behind our safety work. And she came up with a paper on real time failure detection and monitoring of agents. And what I really like about that paper is it talks about a kind of tiered approach to assurance as well. So when you think about agent deployments, do you need to be thinking about assurance based on the risks or the stakes at hand? So is it in the financial services sector? Is it in making about making medical decisions? So how do you tie it as close to the use case and the risks?

And that needs to be also linked to reversibility. What’s the possibility around reversibility of actions and the consequences of that? And then third, we have affordances. What are the kind of affordances you give to the agents? How much autonomy do they have? And so how do you design an assurance ecosystem with all of these different components in mind and a kind of tiered approach? And the more that we can advise, you know, the USKC and a lot of policymakers who clearly are trying to make decisions in this area, I think that’s what success would look like for us.

This was totally not planned. Steph plugging our work here, but I can’t imagine a better note to end on this. It’s a field wide challenge, but I just want to emphasize the field wide opportunity. No, you know, no one single organization can get this right. So hopefully that’s a helpful reminder as we end with this summit and move on to the next iteration. So thank you, everyone. Hope you have a great. safe flight back home. Fred, that’s tonight for you. And for a closing keynote, I’m going to welcome Natasha Crampton, who’s a Chief Responsible AI Officer at Microsoft. And post that, we’ll hear from Chris, who’s the CEO of FMF. Thanks, everyone. Do you want to give it?

Okay, so we’re going to get mementos. Sorry, you might want to come back. You don’t want to miss this. Thank you very much.

Thanks so much, Madhu, and to all of our panellists for what was, I think, a very rich and grounded and also at times humorous discussion. Thank you. One of the things that came across clearly for me today is that we need AI assurance to no longer just be a theoretical exercise, but we actually need to build it into an operational discipline. And that’s a discipline that really needs to work across borders, across languages and cultures, and I think increasingly across agentic systems, systems that don’t just generate outputs but actually take action. I heard this panelist focus on the fact that assurance is pretty uneven today. It’s often strongest where there’s access to compute and data and evaluation infrastructure, and weakest where those things are scarce.

And as several of our panelists emphasized, if we don’t address that gap deliberately, the shift towards AI agents is only going to make that divide even worse. Rather than closing it. When I think about the nature of assurance, I think with agentic systems, it does need to change in its emphasis somewhat. Pre -deployment testing has always been necessary for all types of systems, and so too has post -deployment testing, of course. But post -deployment testing in an agentic world takes on an even greater level of importance, in my view. When systems can plan and they can chain actions, they can interact with tools, they can adapt over time, assurance really has to move towards continuous monitoring, real -time detection, and clear accountabilities for when interventions need to take place.

That can be quite a hard technical problem, but it’s also a governance challenge. So I know that PAI is known for convening communities of not just thinkers, but also doers. And so I wanted to leave everyone with a couple of ideas of implications that really follow from some of the insights that we heard today. The first is that it’s really important that we build assurance into systems as part of the system development lifecycle. And we don’t just seek to bolt it on at the end. So that means that we need to design systems so that they can be observed and audited and constrained in practice, not just in policy documents. Second, assurance has to be interoperable.

We heard Prime Minister Modi speak yesterday about building in India and delivering to the world. That, I think, is absolutely an aspiration that we should strive towards. But that can only work if we have evidence. Evaluation methods and documents and signals of risk that are usable across regions. Thank you. and adaptable to local languages, cultures, and deployment realities. Third, assurance has to be shared. No single company or government or institution can do this alone. And that’s especially true for agents, given how pervasive they are expected to become across the economy. We need shared evaluation infrastructure, shared taxonomies, and shared investment in capacity, particularly in the global south. So for me, this is why organizations like the Partnership on AI, as well as the many collaborators that have come here together in this week’s India AI Impact Summit, as well as open engagement across the community to make sure that we get this right.

It’s a really foundational area for collaboration for all of us. Now, my view is that if we do get assurance, and by right, I mean it needs to be global and inclusive and also dynamic. I think it really does become an enabler of trust and adoption, as Minister Teo said, not a break on progress. One of the key things that I think we need to do as a community is really to treat assurance as infrastructure, infrastructure that we need to build together and put into practice together. Thanks very much.

Well, what a phenomenal session from the opening and closing keynotes to a really rich and dynamic panel. I cannot think of a better way to close out what has been an extraordinarily rich and dynamic summit as well. I have the impossible task of trying to summarize everything that was just said here. So if you’ll bear with me, I’ll just offer kind of three core themes that seem to jump out to me. One is that we need to evolve and mature our understanding of assurance. There’s a lot of reference to agents here, the kind of coming prospect of multi -agent environments as well. We need from evals to mitigations, we need to have a better kind of an evolving understanding of how to do assurance.

Second, and probably more importantly, we also heard a lot about assurance as a global effort. Here I love Steph’s point about the need for greater north -south collaboration. There’s a lot of discussion from Fred and others about the need for global standards and harmonizing those standards and making them interoperable. And then there was also a lot of reference to some of the new institutions that we’ve evolved to enable that global dialogue to happen, whether it’s the institution that was announced literally just before this session an hour ago for the kind of global network or the international network of ACs that have also been kind of revitalized recently as well. And then kind of the last point that really jumped out at me was the assurance as a shared responsibility.

And, Fikosi, I love the point about kind of assurance as a bottom -up effort, and I think it’s one that, you know, we all have a role to play here, regardless of which sector you are in, regardless of what aspect of assurance you’re taking part in, there’s a role for all of us. So with that, I’m going to leave you with just one kind of final call to action, and that is to get involved, right? You know, if we want this technology to be safe and secure and trusted, we all have a role to play. So download the reports, very important thing. Download the great reports that have just come out on this topic. Get involved.

Look at the work that PAI and others are doing as well, and become a part of the conversation about how we’re going to take this amazing technology, but really make sure that it’s safe and secure and that we have a way to trust it. You know, in the opening remarks, Rameca, kind of used this great metaphor of the seed, right? Like one of the goals of the reports that they put out and the conversation in this panel. was to try and plant the seed about, you know, to watch kind of assurance grow. So I guess the parting thought I would give you is to say let’s all kind of roll up our sleeves and get to work and make sure that the seed grows.

So with that, thank you. And thank you as well for our panelists and speakers. Thank you. Thank you. Thank you.