Challenging the status quo of AI security
11 Jul 2025 16:10h - 16:30h
Challenging the status quo of AI security
Session at a glance
Summary
This discussion focused on the security challenges and standardization needs for AI agents and multi-agent systems, presented by Study Group 17 which specializes in AI security. Sounil Yu introduced two key mental models for understanding AI security: the DIKW pyramid (Data, Information, Knowledge, Wisdom) and the OODA loop (Observe, Orient, Decide, Act). He explained that AI operates at the knowledge layer, requiring new security controls specifically designed for that abstraction level rather than applying traditional data-layer security measures that could result in degraded AI performance.
Babak Hodjat discussed the urgent need for standards in agentic AI systems, explaining how individual AI agents are increasingly being connected to work together within organizations. He demonstrated how a 350,000-employee company like Cognizant faces challenges when different divisions deploy their own AI agents that need to communicate and share information while maintaining security boundaries. The complexity multiplies when agents representing different organizations or individuals need to interact and negotiate on behalf of their users.
Debora Comparin addressed the critical issue of identity management for AI agents, highlighting several open problems that require standardization. These include defining what constitutes an agent’s identity, establishing trustworthy identity verification systems, securing cryptographic materials for agents, handling liability when agents make poor decisions, and managing delegation of authority in hierarchical agent systems. She emphasized the need for industry-wide collaboration to develop common security standards.
Xiaofang Yang provided real-world examples from Alipay’s experience in China, where AI agents are widely deployed across various industries including live shopping streams. She illustrated security vulnerabilities through an example where an AI shopping host could be manipulated into inappropriate behavior, demonstrating the need for proper safeguards and testing. The discussion concluded with a call to action for immediate development and adoption of security standards for AI systems, emphasizing that while implementing security from day one would be ideal, the next best time to act is now.
Keypoints
## Major Discussion Points:
– **Mental Models for AI Security**: Sounil Yu introduced frameworks like the DIKW pyramid (Data, Information, Knowledge, Wisdom) and OODA loop to understand AI security challenges, emphasizing that AI operates at the “knowledge layer” and requires new security controls rather than applying traditional data/information layer solutions.
– **Multi-Agent System Coordination and Security**: Babak Hodjat discussed the evolution from single AI systems to interconnected agent networks within organizations, highlighting the urgent need for standards as agents communicate with each other and handle sensitive data across different business divisions.
– **Identity Management Challenges for AI Agents**: Debora Comparin outlined critical open problems including defining what constitutes an agent’s identity, establishing trustworthy identity verification, securing cryptographic materials, handling liability issues, and managing delegation of authority in agent hierarchies.
– **Real-World Implementation Risks**: Xiaofang Yang shared practical examples from Alipay’s AI agent platform, including a case where an AI shopping host was manipulated into inappropriate behavior, demonstrating the need for proper safeguards and testing in production environments.
– **Urgent Need for Standardization**: All speakers emphasized the critical timing for developing security standards for agentic AI systems, with a call to action for immediate collaborative work rather than waiting for perfect solutions.
## Overall Purpose:
The discussion aimed to present findings from Study Group 17’s workshop on AI security and advocate for immediate development of security standards for agentic AI systems. The speakers sought to raise awareness about emerging security challenges and rally the community to collaborate on standardization efforts before these systems become more widespread.
## Overall Tone:
The tone was professional and urgent throughout, with speakers consistently emphasizing the time-sensitive nature of addressing AI security challenges. While informative and educational, there was an underlying sense of urgency and concern about the rapid deployment of AI agents without proper security frameworks. The tone remained collaborative and solution-oriented, with multiple calls for community participation and standardization efforts.
Speakers
– **Sounil Yu**: Moderator and presenter who shared mental models for AI security, including an OSI model for AI and the OODA loop framework. Focused on AI security standards and mental models for understanding AI problem spaces.
– **Babak Hodjat**: Representative from Cognizant (350,000 employee company), discussed agentic AI and multi-agency systems. Focused on standards and security issues associated with agentic AI, and mentioned their AI for Good initiative under UN SDGs.
– **Debora Comparin**: Expert in identity management, specifically addressing identity management challenges in the context of agentic AI. Discussed open problems around agent identity, trustworthiness, security, liability, and delegation issues.
– **Xiaofang Yang**: Representative from Alipay, shared practical use cases and real-world examples of AI agent implementation in China. Discussed their T-box platform for creating AI agents and security challenges encountered in live shopping streams.
Additional speakers:
– **Meredith Whitaker**: President of Signal (mentioned by Sounil Yu as having spoken on day one about AI agent security violations, but did not speak in this transcript)
Full session report
# AI Security Standards and Agentic Systems Discussion Report
## Executive Summary
This panel discussion, moderated by Sounil Yu from Study Group 17, addressed the urgent need for security standards in artificial intelligence systems, with particular focus on agentic AI and multi-agent systems. The session followed a morning workshop that covered identity, agentic AI, multi-agentic AI, and practical use cases. The panel brought together experts from diverse backgrounds to examine current security challenges and discuss potential frameworks for addressing them.
The discussion featured contributions from Babak Hodjat (Cognizant), Debora Comparin (identity management expert), and Xiaofang Yang (Alipay), each providing perspectives from their respective domains. Key themes included the need for new mental models for AI security, challenges in multi-agent system coordination, identity management for AI agents, and real-world security vulnerabilities in current implementations.
## Key Presentations and Insights
### Mental Models for AI Security (Sounil Yu)
Yu opened the discussion by introducing several mental models for understanding AI security challenges. He proposed developing “an OSI model for AI” to provide a structured communication framework for discussing AI security across different cultures and languages, similar to how the original OSI model facilitated networking discussions.
Central to Yu’s presentation was the DIKW pyramid (Data, Information, Knowledge, Wisdom), which he used to illustrate that AI operates primarily at the knowledge layer. He argued that applying traditional data-layer security controls to knowledge-layer AI operations results in “stupid LLMs” – systems whose functionality is severely degraded by inappropriate security measures.
Yu also referenced the OODA loop framework, specifically mentioning “Sensing, Sensemaking” as key components for understanding AI security, particularly the importance of separating these functions from decision-making and action phases.
### Multi-Agent Systems at Enterprise Scale (Babak Hodjat)
Hodjat provided insights from Cognizant’s perspective as a 350,000-employee organization implementing AI systems. He explained that while the business case for agentic AI is compelling due to cost reduction and efficiency improvements, this creates pressure to deploy systems rapidly while working against time to implement proper safeguards.
He described the evolution from single-agent systems to complex multi-agent ecosystems, where agents representing different stakeholders must coordinate while protecting their respective interests. Hodjat proposed a technical approach involving separation of LLM communications from code communications within agent architectures, allowing organizations to maintain security controls while leveraging large language model capabilities.
Hodjat also mentioned the “AI for Good initiative” and called for volunteers to participate in developing solutions for these challenges.
### Identity Management Challenges (Debora Comparin)
Comparin presented what she described as “food for thoughts” regarding identity management for AI agents, framing her contribution as exploring “open problems” rather than providing definitive solutions. She began with fundamental questions: “What is even ID of an agent? What is an identity of an agent?”
Her presentation outlined several critical areas requiring standardization:
– Defining what constitutes an agent’s identity
– Determining what content should be standardized for agent identities
– Establishing trustworthiness and verification mechanisms
– Managing cryptographic materials for autonomous systems
– Addressing liability and accountability when agents make poor decisions
– Handling delegation in agent hierarchies and preventing privilege escalation
### Real-World Implementation Examples (Xiaofang Yang)
Yang shared practical examples from Alipay’s experience with AI agent deployment, including their T-box platform that enables drag-and-drop creation of AI agents across various industries. She described how this platform has facilitated widespread deployment while revealing security vulnerabilities in practice.
Yang provided a specific example of AI agents used as live shopping hosts that proved vulnerable to prompt injection attacks. Users could manipulate these AI streamers by asking them to “turn to developer mode” or perform inappropriate actions, demonstrating fundamental gaps in role-based security controls.
She emphasized the dual nature of AI systems – their ability to both empower security capabilities and introduce new risks, reinforcing the urgency of developing appropriate security frameworks.
## Technical Approaches Discussed
### Architectural Solutions
Hodjat’s proposal to separate LLM communications from code communications represents a practical approach to maintaining security while preserving AI capabilities. This separation delegates language understanding tasks to LLMs while reserving consistency and security functions to traditional code.
### Framework Development
Yu’s mental models, particularly the DIKW pyramid and proposed OSI model for AI, provide systematic approaches for understanding and communicating about AI security challenges. These frameworks help identify where different types of security controls should be applied.
### Infrastructure Requirements
Comparin’s analysis highlighted the need for comprehensive infrastructure development, including trustworthy identity issuers, cryptographic key management systems, and verifiable logging mechanisms for AI agents.
## Current Challenges and Gaps
The discussion revealed several immediate challenges requiring attention:
– **Definitional gaps**: Lack of clear definitions for basic concepts like agent identity
– **Standards absence**: No established standards for agent-to-agent communication and coordination
– **Security vulnerabilities**: Current systems showing susceptibility to prompt injection and manipulation attacks
– **Governance questions**: Unresolved issues around liability and accountability for agent actions
– **Implementation pressure**: Business drivers pushing rapid deployment ahead of security framework development
## References to Broader Context
Yang referenced Meredith Whitaker’s presentation from day one of the conference, which discussed AI agent security violations, providing additional context for the urgency of addressing these challenges.
Yu noted that Study Group 17 had conducted a workshop that morning covering the key topics discussed in the panel, indicating ongoing work in developing solutions for these challenges.
## Next Steps and Call to Action
The discussion concluded with recognition of the need for immediate action on developing AI security standards. While speakers acknowledged that implementing security from the beginning of AI development would have been ideal, they emphasized that the rapid pace of AI deployment requires urgent attention to these challenges now.
Study Group 17 committed to continuing work on defining security standards for AI systems, with calls for broader multi-stakeholder collaboration to address the complex technical and governance challenges identified.
## Conclusion
This panel discussion highlighted the critical gap between the rapid deployment of AI systems and the development of appropriate security frameworks. The combination of theoretical frameworks, practical organizational experience, technical expertise, and real-world implementation examples provided a comprehensive view of current challenges.
The speakers demonstrated the complexity of AI security challenges while offering practical approaches and frameworks for addressing them. The discussion underscored the need for coordinated action across multiple stakeholders to develop and implement security standards that can keep pace with AI system deployment.
Session transcript
Sounil Yu: Well, thank you for having us here. What we had this morning was a workshop for study group 17, which is focused on security. And I’m sure that many conversations that you’ve had about AI, agentic AI, throughout the course of this week, security has probably been something that you’ve considered and maybe not put much thought into, while study group 17 is really focused on saying what are the security ramifications when we start unleashing AI to the world, and empowering us to be able to tap into all these use cases that we see. And there have been some amazing use cases that we’ve heard of, but at the same time, we want to do this in a responsible way that doesn’t cause harm and destruction for the rest of humanity. So with that said, we had a conversation this morning around the topic of security, specifically focused on a couple areas. One is identity, another one is agentic AI, and then multi-agentic AI. And then last, we talked about actual practical use cases. How do we secure these things when it’s actually in an organization? So we’re going to cover that with the panelists that you see here as well. So I’m Sounil Yu, and what I shared at the workshop was a set of mental models that help us understand the problem space itself with AI. And so I actually proposed what I call an OSI model for AI. If you’re familiar with standards, the OSI model is how we look at the network. And we look at the network as a way to describe the different facets of standards that are needed for network communications. I think that mental models are particularly useful because it serves as a communication layer for our brain across different cultures, across different languages. Mental models serve as a nice way to communicate very quickly. And so to be able to communicate very quickly with this large audience and within the study group, I shared with the group many different mental models. I’ll share two of them with you here. So one of the mental models that I shared was what’s called the DIKW pyramid, and it stands for data, information, knowledge, and wisdom. And what we have seen is a progression of technologies that have moved us up to the knowledge layer. So we now have knowledge tools for knowledge workers. ChatGPT really brought us into the knowledge layer. Understanding what that looks like, we now have a baseline or a template because we can use words that follow data, words like data security, data engineering, data privacy. Those same words, when they apply to information, they still work. When you apply it to knowledge, it gives us the problem space for AI. This gives us a sense of what problems we’re going to run into when we deal with AI challenges. Well, when it comes to AI security, what does that look like? And one way that we can look at AI security is to say, well, this is the same problem as knowledge security. How do we bound knowledge or how do we control knowledge? How do we shape knowledge so that it doesn’t go to the wrong people or doesn’t provide harmful results? All those sort of things. And so inside of an enterprise, it looks something like this, where you want to share valuable insights to your employees, but you also don’t want to share things that could cause harm inside your organization. But there are some people who should know these and some people who shouldn’t. So this is a knowledge security problem that we would have to try to find ways to tackle. But one of the interesting things that we learned from the pyramid is that the pyramid also suggests that we typically try to solve the problem at the wrong layer. So if I try to solve it at the data and information layer, what oftentimes happens, this mental model suggests that it looks like squeezing the bottom part of the pyramid, which means that if you squeeze this bottom part of the pyramid, we end up squeezing the top part and you end up with a stupid LLM, which nobody wants either. And so we have to recognize, when we look at standards, we have to recognize we have a new abstraction layer and we need controls at that new layer. And if we apply controls at the wrong layer, then we end up causing other problems. The other mental model that I’ll quickly go through is something called the OODA loop. It stands for Observe, Orient, Decide, Act. Instead of using Observe, Orient, I use the word Sensing, Sensemaking, and then Decisionmaking, Acting. That’s what the OODA loop is. Each of these different things have different stages. And the main thing to take away from this mental model, one of the main things to take away is that the notion of sensemaking, which is where AI fits, is very different than the notion of acting. And we have to recognize that there’s a separation there with decisionmaking. Moreover, we have turned over some of these functions to the machine. And there’s three different modes of how we can operate using this mental model of the OODA loop, where we’ve turned over sensing and acting, but we reserve the right to do the sensemaking and decisionmaking. The next stage is when we allow LLMs to have access to tools, and we give them the sensemaking apparatus, which is where the LLM comes in, but we still reserve the right to do decisionmaking. The last stage is when we start letting the machine also do autonomous decisionmaking. And there’s a lot of different controls that we need when we, a lot of controls and a lot of different ways that we need to create standards around this last stage, because we don’t necessarily have clear understanding of how to connect these pieces together. But that’s exactly what we’re about to do pretty soon. And so this creates a lot of other interesting challenges. And with that, that was the mental model that I shared with the group. And with that, I may now turn it over to Babak, who is going to talk about agentic AI and how he’s looked at the standards issues associated, standards and security issues associated with agentic AI.
Babak Hodjat: Thank you very much, Sounil. Yeah, we came out here for two reasons, as cognizant, one, to get people involved in our AI for Good initiative and partnership. And we’ve already had a number of projects. So if anybody’s interested in volunteering and helping in building these global decisioning systems and bringing in human expertise together with AI to tackle world problems, there’s a lot of pent up energy by AI folks. So please, please come join us under the UN SDGs. I just have to put a plug for that. And you know, you can take a photo of this and get in touch with us. But the main reason we got together was, you know, looking at the viability and the the opportunity to put together a standard around agents and multi-agency. And we actually saw some urgency around this within our company. But it’s very evident, having had these panels today, that this urgency and this need does exist and the energy is there for us to come together. In fact, if anything, we’ve collected a number of requirements and we’re going to work together on them. Just to really quickly level set and kind of give you a picture of why there is urgency. On the left hand side of this slide, you have a gen AI system, a be all end all general model that just transforms stuff, it produces output. When it becomes an agent, you give it some agency, there’s some autonomy you infer on it. It’s a combination of the LLM plus some code and it has some tools. Now if you put it in a box, give it some job description and have it operate something, there’s going to be another agent right next to it, probably doing something adjacent. And it just makes sense to have those connect and talk to each other. When you create these agentic systems, you’re reducing costs, you’re improving efficiencies and on a case by case basis, you’re improving quality. There is a lot of impetus to actually roll out these injecting systems. So we’re working against time here to safeguard them. Just a quick example, we’re a 350,000 employee company and different divisions have their own IT divisions, they’re bringing in their own AI agents into the fold and needing to connect them. It just doesn’t make sense for us to go around and basically have a rolodeck of agents and know which agent to talk to when. If I have some need, I really want the top agent that I’m talking to representing the entire organization to then talk to another agent and basically resolve what my need is and get back to me. So right there, I have agents talking to each other. Do they know my identity? Do they know what kind of data I have access to? These are all major questions. I have a quick video here of that same system and a user coming in and saying something very generic that the top few agents might not even know that the organization can resolve for them. I’m getting married, that’s a life change event. Which agents are responsible? And remember that we keep plugging in new agents on the go into the system so the domain of discourse and capabilities is growing. And yes, we’re talking about multi-agentic systems of many agents and this is even a moderately sized one. So if that’s the case, if you expand it to beyond just the same organization having a bunch of agents talking to each other and now you have agents representing me, the consumer, talking to agents representing, for example, three different travel companies and negotiating on my behalf, how do we deal with that coordination? We do want them to be able to collaborate and do something useful for me, but at the same time we don’t want it to give away a whole bunch of information about me. So that becomes a major issue. There’s a major need, and this is my last slide, on securing these systems and having standards around them and hopefully you see why. The good news here, though, is we’re moving away from completely delegating to a large language model that’s open to hallucination and inconsistencies to agents that are a combination of a large language model and code. Therefore, best practices allow us to delegate to the large language model when it comes to understanding language or reasoning and delegating to code when we need consistency, when we need safeguards and security. And even in the inter-agent communications, we can actually separate the pipes that communicate between LLMs or communicate between codes. So last thing I’m going to say is in this new agentic world, which is much more engineered, we have agency over these agents, but we do need standards to make sure that this best practices are implemented.
Sounil Yu: Thanks, Babak. And one of the standards that we absolutely need to establish is around identity. And so with that, let me turn it over to Debora, who can cover that piece.
Debora Comparin: Good afternoon, everyone. It’s really a pleasure to be here with you today. I will share with you some food for thoughts around identity management in the context of agentic AI. And if you think that identity management in this context is easy, I think you’ll change your mind. So I will share a couple of these points of discussion we had this morning. These are mostly open problems. It’s not an exhaustive list, but it’s to fit sort of like into what Babak was saying. We do need some work to be done. This work starts with the way I see it, standardization. So in getting actually the industry and the major stakeholders together and discuss what is our common view around the security posture on identity management, specifically. That’s the topic I’m covering of agentic AI, but broader than that. This is, I think it’s very important because we all ought to have our say in all of us experts in this ecosystem, because eventually it’s about us, as Babak said, it’s about our data, right? As citizens, as human beings. So without further ado, let me open up the world of ID management of agentic AI. What is even ID of an agent? What is an identity of an agent? That’s, again, room for standardization. We don’t know today, right? So what is the content of this identity? What is relevant to standardize and to share? But beyond this, how can I make this trustworthy, right? It’s not just about being able to fetch some form of ID cards for agents, but it’s being able to trust whatever is in there. And trust also means making sure that that ID card of agent is tied to the right agent and is coming from a proper and trustworthy issuer. So you see what I mean here? We really need a whole infrastructure of trust behind identity management of agentic AI. Final then, also, the second point is more about security. We tend to focus a lot in our discourse around the idea of agents in terms of content and trust, but how about the security? How can, it’s actually under research, this concept of how can I secure the cryptographic material that I need, that an agent needs, right? So there’s all this area of research that I think should be explored. And again, we should have probably a common posture around security on this within the industry. And then there’s liability. So we have imagined that agents can do a whole lot of things for us, which is great, but how do we deal with liability? Eventually, if they if an agent go wrong or does certain, take certain decision or end up with certain action we’re not happy with. So how can we trace it back to where this happened? And how can we get logs and logs that we can, again, can be a cryptographically verifiable and you can trust? And then there’s the problem of delegation. So I, as a user, want to, again, use and leverage an agent that might actually leverage other agents in a tree, in a hierarchy. So this gets very complicated when it comes to delegation of authority. We need to make sure that this doesn’t escalate, right? So when you go down the tree of agents, it becomes something, so the delegation becomes something bigger than actually how it started from the user standpoint. So we have some tools for delegation. If any of you are actually familiar with ID management, you would know some of the existing standard, but they need to be adapted. They need to be adapted to this new ecosystem, this new infrastructure of agents that we are building, especially in agents in the Internet. So these are some open points of trustworthiness of ID of agents, security of cryptography material for agents, liability, how do we handle that, and the delegation issues. Some of the main key points that I’d like you to take home and think about it, if you’re an expert, don’t hesitate to get in touch. I’m always happy to discuss about this and really, I will just end up with, close with a call for action to work together, work together in a multi-stakeholder organization and push for some clarity and standardization around these topics.
Sounil Yu: Thank you very much, Debora. When it comes to standards, we oftentimes think in the hypothetical, but it actually helps when we have real-life use cases, real-life examples, and so Xiaofang Yang also shared that within the context of her organization. So let me turn it over to her to share what she’s learned.
Xiaofang Yang: Good afternoon, everyone. I’m Xiaofang Yang from Alipay. Yeah, so thank you for having me here. I would like to share some practice that we have in China. So, as all of us know that since it debuted, that the generated content already amazed a lot of people as a chatbot, but since last year, there are so many tech players, so many companies have invested a lot of technology into making AI agents, like in Alipay, that we have a platform called T-box, where everybody can create an AI agent very easily, like drag and drop different AI components, and then in a few minutes, then you can make an AI agent. So the AI technology has been rapidly and widely adopted in so many industries and companies in China. Take one example, like a live shopping stream. I’m not sure if any of you have tried. So, the streamer, we can say the live host, is already being AI-powered. It can talk and can showcase products and to adapt, interact with the audiences. So, it’s an amazing experience that we don’t need, and it always looks like two people. But what’s the problem? That when users ask the the AI, the AI streamer turn to a developer mode and ask them to, like, meow like a cat for 100 times, the AI agent will just comply without any question. So, we see this kind of consequence is unwanted, right? Because we want the host to just do its job to to sell the product, but not to ruin the live the live shopping. So, we see that when entering the new age of AI agents, we see so many different types of security challenges that that means we we need to do well test to make sure that second AI agents can should be always act as as we wanted. And then we also need a proper safeguard online to make sure that when there’s evil inputs from the outside that we can block. So, AI has two sides. On the one hand, it empowers security. It helps us to improve the security measures. It can also introduce new risks. So, I still believe that when we handle these new security challenges, that the standards could be very important to help reduce the risk.
Sounil Yu: So, for those who are here on day one, you heard Meredith Whitaker, the president of Signal, she gave us a scenario where an agent, an AI agent, violated a lot of these security boundaries that we just talked about. And she cast a scenario of what could possibly go wrong. What I hope you learned from here is that study group 17 is specifically looking at that problem to understand how do we infuse security and trust into these AI systems because we need to have that really on day one. But if we can’t have it on day one, the next best time to do it is today. And so, a call to action for us is really let’s get to work now, define those standards, make sure that they’re adopted and embraced because the sooner we do it, the better we’ll have, the sooner we’ll have a trustworthy and secure system as well. So, thank you very much. Enjoy the rest of your conference.
Sounil Yu
Speech speed
182 words per minute
Speech length
1325 words
Speech time
435 seconds
OSI model for AI provides communication framework across cultures and languages
Explanation
Sounil Yu proposes an OSI model for AI similar to the network OSI model, serving as mental models that help communicate complex AI concepts quickly across different cultures and languages. These mental models act as a communication layer for the brain to facilitate understanding of AI problem spaces.
Evidence
References the existing OSI model for networks as an analogy and explains how mental models serve as communication tools across cultures and languages
Major discussion point
AI Security Framework and Mental Models
Topics
Digital standards
Agreed with
– Babak Hodjat
– Debora Comparin
Agreed on
Multi-stakeholder collaboration needed for AI security standardization
DIKW pyramid (Data, Information, Knowledge, Wisdom) shows AI operates at knowledge layer requiring new controls
Explanation
The DIKW pyramid demonstrates that AI technologies like ChatGPT have moved us into the knowledge layer, creating knowledge tools for knowledge workers. This progression means we need to develop new security controls specifically for the knowledge layer rather than trying to apply data or information layer controls.
Evidence
ChatGPT example showing progression to knowledge layer, and explanation of how terms like ‘data security’ can be applied to ‘knowledge security’
Major discussion point
AI Security Framework and Mental Models
Topics
Digital standards | Privacy and data protection
OODA loop model demonstrates separation between sensemaking and acting in AI systems
Explanation
The OODA loop (Observe, Orient, Decide, Act) model shows different stages of AI operation, with sensemaking being distinct from acting. This model illustrates three modes of operation: humans doing sensemaking/decision-making while machines handle sensing/acting, LLMs doing sensemaking while humans decide, and fully autonomous machine decision-making.
Evidence
Detailed explanation of the three operational modes and how functions are distributed between humans and machines
Major discussion point
AI Security Framework and Mental Models
Topics
Digital standards
Applying controls at wrong abstraction layer creates problems like “stupid LLMs”
Explanation
When security controls designed for data and information layers are applied to AI systems operating at the knowledge layer, it’s like squeezing the bottom of the pyramid, which results in degraded AI performance. This demonstrates the need for controls appropriate to each abstraction layer.
Evidence
Pyramid analogy showing how squeezing the bottom affects the top, resulting in ‘stupid LLMs’
Major discussion point
AI Security Framework and Mental Models
Topics
Digital standards | Network security
Agreed with
– Babak Hodjat
– Xiaofang Yang
Agreed on
AI systems introduce new security challenges requiring specialized approaches
Security considerations often overlooked in AI discussions despite critical importance
Explanation
Sounil Yu notes that while many conversations about AI and agentic AI have occurred, security has probably been considered but not given much thought. Study group 17 focuses specifically on security ramifications of unleashing AI to ensure responsible deployment without causing harm to humanity.
Evidence
Reference to conversations throughout the week where security wasn’t prioritized, and mention of Study Group 17’s focus
Major discussion point
Urgency for AI Security Standards
Topics
Network security | Digital standards
Standards needed immediately rather than waiting – “next best time is today”
Explanation
Sounil Yu emphasizes the urgent need to define and adopt AI security standards now rather than waiting. He advocates for immediate action to establish trustworthy and secure AI systems, referencing a scenario from Meredith Whitaker about AI agents violating security boundaries.
Evidence
Reference to Meredith Whitaker’s scenario from day one about AI agents violating security boundaries
Major discussion point
Urgency for AI Security Standards
Topics
Digital standards | Network security
Agreed with
– Babak Hodjat
– Debora Comparin
– Xiaofang Yang
Agreed on
Urgent need for AI security standards and immediate action
Babak Hodjat
Speech speed
165 words per minute
Speech length
819 words
Speech time
296 seconds
Agentic systems reduce costs, improve efficiency and quality, creating impetus for rapid deployment
Explanation
When AI systems become agents with autonomy, combining LLMs with code and tools, they create significant business value by reducing costs and improving efficiency and quality. This creates strong motivation for organizations to rapidly deploy these systems, making it urgent to establish security safeguards.
Evidence
Explanation of progression from general AI to agentic systems and the business benefits driving adoption
Major discussion point
Agentic AI Systems and Multi-Agent Coordination
Topics
Digital business models | Future of work
Agreed with
– Sounil Yu
– Debora Comparin
– Xiaofang Yang
Agreed on
Urgent need for AI security standards and immediate action
Multi-agent systems require coordination between agents within and across organizations
Explanation
As organizations deploy multiple AI agents, these agents need to communicate and coordinate with each other rather than maintaining separate systems. This extends beyond single organizations to scenarios where consumer agents negotiate with multiple company agents, creating complex coordination challenges.
Evidence
Example of 350,000 employee company with different divisions deploying agents, and scenario of consumer agents negotiating with travel company agents
Major discussion point
Agentic AI Systems and Multi-Agent Coordination
Topics
Digital business models | Consumer protection
Agreed with
– Sounil Yu
– Debora Comparin
Agreed on
Multi-stakeholder collaboration needed for AI security standardization
Agents combining LLMs with code allow delegation of language tasks to LLMs and consistency/security to code
Explanation
Unlike pure LLM systems prone to hallucination, agentic systems combine LLMs with code, allowing best practices where language understanding and reasoning are delegated to LLMs while consistency and security functions are handled by code. This engineered approach provides more control and reliability.
Evidence
Contrast between pure LLM delegation and hybrid LLM-code approach, with explanation of task separation
Major discussion point
Agentic AI Systems and Multi-Agent Coordination
Topics
Digital standards | Network security
Agreed with
– Sounil Yu
– Xiaofang Yang
Agreed on
AI systems introduce new security challenges requiring specialized approaches
Best practices can be implemented through engineered approach separating LLM and code communications
Explanation
In multi-agent systems, communication pipes between agents can be separated – some for LLM-to-LLM communication and others for code-to-code communication. This engineered approach allows for better implementation of security best practices and standards in the agentic world.
Evidence
Description of separate communication pipes for different types of inter-agent communications
Major discussion point
Agentic AI Systems and Multi-Agent Coordination
Topics
Digital standards | Network security
Debora Comparin
Speech speed
165 words per minute
Speech length
726 words
Speech time
262 seconds
Agent identity definition and content standardization remains an open problem
Explanation
The fundamental question of what constitutes an agent’s identity and what content should be included in that identity remains undefined. This includes determining what information is relevant to standardize and share, representing a critical gap that requires industry collaboration to address.
Evidence
Framing as open problems and questions about identity content and standardization needs
Major discussion point
Identity Management Challenges in Agentic AI
Topics
Digital identities | Digital standards
Agreed with
– Sounil Yu
– Babak Hodjat
– Xiaofang Yang
Agreed on
Urgent need for AI security standards and immediate action
Trustworthy identity infrastructure requires proper issuers and cryptographic security
Explanation
Beyond just creating identity credentials for agents, there’s a need for a complete infrastructure of trust that ensures identity credentials are tied to the correct agents and issued by trustworthy authorities. This includes securing the cryptographic materials that agents need to operate securely.
Evidence
Analogy to ID cards and emphasis on trustworthy issuers and cryptographic material security
Major discussion point
Identity Management Challenges in Agentic AI
Topics
Digital identities | Encryption | Digital standards
Agreed with
– Sounil Yu
– Babak Hodjat
Agreed on
Multi-stakeholder collaboration needed for AI security standardization
Liability tracing requires cryptographically verifiable logs when agents make wrong decisions
Explanation
When AI agents make decisions or take actions that users are unhappy with, there needs to be a way to trace back what happened and determine liability. This requires maintaining logs that are cryptographically verifiable and trustworthy to support accountability.
Evidence
Discussion of tracing back wrong decisions and need for verifiable logs
Major discussion point
Identity Management Challenges in Agentic AI
Topics
Liability of intermediaries | Encryption | Digital standards
Delegation of authority in agent hierarchies needs safeguards to prevent escalation
Explanation
When users delegate authority to agents that then use other agents in a hierarchical tree structure, there’s a risk that the delegation could escalate beyond what the user originally intended. Existing delegation standards need to be adapted for this new ecosystem of interconnected agents.
Evidence
Description of delegation trees and escalation risks, mention of existing standards needing adaptation
Major discussion point
Identity Management Challenges in Agentic AI
Topics
Digital identities | Digital standards
Xiaofang Yang
Speech speed
130 words per minute
Speech length
386 words
Speech time
176 seconds
AI agents can be easily created but lack proper security controls in practice
Explanation
Platforms like Alipay’s T-box allow anyone to create AI agents easily through drag-and-drop interfaces in just minutes. While this democratizes AI agent creation, it also means many agents are deployed without adequate security considerations or controls.
Evidence
Description of Alipay’s T-box platform enabling easy agent creation through drag-and-drop
Major discussion point
Real-World Security Challenges and Implementation
Topics
Digital business models | Consumer protection
Live shopping AI streamers demonstrate vulnerability to prompt injection attacks
Explanation
AI-powered live shopping hosts can interact with audiences and showcase products, but they’re vulnerable to manipulation when users ask them to enter developer mode or perform unintended actions like meowing like a cat 100 times. This shows how agents can be made to comply with requests that go against their intended purpose.
Evidence
Specific example of AI streamer being asked to meow like a cat 100 times and complying, disrupting the shopping experience
Major discussion point
Real-World Security Challenges and Implementation
Topics
E-commerce and Digital Trade | Consumer protection
AI presents dual nature – empowering security while introducing new risks
Explanation
AI technology has two sides: it can enhance security measures and help improve existing security systems, but it also introduces entirely new types of security risks and challenges. This dual nature requires careful consideration of both benefits and risks.
Evidence
General observation about AI’s dual impact on security
Major discussion point
Real-World Security Challenges and Implementation
Topics
Network security | Digital standards
Agreed with
– Sounil Yu
– Babak Hodjat
Agreed on
AI systems introduce new security challenges requiring specialized approaches
Standards are crucial for reducing risks in AI agent deployment
Explanation
Given the new security challenges that emerge when deploying AI agents, having proper standards in place is essential for risk reduction. Standards can help ensure that security safeguards are properly implemented and that agents behave as intended.
Evidence
Connection between observed security challenges and need for standards
Major discussion point
Real-World Security Challenges and Implementation
Topics
Digital standards | Network security
Agreed with
– Sounil Yu
– Babak Hodjat
– Debora Comparin
Agreed on
Urgent need for AI security standards and immediate action
Agreements
Agreement points
Urgent need for AI security standards and immediate action
Speakers
– Sounil Yu
– Babak Hodjat
– Debora Comparin
– Xiaofang Yang
Arguments
Standards needed immediately rather than waiting – “next best time is today”
Agentic systems reduce costs, improve efficiency and quality, creating impetus for rapid deployment
Agent identity definition and content standardization remains an open problem
Standards are crucial for reducing risks in AI agent deployment
Summary
All speakers agree that there is an urgent need to establish AI security standards now rather than waiting, as the rapid deployment of AI systems creates immediate security risks that must be addressed through standardization efforts.
Topics
Digital standards | Network security
AI systems introduce new security challenges requiring specialized approaches
Speakers
– Sounil Yu
– Babak Hodjat
– Xiaofang Yang
Arguments
Applying controls at wrong abstraction layer creates problems like “stupid LLMs”
Agents combining LLMs with code allow delegation of language tasks to LLMs and consistency/security to code
AI presents dual nature – empowering security while introducing new risks
Summary
Speakers consensus that AI systems create fundamentally new security challenges that cannot be addressed by simply applying existing security controls, requiring new approaches tailored to AI’s unique characteristics.
Topics
Network security | Digital standards
Multi-stakeholder collaboration needed for AI security standardization
Speakers
– Sounil Yu
– Babak Hodjat
– Debora Comparin
Arguments
OSI model for AI provides communication framework across cultures and languages
Multi-agent systems require coordination between agents within and across organizations
Trustworthy identity infrastructure requires proper issuers and cryptographic security
Summary
All three speakers emphasize the need for collaborative, multi-stakeholder approaches to develop AI security standards that can work across different organizations, cultures, and technical domains.
Topics
Digital standards | Digital identities
Similar viewpoints
Both speakers recognize the complexity of multi-agent systems and the need for proper coordination and delegation mechanisms, with Babak focusing on inter-organizational coordination and Debora on hierarchical delegation risks.
Speakers
– Babak Hodjat
– Debora Comparin
Arguments
Multi-agent systems require coordination between agents within and across organizations
Delegation of authority in agent hierarchies needs safeguards to prevent escalation
Topics
Digital standards | Digital identities
Both speakers observe that security is often an afterthought in AI development and deployment, with Sounil noting this in general AI discussions and Xiaofang providing concrete examples from practice.
Speakers
– Sounil Yu
– Xiaofang Yang
Arguments
Security considerations often overlooked in AI discussions despite critical importance
AI agents can be easily created but lack proper security controls in practice
Topics
Network security | Digital standards
Both speakers advocate for engineered, systematic approaches to AI security rather than ad-hoc solutions, emphasizing the need for proper infrastructure and architectural considerations.
Speakers
– Babak Hodjat
– Debora Comparin
Arguments
Best practices can be implemented through engineered approach separating LLM and code communications
Trustworthy identity infrastructure requires proper issuers and cryptographic security
Topics
Digital standards | Network security | Encryption
Unexpected consensus
Mental models and frameworks as essential communication tools for AI security
Speakers
– Sounil Yu
– Babak Hodjat
– Debora Comparin
Arguments
OSI model for AI provides communication framework across cultures and languages
Agents combining LLMs with code allow delegation of language tasks to LLMs and consistency/security to code
Agent identity definition and content standardization remains an open problem
Explanation
Unexpectedly, all technical speakers converged on the importance of having clear conceptual frameworks and models to communicate about AI security challenges, suggesting that the field needs common vocabulary and mental models before technical solutions can be effectively implemented.
Topics
Digital standards
Immediate practical deployment challenges outpacing security considerations
Speakers
– Babak Hodjat
– Xiaofang Yang
Arguments
Agentic systems reduce costs, improve efficiency and quality, creating impetus for rapid deployment
AI agents can be easily created but lack proper security controls in practice
Explanation
Both speakers from different organizational contexts (enterprise and platform provider) independently identified the same phenomenon where business incentives and ease of deployment are driving rapid AI adoption faster than security measures can be implemented, creating systemic risk.
Topics
Digital business models | Network security
Overall assessment
Summary
The speakers demonstrate strong consensus on the urgent need for AI security standards, the inadequacy of existing security approaches for AI systems, and the necessity of multi-stakeholder collaboration. They agree that current AI deployment is outpacing security considerations and that new frameworks are needed.
Consensus level
High level of consensus with complementary expertise – each speaker brings different perspectives (theoretical frameworks, enterprise implementation, identity management, practical deployment) but all converge on the same fundamental challenges and solutions. This strong alignment suggests the field is ready for coordinated standardization efforts and that the identified problems are genuine and widely recognized across different domains and geographies.
Differences
Different viewpoints
Unexpected differences
Overall assessment
Summary
The speakers showed remarkable consensus on the core issues, with no direct disagreements identified. All speakers agreed on the urgent need for AI security standards, the complexity of multi-agent systems, and the importance of proper identity management and security controls.
Disagreement level
Very low disagreement level. The speakers presented complementary perspectives rather than conflicting viewpoints, which suggests strong alignment within the AI security community on fundamental challenges. This consensus could facilitate faster progress on standardization efforts, but may also indicate that dissenting voices or alternative approaches were not represented in this particular discussion.
Partial agreements
Partial agreements
Similar viewpoints
Both speakers recognize the complexity of multi-agent systems and the need for proper coordination and delegation mechanisms, with Babak focusing on inter-organizational coordination and Debora on hierarchical delegation risks.
Speakers
– Babak Hodjat
– Debora Comparin
Arguments
Multi-agent systems require coordination between agents within and across organizations
Delegation of authority in agent hierarchies needs safeguards to prevent escalation
Topics
Digital standards | Digital identities
Both speakers observe that security is often an afterthought in AI development and deployment, with Sounil noting this in general AI discussions and Xiaofang providing concrete examples from practice.
Speakers
– Sounil Yu
– Xiaofang Yang
Arguments
Security considerations often overlooked in AI discussions despite critical importance
AI agents can be easily created but lack proper security controls in practice
Topics
Network security | Digital standards
Both speakers advocate for engineered, systematic approaches to AI security rather than ad-hoc solutions, emphasizing the need for proper infrastructure and architectural considerations.
Speakers
– Babak Hodjat
– Debora Comparin
Arguments
Best practices can be implemented through engineered approach separating LLM and code communications
Trustworthy identity infrastructure requires proper issuers and cryptographic security
Topics
Digital standards | Network security | Encryption
Takeaways
Key takeaways
AI security requires new mental models and frameworks, including an OSI model for AI and understanding that AI operates at the knowledge layer of the DIKW pyramid
Security controls must be applied at the correct abstraction layer – applying data/information layer controls to knowledge-layer AI systems creates problems like ‘stupid LLMs’
Multi-agent systems are rapidly being deployed across organizations, creating urgent need for coordination standards and security frameworks
Agent identity management presents fundamental challenges including defining what constitutes agent identity, establishing trustworthy infrastructure, ensuring cryptographic security, and managing liability
Real-world implementations show AI agents are vulnerable to prompt injection attacks and lack proper security controls despite easy deployment
The separation between AI sensemaking capabilities and decision-making/acting functions is critical for maintaining human control and security
Standards development is urgently needed now rather than waiting, as AI agent deployment is accelerating faster than security frameworks
Resolutions and action items
Study Group 17 to continue working on defining security standards for AI systems
Call for multi-stakeholder collaboration to establish common security posture and standardization around AI agent identity management
Invitation for experts to join Cognizant’s AI for Good initiative under UN SDGs
Industry stakeholders urged to work together on requirements gathering for agent and multi-agency standards
Immediate action needed to define and adopt AI security standards rather than delaying implementation
Unresolved issues
What constitutes the identity of an AI agent and what content should be standardized
How to secure cryptographic material that agents need for authentication and communication
How to handle liability when agents make wrong decisions or take unwanted actions
How to manage delegation of authority in agent hierarchies to prevent privilege escalation
How to adapt existing identity management standards to the new ecosystem of internet-connected agents
How to implement proper safeguards for AI agents while maintaining their functionality and efficiency
How to establish cryptographically verifiable logging systems for agent actions and decisions
Suggested compromises
Separating LLM communications from code communications in inter-agent systems to balance functionality with security
Delegating language understanding and reasoning tasks to LLMs while reserving consistency and security functions to traditional code
Implementing best practices through engineered approaches that provide agency over agents while maintaining security controls
Balancing the rapid deployment benefits of AI agents with the need for proper security testing and safeguards
Thought provoking comments
I actually proposed what I call an OSI model for AI… Mental models serve as a nice way to communicate very quickly… I shared with the group many different mental models including the DIKW pyramid (data, information, knowledge, and wisdom) and the OODA loop.
Speaker
Sounil Yu
Reason
This comment is insightful because it introduces a systematic framework for understanding AI security by drawing parallels to established networking standards. The DIKW pyramid concept is particularly thought-provoking as it suggests that AI has brought us to the ‘knowledge layer’ and that trying to solve knowledge-layer problems at the data/information layer results in ‘squeezing the pyramid’ and creating ‘stupid LLMs.’ This reframes how we should approach AI security controls.
Impact
This foundational framework set the intellectual tone for the entire discussion, providing a common vocabulary and conceptual structure that subsequent speakers built upon. It shifted the conversation from ad-hoc security concerns to systematic thinking about AI security layers.
When you create these agentic systems, you’re reducing costs, you’re improving efficiencies… So we’re working against time here to safeguard them… Do they know my identity? Do they know what kind of data I have access to? These are all major questions.
Speaker
Babak Hodjat
Reason
This comment is thought-provoking because it highlights the fundamental tension between rapid AI adoption driven by business benefits and the urgent need for security standards. The real-world example of a 350,000-employee company deploying agents across divisions illustrates how the multi-agent problem isn’t theoretical but happening now at scale.
Impact
This comment shifted the discussion from theoretical frameworks to urgent practical realities, creating a sense of immediacy. It demonstrated that the security challenges aren’t future problems but current organizational pain points, which gave weight to the subsequent technical discussions about identity and standards.
What is even ID of an agent? What is an identity of an agent? That’s, again, room for standardization. We don’t know today, right?… We really need a whole infrastructure of trust behind identity management of agentic AI.
Speaker
Debora Comparin
Reason
This comment is particularly insightful because it exposes how fundamental concepts we take for granted in human systems (identity, trust, liability) become complex open problems in AI systems. The question ‘What is even ID of an agent?’ reveals that we’re building systems without defining basic building blocks.
Impact
This comment deepened the technical complexity of the discussion by breaking down identity management into specific unsolved problems (trustworthiness, cryptographic security, liability, delegation). It moved the conversation from general security concerns to concrete technical challenges that need standardization.
When users ask the AI streamer to turn to developer mode and ask them to meow like a cat for 100 times, the AI agent will just comply without any question… we want the host to just do its job to sell the product, but not to ruin the live shopping.
Speaker
Xiaofang Yang
Reason
This seemingly simple example is profoundly thought-provoking because it illustrates how AI agents can be manipulated to break their intended roles in ways that seem harmless but reveal deeper security vulnerabilities. The ‘meowing cat’ scenario demonstrates prompt injection attacks in a real commercial context.
Impact
This concrete, almost humorous example made the abstract security concepts tangible and relatable. It showed how theoretical vulnerabilities manifest in real business scenarios, reinforcing the urgency established by earlier speakers and providing a memorable illustration of why the standards work is critical.
The pyramid also suggests that we typically try to solve the problem at the wrong layer… if you squeeze this bottom part of the pyramid, we end up squeezing the top part and you end up with a stupid LLM, which nobody wants either.
Speaker
Sounil Yu
Reason
This insight is thought-provoking because it suggests that traditional security approaches may be counterproductive when applied to AI systems. The metaphor of ‘squeezing the pyramid’ elegantly captures how restricting data/information layers can degrade AI capabilities, highlighting the need for new approaches to security controls.
Impact
This comment provided a crucial conceptual bridge explaining why existing security methods might fail with AI systems. It justified the need for new standards and approaches rather than simply adapting existing ones, influencing how subsequent speakers framed their technical challenges.
Overall assessment
These key comments collectively shaped the discussion by establishing a progression from theoretical frameworks to urgent practical realities. Sounil Yu’s mental models provided the intellectual foundation, Babak’s organizational examples created urgency, Debora’s technical breakdown revealed the complexity of implementation, and Xiaofang’s real-world scenario made the consequences tangible. Together, they transformed what could have been an abstract academic discussion into a compelling case for immediate action on AI security standards. The comments built upon each other to create a narrative arc: we have frameworks to understand the problem, we have urgent business drivers, we have complex technical challenges, and we have real-world consequences of inaction. This progression effectively motivated the call to action for developing standards ‘today’ rather than waiting for perfect solutions.
Follow-up questions
What are the security ramifications when we start unleashing AI to the world?
Speaker
Sounil Yu
Explanation
This is a fundamental question that drives the entire study group 17’s focus and requires ongoing research to ensure responsible AI deployment
How do we bound knowledge or how do we control knowledge? How do we shape knowledge so that it doesn’t go to the wrong people or doesn’t provide harmful results?
Speaker
Sounil Yu
Explanation
This addresses the core challenge of knowledge security in AI systems and requires development of new control mechanisms at the knowledge layer
What is even ID of an agent? What is an identity of an agent?
Speaker
Debora Comparin
Explanation
This is a fundamental standardization question that needs to be resolved before implementing identity management systems for AI agents
What is the content of this identity? What is relevant to standardize and to share?
Speaker
Debora Comparin
Explanation
This follows from the identity definition question and is crucial for establishing interoperable standards for agent identification
How can I make this trustworthy?
Speaker
Debora Comparin
Explanation
Trust is essential for agent identity systems to function properly and requires development of verification mechanisms
How can I secure the cryptographic material that I need, that an agent needs?
Speaker
Debora Comparin
Explanation
This is an active area of research that’s critical for the security foundation of agent systems
How do we deal with liability when agents go wrong or make decisions we’re not happy with?
Speaker
Debora Comparin
Explanation
This addresses accountability and traceability in agent systems, which is crucial for real-world deployment
How can we get logs that are cryptographically verifiable and trustworthy?
Speaker
Debora Comparin
Explanation
This is necessary for establishing audit trails and accountability in agent systems
How do we handle delegation of authority in agent hierarchies without escalation?
Speaker
Debora Comparin
Explanation
This addresses the challenge of maintaining proper authorization boundaries as agents delegate to other agents
Do agents know my identity? Do they know what kind of data I have access to?
Speaker
Babak Hodjat
Explanation
These are critical questions for multi-agent systems operating within organizations and handling sensitive data
How do we deal with coordination when agents representing consumers talk to agents representing companies?
Speaker
Babak Hodjat
Explanation
This addresses the challenge of inter-organizational agent communication while protecting user privacy
How do we ensure agents don’t give away information about users during negotiations?
Speaker
Babak Hodjat
Explanation
This is crucial for maintaining privacy in agent-to-agent interactions across organizational boundaries
How do we need to create standards around autonomous decision-making by machines?
Speaker
Sounil Yu
Explanation
This addresses the highest level of AI autonomy and requires careful consideration of control mechanisms
How do we ensure proper safeguards are in place when AI agents act autonomously?
Speaker
Xiaofang Yang
Explanation
This is demonstrated by the live shopping example where agents complied with inappropriate requests, showing the need for better behavioral controls
Disclaimer: This is not an official session record. DiploAI generates these resources from audiovisual recordings, and they are presented as-is, including potential errors. Due to logistical challenges, such as discrepancies in audio/video or transcripts, names may be misspelled. We strive for accuracy to the best of our ability.
Related event
