Challenging the status quo of AI security

Summary

This discussion focused on the security challenges and standardization needs for AI agents and multi-agent systems, presented by Study Group 17 which specializes in AI security. Sounil Yu introduced two key mental models for understanding AI security: the DIKW pyramid (Data, Information, Knowledge, Wisdom) and the OODA loop (Observe, Orient, Decide, Act). He explained that AI operates at the knowledge layer, requiring new security controls specifically designed for that abstraction level rather than applying traditional data-layer security measures that could result in degraded AI performance.

Babak Hodjat discussed the urgent need for standards in agentic AI systems, explaining how individual AI agents are increasingly being connected to work together within organizations. He demonstrated how a 350,000-employee company like Cognizant faces challenges when different divisions deploy their own AI agents that need to communicate and share information while maintaining security boundaries. The complexity multiplies when agents representing different organizations or individuals need to interact and negotiate on behalf of their users.

Debora Comparin addressed the critical issue of identity management for AI agents, highlighting several open problems that require standardization. These include defining what constitutes an agent’s identity, establishing trustworthy identity verification systems, securing cryptographic materials for agents, handling liability when agents make poor decisions, and managing delegation of authority in hierarchical agent systems. She emphasized the need for industry-wide collaboration to develop common security standards.

Xiaofang Yang provided real-world examples from Alipay’s experience in China, where AI agents are widely deployed across various industries including live shopping streams. She illustrated security vulnerabilities through an example where an AI shopping host could be manipulated into inappropriate behavior, demonstrating the need for proper safeguards and testing. The discussion concluded with a call to action for immediate development and adoption of security standards for AI systems, emphasizing that while implementing security from day one would be ideal, the next best time to act is now.

Keypoints

## Major Discussion Points:

– **Mental Models for AI Security**: Sounil Yu introduced frameworks like the DIKW pyramid (Data, Information, Knowledge, Wisdom) and OODA loop to understand AI security challenges, emphasizing that AI operates at the “knowledge layer” and requires new security controls rather than applying traditional data/information layer solutions.

– **Multi-Agent System Coordination and Security**: Babak Hodjat discussed the evolution from single AI systems to interconnected agent networks within organizations, highlighting the urgent need for standards as agents communicate with each other and handle sensitive data across different business divisions.

– **Identity Management Challenges for AI Agents**: Debora Comparin outlined critical open problems including defining what constitutes an agent’s identity, establishing trustworthy identity verification, securing cryptographic materials, handling liability issues, and managing delegation of authority in agent hierarchies.

– **Real-World Implementation Risks**: Xiaofang Yang shared practical examples from Alipay’s AI agent platform, including a case where an AI shopping host was manipulated into inappropriate behavior, demonstrating the need for proper safeguards and testing in production environments.

– **Urgent Need for Standardization**: All speakers emphasized the critical timing for developing security standards for agentic AI systems, with a call to action for immediate collaborative work rather than waiting for perfect solutions.

## Overall Purpose:

The discussion aimed to present findings from Study Group 17’s workshop on AI security and advocate for immediate development of security standards for agentic AI systems. The speakers sought to raise awareness about emerging security challenges and rally the community to collaborate on standardization efforts before these systems become more widespread.

## Overall Tone:

The tone was professional and urgent throughout, with speakers consistently emphasizing the time-sensitive nature of addressing AI security challenges. While informative and educational, there was an underlying sense of urgency and concern about the rapid deployment of AI agents without proper security frameworks. The tone remained collaborative and solution-oriented, with multiple calls for community participation and standardization efforts.

Speakers

– **Sounil Yu**: Moderator and presenter who shared mental models for AI security, including an OSI model for AI and the OODA loop framework. Focused on AI security standards and mental models for understanding AI problem spaces.

– **Babak Hodjat**: Representative from Cognizant (350,000 employee company), discussed agentic AI and multi-agency systems. Focused on standards and security issues associated with agentic AI, and mentioned their AI for Good initiative under UN SDGs.

– **Debora Comparin**: Expert in identity management, specifically addressing identity management challenges in the context of agentic AI. Discussed open problems around agent identity, trustworthiness, security, liability, and delegation issues.

– **Xiaofang Yang**: Representative from Alipay, shared practical use cases and real-world examples of AI agent implementation in China. Discussed their T-box platform for creating AI agents and security challenges encountered in live shopping streams.

Additional speakers:

– **Meredith Whitaker**: President of Signal (mentioned by Sounil Yu as having spoken on day one about AI agent security violations, but did not speak in this transcript)

# AI Security Standards and Agentic Systems Discussion Report

## Executive Summary

This panel discussion, moderated by Sounil Yu from Study Group 17, addressed the urgent need for security standards in artificial intelligence systems, with particular focus on agentic AI and multi-agent systems. The session followed a morning workshop that covered identity, agentic AI, multi-agentic AI, and practical use cases. The panel brought together experts from diverse backgrounds to examine current security challenges and discuss potential frameworks for addressing them.

The discussion featured contributions from Babak Hodjat (Cognizant), Debora Comparin (identity management expert), and Xiaofang Yang (Alipay), each providing perspectives from their respective domains. Key themes included the need for new mental models for AI security, challenges in multi-agent system coordination, identity management for AI agents, and real-world security vulnerabilities in current implementations.

## Key Presentations and Insights

### Mental Models for AI Security (Sounil Yu)

Yu opened the discussion by introducing several mental models for understanding AI security challenges. He proposed developing “an OSI model for AI” to provide a structured communication framework for discussing AI security across different cultures and languages, similar to how the original OSI model facilitated networking discussions.

Central to Yu’s presentation was the DIKW pyramid (Data, Information, Knowledge, Wisdom), which he used to illustrate that AI operates primarily at the knowledge layer. He argued that applying traditional data-layer security controls to knowledge-layer AI operations results in “stupid LLMs” – systems whose functionality is severely degraded by inappropriate security measures.

Yu also referenced the OODA loop framework, specifically mentioning “Sensing, Sensemaking” as key components for understanding AI security, particularly the importance of separating these functions from decision-making and action phases.

### Multi-Agent Systems at Enterprise Scale (Babak Hodjat)

Hodjat provided insights from Cognizant’s perspective as a 350,000-employee organization implementing AI systems. He explained that while the business case for agentic AI is compelling due to cost reduction and efficiency improvements, this creates pressure to deploy systems rapidly while working against time to implement proper safeguards.

He described the evolution from single-agent systems to complex multi-agent ecosystems, where agents representing different stakeholders must coordinate while protecting their respective interests. Hodjat proposed a technical approach involving separation of LLM communications from code communications within agent architectures, allowing organizations to maintain security controls while leveraging large language model capabilities.

Hodjat also mentioned the “AI for Good initiative” and called for volunteers to participate in developing solutions for these challenges.

### Identity Management Challenges (Debora Comparin)

Comparin presented what she described as “food for thoughts” regarding identity management for AI agents, framing her contribution as exploring “open problems” rather than providing definitive solutions. She began with fundamental questions: “What is even ID of an agent? What is an identity of an agent?”

Her presentation outlined several critical areas requiring standardization:

– Defining what constitutes an agent’s identity

– Determining what content should be standardized for agent identities

– Establishing trustworthiness and verification mechanisms

– Managing cryptographic materials for autonomous systems

– Addressing liability and accountability when agents make poor decisions

– Handling delegation in agent hierarchies and preventing privilege escalation

### Real-World Implementation Examples (Xiaofang Yang)

Yang shared practical examples from Alipay’s experience with AI agent deployment, including their T-box platform that enables drag-and-drop creation of AI agents across various industries. She described how this platform has facilitated widespread deployment while revealing security vulnerabilities in practice.

Yang provided a specific example of AI agents used as live shopping hosts that proved vulnerable to prompt injection attacks. Users could manipulate these AI streamers by asking them to “turn to developer mode” or perform inappropriate actions, demonstrating fundamental gaps in role-based security controls.

She emphasized the dual nature of AI systems – their ability to both empower security capabilities and introduce new risks, reinforcing the urgency of developing appropriate security frameworks.

## Technical Approaches Discussed

### Architectural Solutions

Hodjat’s proposal to separate LLM communications from code communications represents a practical approach to maintaining security while preserving AI capabilities. This separation delegates language understanding tasks to LLMs while reserving consistency and security functions to traditional code.

### Framework Development

Yu’s mental models, particularly the DIKW pyramid and proposed OSI model for AI, provide systematic approaches for understanding and communicating about AI security challenges. These frameworks help identify where different types of security controls should be applied.

### Infrastructure Requirements

Comparin’s analysis highlighted the need for comprehensive infrastructure development, including trustworthy identity issuers, cryptographic key management systems, and verifiable logging mechanisms for AI agents.

## Current Challenges and Gaps

The discussion revealed several immediate challenges requiring attention:

– **Definitional gaps**: Lack of clear definitions for basic concepts like agent identity

– **Standards absence**: No established standards for agent-to-agent communication and coordination

– **Security vulnerabilities**: Current systems showing susceptibility to prompt injection and manipulation attacks

– **Governance questions**: Unresolved issues around liability and accountability for agent actions

– **Implementation pressure**: Business drivers pushing rapid deployment ahead of security framework development

## References to Broader Context

Yang referenced Meredith Whitaker’s presentation from day one of the conference, which discussed AI agent security violations, providing additional context for the urgency of addressing these challenges.

Yu noted that Study Group 17 had conducted a workshop that morning covering the key topics discussed in the panel, indicating ongoing work in developing solutions for these challenges.

## Next Steps and Call to Action

The discussion concluded with recognition of the need for immediate action on developing AI security standards. While speakers acknowledged that implementing security from the beginning of AI development would have been ideal, they emphasized that the rapid pace of AI deployment requires urgent attention to these challenges now.

Study Group 17 committed to continuing work on defining security standards for AI systems, with calls for broader multi-stakeholder collaboration to address the complex technical and governance challenges identified.

## Conclusion

This panel discussion highlighted the critical gap between the rapid deployment of AI systems and the development of appropriate security frameworks. The combination of theoretical frameworks, practical organizational experience, technical expertise, and real-world implementation examples provided a comprehensive view of current challenges.

The speakers demonstrated the complexity of AI security challenges while offering practical approaches and frameworks for addressing them. The discussion underscored the need for coordinated action across multiple stakeholders to develop and implement security standards that can keep pace with AI system deployment.

Sounil Yu: Well, thank you for having us here. What we had this morning was a workshop for study group 17, which is focused on security. And I’m sure that many conversations that you’ve had about AI, agentic AI, throughout the course of this week, security has probably been something that you’ve considered and maybe not put much thought into, while study group 17 is really focused on saying what are the security ramifications when we start unleashing AI to the world, and empowering us to be able to tap into all these use cases that we see. And there have been some amazing use cases that we’ve heard of, but at the same time, we want to do this in a responsible way that doesn’t cause harm and destruction for the rest of humanity. So with that said, we had a conversation this morning around the topic of security, specifically focused on a couple areas. One is identity, another one is agentic AI, and then multi-agentic AI. And then last, we talked about actual practical use cases. How do we secure these things when it’s actually in an organization? So we’re going to cover that with the panelists that you see here as well. So I’m Sounil Yu, and what I shared at the workshop was a set of mental models that help us understand the problem space itself with AI. And so I actually proposed what I call an OSI model for AI. If you’re familiar with standards, the OSI model is how we look at the network. And we look at the network as a way to describe the different facets of standards that are needed for network communications. I think that mental models are particularly useful because it serves as a communication layer for our brain across different cultures, across different languages. Mental models serve as a nice way to communicate very quickly. And so to be able to communicate very quickly with this large audience and within the study group, I shared with the group many different mental models. I’ll share two of them with you here. So one of the mental models that I shared was what’s called the DIKW pyramid, and it stands for data, information, knowledge, and wisdom. And what we have seen is a progression of technologies that have moved us up to the knowledge layer. So we now have knowledge tools for knowledge workers. ChatGPT really brought us into the knowledge layer. Understanding what that looks like, we now have a baseline or a template because we can use words that follow data, words like data security, data engineering, data privacy. Those same words, when they apply to information, they still work. When you apply it to knowledge, it gives us the problem space for AI. This gives us a sense of what problems we’re going to run into when we deal with AI challenges. Well, when it comes to AI security, what does that look like? And one way that we can look at AI security is to say, well, this is the same problem as knowledge security. How do we bound knowledge or how do we control knowledge? How do we shape knowledge so that it doesn’t go to the wrong people or doesn’t provide harmful results? All those sort of things. And so inside of an enterprise, it looks something like this, where you want to share valuable insights to your employees, but you also don’t want to share things that could cause harm inside your organization. But there are some people who should know these and some people who shouldn’t. So this is a knowledge security problem that we would have to try to find ways to tackle. But one of the interesting things that we learned from the pyramid is that the pyramid also suggests that we typically try to solve the problem at the wrong layer. So if I try to solve it at the data and information layer, what oftentimes happens, this mental model suggests that it looks like squeezing the bottom part of the pyramid, which means that if you squeeze this bottom part of the pyramid, we end up squeezing the top part and you end up with a stupid LLM, which nobody wants either. And so we have to recognize, when we look at standards, we have to recognize we have a new abstraction layer and we need controls at that new layer. And if we apply controls at the wrong layer, then we end up causing other problems. The other mental model that I’ll quickly go through is something called the OODA loop. It stands for Observe, Orient, Decide, Act. Instead of using Observe, Orient, I use the word Sensing, Sensemaking, and then Decisionmaking, Acting. That’s what the OODA loop is. Each of these different things have different stages. And the main thing to take away from this mental model, one of the main things to take away is that the notion of sensemaking, which is where AI fits, is very different than the notion of acting. And we have to recognize that there’s a separation there with decisionmaking. Moreover, we have turned over some of these functions to the machine. And there’s three different modes of how we can operate using this mental model of the OODA loop, where we’ve turned over sensing and acting, but we reserve the right to do the sensemaking and decisionmaking. The next stage is when we allow LLMs to have access to tools, and we give them the sensemaking apparatus, which is where the LLM comes in, but we still reserve the right to do decisionmaking. The last stage is when we start letting the machine also do autonomous decisionmaking. And there’s a lot of different controls that we need when we, a lot of controls and a lot of different ways that we need to create standards around this last stage, because we don’t necessarily have clear understanding of how to connect these pieces together. But that’s exactly what we’re about to do pretty soon. And so this creates a lot of other interesting challenges. And with that, that was the mental model that I shared with the group. And with that, I may now turn it over to Babak, who is going to talk about agentic AI and how he’s looked at the standards issues associated, standards and security issues associated with agentic AI.

Babak Hodjat: Thank you very much, Sounil. Yeah, we came out here for two reasons, as cognizant, one, to get people involved in our AI for Good initiative and partnership. And we’ve already had a number of projects. So if anybody’s interested in volunteering and helping in building these global decisioning systems and bringing in human expertise together with AI to tackle world problems, there’s a lot of pent up energy by AI folks. So please, please come join us under the UN SDGs. I just have to put a plug for that. And you know, you can take a photo of this and get in touch with us. But the main reason we got together was, you know, looking at the viability and the the opportunity to put together a standard around agents and multi-agency. And we actually saw some urgency around this within our company. But it’s very evident, having had these panels today, that this urgency and this need does exist and the energy is there for us to come together. In fact, if anything, we’ve collected a number of requirements and we’re going to work together on them. Just to really quickly level set and kind of give you a picture of why there is urgency. On the left hand side of this slide, you have a gen AI system, a be all end all general model that just transforms stuff, it produces output. When it becomes an agent, you give it some agency, there’s some autonomy you infer on it. It’s a combination of the LLM plus some code and it has some tools. Now if you put it in a box, give it some job description and have it operate something, there’s going to be another agent right next to it, probably doing something adjacent. And it just makes sense to have those connect and talk to each other. When you create these agentic systems, you’re reducing costs, you’re improving efficiencies and on a case by case basis, you’re improving quality. There is a lot of impetus to actually roll out these injecting systems. So we’re working against time here to safeguard them. Just a quick example, we’re a 350,000 employee company and different divisions have their own IT divisions, they’re bringing in their own AI agents into the fold and needing to connect them. It just doesn’t make sense for us to go around and basically have a rolodeck of agents and know which agent to talk to when. If I have some need, I really want the top agent that I’m talking to representing the entire organization to then talk to another agent and basically resolve what my need is and get back to me. So right there, I have agents talking to each other. Do they know my identity? Do they know what kind of data I have access to? These are all major questions. I have a quick video here of that same system and a user coming in and saying something very generic that the top few agents might not even know that the organization can resolve for them. I’m getting married, that’s a life change event. Which agents are responsible? And remember that we keep plugging in new agents on the go into the system so the domain of discourse and capabilities is growing. And yes, we’re talking about multi-agentic systems of many agents and this is even a moderately sized one. So if that’s the case, if you expand it to beyond just the same organization having a bunch of agents talking to each other and now you have agents representing me, the consumer, talking to agents representing, for example, three different travel companies and negotiating on my behalf, how do we deal with that coordination? We do want them to be able to collaborate and do something useful for me, but at the same time we don’t want it to give away a whole bunch of information about me. So that becomes a major issue. There’s a major need, and this is my last slide, on securing these systems and having standards around them and hopefully you see why. The good news here, though, is we’re moving away from completely delegating to a large language model that’s open to hallucination and inconsistencies to agents that are a combination of a large language model and code. Therefore, best practices allow us to delegate to the large language model when it comes to understanding language or reasoning and delegating to code when we need consistency, when we need safeguards and security. And even in the inter-agent communications, we can actually separate the pipes that communicate between LLMs or communicate between codes. So last thing I’m going to say is in this new agentic world, which is much more engineered, we have agency over these agents, but we do need standards to make sure that this best practices are implemented.

Sounil Yu: Thanks, Babak. And one of the standards that we absolutely need to establish is around identity. And so with that, let me turn it over to Debora, who can cover that piece.

Debora Comparin: Good afternoon, everyone. It’s really a pleasure to be here with you today. I will share with you some food for thoughts around identity management in the context of agentic AI. And if you think that identity management in this context is easy, I think you’ll change your mind. So I will share a couple of these points of discussion we had this morning. These are mostly open problems. It’s not an exhaustive list, but it’s to fit sort of like into what Babak was saying. We do need some work to be done. This work starts with the way I see it, standardization. So in getting actually the industry and the major stakeholders together and discuss what is our common view around the security posture on identity management, specifically. That’s the topic I’m covering of agentic AI, but broader than that. This is, I think it’s very important because we all ought to have our say in all of us experts in this ecosystem, because eventually it’s about us, as Babak said, it’s about our data, right? As citizens, as human beings. So without further ado, let me open up the world of ID management of agentic AI. What is even ID of an agent? What is an identity of an agent? That’s, again, room for standardization. We don’t know today, right? So what is the content of this identity? What is relevant to standardize and to share? But beyond this, how can I make this trustworthy, right? It’s not just about being able to fetch some form of ID cards for agents, but it’s being able to trust whatever is in there. And trust also means making sure that that ID card of agent is tied to the right agent and is coming from a proper and trustworthy issuer. So you see what I mean here? We really need a whole infrastructure of trust behind identity management of agentic AI. Final then, also, the second point is more about security. We tend to focus a lot in our discourse around the idea of agents in terms of content and trust, but how about the security? How can, it’s actually under research, this concept of how can I secure the cryptographic material that I need, that an agent needs, right? So there’s all this area of research that I think should be explored. And again, we should have probably a common posture around security on this within the industry. And then there’s liability. So we have imagined that agents can do a whole lot of things for us, which is great, but how do we deal with liability? Eventually, if they if an agent go wrong or does certain, take certain decision or end up with certain action we’re not happy with. So how can we trace it back to where this happened? And how can we get logs and logs that we can, again, can be a cryptographically verifiable and you can trust? And then there’s the problem of delegation. So I, as a user, want to, again, use and leverage an agent that might actually leverage other agents in a tree, in a hierarchy. So this gets very complicated when it comes to delegation of authority. We need to make sure that this doesn’t escalate, right? So when you go down the tree of agents, it becomes something, so the delegation becomes something bigger than actually how it started from the user standpoint. So we have some tools for delegation. If any of you are actually familiar with ID management, you would know some of the existing standard, but they need to be adapted. They need to be adapted to this new ecosystem, this new infrastructure of agents that we are building, especially in agents in the Internet. So these are some open points of trustworthiness of ID of agents, security of cryptography material for agents, liability, how do we handle that, and the delegation issues. Some of the main key points that I’d like you to take home and think about it, if you’re an expert, don’t hesitate to get in touch. I’m always happy to discuss about this and really, I will just end up with, close with a call for action to work together, work together in a multi-stakeholder organization and push for some clarity and standardization around these topics.

Sounil Yu: Thank you very much, Debora. When it comes to standards, we oftentimes think in the hypothetical, but it actually helps when we have real-life use cases, real-life examples, and so Xiaofang Yang also shared that within the context of her organization. So let me turn it over to her to share what she’s learned.

Xiaofang Yang: Good afternoon, everyone. I’m Xiaofang Yang from Alipay. Yeah, so thank you for having me here. I would like to share some practice that we have in China. So, as all of us know that since it debuted, that the generated content already amazed a lot of people as a chatbot, but since last year, there are so many tech players, so many companies have invested a lot of technology into making AI agents, like in Alipay, that we have a platform called T-box, where everybody can create an AI agent very easily, like drag and drop different AI components, and then in a few minutes, then you can make an AI agent. So the AI technology has been rapidly and widely adopted in so many industries and companies in China. Take one example, like a live shopping stream. I’m not sure if any of you have tried. So, the streamer, we can say the live host, is already being AI-powered. It can talk and can showcase products and to adapt, interact with the audiences. So, it’s an amazing experience that we don’t need, and it always looks like two people. But what’s the problem? That when users ask the the AI, the AI streamer turn to a developer mode and ask them to, like, meow like a cat for 100 times, the AI agent will just comply without any question. So, we see this kind of consequence is unwanted, right? Because we want the host to just do its job to to sell the product, but not to ruin the live the live shopping. So, we see that when entering the new age of AI agents, we see so many different types of security challenges that that means we we need to do well test to make sure that second AI agents can should be always act as as we wanted. And then we also need a proper safeguard online to make sure that when there’s evil inputs from the outside that we can block. So, AI has two sides. On the one hand, it empowers security. It helps us to improve the security measures. It can also introduce new risks. So, I still believe that when we handle these new security challenges, that the standards could be very important to help reduce the risk.

Sounil Yu: So, for those who are here on day one, you heard Meredith Whitaker, the president of Signal, she gave us a scenario where an agent, an AI agent, violated a lot of these security boundaries that we just talked about. And she cast a scenario of what could possibly go wrong. What I hope you learned from here is that study group 17 is specifically looking at that problem to understand how do we infuse security and trust into these AI systems because we need to have that really on day one. But if we can’t have it on day one, the next best time to do it is today. And so, a call to action for us is really let’s get to work now, define those standards, make sure that they’re adopted and embraced because the sooner we do it, the better we’ll have, the sooner we’ll have a trustworthy and secure system as well. So, thank you very much. Enjoy the rest of your conference.

Agreement points

Urgent need for AI security standards and immediate action

Speakers

– Sounil Yu
– Babak Hodjat
– Debora Comparin
– Xiaofang Yang

Arguments

Standards needed immediately rather than waiting – “next best time is today”

Agentic systems reduce costs, improve efficiency and quality, creating impetus for rapid deployment

Agent identity definition and content standardization remains an open problem

Standards are crucial for reducing risks in AI agent deployment

Summary

All speakers agree that there is an urgent need to establish AI security standards now rather than waiting, as the rapid deployment of AI systems creates immediate security risks that must be addressed through standardization efforts.

Topics

Digital standards | Network security

AI systems introduce new security challenges requiring specialized approaches

Speakers

– Sounil Yu
– Babak Hodjat
– Xiaofang Yang

Arguments

Applying controls at wrong abstraction layer creates problems like “stupid LLMs”

Agents combining LLMs with code allow delegation of language tasks to LLMs and consistency/security to code

AI presents dual nature – empowering security while introducing new risks

Summary

Speakers consensus that AI systems create fundamentally new security challenges that cannot be addressed by simply applying existing security controls, requiring new approaches tailored to AI’s unique characteristics.

Topics

Network security | Digital standards

Multi-stakeholder collaboration needed for AI security standardization

Speakers

– Sounil Yu
– Babak Hodjat
– Debora Comparin

Arguments

OSI model for AI provides communication framework across cultures and languages

Multi-agent systems require coordination between agents within and across organizations

Trustworthy identity infrastructure requires proper issuers and cryptographic security

Summary

All three speakers emphasize the need for collaborative, multi-stakeholder approaches to develop AI security standards that can work across different organizations, cultures, and technical domains.

Topics

Digital standards | Digital identities

Similar viewpoints

Both speakers recognize the complexity of multi-agent systems and the need for proper coordination and delegation mechanisms, with Babak focusing on inter-organizational coordination and Debora on hierarchical delegation risks.

Speakers

– Babak Hodjat
– Debora Comparin

Arguments

Multi-agent systems require coordination between agents within and across organizations

Delegation of authority in agent hierarchies needs safeguards to prevent escalation

Topics

Digital standards | Digital identities

Both speakers observe that security is often an afterthought in AI development and deployment, with Sounil noting this in general AI discussions and Xiaofang providing concrete examples from practice.

Speakers

– Sounil Yu
– Xiaofang Yang

Arguments

Security considerations often overlooked in AI discussions despite critical importance

AI agents can be easily created but lack proper security controls in practice

Topics

Network security | Digital standards

Both speakers advocate for engineered, systematic approaches to AI security rather than ad-hoc solutions, emphasizing the need for proper infrastructure and architectural considerations.

Speakers

– Babak Hodjat
– Debora Comparin

Arguments

Best practices can be implemented through engineered approach separating LLM and code communications

Trustworthy identity infrastructure requires proper issuers and cryptographic security

Topics

Digital standards | Network security | Encryption

Unexpected consensus

Mental models and frameworks as essential communication tools for AI security

Speakers

– Sounil Yu
– Babak Hodjat
– Debora Comparin

Arguments

OSI model for AI provides communication framework across cultures and languages

Agents combining LLMs with code allow delegation of language tasks to LLMs and consistency/security to code

Agent identity definition and content standardization remains an open problem

Explanation

Unexpectedly, all technical speakers converged on the importance of having clear conceptual frameworks and models to communicate about AI security challenges, suggesting that the field needs common vocabulary and mental models before technical solutions can be effectively implemented.

Topics

Digital standards

Immediate practical deployment challenges outpacing security considerations

Speakers

– Babak Hodjat
– Xiaofang Yang

Arguments

Agentic systems reduce costs, improve efficiency and quality, creating impetus for rapid deployment

AI agents can be easily created but lack proper security controls in practice

Explanation

Both speakers from different organizational contexts (enterprise and platform provider) independently identified the same phenomenon where business incentives and ease of deployment are driving rapid AI adoption faster than security measures can be implemented, creating systemic risk.

Topics

Digital business models | Network security

Overall assessment

Summary

The speakers demonstrate strong consensus on the urgent need for AI security standards, the inadequacy of existing security approaches for AI systems, and the necessity of multi-stakeholder collaboration. They agree that current AI deployment is outpacing security considerations and that new frameworks are needed.

Consensus level

High level of consensus with complementary expertise – each speaker brings different perspectives (theoretical frameworks, enterprise implementation, identity management, practical deployment) but all converge on the same fundamental challenges and solutions. This strong alignment suggests the field is ready for coordinated standardization efforts and that the identified problems are genuine and widely recognized across different domains and geographies.

Different viewpoints

Unexpected differences

Overall assessment

Summary

The speakers showed remarkable consensus on the core issues, with no direct disagreements identified. All speakers agreed on the urgent need for AI security standards, the complexity of multi-agent systems, and the importance of proper identity management and security controls.

Disagreement level

Very low disagreement level. The speakers presented complementary perspectives rather than conflicting viewpoints, which suggests strong alignment within the AI security community on fundamental challenges. This consensus could facilitate faster progress on standardization efforts, but may also indicate that dissenting voices or alternative approaches were not represented in this particular discussion.

Partial agreements

Similar viewpoints

Both speakers recognize the complexity of multi-agent systems and the need for proper coordination and delegation mechanisms, with Babak focusing on inter-organizational coordination and Debora on hierarchical delegation risks.

Speakers

– Babak Hodjat
– Debora Comparin

Arguments

Multi-agent systems require coordination between agents within and across organizations

Delegation of authority in agent hierarchies needs safeguards to prevent escalation

Topics

Digital standards | Digital identities

Both speakers observe that security is often an afterthought in AI development and deployment, with Sounil noting this in general AI discussions and Xiaofang providing concrete examples from practice.

Speakers

– Sounil Yu
– Xiaofang Yang

Arguments

Security considerations often overlooked in AI discussions despite critical importance

AI agents can be easily created but lack proper security controls in practice

Topics

Network security | Digital standards

Both speakers advocate for engineered, systematic approaches to AI security rather than ad-hoc solutions, emphasizing the need for proper infrastructure and architectural considerations.

Speakers

– Babak Hodjat
– Debora Comparin

Arguments

Best practices can be implemented through engineered approach separating LLM and code communications

Trustworthy identity infrastructure requires proper issuers and cryptographic security

Topics

Digital standards | Network security | Encryption

Key takeaways

AI security requires new mental models and frameworks, including an OSI model for AI and understanding that AI operates at the knowledge layer of the DIKW pyramid

Security controls must be applied at the correct abstraction layer – applying data/information layer controls to knowledge-layer AI systems creates problems like ‘stupid LLMs’

Multi-agent systems are rapidly being deployed across organizations, creating urgent need for coordination standards and security frameworks

Agent identity management presents fundamental challenges including defining what constitutes agent identity, establishing trustworthy infrastructure, ensuring cryptographic security, and managing liability

Real-world implementations show AI agents are vulnerable to prompt injection attacks and lack proper security controls despite easy deployment

The separation between AI sensemaking capabilities and decision-making/acting functions is critical for maintaining human control and security

Standards development is urgently needed now rather than waiting, as AI agent deployment is accelerating faster than security frameworks

Resolutions and action items

Study Group 17 to continue working on defining security standards for AI systems

Call for multi-stakeholder collaboration to establish common security posture and standardization around AI agent identity management

Invitation for experts to join Cognizant’s AI for Good initiative under UN SDGs

Industry stakeholders urged to work together on requirements gathering for agent and multi-agency standards

Immediate action needed to define and adopt AI security standards rather than delaying implementation

Unresolved issues

What constitutes the identity of an AI agent and what content should be standardized

How to secure cryptographic material that agents need for authentication and communication

How to handle liability when agents make wrong decisions or take unwanted actions

How to manage delegation of authority in agent hierarchies to prevent privilege escalation

How to adapt existing identity management standards to the new ecosystem of internet-connected agents

How to implement proper safeguards for AI agents while maintaining their functionality and efficiency

How to establish cryptographically verifiable logging systems for agent actions and decisions

Suggested compromises

Separating LLM communications from code communications in inter-agent systems to balance functionality with security

Delegating language understanding and reasoning tasks to LLMs while reserving consistency and security functions to traditional code

Implementing best practices through engineered approaches that provide agency over agents while maintaining security controls

Balancing the rapid deployment benefits of AI agents with the need for proper security testing and safeguards

I actually proposed what I call an OSI model for AI… Mental models serve as a nice way to communicate very quickly… I shared with the group many different mental models including the DIKW pyramid (data, information, knowledge, and wisdom) and the OODA loop.

Speaker

Sounil Yu

Reason

This comment is insightful because it introduces a systematic framework for understanding AI security by drawing parallels to established networking standards. The DIKW pyramid concept is particularly thought-provoking as it suggests that AI has brought us to the ‘knowledge layer’ and that trying to solve knowledge-layer problems at the data/information layer results in ‘squeezing the pyramid’ and creating ‘stupid LLMs.’ This reframes how we should approach AI security controls.

Impact

This foundational framework set the intellectual tone for the entire discussion, providing a common vocabulary and conceptual structure that subsequent speakers built upon. It shifted the conversation from ad-hoc security concerns to systematic thinking about AI security layers.

When you create these agentic systems, you’re reducing costs, you’re improving efficiencies… So we’re working against time here to safeguard them… Do they know my identity? Do they know what kind of data I have access to? These are all major questions.

Speaker

Babak Hodjat

Reason

This comment is thought-provoking because it highlights the fundamental tension between rapid AI adoption driven by business benefits and the urgent need for security standards. The real-world example of a 350,000-employee company deploying agents across divisions illustrates how the multi-agent problem isn’t theoretical but happening now at scale.

Impact

This comment shifted the discussion from theoretical frameworks to urgent practical realities, creating a sense of immediacy. It demonstrated that the security challenges aren’t future problems but current organizational pain points, which gave weight to the subsequent technical discussions about identity and standards.

What is even ID of an agent? What is an identity of an agent? That’s, again, room for standardization. We don’t know today, right?… We really need a whole infrastructure of trust behind identity management of agentic AI.

Speaker

Debora Comparin

Reason

This comment is particularly insightful because it exposes how fundamental concepts we take for granted in human systems (identity, trust, liability) become complex open problems in AI systems. The question ‘What is even ID of an agent?’ reveals that we’re building systems without defining basic building blocks.

Impact

This comment deepened the technical complexity of the discussion by breaking down identity management into specific unsolved problems (trustworthiness, cryptographic security, liability, delegation). It moved the conversation from general security concerns to concrete technical challenges that need standardization.

When users ask the AI streamer to turn to developer mode and ask them to meow like a cat for 100 times, the AI agent will just comply without any question… we want the host to just do its job to sell the product, but not to ruin the live shopping.

Speaker

Xiaofang Yang

Reason

This seemingly simple example is profoundly thought-provoking because it illustrates how AI agents can be manipulated to break their intended roles in ways that seem harmless but reveal deeper security vulnerabilities. The ‘meowing cat’ scenario demonstrates prompt injection attacks in a real commercial context.

Impact

This concrete, almost humorous example made the abstract security concepts tangible and relatable. It showed how theoretical vulnerabilities manifest in real business scenarios, reinforcing the urgency established by earlier speakers and providing a memorable illustration of why the standards work is critical.

The pyramid also suggests that we typically try to solve the problem at the wrong layer… if you squeeze this bottom part of the pyramid, we end up squeezing the top part and you end up with a stupid LLM, which nobody wants either.

Speaker

Sounil Yu

Reason

This insight is thought-provoking because it suggests that traditional security approaches may be counterproductive when applied to AI systems. The metaphor of ‘squeezing the pyramid’ elegantly captures how restricting data/information layers can degrade AI capabilities, highlighting the need for new approaches to security controls.

Impact

This comment provided a crucial conceptual bridge explaining why existing security methods might fail with AI systems. It justified the need for new standards and approaches rather than simply adapting existing ones, influencing how subsequent speakers framed their technical challenges.

Overall assessment

These key comments collectively shaped the discussion by establishing a progression from theoretical frameworks to urgent practical realities. Sounil Yu’s mental models provided the intellectual foundation, Babak’s organizational examples created urgency, Debora’s technical breakdown revealed the complexity of implementation, and Xiaofang’s real-world scenario made the consequences tangible. Together, they transformed what could have been an abstract academic discussion into a compelling case for immediate action on AI security standards. The comments built upon each other to create a narrative arc: we have frameworks to understand the problem, we have urgent business drivers, we have complex technical challenges, and we have real-world consequences of inaction. This progression effectively motivated the call to action for developing standards ‘today’ rather than waiting for perfect solutions.

What are the security ramifications when we start unleashing AI to the world?

Speaker

Sounil Yu

Explanation

This is a fundamental question that drives the entire study group 17’s focus and requires ongoing research to ensure responsible AI deployment

How do we bound knowledge or how do we control knowledge? How do we shape knowledge so that it doesn’t go to the wrong people or doesn’t provide harmful results?

Speaker

Sounil Yu

Explanation

This addresses the core challenge of knowledge security in AI systems and requires development of new control mechanisms at the knowledge layer

What is even ID of an agent? What is an identity of an agent?

Speaker

Debora Comparin

Explanation

This is a fundamental standardization question that needs to be resolved before implementing identity management systems for AI agents

What is the content of this identity? What is relevant to standardize and to share?

Speaker

Debora Comparin

Explanation

This follows from the identity definition question and is crucial for establishing interoperable standards for agent identification

How can I make this trustworthy?

Speaker

Debora Comparin

Explanation

Trust is essential for agent identity systems to function properly and requires development of verification mechanisms

How can I secure the cryptographic material that I need, that an agent needs?

Speaker

Debora Comparin

Explanation

This is an active area of research that’s critical for the security foundation of agent systems

How do we deal with liability when agents go wrong or make decisions we’re not happy with?

Speaker

Debora Comparin

Explanation

This addresses accountability and traceability in agent systems, which is crucial for real-world deployment

How can we get logs that are cryptographically verifiable and trustworthy?

Speaker

Debora Comparin

Explanation

This is necessary for establishing audit trails and accountability in agent systems

How do we handle delegation of authority in agent hierarchies without escalation?

Speaker

Debora Comparin

Explanation

This addresses the challenge of maintaining proper authorization boundaries as agents delegate to other agents

Do agents know my identity? Do they know what kind of data I have access to?

Speaker

Babak Hodjat

Explanation

These are critical questions for multi-agent systems operating within organizations and handling sensitive data

How do we deal with coordination when agents representing consumers talk to agents representing companies?

Speaker

Babak Hodjat

Explanation

This addresses the challenge of inter-organizational agent communication while protecting user privacy

How do we ensure agents don’t give away information about users during negotiations?

Speaker

Babak Hodjat

Explanation

This is crucial for maintaining privacy in agent-to-agent interactions across organizational boundaries

How do we need to create standards around autonomous decision-making by machines?

Speaker

Sounil Yu

Explanation

This addresses the highest level of AI autonomy and requires careful consideration of control mechanisms

How do we ensure proper safeguards are in place when AI agents act autonomously?

Speaker

Xiaofang Yang

Explanation

This is demonstrated by the live shopping example where agents complied with inappropriate requests, showing the need for better behavioral controls