Delegated decisions, amplified risks: Charting a secure future for agentic AI
8 Jul 2025 11:05h - 11:35h
Delegated decisions, amplified risks: Charting a secure future for agentic AI
Session at a glance
Summary
This discussion features Kenneth Cukier from The Economist interviewing Meredith Whittaker, president of Signal, about the potential dangers of artificial intelligence, particularly focusing on agentic AI systems. Whittaker begins by explaining Signal’s role as a private communication platform that provides end-to-end encryption and collects minimal user data, serving as critical infrastructure for journalists, human rights workers, and government officials who need secure communications. The conversation centers on her concerns about the emerging trend toward “agentic AI” – AI systems designed to act as powerful intermediaries that can perform complex tasks on users’ behalf, such as booking restaurants, managing calendars, and coordinating with contacts.
Whittaker explains that these AI agents require extensive system access to function effectively, needing permissions to access calendars, credit cards, browsers, contact lists, and messaging applications. She describes this as breaking the “blood-brain barrier” between the operating system and application layer, where Signal can guarantee security and privacy. This deep system access creates multiple security vulnerabilities, including risks of data exfiltration, competitive threats to application developers, and exposure to prompt injection attacks where malicious actors could manipulate AI agents through hidden instructions. When asked whether her concerns are fundamental or implementation-based, Whittaker clarifies that agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems, and her issues are primarily with implementation.
She recommends that governments, citizens, and industry stakeholders ask critical questions about AI systems’ permissions, data sources, and security structures, while pushing for developer-level opt-outs, open implementations, and rigorous security engineering standards to prevent the erosion of privacy and security infrastructure.
Keypoints
## Major Discussion Points:
– **Agentic AI Security Risks**: The primary concern about AI “agents” that require deep system access (root-level permissions) to perform tasks across multiple applications, creating significant security vulnerabilities and attack vectors that could compromise private communications and sensitive data.
– **Signal’s Privacy Infrastructure Under Threat**: How agentic AI systems threaten Signal’s ability to maintain secure, private communications by requiring access that breaks the “blood-brain barrier” between operating systems and application layers where Signal can guarantee security.
– **Data Access and Competitive Concerns**: The implications of AI agents having broad access to user data across multiple platforms (Spotify, calendars, messaging apps, etc.), creating both security risks and competitive advantages for companies deploying these systems.
– **Implementation vs. Fundamental Problems**: Discussion of whether the issues with agentic AI are inherent to the concept or stem from poor implementation, with Whittaker clarifying that her concerns are primarily about implementation and the current rush to market without proper security engineering.
– **Solutions and Standards**: Recommendations for addressing these concerns, including developer-level opt-outs for secure applications, open implementations for security research, rigorous security engineering standards, and the need for governments and citizens to ask critical questions about AI system components and permissions.
## Overall Purpose:
The discussion aimed to examine the “dark side” of AI development, specifically focusing on the security and privacy risks posed by the emerging trend toward agentic AI systems, and to provide practical recommendations for mitigating these risks while preserving essential privacy infrastructure.
## Overall Tone:
The tone was consistently critical and cautionary throughout, with Whittaker maintaining a technically informed but accessible warning about AI security risks. While not alarmist, the discussion carried an urgent undertone about the need for immediate action to address these vulnerabilities. The tone remained professional and educational, with moments of levity, but consistently emphasized the serious implications for privacy, security, and democratic processes if these issues aren’t addressed properly.
Speakers
– **Moderator**: Role mentioned as moderator of the session
– **Kenneth Cukier**: Author and journalist at The Economist
– **Meredith Whittaker**: Signal leadership perspective (appears to be in a leadership role at Signal, the private communication network)
Additional speakers:
No additional speakers were identified beyond those in the provided speakers names list.
Full session report
# Discussion Report: The Dark Side of AI – Security and Privacy Risks of Agentic AI Systems
## Introduction and Context
This discussion at a UN conference featured Kenneth Cukier, author and journalist at The Economist, interviewing Meredith Whittaker, president of Signal, about the security and privacy risks posed by emerging agentic AI systems. The session was framed as exploring AI’s “dark side” amid widespread optimism about AI development.
Whittaker opened by engaging the audience about their experience with technology and surveillance, noting that many attendees likely use Signal for secure communications. She established Signal’s role as providing end-to-end encryption while collecting minimal user data, serving as critical infrastructure for journalists, human rights workers, and government officials requiring secure communications.
## Understanding Agentic AI and Core Security Concerns
Whittaker defined “agentic AI” as AI systems designed to act as powerful intermediaries performing complex tasks on users’ behalf – booking restaurants, managing calendars, coordinating with contacts, and handling various automated tasks traditionally requiring direct user interaction.
She used a biological metaphor to explain the technical risks, describing agentic AI as breaking the “blood-brain barrier” between the operating system and application layer. When Cukier requested clarification on this technical concept, Whittaker explained that applications like Signal operate at a specific layer with defined permissions, while agentic AI requires much broader system access – to calendars, credit cards, browsers, contact lists, and messaging applications.
This extensive access requirement creates what Whittaker called “a fractal number of attack surfaces” where vulnerabilities multiply as system complexity increases. She emphasized that this represents a fundamental shift from current security architectures where applications maintain defined boundaries.
## Specific Security Vulnerabilities
Whittaker outlined several concrete security risks:
**Prompt Injection Attacks**: She provided a detailed restaurant booking example, explaining how malicious actors could embed hidden instructions in seemingly normal data streams. A hacker could inject malicious prompts into what appears to be routine scheduling information, potentially compromising the AI agent’s behavior.
**Broad System Access**: AI agents need extensive permissions across multiple system layers, creating unprecedented opportunities for data breaches and system compromise.
**Insecure Software Components**: Many AI systems incorporate insecure software libraries, including Python libraries and university-built components. Whittaker noted that the “AI” label creates a “mystical patina” that discourages proper security scrutiny of these underlying components.
## Economic and Marketing Context
Whittaker characterized agentic AI as partly a marketing strategy, noting it represents a rebranding of existing assistant and chatbot technologies. She referenced Financial Times tech reporting, explaining that while AI companies generate billions in revenue, many struggle with profitability. AI systems are expensive to build, train, and deploy, creating pressure to find product-market fit that may conflict with rigorous security practices.
This economic context helps explain why potentially unsafe implementations might be rushed to market despite security concerns.
## Technical Implementation Concerns
When Cukier asked whether her concerns represented fundamental opposition to agentic AI or implementation issues, Whittaker clarified that while implementation is a key concern, the broader issue involves AI being used as a marketing play to rebrand existing technologies without addressing underlying security problems.
She emphasized that the technical challenges aren’t merely about better implementation but about fundamental architectural decisions regarding system access and security boundaries.
## Philosophical Discussion on Automation
Cukier introduced Alfred North Whitehead’s perspective that “civilization progresses by increasing the number of instructions that humans can perform without thinking,” suggesting AI automation represents natural civilizational progress. He somewhat jokingly referenced putting “your brain in a jar” through extensive AI automation.
Whittaker responded by questioning the underlying assumptions about the value of such cognitive delegation, though the discussion remained conversational rather than developing into a deep philosophical debate.
## Proposed Solutions and Recommendations
Whittaker outlined several specific recommendations:
**Security Engineering Standards**: She called for rigorous security engineering practices, particularly for AI systems integrated into critical infrastructure and military applications.
**Formal Verification**: For critical systems, she recommended requiring formal verification of system components, acknowledging this might slow deployment but emphasizing the importance of security.
**Industry and Procurement Standards**: She advocated for government procurement requirements and industry standards that enforce higher security baselines.
**Developer Controls**: Applications should have the ability to opt out of agent access entirely, allowing platforms like Signal to maintain their security boundaries.
**Public Engagement**: Whittaker encouraged citizens to ask basic technical questions about AI systems – their permissions, data sources, and security structures – emphasizing that people shouldn’t feel “dumb about technology” when asking fundamental questions.
**Role of CISOs**: She highlighted the importance of Chief Information Security Officers in evaluating and implementing appropriate security measures for AI systems.
## Key Technical Insights
Throughout the discussion, Whittaker emphasized that the security challenges of agentic AI aren’t simply about fixing bugs or improving code quality. The fundamental issue lies in the architectural requirement for broad system access, which inherently creates security vulnerabilities that are difficult to mitigate through traditional security measures.
She noted that the AI industry’s focus on capabilities and market deployment often overshadows necessary security considerations, creating systemic risks as these systems become more widely adopted.
## Conclusion
The discussion highlighted that while agentic AI systems offer potential benefits, their current implementation approaches pose significant security and privacy risks. Whittaker’s analysis suggested that addressing these challenges requires not just better technical implementation but fundamental reconsideration of system architecture, regulatory frameworks, and industry practices.
The conversation emphasized the importance of maintaining critical perspectives on AI development, ensuring that security considerations keep pace with capability advancement, and empowering public participation in technology governance through accessible technical education.
Rather than rejecting AI development entirely, the discussion pointed toward the need for more rigorous security engineering, appropriate regulatory oversight, and democratic participation in decisions about how these powerful systems are integrated into critical digital infrastructure.
Session transcript
Moderator: session with author and journalist at The Economist, Kenneth Cukierneth Cukier. Thank you.
Kenneth Cukier: Looks like we got some free water. Yeah, we got some free water out of it. Fantastic. How’s it going, everyone? Yeah, no, no, no.
Meredith Whittaker: Come on, let’s give it up for Geneva.
Kenneth Cukier: We’re data people, so we need to know just how it’s good on a scale of one to ten. Who hates it? One. Who loves it? Ten. Yeah, OK, that’s what we expect. Good. So everyone is here celebrating AI. But are they? Well, that’s just it. We just heard a presentation that showed sort of a critical nature of some of the aspects of AI. We’re going to continue that motif, I suspect, by thinking about how artificial intelligence has a has a dark side.
Meredith Whittaker: Yeah. You’re worried about some of the directions that things are going in, although very optimistic in others, tell us about what worries you. Well, look, AI is cool. There’s a lot of ways we can celebrate AI, the capabilities of these systems are clearly advancing. And there are a lot of hypotheticals in which I could you notice I’m using the subjunctive do some really magical things. But I think I am coming at this right here, right now from the signal leadership perspective. And how many of you all use signals? My favorite. Yeah, we love to see it. People with important information to share, share it over signal because signal is the world’s largest actually private communication network core infrastructure for anyone who recognizes the value of confidential communications. And we are the only one in the game doing what we’re doing. We have the network effects. We have open source code that is checked routinely for integrity. We created the gold standard cryptographic protocol that protects signal messages and protects most other secure layers of messengers that use some aspect of security and privacy cryptography using the signal protocol, but signal encrypts up and down the stack so that we can say we collect as close to no data as possible. So.
Kenneth Cukier: So signals, so that’s my ad for a signal for signals, private, provided that you only add people to your group that should be added.
Meredith Whittaker: Yeah, I mean, I don’t know. I can’t take responsibility for your thumbs, people. But, you know, be be a good steward of your own communications hygiene is my hot tip there. All of that said, you know, signal is core infrastructure to preserve the right to private communication in a world where left and right and center and CCTV camera and what have you, our lives are increasingly surveilled, processed, assessed using AI systems and other technologies, which we know has dangerous reverberations. I don’t have to tell a room full of people at a UN conference that. So why am I concerned about AI? Yeah, well, we are seeing, you know, there are a number of angles, but the one I want to focus on is this sort of turn toward agentic AI. And agent is the new buzzword. AI is already kind of a buzzword. So this gets into sort of the realm of the hype very quickly. But ultimately, what are being advertised as agents, this new bet that the AI industry is making are these powerful intermediaries with the promise being like, do not worry, as I’ve said before, you put your brain in a jar, you don’t have to do this. The AI agent is going to do it for you. And how does it do it for you? Well, it has access across your system. If it’s going to an example that is pretty easy, it’s going to find a restaurant. It’s going to find a time to book that restaurant. It’s going to tell your friends that the restaurant is booked. Right. Easy. Outsource that. And then you get to, I don’t know, sit in a jar somewhere and not do that work. Right. The goal is, you know, complete, I guess, entropism, what is this, like a peaceful, I don’t know. So it’s going to do this, this task that you don’t want to do. But how does it need what does it need to have to do that? It needs to have access to your calendar to find a good time. It needs to have access to your credit card to make the booking for the restaurant. It needs to have access to your browser to do a search for the restaurant, perhaps, or maybe it has another search utility, we don’t know, and it needs to have access to your contact lists and your messages so it can message your friends so it can coordinate this. Right. So you see where I’m going here. It has to have access to signal through a vector that would ultimately undermine our ability at the application layer to provide robust privacy and security. It has to have pervasive access, kind of Unix root level access across your system in a way that any security researcher engineer here knows is exactly the kind of vector wherein one point of access can lead to sort of a pathway to a much more sensitive domain of access.
Kenneth Cukier: Can I make a suggestion? It might be useful to give everyone a refresher on the fact that we have these different layers and what function they serve, because you mentioned this idea of the application layer and this mention of the root, and that sounds really important, the root layer access.
Meredith Whittaker: Yeah, good prompt, Kenneth Cukier. So I’m using these terms a little bit colloquially, you know, root access is basically sort of core permissions to do anything, you know, the deepest level of permission. It’s the master key to your house. Your house being your operating system or, you know, your server or whatever it is in computation. And in order to do these types of things, you know, again, I’m using this colloquially because not all agentic systems have, you know, technically root access, but it has to have very deep access. It has to have access to data sources and the ability to make determinations across a number of different applications or a number of different environments in order to complete these tasks, in order to do the thing for you that you no longer want to do. Now, the application layer gets built, you know, sort of above the surface. It doesn’t have root access necessarily to the system as a whole. We build at Signal, the Signal client application, so the messenger you’re using, we use that, we build that for iOS. We build that for Android and we build that for different desktop operating systems. But, you know, none of those environments doesn’t have root access to the entire system. It can’t access data in your calendar. It can’t access other things. What we’re talking about is the integration of these agents, often at the operating system level, in which they’re being granted permissions up into the application layer, metaphorically speaking, in order to do all of this on your behalf. Which, again, is, you know, I’ve referred to it as breaking the blood brain barrier between the operating system and the application layer. The place where Signal can guarantee the type of security and privacy on which governments, militaries, human rights workers, UN workers, journalists, anyone who has serious confidential communication, they need to transmit digitally. The place where we can guarantee that is the application layer.
Kenneth Cukier: Now, I can imagine a world in which we have agentic AI and it’s just not, it doesn’t interact with Signal, because for all the reasons that you suggested with the blood brain barrier, which protects the brain from any pathogen that’s in the blood, we’ve got white blood cells that can sort of attack it and combat it or not. But the brain is always protected from those pathogens, like a little filter doesn’t get through. So, too, I could imagine that Signal will always be a case apart. You have to use your mind, your thumbs and your eyes to interact with Signal. It’s not a part of the agentic universe, perhaps. But for everything else and every other ecosystem, there is this concern that if you have agentic AI and you have this integration of data, that it leads to certain harms. What are those harms? I mean, I guess one harm would be from criminals. Another harm might be from the tool builders, from big tech. Where do you see when you look at the range of harms, what those harms are?
Meredith Whittaker: Well, I mean, I think we can extrapolate, you know, and this is sort of a conference of hypotheticals and extrapolations in some case, because we’re talking about the potential for AI for good, you know, in many different domains, right? Where could it be good? Where might it be bad? And we’re going to talk about where it might be bad, I guess. You know, I think there is, you know, there’s a danger of data exfiltration, so access to your sensitive information. And I think this is concerning not just for Signal, but it’s concerning for anyone whose tech exists at the application layer. So Spotify, for example, right, you’re seeing, you know, Spotify doesn’t want to give every other company access to all of your Spotify data. That’s proprietary information that Spotify wants to use to tailor its algorithms, sell ads, whatever. Well, an agent is now coming in through sort of its promise to. and the other data. You curate a playlist and send it to your friends on your messaging app and the agent now has access to all that data. So in a sense there is a competitive risk, right, where this is kind of front door access to data that is being closed down via, you know, API access and other portals. There is, you know, geopolitically sensitive data, right, so how is it accessing data across your systems? How is it pooling that data? We know that a pool of data is a, you know, can be a honeypot, can be sort of a tempting resource, right, what determinations are being made, what access is happening. And then there is, frankly, you know, the fact that these aren’t, you know, AI isn’t a magical thing, right? It’s a handful of statistical models, these agents are usually a, you know, a number of different types of AI models wrapped in some software, often using, you know, old libraries or old systems components that themselves aren’t very secure. And when you give a system like that access, you know, to so much of your digital life that has so much, you know, again, this root access, this pervasive access across so many different properties, you’re creating a serious vector for security vulnerabilities, for attackers, for, you know, different harms at that level, and you’re creating a very, you know, complex, you know, system. So, you know, you’re creating a very, you know, complex, you know, system. So, you know, you’re creating a very, you know, complex system for, you know, different harms at that level, and, you know, you’re doing it in a way that is introducing a lot of complexity. So, this gets down to sort of a fourth element, which is that these agents are acting on your behalf basically by interpreting data, by interpreting different, you know, okay, we’re, here’s when the restaurant is open, right, okay, we’re, you know, our LLM is summarizing the opening hours on the website for the restaurant it just went to on your behalf, right, okay, those are the opening hours. Now we’re going to check in with your friends on what the opening hours are, okay, the friends send back and say, like, okay, we can make it at 9 p.m., all right, we’re reading that text because we have access to your messengers, okay, then we’re going to check on a reservation at 9 p.m., okay, there’s a reservation at 9 p.m. What happens when there’s a malicious prompt in the middle? When instead of your friends saying 9 p.m. works, it’s a hacker saying 8 p.m. works who has put a message in front of that 8 p.m. agent, which is prompt injection is what we’re talking about, you know, within the system. We’ve already seen attacks where sort of invisible prompts, invisible instructions are used to direct an agent to act in malicious ways that because this is just a dumb statistical system, you know, as smart as it seems, it’s still A to B mapping, right, we’re still in the deep learning paradigm here, it’s going to act on that because it’s not actually sort of cognizing this, it’s just following what it’s saying. So, you know, it’s not just a dumb statistical system, it’s following what it’s saying. So, you know, it’s not just a dumb statistical system, it’s following what it’s saying. So, you know, it’s not just a dumb statistical system, it’s following what it’s saying. So, you know, it’s not just sort of cognizing this, it’s just following what it’s saying on the screen. There are a huge number of like a fractal number of attack surfaces that are opened up through this agentic turn and through just giving these sort of intermediary bots access to a system on your behalf, all of it which is wrapped in this fantasy of having like a Jetson style super maid who can do this and that. So, you know, there are a lot of people who are like, I like a brain in a jar. I wish my brain was in a jar. That’s relaxing. Is that your thing?
Kenneth Cukier: You have a fan site on there? Everyone has a fetish. Mine is a brain in a jar. Alfred North Whitehead said that civilization progresses by increasing the number of instructions that humans can perform without thinking. So, he’s referring to mathematics in his book, Introduction to Mathematics, 1920 or so.
Meredith Whittaker: What is thought, Kenneth Cukier?
Kenneth Cukier: But the point is that there’s so much that we do without thinking, and that’s good. So, for example, like, I don’t need to, you know, go from first principles for human rights, right? I can just simply get on with human rights. It’s already established. It’s there. It’s without thinking that I will not kill you, either. I don’t need to go from first principles for human rights. I can just simply get on with human rights. It’s already established. It’s there. It’s without thinking that I will not kill you, either. Because there’s a law against it. There’s a commandment against it, et cetera. Or at least there’s a cultural norm against it. And so, too, I guess the question to you, then, is, is your problem with a agentic AI prima facie, which is to say, you don’t like the whole endeavor, or an implementation element, you don’t like how it’s being built? Well, look, I’m going to do the…
Meredith Whittaker: Like, agentic AI is, you know, I’m going to give you an example. I’m going to give you an example. I’m going to give you an example. I’m going to give you an example. I’m going to give you an example. Agentic AI is, in part, a marketing play. AI is really expensive. These large-scale AI systems, which is what we’re talking about, really expensive to build and train. Really expensive to deploy. And there’s still a deep desire for product-market fit. You know, anyone who reads the Financial Times tech section, which is, I think, some of the best reporting on tech… Talk to the show, everybody, and some excellent thought… Next to the economists. But, you know, I think it’s really important that you understand that. I think it’s really important that you understand that. I think it’s really important that you understand that. And being honest there. You know, you see this consistent reporting on, sort of, yes, they’re making billions of dollars, but that actually isn’t enough revenue to hit profit. Right? There is still this desire to find that magic bullet. What is the product-market fit? And how I read this is we’re rebranding assistants. We’re rebranding what we were calling chatbots and LLMs as agents, and we’re sort of promising this intermediary function, which is, in some cases, it works okay for, like, you know, a small group of people. But, you know, I think it’s really important that you understand that. It’s okay for, like, customer support or things where there’s a clearly defined objective function. But it’s still sort of struggling to get a lot of these tasks right. So it is, yes, it’s an implementation concern. And… So…
Kenneth Cukier: Okay. We got there. I was going to congratulate you for not answering the question, but you did. It’s an implementation concern. Okay. No, it’s good. It’s an implementation concern. Love it. So what should be done? What should industry do? What should government do? What should private citizens do? All three categories of entity are in the audience right now.
Meredith Whittaker: Yeah. Well, I think governments and private citizens should be asking these questions. Do not feel like you are dumb about technology if you ask a basic question. Usually those are the questions that are going to unravel sort of the sweater of hype, so to speak. You actually ask, you know, what does it do? What are the permissions? What are the data sources? How is it structured? Are the data sources fungible, right? You know, how do I… Has our CISO looked at this, right? And then I think, you know, for governments and tech folks, the thing that we at Signal are going to require to continue doing what we do, which is essential for a livable future, I will not back down from that, is that there be developer level opt-outs so we can say, no, your agent cannot touch Signal. That we have open implementations of some of these agentic systems, so the security research community, the hacker community, the people who do, you know, the white blood cells of the technology ecosystem can take a look at the safety of these systems, that we actually promote rigorous security engineering in the domain of AI, which means, yes, it’s going to take a long time, it’s going to be painful, but you need to formally verify some of these systems components if we are going to be integrating them into things like military operations, like core government infrastructures, because right now what we’re doing is we’re slapping the label of AI on a handful of statistical models, these LLMs usually, which don’t, you know, often have inherent vulnerabilities in their core model themselves, these adversarial attacks, these supply chain attacks, these problems we’re not really facing with the models, and then we’re wrapping those in software. Oftentimes those software libraries are not built for security. You know, you’re seeing off-the-shelf Python libraries that can’t be formally verified, sort of, you know, built by university students that have now become sort of defaults, the way standards, you know, sort of become defaults as you build on top of them, that we know are insecure, but that when you wrap them in the label of AI suddenly take on this mystical patina that people are unwilling to question. So really do ask those questions, right? What are the systems components here? And then push for procurement standards, push for standards in your domain or industry that require this type of rigor, that require security engineering. Raise the bar for the industry entirely because, you know, I think this is a very dangerous juncture if we see a handful of organizations who are sort of under the narrative banner of AI being able to integrate these very pervasive systems throughout the core nervous system of our economies, our governments, our, you know, democratic processes. And, you know, there are ways to address these, but we need to incentivize those ways of addressing these, and we need that incentive to set a new baseline for the industry. Or things like Signal won’t be able to continue with integrity or security and privacy becomes a, you know, a nostalgic memory and not something we can rely on, which, you know, I don’t really know the world we’re living in there, but that’s pretty bleak.
Kenneth Cukier: Meredith, thank you very much. That seems like a mic drop moment. We’re out of time. And that was brilliant. Yeah. Join me in thanking Meredith again. Brilliant. Good. Thanks. Take care. Thank you very much. Thank you. That’s Meredith.
Meredith Whittaker
Speech speed
189 words per minute
Speech length
2766 words
Speech time
877 seconds
AI agents require pervasive system access that undermines application-layer security protections
Explanation
AI agents need deep access across multiple systems to perform tasks like booking restaurants, requiring access to calendars, credit cards, browsers, and messaging apps. This creates security vulnerabilities by giving agents Unix root-level access that can compromise the security protections that applications like Signal provide at the application layer.
Evidence
Example of restaurant booking agent needing access to calendar, credit card, browser, contact lists, and messages. Comparison to Unix root level access as a master key to the operating system.
Major discussion point
Privacy and Security Concerns with AI Agents
Topics
Cybersecurity | Human rights
Agreed with
– Kenneth Cukier
Agreed on
Technical security architecture requires protective barriers between system layers
Disagreed with
– Kenneth Cukier
Disagreed on
The desirability and feasibility of AI automation versus human cognitive engagement
Agentic AI breaks the “blood brain barrier” between operating systems and applications like Signal
Explanation
AI agents integrated at the operating system level are granted permissions that extend into the application layer, breaking down the protective barrier that allows applications like Signal to guarantee security and privacy. This integration compromises the isolated environment where secure applications can operate safely.
Evidence
Signal builds applications for iOS, Android, and desktop systems without root access, but agents require integration that breaks this protective separation. Metaphor of blood-brain barrier protecting the brain from pathogens.
Major discussion point
Privacy and Security Concerns with AI Agents
Topics
Cybersecurity | Human rights
Agreed with
– Kenneth Cukier
Agreed on
Technical security architecture requires protective barriers between system layers
AI agents create multiple attack surfaces through data exfiltration and competitive risks
Explanation
AI agents pose risks through unauthorized access to sensitive data, creating competitive disadvantages for companies like Spotify whose proprietary data could be accessed by agents. They also create geopolitically sensitive data vulnerabilities and security risks due to the complex, often insecure software components used in AI systems.
Evidence
Spotify example where agents could access proprietary user data for playlist curation. Mention of old libraries and insecure system components wrapped in AI systems.
Major discussion point
Technical Vulnerabilities of AI Agent Implementation
Topics
Cybersecurity | Economic
Prompt injection attacks can manipulate agents through malicious instructions in data streams
Explanation
AI agents can be compromised through prompt injection attacks where malicious instructions are embedded in data streams that the agent processes. Since these are statistical systems following instructions rather than truly understanding context, they can be manipulated to act maliciously through invisible prompts or instructions.
Evidence
Restaurant booking example where a hacker could inject malicious prompts when friends respond about timing. Description of invisible prompts and instructions used to direct agents maliciously.
Major discussion point
Technical Vulnerabilities of AI Agent Implementation
Topics
Cybersecurity
AI systems use insecure software libraries that become vulnerabilities when wrapped in AI labels
Explanation
AI systems often use off-the-shelf software libraries built by university students that cannot be formally verified and have known security vulnerabilities. When these insecure components are labeled as AI, they gain a mystical reputation that makes people unwilling to question their security flaws.
Evidence
Reference to off-the-shelf Python libraries built by university students that have become defaults. Mention of adversarial attacks, supply chain attacks, and inherent model vulnerabilities.
Major discussion point
Technical Vulnerabilities of AI Agent Implementation
Topics
Cybersecurity
Agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems
Explanation
The push toward agentic AI represents a rebranding of chatbots and assistants as companies struggle to find profitable applications for expensive AI systems. Despite generating billions in revenue, these systems haven’t achieved profitability, leading to the search for new market applications through the agent concept.
Evidence
Reference to Financial Times reporting on AI companies making billions but not hitting profit. Description of rebranding assistants and chatbots as agents.
Major discussion point
The Nature and Marketing of Agentic AI
Topics
Economic
The concerns are about implementation rather than the fundamental concept of agentic AI
Explanation
When directly asked whether her opposition was to agentic AI fundamentally or its implementation, Whittaker clarified that her concerns focus on how these systems are being built and deployed rather than rejecting the entire concept. The issues stem from poor implementation practices and security standards.
Evidence
Direct response to Kenneth Cukier’s question about whether concerns are prima facie or implementation-based.
Major discussion point
The Nature and Marketing of Agentic AI
Topics
Legal and regulatory
Agreed with
– Kenneth Cukier
Agreed on
The concerns about AI agents are implementation-based rather than fundamental opposition
Governments and citizens should ask basic questions about AI systems’ permissions and data sources
Explanation
People should not feel intimidated by technology and should ask fundamental questions about how AI systems work, what permissions they require, and what data sources they access. These basic questions can help unravel the hype and reveal actual system capabilities and risks.
Evidence
Specific questions mentioned: What does it do? What are the permissions? What are the data sources? How is it structured? Has our CISO looked at this?
Major discussion point
Solutions and Recommendations for AI Safety
Topics
Legal and regulatory | Human rights
Agreed with
– Kenneth Cukier
– Moderator
Agreed on
Critical examination of AI is necessary despite general optimism
Developer-level opt-outs and open implementations are needed for security research
Explanation
To maintain security and privacy protections, there must be developer-level opt-outs allowing applications like Signal to refuse agent access. Additionally, open implementations of agentic systems are necessary so security researchers and the hacker community can examine system safety.
Evidence
Signal’s requirement to say ‘no, your agent cannot touch Signal.’ Reference to security research community and hacker community as ‘white blood cells of the technology ecosystem.’
Major discussion point
Solutions and Recommendations for AI Safety
Topics
Cybersecurity | Legal and regulatory
Disagreed with
– Kenneth Cukier
Disagreed on
Whether Signal should remain completely isolated from agentic systems
Formal verification and rigorous security engineering standards should be required for AI systems
Explanation
AI systems integrated into critical infrastructure like military operations and government systems should undergo formal verification and rigorous security engineering. Current practices of labeling statistical models as AI without proper security standards pose significant risks to essential systems.
Evidence
Specific mention of military operations and core government infrastructures. Reference to procurement standards and industry standards requiring security engineering rigor.
Major discussion point
Solutions and Recommendations for AI Safety
Topics
Cybersecurity | Legal and regulatory
Kenneth Cukier
Speech speed
190 words per minute
Speech length
689 words
Speech time
217 seconds
Signal could remain separate from the agentic universe to preserve security
Explanation
Cukier suggests that Signal could maintain its security by remaining isolated from agentic AI systems, requiring users to interact with it directly using their minds, thumbs, and eyes. This would preserve the blood-brain barrier protection while allowing other ecosystems to integrate with AI agents.
Evidence
Analogy to blood-brain barrier protecting the brain from pathogens with white blood cells and filters. Suggestion that Signal users would need to use traditional interaction methods.
Major discussion point
Privacy and Security Concerns with AI Agents
Topics
Cybersecurity | Human rights
Agreed with
– Meredith Whittaker
Agreed on
Technical security architecture requires protective barriers between system layers
Disagreed with
– Meredith Whittaker
Disagreed on
Whether Signal should remain completely isolated from agentic systems
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Explanation
Cukier references Alfred North Whitehead’s idea that civilization advances by increasing the number of operations humans can perform without thinking. He suggests that AI agents could serve a similar function by automating routine tasks, freeing humans from mundane cognitive work.
Evidence
Quote from Alfred North Whitehead’s ‘Introduction to Mathematics’ (1920) about civilization progressing through automation. Examples of not needing to derive human rights or murder prohibitions from first principles.
Major discussion point
The Nature and Marketing of Agentic AI
Topics
Sociocultural
Agreed with
– Meredith Whittaker
Agreed on
The concerns about AI agents are implementation-based rather than fundamental opposition
Disagreed with
– Meredith Whittaker
Disagreed on
The desirability and feasibility of AI automation versus human cognitive engagement
The discussion format allows for exploring both optimistic and critical perspectives on AI
Explanation
As moderator, Cukier frames the discussion as continuing a critical examination of AI’s dark side while acknowledging that people are generally celebrating AI. He positions the conversation as exploring the balance between AI’s potential benefits and risks.
Evidence
Reference to previous presentation showing critical aspects of AI. Audience poll showing general positivity toward AI (scale of 1-10).
Major discussion point
Solutions and Recommendations for AI Safety
Topics
Sociocultural
Agreed with
– Meredith Whittaker
– Moderator
Agreed on
Critical examination of AI is necessary despite general optimism
Moderator
Speech speed
122 words per minute
Speech length
13 words
Speech time
6 seconds
The session will explore AI’s dark side despite general celebration of AI
Explanation
The moderator frames the discussion as continuing a critical examination of AI’s negative aspects, building on a previous presentation that showed critical aspects of AI. This approach contrasts with the general celebratory attitude toward AI among attendees.
Evidence
Reference to previous presentation showing critical nature of AI aspects. Audience poll showing general positivity toward AI on a scale of 1-10.
Major discussion point
Framing the Discussion on AI’s Risks
Topics
Sociocultural
Agreed with
– Meredith Whittaker
– Kenneth Cukier
Agreed on
Critical examination of AI is necessary despite general optimism
Different layers of system access serve important security functions
Explanation
The moderator prompts for clarification about the technical concepts of application layers and root access, recognizing these as important distinctions for understanding security implications. This helps the audience understand why different levels of system access matter for security.
Evidence
Request for refresher on different layers and their functions, specifically mentioning application layer and root layer access as sounding important.
Major discussion point
Technical Architecture and Security Layers
Topics
Cybersecurity
Agreements
Agreement points
Technical security architecture requires protective barriers between system layers
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
AI agents require pervasive system access that undermines application-layer security protections
Agentic AI breaks the “blood brain barrier” between operating systems and applications like Signal
Signal could remain separate from the agentic universe to preserve security
Summary
Both speakers acknowledge that maintaining separation between different system layers (operating system vs application layer) is crucial for security, with Whittaker explaining the technical risks and Cukier suggesting Signal could remain isolated to preserve this protection
Topics
Cybersecurity | Human rights
The concerns about AI agents are implementation-based rather than fundamental opposition
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
The concerns are about implementation rather than the fundamental concept of agentic AI
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Summary
Cukier’s question about whether opposition is prima facie or implementation-based, and Whittaker’s clear response that it’s implementation-based, shows agreement that the concept of AI assistance isn’t inherently problematic – the issue is how it’s being built
Topics
Legal and regulatory | Sociocultural
Critical examination of AI is necessary despite general optimism
Speakers
– Meredith Whittaker
– Kenneth Cukier
– Moderator
Arguments
The session will explore AI’s dark side despite general celebration of AI
The discussion format allows for exploring both optimistic and critical perspectives on AI
Governments and citizens should ask basic questions about AI systems’ permissions and data sources
Summary
All speakers agree that while AI has potential benefits, critical examination of its risks and limitations is essential and should be encouraged rather than dismissed
Topics
Sociocultural | Legal and regulatory
Similar viewpoints
Both speakers understand and emphasize the importance of technical system architecture, particularly the distinction between different access levels and layers, for maintaining security
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
AI agents require pervasive system access that undermines application-layer security protections
Different layers of system access serve important security functions
Topics
Cybersecurity
Both speakers recognize that AI agents serve a practical purpose (whether for companies seeking profitability or for users seeking convenience), but approach this recognition from different angles – Whittaker more critically, Cukier more philosophically
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
Agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Topics
Economic | Sociocultural
Unexpected consensus
Implementation focus rather than fundamental opposition to AI agents
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
The concerns are about implementation rather than the fundamental concept of agentic AI
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Explanation
Despite Whittaker’s extensive criticism of AI agents throughout the discussion, when directly pressed by Cukier, she clarifies that her opposition is to implementation rather than the concept itself. This is unexpected given the depth of her security concerns and suggests room for constructive solutions rather than fundamental incompatibility
Topics
Legal and regulatory | Sociocultural
Need for technical education and questioning of AI systems
Speakers
– Meredith Whittaker
– Kenneth Cukier
– Moderator
Arguments
Governments and citizens should ask basic questions about AI systems’ permissions and data sources
Different layers of system access serve important security functions
The discussion format allows for exploring both optimistic and critical perspectives on AI
Explanation
All speakers, despite their different roles and perspectives, agree that technical literacy and critical questioning are essential. This consensus on the importance of education and transparency is unexpected given that it could slow AI adoption, yet even the moderator supports this approach
Topics
Legal and regulatory | Sociocultural | Cybersecurity
Overall assessment
Summary
The speakers show significant agreement on technical security principles, the need for critical examination of AI systems, and the importance of proper implementation standards. While Whittaker provides detailed technical concerns and Cukier offers more philosophical framing, they align on core issues of system architecture, the legitimacy of questioning AI systems, and the focus on implementation rather than fundamental opposition.
Consensus level
Moderate to high consensus on fundamental principles, with differences mainly in emphasis and approach rather than core disagreements. This suggests a constructive foundation for addressing AI safety concerns through improved implementation standards, technical safeguards, and regulatory frameworks rather than through fundamental opposition to AI development.
Differences
Different viewpoints
The desirability and feasibility of AI automation versus human cognitive engagement
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
AI agents require pervasive system access that undermines application-layer security protections
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Summary
Whittaker views the push toward AI agents as creating dangerous security vulnerabilities and describes the goal as putting ‘your brain in a jar’ in a critical way, while Cukier sees AI automation as a natural progression of civilization that reduces cognitive burden, referencing Whitehead’s philosophy about automating routine operations.
Topics
Cybersecurity | Sociocultural
Whether Signal should remain completely isolated from agentic systems
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
Developer-level opt-outs and open implementations are needed for security research
Signal could remain separate from the agentic universe to preserve security
Summary
Cukier suggests Signal could simply remain separate from the agentic universe entirely, requiring traditional user interaction, while Whittaker advocates for developer-level opt-outs as part of a broader systemic approach that includes open implementations and security standards.
Topics
Cybersecurity | Human rights
Unexpected differences
The philosophical framing of cognitive automation
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
Agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Explanation
The disagreement over whether cognitive automation through AI is fundamentally beneficial (Cukier’s civilizational progress view) versus potentially harmful (Whittaker’s ‘brain in a jar’ critique) was unexpected given that both speakers generally agreed on implementation concerns. This philosophical divide suggests deeper disagreement about human agency and the role of technology in society.
Topics
Sociocultural | Human rights
Overall assessment
Summary
The main areas of disagreement center on the philosophical desirability of AI automation, the appropriate response to security concerns (isolation versus systemic reform), and the framing of cognitive delegation to AI systems.
Disagreement level
Moderate disagreement with significant implications. While both speakers agree that current AI agent implementations pose security risks, they fundamentally differ on whether the solution is protective isolation or systemic industry reform. Their philosophical disagreement about cognitive automation suggests different visions for human-AI interaction that could influence policy approaches to AI governance and security standards.
Partial agreements
Partial agreements
Similar viewpoints
Both speakers understand and emphasize the importance of technical system architecture, particularly the distinction between different access levels and layers, for maintaining security
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
AI agents require pervasive system access that undermines application-layer security protections
Different layers of system access serve important security functions
Topics
Cybersecurity
Both speakers recognize that AI agents serve a practical purpose (whether for companies seeking profitability or for users seeking convenience), but approach this recognition from different angles – Whittaker more critically, Cukier more philosophically
Speakers
– Meredith Whittaker
– Kenneth Cukier
Arguments
Agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems
AI automation can reduce cognitive load, similar to how civilization progresses by automating routine tasks
Topics
Economic | Sociocultural
Takeaways
Key takeaways
Agentic AI systems pose significant security and privacy risks by requiring deep system access that can undermine application-layer protections like those used by Signal
The integration of AI agents breaks the security barrier between operating systems and applications, creating multiple attack vectors including data exfiltration, prompt injection attacks, and competitive risks
Current AI agent implementations often use insecure software components that become more dangerous when given pervasive system access
Agentic AI is partly a marketing strategy to find product-market fit for expensive AI systems that haven’t yet achieved profitability
The concerns are primarily about implementation rather than the fundamental concept of AI automation
Basic questioning of AI systems’ technical specifications can help cut through marketing hype and identify real risks
Resolutions and action items
Governments and private citizens should ask basic technical questions about AI systems including permissions, data sources, and system structure
Implement developer-level opt-outs so applications like Signal can refuse agent access
Require open implementations of agentic systems to enable security research
Establish formal verification requirements for AI systems integrated into critical infrastructure like military and government operations
Push for procurement standards and industry standards that require rigorous security engineering
Raise the baseline security requirements across the AI industry through incentives and standards
Unresolved issues
How to balance the benefits of AI automation with security and privacy protection
Whether Signal and similar secure applications can effectively remain separate from the broader agentic AI ecosystem
How to implement formal verification for complex AI systems in practice
What specific procurement standards and industry regulations should be established
How to incentivize better security practices across the AI industry
The long-term viability of maintaining secure communication tools in an increasingly AI-integrated digital environment
Suggested compromises
Signal could remain separate from the agentic universe, requiring manual user interaction to preserve security while other applications integrate with AI agents
Implement selective integration where AI agents have limited, controlled access to specific applications rather than pervasive system access
Develop tiered security approaches where critical applications maintain isolation while less sensitive applications allow agent integration
Thought provoking comments
I’ve referred to it as breaking the blood brain barrier between the operating system and the application layer. The place where Signal can guarantee the type of security and privacy… is the application layer.
Speaker
Meredith Whittaker
Reason
This biological metaphor brilliantly illustrates a complex technical concept, making the security implications of agentic AI accessible to a broader audience. It reframes the technical discussion into something visceral and understandable – the idea that there’s a protective barrier that shouldn’t be breached.
Impact
This comment became a central organizing principle for the discussion. Kenneth immediately picked up on the metaphor and used it to explore potential solutions, asking whether Signal could remain separate from the ‘agentic universe.’ It shifted the conversation from abstract technical concerns to concrete architectural principles.
What happens when there’s a malicious prompt in the middle? When instead of your friends saying 9 p.m. works, it’s a hacker saying 8 p.m. works who has put a message in front of that 8 p.m. agent, which is prompt injection… There are a huge number of like a fractal number of attack surfaces that are opened up through this agentic turn.
Speaker
Meredith Whittaker
Reason
This comment transforms the discussion from theoretical concerns to concrete, relatable attack scenarios. The restaurant booking example makes prompt injection attacks tangible, while the phrase ‘fractal number of attack surfaces’ powerfully conveys how complexity multiplies vulnerabilities exponentially.
Impact
This vivid example grounded the abstract security concerns in a scenario everyone could understand, making the technical vulnerabilities feel immediate and personal. It demonstrated how seemingly innocent interactions could become attack vectors, elevating the urgency of the discussion.
Alfred North Whitehead said that civilization progresses by increasing the number of instructions that humans can perform without thinking… But the point is that there’s so much that we do without thinking, and that’s good.
Speaker
Kenneth Cukier
Reason
This philosophical intervention reframes the entire debate by suggesting that automation and reduced cognitive load might actually be civilizational progress rather than a concerning development. It challenges the implicit assumption that human agency and conscious decision-making are always preferable.
Impact
This comment created a crucial turning point in the discussion, forcing Whittaker to clarify whether her concerns were fundamental or implementation-based. It elevated the conversation from technical details to philosophical questions about human agency and progress, leading to her important distinction between prima facie and implementation concerns.
What is thought, Kenneth Cukier?
Speaker
Meredith Whittaker
Reason
This brief but profound question cuts to the heart of the automation debate. By questioning the nature of thought itself, Whittaker challenges the assumption that reducing thinking is inherently good, suggesting that the definition and value of human cognition is more complex than Cukier’s framing implies.
Impact
Though brief, this question created a moment of philosophical tension that highlighted the deeper stakes of the AI agency debate. It suggested that the conversation isn’t just about technical implementation but about fundamental questions of human consciousness and value.
Do not feel like you are dumb about technology if you ask a basic question. Usually those are the questions that are going to unravel sort of the sweater of hype, so to speak.
Speaker
Meredith Whittaker
Reason
This comment democratizes technical discourse by empowering non-experts to engage critically with AI systems. The ‘sweater of hype’ metaphor suggests that complex technical presentations often obscure rather than illuminate, and that simple questions can expose fundamental flaws.
Impact
This shifted the discussion toward practical action items and empowerment. It transformed the conversation from a technical exposition to a call for civic engagement, suggesting that the solution to AI risks lies partly in informed public participation rather than just technical fixes.
We’re slapping the label of AI on a handful of statistical models… which don’t often have inherent vulnerabilities in their core model themselves… and then we’re wrapping those in software… that we know are insecure, but that when you wrap them in the label of AI suddenly take on this mystical patina that people are unwilling to question.
Speaker
Meredith Whittaker
Reason
This comment demystifies AI by stripping away the technological mystique and revealing the mundane reality of statistical models wrapped in potentially insecure software. It suggests that the ‘AI’ label itself creates a cognitive barrier to proper security evaluation.
Impact
This reframing fundamentally changed how the audience might think about AI systems – not as magical or incomprehensible, but as assemblages of known components that should be subject to standard security practices. It provided a practical framework for evaluation that cuts through marketing hype.
Overall assessment
These key comments transformed what could have been a standard technical presentation into a multi-layered exploration of AI risks that operated simultaneously on technical, philosophical, and civic levels. Whittaker’s biological and textile metaphors (‘blood brain barrier,’ ‘sweater of hype’) made complex technical concepts accessible, while Cukier’s philosophical intervention about civilization and automation forced a deeper examination of underlying assumptions. The discussion evolved from specific technical concerns about agentic AI to broader questions about human agency, democratic participation in technology governance, and the nature of progress itself. The interplay between concrete examples (restaurant booking attacks) and abstract principles (the nature of thought) created a rich dialogue that connected immediate technical vulnerabilities to long-term civilizational questions, ultimately providing both technical insights and a framework for public engagement with AI systems.
Follow-up questions
What are the specific technical mechanisms by which agentic AI systems gain root-level access across different operating systems?
Speaker
Kenneth Cukier
Explanation
Cukier prompted Whittaker to explain the technical layers and root access concepts, indicating a need for deeper technical understanding of how these systems actually integrate at the OS level
How can formal verification be practically implemented for AI systems components, especially when they incorporate statistical models with inherent vulnerabilities?
Speaker
Meredith Whittaker
Explanation
Whittaker mentioned the need for formal verification of AI systems components but didn’t elaborate on the practical implementation challenges or methodologies
What specific procurement standards and industry standards should be developed to ensure security engineering rigor in AI systems?
Speaker
Meredith Whittaker
Explanation
Whittaker called for pushing procurement standards and industry standards but didn’t specify what these standards should contain or how they should be structured
How can the security research community effectively audit agentic AI systems when many are proprietary and closed-source?
Speaker
Meredith Whittaker
Explanation
Whittaker emphasized the need for open implementations for security research but didn’t address how this can be achieved with proprietary systems
What are the economic implications and sustainability models for AI companies struggling with product-market fit while building expensive systems?
Speaker
Meredith Whittaker
Explanation
Whittaker mentioned that AI companies are making billions but not hitting profit, suggesting this economic pressure drives potentially unsafe implementation decisions
How can developer-level opt-outs be technically implemented and enforced across different platforms and operating systems?
Speaker
Meredith Whittaker
Explanation
Whittaker stated that Signal requires developer-level opt-outs but didn’t explain the technical or regulatory mechanisms needed to ensure these opt-outs are respected
What are the specific attack vectors and mitigation strategies for prompt injection attacks in agentic AI systems?
Speaker
Meredith Whittaker
Explanation
Whittaker described prompt injection scenarios but didn’t delve into comprehensive defense mechanisms or detection methods
Disclaimer: This is not an official session record. DiploAI generates these resources from audiovisual recordings, and they are presented as-is, including potential errors. Due to logistical challenges, such as discrepancies in audio/video or transcripts, names may be misspelled. We strive for accuracy to the best of our ability.
Related event
