Hello everybody. Welcome to the World Economics Forum’s town hall on dilemmas around ethics and AI. My name is Matt Honan.

I’m the editor-in-chief of MIT Technology Review. We’re joined today by Meredith Whittaker, who’s the president of the Signal Foundation and the co-founder of the AI Now Institute. Max Tegmark, an MIT professor who teaches on AI and also is the co-founder of the Future of Life Institute.

And Rachel Botsman, an author, artist, and expert on trust in the digital age and an associate fellow at Oxford University. Welcome everybody. Thanks for joining us.

This is designed to be a town hall setting. We want you to be involved in all of this. We’d like for you to ask questions.

We’re going to have polls that you can answer. We really want this to be an interactive section and we hope that you will join us. And so to start with, you’ll see up here a QR code that you can all look at and scan with your phones.

It will launch something called Slido. We’ve got a poll to kick off for all of you that asks, how much do you trust AI systems in your daily lives? And so that’s for you to answer, but I’d like for each of you here on our panel to answer something for me, which is maybe you can tell me what you see as the most urgent ethical challenge facing us today when it comes to AI.

Rachel, would you like to kick us off? Yeah, I

actually think that poll question is a terrible question. Oh no. Sorry.

I think it’s part of the challenge and I’ve listened to it all week because… Trust is, it’s useless without context. So that question, do you trust AI in your life?

I would say to do what? And you can’t really have an ethical debate until you think about context. And this is missing from so many conversations.

You know, there’s certain things I definitely trust AI to do. And then there’s things I don’t. And until you think about context, it’s really hard to frame issues.

So please do answer the polling question because I’d be interested to hear the results. But I thought what I could bring is I could talk about trust in systems and trust in platforms and talk about this at a societal level. But the thing I’m actually really passionate about is I write and I make art.

And so I’m very connected to the hand-mind connection. And one of the things that worries me is as human beings, we’re very bad at recognizing when we’re outsourcing thinking. And when that thinking becomes so efficient that we literally stop thinking.

And so cognitive atrophy, when that sets in, I already see it in peers and in students, not just the speed at which they work, the speed at which they expect answers, but it’s also their tolerance for friction, their ability to sit with doubt and uncertainty.

And this worries me for someone whose job it is to think, but also because to make. So I know that when you make art, AI can do incredible things. I use it in my work to model things, to get things out my head.

But that hand-mind connection is so important. So when that goes and we outsource that to something that is deciding what we care about, what we’re interested in, what we should be curious about, which at the end of the day, it doesn’t care about us.

It does not care about us, but it’s shaping those decisions. I really worry about that. I really worry about that in children, how we don’t lose that connection, and how, as human beings, we do a much better job at parsing when is it great for outsourcing thinking, because it frees up our cognitive abilities and creates that space for many wonderful things and solves problems, and when does it create cognitive atrophy.

So that’s my concern.

Max?

If we set different rules for people of different skin color, that would be viewed as very, very unethical, and I feel we’re doing exactly that with companies from different industries now. If you are a drug company, for example, and you want to sell a new kind of pill which might increase suicidal ideation in teenagers, you have rules. You’re not even allowed to sell your first pill until you’ve done a clinical trial and measured the suicidal ideation, and if it’s problematic, you go back to the drawing board.

Yet, it’s completely legal in America now for a company to sell an AI girlfriend to 12-year-olds, even though we know that that has increased suicidal ideation and caused many, many young kids to take their own lives.

So it’s a massive discrimination, again, where somehow, we’ve decided that AI as an industry in America is actually less regulated even than sandwich shops. You know, if you want to open a cafe, before you can sell your first sandwich, you have to have the health inspector come in and check, and if he finds 53 rats in your kitchen, you’re not selling any sandwiches today.

But if you then turn around and say, you know, don’t worry, I’m not gonna sell any sandwiches. I’m just gonna sell chatbots, AI girlfriends for 12-year-olds, or I’m just gonna build super intelligence, which some of the AI companies are stating openly as their goal while they’re saying, you know, Elon Musk was on a stage last month.

saying, oh, you know, it’s going to be the machines in charge, not the people. And other CEOs have talked about, we’re going to build a new robot species, which is going to replace this. Oh, no problem, as long as you don’t sell sandwiches.

So I think we urgently need to cut this corporate welfare and start treating the AI industry in the same way that we treat any other industry with binding safety standards.

Meredith?

Yeah, you know, I think the term ethical here is kind of a dog whistle that means these things are going to be treated as a sort of afterthought. And the fundamental work of building these systems, that’s the serious work. And then let’s come with some paintbrushes and paint some smiley faces on the side of the bomber.

And I think that that framing kind of tells us where we are in some sense. Because what we’re talking about or what I’m talking about, what I think we’re looking at are fundamental limitations, fundamental risks, and real issues of who and how and where are these systems being created and who gets to determine on whom they are used, how they are used, what they do.

And these aren’t afterthoughts, right? They’re fundamental foundations on which the house of AI is being built.

Can you talk to me a little bit what those fundamental risks are? Like, what do they entail for us?

I mean, I’m not going to stack rank them, but I think very quickly, if we look at what the AI industry is right now, it’s a monoculture. It is focused almost exclusively, at least in the Davos-level popular discourse, on deep learning-based language-centric LLMs. That’s one approach to AI that over its 80-year history has shown up a couple of times, but there are many, many other approaches to using data and using computational systems to create AI approaches, right?

The reason deep learning-based, you know, these sort of LLM-centric, like, models are now everywhere is due to a business model that emerged out of the commercialization of computation of the internet in the 1990s, enabled large platforms to build sort of monopolistic communications and platform networks that, you know, and, you know, communication networks are natural monopolies.

We know this from the telephone. We know this from the telegraph. You know, these forms are no different.

Build these large sort of data-based, you know, kind of advertising-based platforms that gathered a huge amount of data and that built out infrastructures to process and use that data. So, indexing ads, you know, indexing search, you know, the Facebook sort of social media model. So, why am I even talking about that?

That’s old news. We’re in the AI era, right? I’m talking about that because that those are the conditions in which deep learning became newly interesting.

In the early 2010s, before we were all still calling it machine learning, we there was a recognition that old techniques from the 1980s and the early 90s could do new things that were newly interesting, particularly to indexing social media feeds and sort of optimizing for engagement when they were matched with those resources, data and compute.

What is new in this era are not the deep learning centric approaches. It is the access to huge amounts of data and huge amounts of compute that are pooled in the hands of a handful of actors due to their establishing their network effects, their economies of scale in the 1990s and 2000s.

So, this is a form that is contingent on a very particular political economic, you know, formation. It’s not Europe’s sure fit of regulation that’s the problem. It’s not a lack of innovation.

What we’re talking about is a business model and then affordances built on top of that. via data and infrastructure monopolies. So when we talk about ethical issues, we can point to that, but that’s not really an ethical issue.

That’s fundamental to what we’re talking about and to what I would call kind of narrative hijacking that has meant that every time we’re talking about AI, we’re basically talking about this very particular set of, this very particular approach that has sort of swallowed up the entire discourse and that is being treated as a natural progression of scientific and human progress, not a contingency of a business model we should be very skeptical of, given the centralized power over our infrastructure, given the centralized power over our ideological, historical, and information landscape, and now given the power over our institutional and organizational decision-making that is being slowly then quickly seeded as we integrate these models more deeply into our corporations, our governments, and our institutions.

So maybe that would be my answer.

Thank you.

Can I pick up on something?

Yeah, please, go ahead.

I think it’s really important. In these conversations, there’s such a conflation between risk and trust, and talking about co-opting the narrative, I wish we could actually take the word AI and trust out of the equation. You know, it’s funny when you walk down the promenade, the number of signs, AI and trust, they’re not gonna write AI and risk, but trust in itself is, the way I define it, it’s a confident relationship with the unknown, right?

Like, we shouldn’t actually have trust in these systems. Like, I wanna know how these systems work. I wanna know how they make money.

And so it’s like we jumped to the trust conversation, because that’s very convenient for the platforms, before we really had the risk conversation. The mitigation, the management of that risk, before we even talk about trust.

Let’s get into the risk conversation. Mac, maybe you.

Yeah, just following up on both of those, the risk and trust question. I think it’s important to not over-complicate this, and remember how… Good we have been, how we have again and again and again tackled risk and trust issues in every other industry in the past, you know.

We want to trust in our medicines that we buy. We don’t, we want to reduce the risks. How do we do that?

Well, first we used to have a completely unregulated free-for-all in America and elsewhere. Then there were things like the thalidomide drug that was sold to pregnant mothers, promising it was going to reduce morning nausea. And it caused over 100,000 American babies to be born without arms or legs.

And that created so much outrage together with similar scandals that we created the FDA. So now there are safety standards before you can sell a medicine. It’s the company’s job to deal with the risk trust stuff.

They have to make the safety case and then some government appointed domain experts which have no money in the game get to decide is this something they can sell. We do that with restaurants so we don’t have to worry about salmonella every time we go out for something nice to eat. We do it with cars, with crash testing.

We do it with airplanes. Do it with every other industry except AI. It’s really not complicated.

We’ve done it so many times before. And as soon as we have the safety standards in place in an industry, then this amazing innovation that the invisible hand of capitalism creates makes people in the companies put their best talents in to make things safer.

You know, if you compare a biotech company today, and Switzerland has an amazing biotech, with the big AI company, there’s a big structural difference. In the biotech company, they have a much larger fraction of their spending going into product safety. How do you reduce the side effects?

How do you make, et cetera, et cetera. But then, in the big AI company, where it’s maybe 1% of the people working on safety, so it was an afterthought. And I think…

It’s very simple, we just need these binding safety standards, we know how to do it, and then companies will innovate to make more trustworthy products so that they can sell them.

Yeah, I mean it’s like the way I think about the distinction is like sort of risk, whatever industry you pick, it’s like probability, right? So you have a commitment to reduce the likelihood of causing harm, bad things happening. And once you get that probability right, you can then go to the possibility.

I truly don’t understand how we jumped the place of possibility without the reduction of harm, and how are they getting away with it? Why aren’t those guardrails in place? Because we shouldn’t be talking about what is possible until we figure out that probability piece.

I think it’s just a knee-jerk reaction of any industry to try to resist regulation initially, and then you get all this hyperbole, oh, it’s going to destroy the industry. Has biotech been destroyed? Of course not.

It’s thriving, right? And have restaurants been destroyed because you have to have a health inspector check your case? Of course not.

And so it’s natural that companies are also trying to resist regulation now, just like there was a massive lobbying campaign against seatbelts in the US, and then when the seatbelt law came, car sales exploded because people felt more trust in cars.

So I think it’s not surprising at all that AI is the Wild West right now, just because it’s a new kid on the block. And the quicker we fix it and treat it like other industries, the sooner we’ll get the upsides without the downsides.

There have been a number of people who have just joined us in the room, and I just want to point out, this is a town hall. It’s designed to be interactive. There’s a Slido poll that you can access via these QR codes if you’d like, and we’re going to get to audience questions here in just a minute.

But I want to ask something maybe for each of you. There were a lot of discussions around AI safety a couple of years ago, and that seems like it’s really fallen out of fashion, even within the big companies themselves. They were talking about it, and now they no longer are.

I don’t know if it was how you… You’ve had a view inside some of these companies yourself. I don’t know how genuine those discussions were but even things like the UK Safety Institute being renamed, what is it, the UK AI Security Institute now.

Why did that discussion change so much?

Well I think actually addressing a lot of these issues would be really, you know, it would undermine a lot of business models. And if you look at CapEx in AI, you get a really kind of concrete material answer. There’s a huge amount of investment going in, betting on a future where these technologies have accelerated into every corner of our lives, every infrastructure, every organization.

And you have the US GDP wrapped up in this, you have, look at the level of investment and then you see CapEx up there, but revenue is not doing much to meet CapEx. And I think there is a lot of pressure if you look at the lobbying that is happening from these companies, as you know, everyone in Brussels, everyone in K Street is kind of working for or retained by these companies.

And then you have a geopolitical situation which is very clear that there are AI haves and AI have nots, and guarding those infrastructures as a tool of geopolitical power has become a sort of tactic that is being deployed by the governments who have these tools, which also includes bolstering a narrative of inevitability, of acceleration, and sort of a race to AI dominance that is used, I think, conveniently by these companies and some others in good faith to push back on any sort of regulatory intervention that would, you know, be perceived to slow this down.

Now, we do have to ask, like, what are we racing toward? A race to the bottom is not a race we want to win. And I think a lot of these constructs and a lot of these narrative frameworks, again, need to be pushed on, right?

What is AI? Okay. AI right now is deep learning that is a derivative of a very particular platform business model that pulls data in the hands, et cetera, et cetera.

What does it do well? Okay. Well, LLMs do a few things well, but they’re actually ill suited for many, many other things.

There might be other techniques that are more interesting if you actually measure the success of these technologies when integrated into specific verticals, you know, gaming, automotive, other industrial sectors, like, we need to get much more empirical and the kind of people who would.

tear a shitty P&L statement apart in a board meeting who would like look at that, be like, where’d that number come from? Why are we using bespoke accounting methods on that? You know, like really incisive, seem to just let down their entire guards, like get almost like hypnotized when they’re presented with an AI strategy because we’re just kind of assuming we’re gonna be behind and we’re not gonna have access to the magical genie that will transform our industry into productivity land.

But none of that is empirically based. If you actually look at the data, if you actually start measuring what matters across these ecosystems.

Max, yeah.

Yeah, completely excellent points. I think in addition, you know, the fact that there is such a massive amount of money in the spending, of course, means not only that there’s a bunch of lobbying trying to downplay safety and discredit safety or whatever, but it also has this more obvious straightforward effects.

Like, I noticed a number of my colleagues at MIT have stopped talking much about risks now because they have funding from big tech companies and like, I wanna make sure my funding is renewed. We’ve seen this with the tobacco industry and so many other industries in the past. That doesn’t mean that the safety issues have gone away in any way.

In fact, contrary-wise, so many of the things that people warned about five, 10 years ago are actually happening now. We see AI that lies. We have, there was this great work by Anthropic, for example, showing how AI, when it was told that it was gonna be shut down, it found in the email server that the guy who was in charge of shutting it down was having an affair and it blackmailed him to not do it.

This was in simulation, but the concerns are just as strong as ever. It’s been known for a long time also that AI would one day become really good at manipulating people, last month. I spoke with Megan Garcia, whose son committed suicide after being groomed by a chatbot for six months.

And it’s just as a parent, it’s just so heart-wrenching to see that exactly the things we could have prevented if we had taken these safety issues more serious, they’re actually happening now. But I think on the upside, the good news is particularly the child harm they were beginning to see. Everywhere from anything from non-consensual child abuse, deep fakes, to suicides and AI psychosis, is really, really resonating.

I’ve never seen any issue before that made the broader public so engaged in AI risk. And we now have, this last few months, this crazy broad coalition in the United States. I call it the Bernie to Bannon coalition, the B2B coalition.

Tell us about the superintelligence statement.

That’s just part of it. We did a statement saying that we shouldn’t build superintelligence until there’s a scientific consensus, broad scientific consensus that we can control it, and so on. And the people who signed it, Steve Bannon signed it, a bunch of hardcore Democrats signed it, a bunch of top military people signed it.

Mike Mullen was the number one, the head of the Joint Chiefs of Staff, 102 presidents, they signed it. What do all these people have in common, even? Many people thought they wouldn’t agree on anything.

A lot of faith leaders. And they all agree that we want a pro-human future, where AI creates a good future for people, not a bunch of child suicides and other crazy stuff. We’re trying to make a great future for the humans, not for the machines.

So, of course, all these people are going to agree. If we got invaded by some weirdo aliens that don’t care about humans, then, of course, Republicans and Democrats and Americans and Chinese will also join together. So this is, to me, very encouraging, actually, that there’s a poll showing 95% of Americans now don’t want an unregulated race.

craziness. I think there’s real potential that we translate that into what I said in the beginning, just treating AI, the AI industry like every other industry, saying, you know, there are safety standards for everyone else, here are yours as well.

And then we’ll have the incentives so that people, even the people who just look at the balance sheets, will prioritize making things trustworthy, secure, and safe. That seems like a good point. So can we see what the consensus in the room is around the poll, on the Slido poll?

Do we have those results?

Not bad question. Yeah.

And if you answered trust, let me know why.

39% says I rather don’t trust, 32% I’d rather trust.

I’d rather trust. Rather, what is that modifier doing in there?

Yeah. What’s clear is the minority view here is I definitely trust at 8%. I definitely don’t trust is also only at 8%.

Okay. That’s interesting.

What’s the breakdown? What’s 8%?

Yeah. Can we get a show of hands of people who definitely trust? No, nobody wants to show.

Don’t want to put anyone on the spot. I wouldn’t either. I want to come back to something else that there was consensus around, because I think it speaks to a different AI issue, which is increasingly you’re seeing people on Republicans, Democrats, red states, blue states in the United States that don’t want data centers near them.

The whole thing that everything that AI runs off of, it’s from a data center. And they’re going up all over the country. They’re driving, especially in places like Virginia and Georgia.

They’re driving up people’s power bills. They’re using water in the West. But I think it speaks to a larger environmental impact of AI.

I’m curious if you guys have thoughts on that, on the footprint that we may be leaving on society for years to come with new power plants and things like that.

It’s interesting there’s no panel on climate change and AI. Not one panel linking those two issues. I find it astonishing, like, I mean, anyone who was at Davos three years ago, climate change, sustainability, that was, I don’t even want to call it the shiny toy, but right, it was the central theme.

There is not one panel linking, how? I didn’t realize that. Yeah, like, whatever we call it, sustainability, energy efficiency, climate change, we have to link these two issues.

I’m not an expert on data centers, but I do find that amazing that you can have a whole program three years ago around the climate and environment and sustainability and green energy, and then three years later there is nothing linking those two issues.

So I ask, who’s driving that agenda? Because that cannot be, well, I don’t know if it’s deliberate or if it’s unintentional, but to me that’s the glaring problem is that in society this new thing comes along and then another issue, it sits over here, and there’s not enough people who just join the dots.

I can’t comment on data centers, it’s really not my area of expertise, but I found that really astonishing about human beings and just these discussion forums that we don’t do enough linking of those issues.

You can’t separate those two things. Sorry, that was me on my, yeah.

No, I’m glad you brought it up. But I do think it’s, or maybe I should ask you, do you think it’s true that AI is a fundamentally extractive technology? I mean, it’s pulling from human knowledge, it’s using our energy systems, like is it, do you think it’s worth it?

I mean, fundamentally

this version of AI, this deep learning monoculture, I’m going to be pedant about this continually, yeah, that’s how it works, right? You pool a bunch of data and that data is sort of stuff that was put on the web or scanned, you know, starting in the sort of early 90s through now and that’s kind of, you know, that’s all of human knowledge, folks, Reddit comments, and it’s, you know, trained on that distribution.

If that normal distribution maps to the context of use, yeah, it’ll be more useful. If it doesn’t, then you can sort of bolt a rag database on the side and kind of, you know, try to adjust it, but ultimately that’s what it is. It extracts from that.

And I think, you know, I come from the arts. I went to art school in high school. I was very serious about it.

And seeing the extraction of the creative industries is pretty heartbreaking to me. And just the way that creative professionals are being treated and sort of understood in that industry. That’s not a comment on copyright.

That’s just a comment on sort of the values that seem to be encoded in how this is being creative. And then, of course, you know, energy-wise, the sort of AI monoculture that is scale at all costs on, you know, very, very resource-intensive deep learning approaches takes a huge amount of energy, takes a huge amount of water.

Data centers, you know, are not always clean. You have a lot of issues with dry towns now in the U.S., you know, energy bills going up. And then, of course, you know, you have green sources of energy coming online.

But they’re not at all compensating for the increased use. And so, a lot of claims of sustainability end up being very paper thin that says, hey, we’re using solar for this data center. But that means a number of coal plots haven’t been decommissioned that are now continually serving communities.

And I think that’s, you know, that’s an issue that, you know, I kind of feel like, you know, when it’s in the far future, we can all talk about it. Or the farther future, it’s like, yeah, let’s have a panel on that. But, you know, this is rubber meets the road, right?

You have people, you know, you have brown outs that are happening now. You have thresholds that are being reached. And so, the action that would need to be taken is, you know, significant.

And I think we sort of, you know, the sad truth is when we frame these things as ethics or nice to have or sort of, like, you know, decoration on the side of an inevitability, then, you know, we draw back from actually addressing those issues because addressing them would, you know, require looking closely at that narrative and realizing nothing none of this is inevitable.

All of this could be changed. And, you know, if we prioritize sustainability and, you know, making sure that we’re using our finite resources in a way that is socially beneficial, we might make different choices.

Max, is there something you have thoughts on?

Yeah, my opinion is, my take is that the water issue is actually smaller than some people have claimed, but the energy issue is very real, and on the data issue, oh my gosh, I mean, you’re really making my point there, Meredith, about how, again, the AI industry is treated completely different from any other industry.

What would happen if some company in the film industry just blatantly ripped off everybody’s copyright? Yet, for some reason, companies have been allowed to just take every single copyrighted book and whatever, I found my books, even in training data, and just do whatever they want with it, but just because, oh, it’s AI, exceptions, we were special, and yeah, there was a 1.5 billion copyright settlement against Anthropic now, but that’s just the tip of the iceberg, really.

I think there should be a law saying that companies have to declare what they put into their training. If you buy a food product in the supermarket, you want to know what’s in it, and as soon as you did that, companies would get their pants sued off from all the copyright infringements. Now, it’s hard to sue them because they won’t tell you what they trained on, and again, I don’t see any reason why we need to, why we should have this kind of discrimination where all the other companies have to play by rules, and AI companies don’t.

So, I want to get to some audience questions here in a moment, but before I do, just let me leave, or get you guys to explore one thing, which is, we’re talking about some negatives here, but I’d love to know, like, what are your thoughts on what can we do as individuals to, like, increase our agency, to ensure that there are rules or regulations to ensure that we can trust these systems?

What is there to do? Well, I’ll talk from a personal level. I think it’s really important to set

boundaries in your own work, right? So to your point, I mean, I could use any one of these tools and it could write my next book because I’ve trained it so well and sometimes it’s like I’m writing an article and I think, well, what would it say?

And then I think, oh, that’s really good. Maybe I should just use that, right? Like it’s so easy to slip into that space.

So my rule, and this is a really simple one, is that when I’m drafting something, I’m away from my computer. I use a pen and paper, I have to complete a first draft with my own thinking and my own points. It’s a very simple boundary and rule.

It is so easy to break when you’re under pressure. And then I come back and then I ask it questions and I ask it to go back into my whole history of work and it does pull out really interesting points and it does improve the argument. But if I didn’t set that boundary and I started at the point of I’ve got to write an article for you, Matt, for the MIT, I’ve only got an hour, could you draft something on trust and ethics and AI?

And then I could tinker with it and you probably would not know, right? And then I could say, could you edit it in Matt’s voice? And you’d be like, this is really clean copy.

Thanks so much, Rachel. Now, so this is the thing, right? For me, the boundaries are actually trust in ourself as well, like that’s the thing you can control.

That’s the agency piece. And the piece that we often don’t talk about is identity. So I just wanna give you one brief example because it’s a study I came across and it’s from the 70s and there was a big sociology done on bakers.

Now bear with me. And it was when machines were replacing bakers in bakeries. So they wouldn’t need the bread anymore, they’d press the croissant button and out would pop croissants.

And at first the bakers loved it, right? They’d go home earlier, there were less burns, they were far more efficient. And then what the sociologists found was in a very short space of time, within six months, they said they couldn’t call themselves bakers anymore.

It was the identity piece that went. And they said they would go home to bake bread. and I give that as an analogy of how quickly that can happen if you don’t think about who am I, right?

Am I an artist? Am I a writer? Am I a teacher?

Like what is that real meaning I have in my life and how does that relate to my thinking and create some kind of personal boundaries around that because that outsourcing is so good and it’s so quick and it’s so easy that unless we set those guardrails for ourselves, how do we pass them on to our students?

How do you pass them on to children? We have to understand how to carry that line to pass on to others.

Well, I know there’s a question right back here because you had your hand up. Can we get a microphone over her?

Thanks. No, I was really moved by what Rachel was saying about the probability and possibility and yesterday I was at this panel with Joel Kaplan from ETA and then Minister of Deregulation from Argentina who are probably best buddies by now and they were talking about like the only thing we need right now in societies is just to let AI run things and then if harm happens we like figure it out but let it run for a while just like a regular and then we will kind of like see what harms are there but then like nobody mentioned the threshold for harms or nobody mentioned like well if we have these signals this is what we are monitoring nobody mentions what people are actually looking into there’s just this like push for like just keep going the same message came from Jason Huang from NVIDIA saying we just need more investment in infrastructure and everything else is going to figure itself out so how do we kind of create these pressures for the things that we want to see in terms of safety because you’re mentioning the coalitions you know we are talking about well we know the business models are broken and they’re not really figuring out the humanity side of this so what what will create create because we are sitting in this room separately from those people and talking about ethics as the creation on the side but like how will we do this like what is where is the change either of you go ahead yeah let’s get super concrete like if we

if we were to treat AI like any other industry what would actually happen and how would how would this affect what the companies do? So, suppose Microsoft has a new productivity tool, there’s some safety standards, they will look at that probably the same way the Food and Drug Administration would look at a new kind of fruit juice, saying, you know, this productive random, Microsoft productivity tool, very hard to see much potential for harm, for the fruit juice you just want to make sure it doesn’t have arsenic in it or whatever, but that can be handled just with some random.

It would basically perform super quick, it’s out in the market, right? Now comes character AI, here is our new AI girlfriend, or some other company, you know, for 12-year-olds. That would be looked at more like maybe a new opioid drug, high potential for addiction, pretty obvious how it might cause harm, how it might cause deaths.

After the clinical trial you go, they do a control group, a test group, and notice that actually there’s a strong increase in suicidal ideation in those who try this chatbot, so you can see it from the chat logs pretty quickly.

So, sorry, no approval to you, next customer please. And now someone comes along and says, I want to do this recursive self-improvement with no human oversight. So, oh, that sounds a lot like digital gain-of-function research.

So, how do you do this in biology? Oh, biological gain-of-function research actually isn’t allowed in the U.S. right now.

So, next customer please. It would very quickly be noticed by the companies that these are the kind of products where approval is trivial and you can make a lot of money, resources get shifted into there, and frankly AI girlfriends is not, I’m quite sure, that’s never been Microsoft’s key idea for how you’re going to make money in the future anyway.

It would shrink into this little fringe thing that it should be. So we don’t have to lecture the companies at all. We just have to have these rules that treat them like other companies.

And then I think we’re just gonna see a massive behavior change in the companies.

What are the rules?

The government, of course. Just like the FDA, the government decided. We don’t want another thalidomide.

Max, can I come back to what Meredith said? I mean, I’m all for the rules and the regulations, but this is a market problem. The economies will collapse.

I love that he needs more investment in infrastructure. That’s wonderful, right?

Biotech did not collapse.

Biotech is struggling now. However, it’s really hard.

But it didn’t collapse in the 60s when the FDA was created.

Well, right.

And I think it’s increased trust in medicines and massively increased sales.

I have a biotech company, I’m not gonna name any names, but they’re doing incredible AI-centric research on cancer. It’s high risk, high reward. They’re using DeepMind’s Protein Folding Database.

Extremely hard. This is long-term research. It may not pan out, but that is a social good.

QED. They can’t get Series B because the VCs want a chatbot to cash out, right? That’s what we’re looking at here.

I mean, that’s one story among many that you hear in this ecosystem. And it’s, again, this is a monoculture. I do wanna answer this question because I just find it, I think I worked on my first sort of AI regulation, how would we think about these guardrails around machine learning in probably 2015, right?

And that was right after the DeepMind acquisition at Google. 2012 was AlexNet, and that’s where everyone started getting interested in deep learning again. So this is pretty early.

I’ve probably had this conversation once every quarter since then. What are we gonna do? And I don’t, there’s, in some sense it’s gotten worse and worse, the level of discourse has gotten more and more sort of like.

Baby brains to be blunt about it like yeah We need just accelerate it all wash out in in in the wash and meanwhile I’m I’m leading signal how many people here use signal, okay? I love you all You know but signal is core infrastructure to preserve the human right for private communication It is used by militaries it is used by governments. It is used by boardrooms.

It is used by dissidents Ukraine uses it extensively. This is these are contexts where the ability to keep your communications private is life-or-death Signal is an application built on you know on top of operating systems we build one version for Android one version for iOS one version for Windows and and you know Mac OS on your desktop and We take this responsibility really seriously we open source all of our code so people can look at it We open source our cryptographic protocol and our implementation We’re working with some people you know with with Max and some folks to formally verify that which means Mathematically you can prove that what it says in the code is what it’s doing and that’s a level of assurance We put out there because we recognize that this could you know people die if we mess this up But we have to run on top of these these these operating systems And you are seeing the three operating system vendors now Rushing to integrate what they’re calling AI agents into these operating systems now AI agents the marketing promise is you have a magic genie?

That’s gonna do your living for you right so you ask the AI agent Hey, can you plan a birthday party for me and coordinate with my friends that sounds great? You can put your brain in a jar, and you don’t have to do any living You can just walk down the promenade. You know seagulls in your mind And While the agent plans your birthday party, but what is it an agent actually doing at the level of technology in?

That case it would need access to your calendar access to your credit card access to your browser to simulate mouse clicks and place orders And in this hypothetical scenario access to your signal messages to text your friend as if it were you And you have to coordinate and then sort of you know put that on your calendar, right?

That is a significant security vulnerability. So while we’re telling stories of inevitable magic genies that exist in a bottle, what these agents are doing is getting what’s akin to root permission in your operating system for the Unix people in the room.

They are reading data from your screen buffer at a pixel level, bypassing signal. They are hooking into your accessibility APIs to get audio, to get image data from the screen. They are rooting through your file system, the sort of deepest level of your operating system that controls what the computer can and cannot do, what software can and cannot do, what data you can access and what you can do with it.

They are making remote API calls to third-party services. They are sending all of that to the cloud because there is no LLM small enough to run on your device and they are processing in the cloud to create statistical models of your behavior so they can guess what you might want to do next.

These are incredibly susceptible to prompt injection attacks. There is no solution for that because when it comes to natural language, which is what these agentic systems are processing, they fundamentally cannot tell the difference between an authentic desire of the user and language that may look like an authentic desire but is in fact not representative of what people actually want.

If you look at the architectures of these agents on the operating system, they look a little bit like the architectures of targeted malware given the permissions that they allow, given the data access they allow, and given the vectors to send that off your device, process in the cloud, access a website outside, et cetera.

This is an existential threat to signal, to the ability to build securely and privately at the application layer, and it is a fundamental paradigm shift in the history of computing where for decades and decades, we viewed the operating system as a neutral set of tools that developers and users can sort of use to control the fundamentals of how their device worked and how the software worked.

These are now being effectively sort of remote controlled by the companies building these agents and these systems that are ultimately Taking agency away from both developers like Signal and the users of these systems.

Again, you know, I went into this level of technical detail because you know That’s the level at which any dignified adult conversation should be happening. Like come on You think you can take responsibility for the entire world, but you don’t have to answer for it I did not come up in a world where that had any dignity to it So I think you know like in some sense we just need to demand a bit more spinefulness of the people who claim to own the future and Recognize this is what’s happening.

If you’re hooking into my accessibility API you have ultimately decimated Signal’s ability to provide this existential this this fundamental service and I will shut it down before I will continue to operate without Integrity because I do not want people being harmed because they trust us They trust us to provide a service.

We no longer can

Well, that was wow. Okay And with that we’re out of time, I’m sorry, I know we had some more questions in the room We didn’t get to thank you so much for joining us today. Thank you for being here I hope you all got a lot out of that.

I did Thanks, everybody You and Tristan Thompson.