The premise is that most of the progress in AI up to now has been through scaling, more data, more compute, and that that is still useful but there are other better things. So I’m gonna ask each of our three wonderful panelists to talk a little bit about what they’re working on now. By the time we’re done with that, our fourth panelist Yuval Noah Harari will arrive and he’ll join in and try to catch up.

So, Yoshua, you’re working on scientist AI, which is incredible. Explain what it is and how it’s different from previous paradigms of AI.

Thank you, thank you. So what’s motivating the scientist AI and also the new nonprofit I created to engineer it, called Law Zero, is how it addresses the question of reliability of the AI systems we’re building, especially the genetic systems, how it deals with the issue that current AI systems can have goals, sub-goals, that we did not choose and that can go against our instructions.

And this is something that’s already been observed and is even more prevalent in the last year across a number of experimental studies, but also in the deployment of AI, for example, with sycophancy. It’s an issue that is kind of very concerning when you look at the behavior of self-preservation, where AIs don’t want to be shut down and want to evade our oversight, be willing to do things like blackmail in order to escape our control.

So even things like preventing misuse, the companies put monitors and guardrails, but somehow this still doesn’t work really well enough. And the core of our thesis is that we can change the way that AIs are trained. So it could be the same kind of architecture, but the training objective and the way we message the data is going to be such that we obtain guarantees that the system will be honest in a probabilistic sense.

OK, so how do you do that?

How do you do that? So the core of the idea, which is connected to it.

I’m trying to do it with my kids.

Yes. So the core of the idea, which is behind the name, is take as an inspiration not to imitate people, but to imitate what science at an ideal level is trying to do. So think about the laws of physics.

The laws of physics can be turned into predictions, and those predictions will be honest. They don’t care about whether the prediction is going to help one person or another person. So it turns out that it is possible to define training objectives for neural nets so that they will converge to what something like scientific laws would predict.

And then we get something that we can rely. For example, we can rely on to create technical guardrails around agents that we don’t trust. So if an agent is proposing an action for each action that the agent proposes, a honest predictor could tell us whether that action has some probability of creating a particular kind of harm.

And of course, veto that action if that’s the case.

But you still are then going to be required to put in some threshold of when it will take that action, right? If it has a percentage odds of harm of more than 1 in 10 or 1 in 1,000, wherever you put it, you still have some human concern, you still have some potential harm created.

Absolutely. So when we build a nuclear plant, we have to decide where we put the threshold, right? And for nuclear plants, it might be, you know, one in a million years that something bad is going to happen because it’s so severe.

Depending on the kind of harm that we’re trying to prevent, society, not AIs, have to decide where we put those thresholds.

Great. I’ve always thought it was interesting that for most things, we’ll accept like a one in a 10 million chance of nuclear plant exploding, but we continue to build AI even though general predictions that it might wipe out humanity are like 10%.

All right. Yejin, why don’t you talk a little bit about some of your work in continual learning? And you, of course, have been a brilliant critic of scaling laws for a long time, including on a panel last year with Yoshua.

So tell us what you’re working on now.

All right. So let me step back a little bit before I do continual learning. You know, right now, AI is like super impressive, but it’s a little bit jagged intelligence, right?

In that it’s amazing at bar exams and, you know, some of these like really difficult international math Olympiad problems, yet, you know, you’re not going to rely on it for, you know, doing your tax return or even like making some important transactions because you may not be able to click the right button on your computer.

So why is it that is because right now, the way that we train AI, LLMs, general AI, is too data dependent, and it’s a one-time training, and then you deploy it, and it may or may not make a mistake. So here are a few really important things that I think we need to solve in order to get to the next intelligence. So the first, first of all, continual learning.

So, you know, like 101, machine learning 101 is that you separate our training from testing. It’s almost a sin. to mix the two.

But human intelligence is not like that. From the day one a baby is born, it’s in the deployment mode. It has to figure things out.

It’s real life. So humans can learn during the deployment time. And we need to somehow figure out how to ensure AI can learn continuously during the test time.

So it’s test time training that I’m working on. Another angle that’s really important is that currently one of the key reasons in my mind why AI is unreliable sometimes. And we need to worry about the safety concerns as well that, for example, paper clip scenario where you ask LLMs to generate as many paper clips as possible, it might kill all of us in order to produce one more paper clip, right?

So in order to avoid that kind of a silly situation that’s harmful for humans, AI should really figure out how the world works for the sake of learning how the world works as opposed to just passively learning whatever data that’s even given to us.

So I think a fundamental challenge here is that LLMs learn passively as opposed to proactively. It’s not really thinking for itself. It’s just trying to memorize all the text given to us and then try to solve all the math problems given to us as opposed to us humans being curious about how the world works and trying to think for ourselves.

And then lastly, it’s way too data dependent. Whatever data is rich, it works. Whatever data is not rich, it doesn’t work.

That’s how things are right now. And then safety is hard because we have to create all the safety data. And red teaming jailbreaks, these are not area of domain where there’s a lot of data.

So in order to fix this problem, I think we need an entirely different. learning paradigm, where it’s really about thinking for yourself, almost trading off data with the compute. So, you know, learning with way less data, but with more mental efforts.

Quickly, Yejin, but if you have continual learning, doesn’t that open up a whole new spectra of problems? Like, right now you build a model, you run a bunch of tests, eventually you refine it, a few months later you do it again. If it’s continuously learning, how does that not become just suddenly infinite, right?

I mean, if you are learning from every answer and you’re giving feedback and, like, a baby’s, like, in its crib, it’s walking around, it’s contained, but if you have, you know, a few billion people using a model at any time and it’s learning constantly, doesn’t that open up whole new vectors of wonder, but whole new vectors of problems?

Yes and no. So it could, in theory, in a long term, but it’s so far off, in my mind, in the sense that humans can also continually learn, but there’s a limit as to how much we can really reach.

But another problem is that after the system has evolved sufficiently through this continual learning, all the safety tests that we did previously may not be valid anymore. So I think there’s a real safety risk that you’re pointing to.

Yes, so my hope is that if AI is trained correctly from day one, so that it really understands the human norms and values, not just math the problem-solving, but human norms and values, such that it will build its worldview and everything else on top of it, and then it’s going to behave based on that.

And how do you deal with reward hacking? In other words, even if it understands human values, it might have optimized something that is not quite what we want.

I agree. So reward hacking implies that, you know, we just go with the reinforcement learning and that’s all we got. No, it shouldn’t be all we got.

no human being is optimizing for one reward for the rest of their life, right? Like, we have so many different goals that are at us, and we make some sacrifice, you know, I might want to do something, but I might not do it because, you know, I respect other people, right? So AI should be exactly this, that they should understand that values are at us in real life, in human life, and that it needs to know how to make the trade-offs such that it’s going to not violate laws, it’s not going to harm people, and whenever it’s not clear what to do, because there are always situations in which it’s not clear what the gold answer is.

It should consult with the humans and release the decision-making to humans.

All right, we’re gonna move to Eric. Eric, you have recently built a big new model, K2 Think. You’ve had a whole series of innovations in it.

Explain what you’ve done that’s novel and what’s different from the amazing stuff that Yejin and Joshua are working on.

Well, I have, there’s my boot. So yeah, as Nick just mentioned, we at MPCI are among the few, maybe the only university that is actually building those foundation models from scratch, from scratch meaning that, you know, you gather your own data, you implement your own algorithm, you build your own machine, and then you train from top to bottom, and then you release and serve the whole process.

I thought that it’s important for academic to be a player like this so that we can share the knowledge to the public so that people can study many of the nuances, you know, in building this, and also understand the safety and the risk issue.

In fact, I want to say that it is by no means easy. It’s very, very difficult. In fact, I almost want to say that AI systems and the softwares are actually very vulnerable.

They are not very robust, and they are not very powerful. You remove one machine from the cluster, you can crash the whole thing already. Now, what I’m building right now is of course to improve AI performance, but I want to maybe add on your question.

a comment on what do we mean by intelligence and how to break it down. Because if I tell my engineer, say, hey, build a software that is intelligent, they don’t know what to do. So many people have different opinions on intelligence.

There are Nobel Prize winners in the economy who may not do very well in their stock inspection. Their wife may do better than them. That’s actually reflecting on a different level of intelligence and different utilities.

In my opinion, what LM right now is delivering is a limited form of intelligence. I would call them maybe textual intelligence or maybe visual intelligence, which is actually on a piece of paper in the form of language or maybe video. But these are like book knowledge.

If you want to put it on action, I was actually hiking a week ago in the Austria Alps. I do the GPTs, I do the Google, I got all the training guides and even a Google map in my hand. When I walk to the mountain, you still cannot rely on paper.

You have to rely on yourself. You have all these unexpected situations. Snow is too deep and the weather is no good and you cannot see the past anymore.

What do you do? So this requires already a new type of intelligence that is not available right now which we call physical intelligence. And that’s actually where people hear about the topic of world models.

World model is about understanding the world, able to generate plans and the strategies and the sequence of actions purposefully so that you can execute it and you can actually deploy it and also you can adapt to changing environment.

But still, this is not necessarily the smartest thing that we could imagine because I would call the next level beyond physical intelligence would be social intelligence. Right now, we don’t actually see two LMs collaborating yet. They don’t really understand each other in the form that we human do, right?

There is no definition of a self. What is my limitation? what is your limitation?

How can we divide the job into two or 100 so that we can break them into parts? Therefore, you can never ask LM, our model, to help run a company or run a country because they don’t understand these kind of nuances of interactive behaviors. I would put, in fact, also a last layer of intelligence that is still even further, which I would, for the sake of, for the lack of good name, I call them philosophical intelligence, which is that is LM or AI models itself curious to discover the world, to look for data, and to learn things, and then to explain without being asked to explain?

That’s probably where Josh is very, very concerned about because that’s where you start to see definitively some sign of identity and agency. I want to say that we are not there yet. We are very far from there.

Even the current physical model, the world model, is very primitive because it is primarily rely on a wrong architecture that is directly offspring of the LM. So what my work is involving right now is to come up with new architectures which represents the data, do the reasoning, and also do the learning using different ideas. People may heard about the Yann LeCun’s Architecture of Japan, right?

It is a architecture behind many of the current world models. We have a third model called the JLP, which does the following. First of all, your representation of knowledge needs to be richer, need to be containing both continuous and symbolic signals so that you can reason at different level of granularity.

And secondly, you need to have the right architecture which can carry you a long way. People play with the SORA probably have that experience. How many seconds of video can generate?

Maybe 10 seconds, maybe a minute. It’s not because they run out of memory. It’s because going beyond one minute or 10 minute, you don’t have the ability to track consistency, to reason consistently.

In fact, you can try a very interesting experiment. You just ask the Sara, or maybe Jiminy, to generate you 360 degree of round view around you, and then turn back to your degree zero. Did you see the same thing or not?

It is not guaranteed. That’s actually a lack of consistency already in the system. And then stateful representations, you know, and also there are things like a continuous learning paradigm is a problem.

Right now, all models are in the form of what we call passive learning. You feed the data, and then the model will be trained on those data. In machine learning, we knew in the past a new paradigm called active learning or proactive learning where the system should hopefully be able to identify where they want to learn more by using, asking for more data.

But we are not yet there, not along, go out and looking for data and create it themselves. So I think AI, as of now, in my opinion, still is a very primitive age. We have a lot to do to really get it to work.

Well, Yuval, this is the handsome man at the end of the panels, Yuval Milharar. You probably already know that. Welcome.

However, he’s late for the same reason that everything is complicated in Davos, which is geopolitics. Apparently, Macron went late. It pushed his panel back, so here we are.

Yuval, what we’ve been talking about is different paths, new research that Joshua’s been doing, that Yejin’s been doing. You just walked in on what Eric has been doing. Lots of different promising ways to make AI go faster.

I’m gonna just ask you a philosophical question, which is, do you think that as we look for new models of AI, we should be trying to make it more like the human mind or less like the human mind? This is something you’ve written about beautifully, but I haven’t heard you talk about this in the last while.

No, I think it’s completely different from the human mind. The whole question of when will AI reach the same level as human intelligence, this is ridiculous. It’s like asking, when will airplanes finally be like birds?

They will never, ever be like birds.

And they shouldn’t be.

And they shouldn’t be. And they can do many, many things that birds can’t. And this will be the same with AIs and humans.

They are not on the same trajectory behind us. They’re on a completely different trajectory for better or for worse. I’m very happy to hear that it is still, that AIs, again, I’m not sure to what extent we can rely on it, how long it will continue.

But the fact that AIs, for instance, cannot cooperate so far, this is wonderful news. I hope it’s true. I hope it will remain like that.

Otherwise, we are in very, very deep trouble. For me, the lesson from history about intelligence, you don’t need a lot of intelligence to change the world and potentially to cause havoc. You can change the world with relatively little intelligence.

And the other thing we’ve learned from human history about intelligence, I’m not referring to anybody in particular. And the other thing we’ve learned about intelligence is that the most intelligent entities on the planet can also be the most deluded. Human beings are by far, so far, the most intelligent entities on the planet and the most deluded.

We believe ridiculous things that no chimpanzee or dog or pig would ever dream of believing. Like that if you go and kill other people of your species, after you die, you go to heaven and there live blissfully ever after because of the wonderful thing you did, that you killed these other members of your species.

No chimpanzee will believe that, but many humans do, at least where I come from. And when I say that you can change the world with relatively little intelligence, humans have already done much of the hard work for the AIs. Like if you drop an AI in the middle of the African savanna and tell it, take over the world, it can’t.

How it will do it? Impossible. But if first you have these apes who build all these bureaucratic systems, like the financial system.

and then you drop the AI into the existing financial system and you tell it, okay, now take this over, that’s much, much easier. The financial system, you don’t need motor skills, you don’t need even to understand the world. An AI can understand, the financial system is the ideal playground for AI.

It’s a purely informational, like to train AIs to make a million dollars. Create a million AIs, give them some seed money, let’s see you make a million dollars. Now, if you have a few AIs that succeeded in doing that, replicate them.

What happens to the world if more and more of the financial system is shaped by AIs that developed better, even though they can’t walk down the street, they know how to invest money better than humans.

It’s a very, very limited intelligence, but again, think about social media. Social media is run to some extent by extremely primitive AIs, these algorithms that control our newsfeed and so forth. Look what they did in 10 years.

We created a human system, media, and then we introduced the AIs into our system and it’s an informational system and they took it over and they, to a large extent, wrecked the world. They are not the only reason for the mess now in the world, but if you think about what extremely primitive AIs did within the human-created system of media, then…

Well, I’m gonna move this to Yoshua because he, in fact, has invented AIs or is working on inventing AIs that if dropped into the financial system and told to wreck it, would not be able to, correct? That’s the hope. Respond to Yuval.

I wanna add something, going back to your first question, connecting humans and AIs and whether we should build. AIs are our image. And indeed, they are quite different from us.

The problem is we interact with them. Many people interact with them with the false belief that they are like us. And the smarter we make them, the more it’s going to be like this.

And there will be people who want to make them even look like us. So it’s going to be video first, eventually maybe physical form. But it’s not clear that it’s good in many ways in terms of how humanity has developed norms and expectations and psychology that work because we interact with other humans.

But AIs are not really humans. For example, they could be immortal. Once an AI is created, in principle, you could just copy it on more computers.

And we can’t do that with our brain, as Jeff Hinton has been highlighting many times. And many other differences, like they can communicate with each other a billion times faster than we can do between each other with ourselves. And so there’s going to be this illusion that we build machines that are like us, but they’re not.

And this is a dangerous illusion that could lead us to take wrong decisions. Part of the problem is the scientists themselves. In the last 40 years that I’ve been working on AI, in the whole community, really, we took inspiration from human intelligence.

You were talking about continual learning because we’re good at that, and we see that it’s lacking in AI. And that’s fine. That’s how research has been moving.

But I think we also have to think of what’s going to happen when this gets to be deployed in society more and more, and people will anthropomorphize and do weird things.

It’s an amazing question. Let’s move this to a topic that I think connects to this pretty well, which is back to the architectural questions or foundation questions, which is the question of open source. And there’s actually been more and more discussion.

here in Davos, in part because Europe is recognizing they need a counterweight to the USAI models. Eric, you’re building open source models. Yejin, you have strong views on them.

Yoshua, you have strong views on them. Yejin, why don’t we start with you? What do you think of the notion that it would be good if there were many more open source models that we all started to use as much as we use the large foundation models?

Yeah, so the way that I like to think about open source is democratization of generative AI, which is a powerful, powerful tool. And what I mean by democratization of generative AI is that it should be, AI should be of human, for human, by humans. AI is of human because it’s really drawing from the internet data that’s the artifact of human intelligence.

It reflects our values, it reflects our knowledge. By the way, values including horrible value that we do to each other, it happens to be on the internet and so AI picks up on that. There are sci-fi movies in which AI kills us all and as a result, that’s what AI might actually say because it’s written in the internet.

AI should be for humans in that humanity at large and all of the humans, not just some humans who happen to be in power. I deeply care about this, that AI should be really for all humans. And by the way, worse than AI for some humans is AI for humans or humans for AI, even worse.

It’s really good to think about how we build and design AI so that we work on problems that could really make humanity better as opposed to just increase subscriptions and win the leaderboard. And then lastly, AI by humans. What I mean by that is that AI should be created, AI should be able to create it by different countries and different, not just the private sectors but public sectors and non-profit org and even academia.

The reason why I think about this way is that, well, I’m a U.S. citizen now, but I used to be a Korean person, and it’s a very wonderful thing if we know how to create this, even from Korea or from other countries, as opposed to them having to just rely on a country or two providing all the services for them.

But would your goals be satisfied if Korea had a closed foundation model, or do you want there to be a universal open model that everybody is able to contribute to? Eric, do you agree with this?

People can choose to close or open, but the reason why, for the time being, I really support open source is because it takes too much of resources to build something really, really good fast. And so, unless you are capable of, you know, really making very large data centers and on lots of GPUs really fast, it really helps to help each other to share the scientific knowledge and everything so that the development goes much faster.

And by doing so, by the way, we can make small models much more powerful so that a lot of organizations who cannot afford as much can build LLMs that serve just their needs, not like general LLMs that can do everything, but something that really serves a business need really well.

All right, Eric, so you nodded at one point and shook your head at another point, so I need you to respond quickly here.

Yeah, I think open source, you know, isn’t really the goal. It is basically, you know, a philosophy or a way of doing things which come very naturally with science, with any of the scientific research.

So what do you mean it’s not the goal? Like, it’s not, like, you’re not doing it for the sake of open source? You’re doing it because it’s a more efficient way to reach the outcome?

No, no, no. It is really almost like a responsibility or a natural style of doing the research of AI. You know, in fact, there are also pragmatic values.

For example, I often ask, do you prefer there is only one car maker in the world that makes you feel safer, or you actually save money? 60, 10, or 100 is better. Open source basically is about sharing knowledge to the general public so that people can use it, also people can study it, and understand it, and improve it.

Of course, the assumption is that this technology is not evil, it’s not something that you really want to get rid of. I don’t think technology itself, by definition, any technology is evil. It’s really about the people who use it in a wrong way.

But by closing source it, you don’t actually stop that. So open sourcing, the benefit from open sourcing, in my opinion, overweights closing it because first of all, you cannot stop your orders of using it, and secondly, by opening it, you actually are promoting more adoption and more understanding of that.

I also want to go back to the issue that Josh just mentioned about the impersonalization of human ear technology creates the risk. That is how we see it now. It’s kind of also implicitly assuming that the way human being do not learn from the new experiences.

In the past, if you look at the history, there are many magical inventions which may make certain population godlike, but then after some time, people actually get comfortable with it and start to form better judgment and also better understanding.

I think the way of really making people safe and comfortable and co-exist nicely with AI is to use more AI and also get quickly adapt to it. It’s like you are in a natural environment, you have the virus and so forth. Of course, you want to think about stopping it, but sometimes the nature choose to let you co-exist with the virus so that you become stronger.

There are some risks, some casualties, but as a population, as a society, we together evolve stronger.

Yeah, sure, and then I have a question for you, Bob.

As a university professor, I’ve been promoting open source and, of course, open science for. my life. But if you start asking ethical questions, then at some point, you start hitting a problem, which is some knowledge can be dangerous when it is available to everyone.

So I’m going to give you a simple example. Biologists are working on how to create new DNA sequences that can actually create new viruses that don’t exist. And if you know a sequence that gives rise to a virus that could kill half of the planet, should you publish it?

And the answer should be obvious in this case, right? So current AI systems that are open sourced are net positive. It helps safety.

It helps democratization of AI. And I’m as worried as you are about concentration of power. I’ll come back to that.

The problem is, if the capabilities of AI continue to grow along the directions that we’ve been talking about, at some point, we end up with AI systems that are like, well, not the sequence itself, but the machine that can generate the sequence that can kill half of the population.

So when AI reaches that stage, we should not just give it to everyone, because there are a lot of crazy people. There are dangerous people. There are people who want to use it for destroying their enemies in military ways.

So we should be very careful when we reach a level of capability where AI can be weaponized. Now, I agree about the issue of concentration of power, but there are other ways than open sources. When we get to that point where AI can be weaponized, I think, and before we get there, we need to think about it, we should really think of how we can manage a few, not just one, a few AI systems that will be dangerous in the wrong hands and where the power to control these things will be decentralized.

So what we don’t want is one entity, one government, one corporation. to dictate what the world should be. But I think that there are solutions to this and we have experienced this sort of thing in the international arena with international treaties, what we’ve done with nuclear weapons, what Europe has done with the EU and so on.

So I think that there are solutions and we should think about ways to both avoid catastrophic use and abuse of power in the hands of just a few.

But this is, I wanna bring in Yuval here because this is an amazing philosophical question, right? There’s the incredibly powerful technology. Are we safer if everybody’s contributing to it and everybody has a say over it but everybody kind of has access to it?

Or are we safer if a relatively small number of people who can be controlled or answerable to governments and are all here and somewhere in the Congress Center have control of it? Have we ever faced this historically, Yuval? Has there ever been a moment like this?

And what happened?

I think the main point is that we just don’t know. We are at a point when we are conducting this huge historical experiment and we just don’t know. The key question for me, how do we build a self-correcting mechanism into it?

How do we make sure that if we get the answer wrong, we’ll have a second chance? And the model for me is the last big technological revolution, which is the Industrial Revolution. When the Industrial Revolution begins in the early 19th century, nobody has an idea how to build a benign, a good industrial society.

This immense new power, steam engines, railroad, steamships, how do you use them for good? And different people have different ideas and they experiment. And European imperialism was one experiment.

Some people say the only way to build an industrial society is to build an empire. You cannot build an industrial society on the level of one country because you must control the raw material and the markets. You must have an empire.

Then you have people who say it must be a totalitarian society. Only a totalitarian system, like Bolshevism or like Nazism, the immense powers of industry can only be controlled by a totalitarian society. Now looking back from the early 21st century, we can say, oh, we know what the answer was.

We think we know. It took 200 years of terrible wars and hundreds of millions of casualties and, you know, injuries that are not healed even today to find out how to build a benign industrial society. And this was just steam engines.

Now we are dealing with potentially super-intelligent agents. Nobody has any experience with building a hybrid human-AI society. We should be a lot more humble in the way that we think we know how to build it.

No, we don’t. How do we make sure—I don’t know what the answer—the question is, how do we build a self-correcting mechanism so if we take the wrong bet, this is not the end?

I want to bring the conversation from philosophical back to more like a practical part. Because it’s about where the checkpoint should be, right? You talk about a dangerous virus.

Influencing a virus is actually not easy. You know, the idea of a nuclear bomb, for example, is published somewhere. You can Google it.

You cannot build it because you need to get the materials, you need to get the labs. There are a lot of checking points already, you know, we learned from generations and centuries of governance and regulation and the human practices were set in many places already. After all, AI is a piece of software.

It is software living in the computers. And when it does the physical harm, it needs to go out of the computer. That’s already one extra checkpoint.

Humans can do it for the AI and eventually there will be robots that will do it.

And humans are subject to checkpoints as well, right? Virus on the other hand does not.

Eric, let me ask you this, since this panel is all about like how to best construct the next generation of AI. Probably all agree here on this panel that we want lots of checkpoints and good checkpoints. We maybe disagree on whether we have enough right now.

What is the sort of methodology or architecture of AI that has the most checkpoints? Yejin, you got one right there?

Yeah, I mean I have a proposal to handle this situation better. I think fundamentally the problem is that AI is too dumb. It’s going to learn on any data that you give to it and if you happen to give data about how to do cyber attacks or how to generate bioweapons, it’s just go ahead and learn from it, right?

That’s the fundamental challenge we’re dealing with. On the other hand, if we build AI, maybe following Yosha’s AI scientist direction, that really learns, think for itself, and really acquire human norms, understand that that’s what it should really abide by, and then when it reads the training data given by some other human, it refuses to learn.

When it knows that this is illegal, it refuses to learn. And by the way, that’s what humans also do. Like a lot of us, of course there are people who want to do bad things, but a lot of us, if I give you how to kill humans, I mean like through bioweapons, would you internalize it for yourself?

No, because you don’t want to act on it. So I think we may need to rethink about how we design AI training algorithms to such that it has more agency about how to choose what to learn.

Or to just not train on Reddit at all. Yoshua.

I just want to mention that there, because we’ve been talking about the technical aspects of these questions. Right now, the way we design AI systems, there is no boundary between data and instruction. So in normal programming, it’s two different things, right?

So a program will read files, and then there’s the code itself, and the programmers write the code. And they know that whatever is in the files, the behavior is going to be according to the code. With the way that we’re building our AIs, there’s no distinction.

And so that’s the reason why it’s so easy to, in the data, put instructions. That’s how you get jailbreaks, right? And other security issues that we have with AIs.

And so I think that in order to get more safety from the AIs, we need them to understand the distinction between what we want and what is instructed in a way that’s been socially regulated. So who decides what the norms are and so on? And what it reads as data.

When it has an interaction with a user, we don’t want the user to be able to make the AI do anything that they want, for example.

So is that like a set of master controls you’re trying to build? Is that like a?

No. So in the scientist AI, the way that we’re doing this is we’re training the AI to make the difference between what people will say, will write, which could be motivated, which the AI should not take as truth or what it should be necessarily doing, and other forms of information which contains underlying truths or underlying causes of what is being seen.

And that second channel is one that is trustworthy, where we don’t necessarily give that access to anybody using the AI, for example. But it’s also a way to make sure we get AIs that understand the difference between what people will say and what is actually the cause of what they say and if what they say is true. So they get honesty.

So we have just a few minutes left. And we’ve talked about a lot of new architectures. We’ve talked about some new agentic systems.

We’ve talked a lot about open source. We’ve talked about continual learning. We’ve talked about different ways of looking at data.

And we’ve kind of talked about all the new systems as though they’re good. Are there any sort of new architectures or methodologies that people are excited about? Maybe ones that we’ve talked about on stage.

that you think are actively bad and that we should not pursue? Maybe Eric and Yijun?

What do you mean by a bad architecture? That the consequences are bad or the performance are bad?

Either works fine.

I think, in fact, maybe that is even compatible with what Josh is worried about. Building a system that is not in the closed-loop fashion, that you purely do thought experiments and embedding internally in some kind of latent representations and complete all the training before emerging to the real world to validate, in my opinion, is a bad system.

Because first of all, performance-wise, there isn’t really enough checkpoints to even control and visualize or understand any of the risky points. And also, it is very hard to connect a system to an action-conditioning point so that you can steer it, you can navigate, you can manipulate. On the other hand, it is going to consume data and energy and the resource and the money for too long before you actually see the end outcome.

I’m not going to name any specific instance of this architecture. But it’s actually pretty prevalent that people sometimes believe that I don’t need to really compare the content from an AI system with real-world data constantly before I achieve superintelligence. And secondly, I also think that the current learning paradigm, which I totally agree with Yoshua and Yejin about, is a very, very primitive and maybe an unproductive one.

The data right now is really the master of the algorithm and of the system. And the system itself is basically a one-shot learning, in a sense. You train it, and then when now I’m using GPT.

or any models, they don’t actually learn from that experiences. And just like ourself, when we’re in the conversation, I’m already learning from both of you and all of you, new points, I enjoy that. And the AI system isn’t built for that kind of functionality yet.

And can you imagine that a system of that kind of dumbness can become super intelligent and come back and go after us? I just don’t feel that us can connect. It doesn’t have that kind of task oriented type of data that guides you beyond just pattern matching, but actually do the reasoning and so forth.

So if our goal is to build smarter and a more powerful system, there are needs to explore new architectures. Of course, there is a separate issue about how do we measure the risk? I don’t really know the exact answer, but I want to actually hear, Josh, your opinion.

Is the solution to not doing that or do that with a very, very conscious and quantitative kind of approach to measure the risk, to experiment with all the scenarios? And then to set up.

Yes, we need to measure the risk on the fly, not just once when we evaluate those models. And we need to make sure that we also have the right societal infrastructure. So even if we knew how to build really safe systems, there are lots of bad things that can happen because humans are humans.

And so we need technical guardrails and we need societal guardrails.

Absolutely, yeah.

All right, let’s wrap this up, Yuval. Your book came out, Nexus came out about a year and a half ago. You had some real concerns about AI.

You’ve just been here with three of the smartest, most influential AI researchers in the world. They’ve won every prize imaginable. Do you feel like we’re getting onto the right track or do you not?

I think we are thinking on different timescales. That when people, a lot of the conversations here in Davos, when they think, when they say long-term. they mean like two years.

When I say long-term, I mean like 200 years. It’s like, again, it’s an industrial revolution. The first commercial railway has been opened between Manchester and Liverpool in 1830.

This is now 1834, 1835, and we are having this discussion. People are saying, the industrial revolution is moving so slowly. They told us that railways and steam engines will change the world.

So what? So a few people are going between Manchester and Liverpool. Didn’t change anything.

This is all science fiction. Because the timescale that we have no idea, even if all progress in AI stops today, the stone have been thrown into the pool, but it just hit the water. We have no idea what are the waves created, even by the AIs that already have been deployed, say a year or two ago.

Social consequences are a completely different thing. You cannot run history in a laboratory and see what are the social consequences of invent, you can test for accidents. You create the first steam engine, you can test for accidents.

You cannot test what will be the geopolitical implications of the cultural implications of steam engine in the laboratory. It’s the same with AI. So it’s just far too soon to know.

And I’m mainly concerned about the lack of concern. That we are creating, we are deploying the most, maybe the most powerful technology in human history. And a lot of very smart and powerful people are worried about what will the investors say in the next quarterly report.

They think in terms of a few months or a year or two.

Joshua.

Just quickly, I wanna thank. thank Yuval Harari because he’s talking about lack of concern. And I’ve started a new organization, a nonprofit, that’s trying to implement the scientist AI.

And Yuval has graciously accepted to be on the board. We need people like him to look with an independent oversight on what we’ll be doing with AI with society in the coming years.

All right, timescale, 0, 0, 0, 0, 0. Thank you so much. This was an amazing panel.

You’re all absolutely wonderful. Thank you for the work you do, and thank you for participating here. Thank you.

Thank you. Thank you.