Chris: For our audience. Singularity. Why is it called that? It refers to the idea that technological progress accelerates, so eventually it'll accelerate to the point where knowing what was going on this morning is irrelevant to this afternoon, which I know, like, kind of already feels that way. But a big part of this is because the singularity will allegedly be the result of an intelligence explosion, which is what will happen when we make an AI smarter than us. And then it wants to make an AI smarter than it, which wants to make an even smarter AI, and so on, until you wind up with an AI of God like powers that brings about transformative change to humanity, which, as you mentioned, absolutely is a millenarian ethos. And if transformative to humanity sounds familiar, that's because that's a transhumanist goal. Hello again. Welcome to Kel.
Kayla: What? Welcome to Celtics, Celts or just weird?
Chris: Celts or just weird? A spin off podcast where we talk about celtic peoples and are they weird?
Kayla: And the Boston Celtics.
Chris: Boston Celtics. Kayla.
Kayla: Sorry.
Chris: That's how you pronounce it correctly. I'm Chris. I am a game designer. I pronounce things wrong. And I also occasionally do some data science, analytics type stuff.
Kayla: I'm Kayla. I'm a television writer, and my co host is a dirty bostonite.
Chris: No, I'm not. I moved away from Massachusetts when I was, like, five.
Kayla: You can't escape your roots, Chris.
Chris: I was born in Danvers Hospital, but not that Danvers hospital.
Kayla: Not the Arkham asylum one.
Chris: Not the Arkham asylum. Not the one that Arkham Asylum was based on, I think. All right, Kayla, I know how much you like talking about weird hypothetical scenarios, so I've got one for you.
Kayla: Is it a lot, or is it a little? Because it's both. It's. I love it, and I hate it so much.
Chris: Don't stop me. Actually, if you've heard this before, because this is for the show, consider the following paradox. And this is called Newcomb's paradox.
Kayla: I. I don't want to do this.
Chris: That's too bad. Cause here we go. Imagine a super intelligent computer algorithm, or AI, or whatever you want, that is extremely proficient at predicting human decision making.
Kayla: Okay?
Chris: It's accurate 99.999% of the time. Or if you prefer, the original way that this was posited is it's infallible. It's always accurate at predicting human decision making. I find that a little distracting because that I'm like, well, nothing's infallible, so I prefer the 99.99% version. Either way, it doesn't really change the way we talk about it. So you have this extremely accurate computer in terms of predicting what humans will decide. Okay, now, imagine you are presented with two boxes. You are given one of two choices. You can take the contents of both boxes, or you can only take the contents of the second box B. What is in the boxes? Box A is transparent and always contains $1,000 guaranteed. Box B is opaque, and it can contain either nothing or $1 million.
Chris: Now, the prediction computer we talked about a minute ago predicted your choice ahead of time, but you don't know what it predicted. What you do know is that if it predicted that you would only take box b, then it put the million dollars in box b. If, on the other hand, it predicted you would take both boxes, it didn't put anything in box b. So the question is, do you take both boxes, or do you only take box b?
Kayla: I don't understand why you wouldn't just take, like, I don't understand. I think I'm too stupid for this. I think I literally am too dumb, because I don't understand.
Chris: Which one do you?
Kayla: I don't understand why you simply would not just take box B. Because if you have a million dollars, who cares about $1,000?
Chris: What's fascinating about this is that your reaction of, like, I don't understand why you wouldn't do blank is extremely common, and it's like 50 people. And this includes, like, professional philosophers, too, by the way. So, like, oh, I don't want to listen to that. I'm just saying, half of people, including the professionals, are like, well, it's obviously box b. And then half the people are like, it's obviously a and b.
Kayla: What do you think.
Chris: By the way, including, we talked about this on our discord, which, oh, we didn't do our call to action this time. Oh, my God.
Kayla: You just started with the anecdote.
Chris: I just started right with the anecdote. Everybody, if you want to support us, go to our Patreon. Patreon.
Kayla: If you want to see the hours long conversation Chris had about this, if.
Chris: You want to debate this with me personally, then come to our discord. The link is in the show notes. One of our lovely discord members completely disagreed with that and also thought that it was kind of like, obviously, why wouldn't you just take both boxes?
Kayla: I don't get it.
Chris: It's a.
Kayla: Is it positive?
Chris: It's a toughie.
Kayla: If the computer predicted that you would take both boxes, then you only get $1,000.
Chris: If the computer predicted you would take both boxes, then it put nothing in the second box, which, therefore, if you take both boxes, then you get $1,000 instead of.
Kayla: That's what I don't understand. Why would you pick $1,000 over a million dollars?
Chris: So I'm gonna try to explain the logic of the two boxer people, which is.
Kayla: Please do. Cause I think there's, like, brain damage going. Ignore.
Chris: Ignore the fact for a second that there was a computer that did a prediction and put stuff in boxes. What should you do in that case?
Kayla: Well, what's the setup then?
Chris: You just. Nobody tells you that a computer did this. What would you do?
Kayla: What's the setup?
Chris: The setup is, do you take one box or two boxes?
Kayla: So if there's a box of a million dollars and a box of $1,000, do I take both or one?
Chris: No, you don't know what's in the second box. It could be zero. It could be a million. The first box, you know, has a thousand. So do you take only the second box? That might be zero, might be a million. Or do you take both boxes?
Kayla: Oh, I see. Okay. So you're saying that.
Chris: What would you do in that scenario? I didn't tell you there was that prediction thing.
Kayla: That's too stupid.
Chris: I just.
Kayla: I'm too dumb for this. I think I am.
Chris: Nobody told you that there is a predicting computer. They just say, like, hey, box B might have one or the other.
Kayla: Then you would take both.
Chris: You would obviously take both. Right. So if that's the case, then why does it matter what the computer did in the past? No matter what the computer did, taking both boxes will always give you $1,000 more.
Kayla: Okay.
Chris: Than taking.
Kayla: So those people are saying that. I guess that's where the.
Chris: That's why it's a paradox.
Kayla: It's not a paradox, though.
Chris: Oh, it totally is.
Kayla: No, it's not.
Chris: The other thing I've noticed about this is it makes people angry for some reason.
Kayla: I'm enraged.
Chris: Yeah. I don't know. It's a weird phenomenon.
Kayla: Okay. I'm like, I can kind of see it. I can kind of understand, I think, the problem.
Chris: The computer made its decision in the past.
Kayla: I know, but you're saying that the computer is a perfect predictor. Yeah.
Chris: Yeah.
Kayla: So if it's a perfect predictor, but people who say take both think that the prediction was made in the past, so it really doesn't bear any weight. Exactly, but.
Chris: Exactly. No, this is why it's a paradox. Because whenever you go down one mode of reasoning, you always kind of go like, but.
Kayla: But what I'm saying is that it does bear weight because it happened and it happened correctly.
Chris: Correct.
Kayla: So if I take both boxes, I'm only getting $1,000. If I take the second box, I'm getting a million dollars.
Chris: Look, I want to know the correlation.
Kayla: I want to know the correlation of what people say versus whether or not they believe in goddess.
Chris: I think that's actually very salient, because I think that there's definitely. And, like, a lot of. I'm not the only one to say this. Like, a lot of people have said this. Like, the premise that there is an infallible predictor is, like.
Kayla: It makes it hard.
Chris: This is a crux here, right? That's something we're not used to, and it's something, frankly, that I don't think is possible. And this paradox is part of why I think that, because my feeling is as written. You definitely take box b. You don't. Two box. That's my feeling.
Kayla: My. Me too.
Chris: You definitely take box b.
Kayla: Wait, but I thought you and I got in an argument about this.
Chris: No, we got an argument about the mode of dust. We're gonna. We'll do that in a future episode. No, we're gonna. Kayla. I'm sorry. This is the podcast. Now. The podcast is just dumb hypotheticals, okay? So the reason I say it's box b is because if you are telling me that this infallibly predicts what I do, then that means that if I pick box b, it'll definitely be a million dollars. And if I pick both, then it'll definitely be $0. Like, you told me that. Now, what that also implies is that there's something going on here called retro causality.
Kayla: Yes. Yes.
Chris: Which is my decision. Retro caused the computer to decide whether it put money in one or the other or whatever. My decision in the present influenced what the computer in the past did. Because of that. Because of that implication of retro causality. That's part of why I think it's impossible to have something that predicts decisions infallibly.
Kayla: Because of the implication.
Chris: Because of the implication.
Kayla: I think that's why I want to know the correlation between people who believe in God and nothing. Because it's like, if you don't believe in God, then you have to take both boxes. Yeah, you fucking take the boxes. Because that's magical. It is. It's magical thinking. It doesn't make any rational, logical sense.
Chris: In the real world.
Kayla: In the real world. But if I'm existing within the context.
Chris: In the confines of the hypothetical, and.
Kayla: I'm taking, then I feel like retro causality exists.
Chris: I agree.
Kayla: And so if you're asking me, you have to ask me two different things. And I will give you two different answers if you ask me this question as is, it's b. If you ask me real life, it's both.
Chris: Agreed. Cause if you told me this was real life, then I would say that you were wrong about the infallible predictor. I would say that's not something that's possible. And part of it is because of the retro causality. And part of it is also like, there's a lot of chaos theory going on here. So my decision as to which box to take affects the computer's prediction, which also affects my decision, which also affects the computer's prediction. So there's a feedback cycle there, right? And feedback cycles are the canary in the coal mine for when you have a chaotic system. And a chaotic system is inherently impossible to predict in any pragmatic way, even for a supercomputer. It's not that. It's not deterministic.
Chris: The chaotic system is deterministic, but it quickly becomes impossible, even for a computer made of all of the atoms in the universe to predict something that has that property with it.
Kayla: That's why the dinosaurs had babies, even though they were all female. Spoiler alert for Jurassic park.
Chris: Life finds a way.
Kayla: Yeah.
Chris: And as a side note here, this isn't like, this is decision theory. Like, this isn't just like, well, some nerds arguing on the Internet. I actually.
Kayla: Please don't put me in the prisoner's dilemma with the computer.
Chris: This is not prisoner's dilemma with the computer. This is prisoner's dilemma between the United States and the Soviet Union. I recently heard an interview. It was actually on. It was one of Dan Carlin's interviews with Annie Jacobson, who wrote a book called Nuclear a scenario. And one of the things they talked about was there was this elderly former director of the United States nuclear Stratcom, which is like the people in charge of doing the nuclear strike, if that ever came to pass, and determining the strategy. And for people who don't know, in the cold War, we had this policy, and I say we. I mean both the US and the USSR, of what's called deterrence. We each produced an insane number of nuclear weapons, all pointed at the other.
Chris: And the reason we know the Russians aren't going to nuke us is because we have stuff pointed at them. And the reason that they know, we're not going to nuke them is because they have stuff pointed at us. Right. There's this mutually assured destruction which deters either side from doing anything.
Kayla: Right.
Chris: By the way, the paradox were talking about was called Nukem's paradox. I just now noticed that. Anyway, so this nuclear.
Kayla: More like, don't. Nukem's paradox, am I right?
Chris: So this former director of nuclear Stratcom was talking as, like, an old man, basically being like, you know, I think back now, and I wonder, like, if somebody had showed me that missiles were inbound to the United States, would I launch my missiles? Would I launch against Russia? And, like, it kind of feels like the answer is no, because at that point, deterrence failed.
Kayla: Yeah.
Chris: So even if we're all gonna die over here, even if the entire north american continent is gonna be glassed, what sense is there in, like, glassing the other hemisphere?
Kayla: Yeah.
Chris: What sense is there and killing everyone when half of everyone's already gonna die? Like, to the point where some people even, like, sort of thought experimenty. But, like, maybe the red button shouldn't be connected to anything.
Kayla: Mmm. Mm.
Chris: It should just be there as, like, a deterrence is the strategy, but if it fails, we shouldn't nuke, I think.
Kayla: But that the nuclear button, is it a button?
Chris: I don't know.
Kayla: Should be inside a human body.
Chris: Yeah.
Kayla: So that if someone's gotta push it, they gotta kill a literal guy. You have to. If you're mister president and you want a nuke, you're gonna have to kill a guy.
Chris: You have to, like, open him up with a knife, like saw style. Yeah. The paradox here, though, is that if you know that the other side is thinking that if both sides know that the other side is like, well, they're not really gonna do it right, then deterrence goes away, and now it's more likely to happen.
Kayla: Yeah. I hate it. Don't talk to me about this.
Chris: It's horrible. So decision theory, which is what this whole newcomb's paradox and what I just talked about with the nuclear exchange, decision theory is a thing, and it's actually, like, not just an esoteric thing, it's a thing at, like, the highest possible stakes for civilization, for the human species. So I just, you know, and I invite you guys in the audience, too, like, ping us on social media or come on, discord and talk about it, because I do get really curious about how people think about these types of paradoxes, and I need to pump the brakes here so that we can move on, because like, God knows we've already spilled like, way too much digital ink over this on discord, as you mentioned. But this thought experiment is one of the favorite thought experiments of less wrongers, and they like thinking about decision theory.
Chris: It's a field they care about a lot. Eliezer Yudkowski even invented his own branch of decision theory, or sloppily reinvented, depending on who you ask, called functional decision theory. If you've recently listened to our season two, episode four on Roko's basilisk, we called it timeless decision theory then, but Mister Yudkowski has since rebranded it anyway. It has to do with what decisions we do make or should make under different conditions. And I think the connection to AI development is kind of obvious here, right? You're literally trying to program a decision machine if you're making AI. And I won't go into all the crazy details that underpin the Roko's basilisk idea because we did that on the other episode. But I will mention that decision theory is a big part of it since we're talking about things less wrongers are into here.
Chris: This may also be a good time to briefly recap what Rokos Basilisk is. So basically it's this thought experiment that somebody posted on less wrong. The poster's name was Roko, hence the name Roko spasilisk. And the idea was like, if there's a future super intelligent AI, even if it's good for humanity, it may decide that it should torture people that knew that it could come into existence but didn't give their resources to help it come into existence. And the reason it will decide that is that might influence people in the past. Sort of like the nukem computer thing. It will retro cause and influence people in the past to try to bring it about sooner. And it will make that calculation based on the utilitarian idea.
Chris: Like every moment I can make myself come into existence sooner is like one more person I can save from cancer because I, as the super intelligent God, like Aih, cured all cancers and did all this cool stuff. And so like every moment that I do that, I push that back. I save x number of lives. So it's totally worth it torture these people that knew that I could be in existence and didn't support it. And that's really scary to these folks.
Kayla: I also want to know now, and I know it's probably zero because I doubt there's a lot of faith in God, and belief in God in less wrong community is my bias assumption. But again, I want to know the correlation of, like, what your stance on Roko's basilisk is and what your stance on God is? I want to know.
Chris: Yeah, I know.
Kayla: And when I say God, I don't mean like, aren't you a Christian? I mean like, do you view the universe as having some sort of sentience or not?
Chris: Right. Did somebody create the universe or is it just all random?
Kayla: I think that my answer and my change in answer has reflected my change in relationship to goddess.
Chris: What's your change in answer?
Kayla: Well, now I'm like, fuck you, Roko's basilisk. Fuck you if you're gonna exist. Fucking torture me, bitch. Like, that's how I feel.
Chris: I mean, under decision theory conditions, that might actually be the right choice because then that means you're uninfluenceable.
Kayla: And in fact, that's who become uninfluenceable.
Chris: Eliezer Yudkowski himself said that basically if you like, pre commit to saying, like, fuck off, then you are uninfluenceable. And then they won't. The super intelligent AI won't torture you because it wouldn't have had any effect.
Kayla: I don't care. I'm just saying you agree with Aliezer. I hate that. Because then it means like, oh, saying fuck you is the right answer. Because then he won't get you. No, get me, get me, bitch. I'm really angry about this because I'm angry about God and the universe.
Chris: Yeah. And I'm with you, man.
Kayla: I just. I have a lot of anger. And so now I don't care about Roko's basilisk anymore.
Chris: To me, I still find it like an interesting thought experiment. That's what it was originally supposed to be. But, like, it really scared a lot of people.
Kayla: It scared the shit out of me.
Chris: It scared the shit out of me. Of you. It scared me. But like, people on less wrong were like having literal nightmares about it. Some people. Some people, not everyone. Some people are like, this scare me, that bit. This is dumb. Some people are having nightmares. I read on rational wiki. Rational wiki has an entry on Roko's basilisk and half of it is devoted to explaining why it's b's to make you feel better and the reason they put that there. They say the reason we put this here is because it wasn't here. We just had. Here's what Roco's vascular is about. And we kept getting emails from less wrong community members saying, like, nobody will let us talk about this on less wrong because it became like, a taboo subject.
Kayla: Right.
Chris: That's in our other episode about how it got Taboo ified there. Nobody will let us talk about this on less wrong. I'm so scared about this and like, I don't have anybody to talk about it with. You guys have this article that seems pretty, you know, well researched. Like, what do you guys think? So apparently they got like, so many emails from like, scared less wrong people that now they have this whole section called. So you're worrying about the basilisk and it's like literally like paragraphs and paragraphs long and like, going into all the different reasons why it's not something to worry about. But that's just like, that's how much effect it had, right? And like, people even like, did donate quite a bit of money to AI research and like, what's called the AI alignment problem because of that.
Chris: Like, now some people donate money because they think that's an existential risk and we should donate money anyway, right? I don't think that's necessarily a bad idea, but some people definitely donated money specifically because it's like, holy shit, I'm gonna get tortured in the future by this, like, God, AI.
Kayla: Oh my God. I hope that this was a marketing psyop.
Chris: Nice. Yeah. Roko was actually working for the organ. By the way, the organization that a lot of people donate to is run by Eliezer Yudkowski.
Kayla: Oh, that seems extremely important to say.
Chris: Yeah, we did say that in our episode.
Kayla: I didn't remember that.
Chris: It's not the only organization that people will donate to, but like, he has an organization dedicated. It was called Miri and then it got changed to Siri or something. Anyway, machine Intelligence research Institute dedicated to the quote unquote the alignment problem. Aligning a super intelligent AI's motivations and desires and personality with human goals and desires.
Kayla: I have a lot of problems with somebody coming up with a thought experiment.
Chris: He didn't come up with it. Roko did.
Kayla: You're right. Thank you. You're right.
Chris: And Eliezer tried to purge you from the site.
Kayla: I'm wrong. I'm wrong there.
Chris: I'm just making you less wrong.
Kayla: Thank you for making me less wrong. That helps me a lot.
Chris: You could counter me that Eliezer, when he was trying to stamp it out and it only made it worse, knew what he was doing. But I don't.
Kayla: No, and I think that. I don't think there's any. I think that the Barbra Streisand effect is extremely difficult to deal with.
Chris: I completely agree. So that whole thing alienated a lot of people in the community and was part of what created this rationalist diaspora is like a lot of people either because they thought it was silly and dumb, or because they were scared of it, or both, sort of detached from less wrong as a website.
Kayla: Fascinating.
Chris: Yeah. Speaking of the basilisk, this brings us to another common trope on less wrong and in subsequent rationalist communities, the idea of stopping existential threats to humanity. I mean, specifically ones where AI kills us all. This community doesn't really care as much about more mundane existential threats like climate change or things like that.
Kayla: Why not?
Chris: To be fair, I'm really generalizing here. There are definitely a lot of people in the community that are worried about the climate crisis. But like, overall, on average, the people that are in this rationalist community care much more about the Sci-Fi type threats like AI and nanotechnology. And you just asked me why not.
Kayla: Yeah, the goo problem.
Chris: Yeah, grey goo. So the reason, the answer to that is they feel like it's much more of an existential threat. Climate change will be disastrous, but it won't potentially kill every single human being and prevent us from ever existing in the future the way that they think AI or nanotechnology might. So they think that it's a bigger problem to solve, even though it's like one problem is kind of here and the other one is like, yeah, I don't know if that's gonna happen, man.
Kayla: That's the problem. Like, when you have like hundreds and hundreds of scientists being like, hello, this is the existential crisis of our time and we must act now versus, like, your friends on the Internet saying, wouldn't it be nutso if like, we invented nanobots and then they, like, replicated and they replicated so hard, they replaced every single, like, thing on planet Earth until planet Earth was just nothing.
Chris: Nanobots. Yeah. So I want you, and also you, our listeners, to keep this in mind because this is going to be a recurring trope, a recurring idea for the next, like, several episodes, is that, like, there's definitely a disconnect between folks in this community and what they care about in terms of this sort of, like, really speculative thing that they, like, think is probably true, though, and is really dangerous and, like, actual stuff that's happening right now in the real world.
Kayla: Right, right.
Chris: That's gonna come up again.
Kayla: Okay.
Chris: Yudkowski himself was even featured in a Time magazine article, which, my opinion, this is last year, this is 2023. I think this is a gigantic mistake on Time's part to platform him this way. But anyway, the whole article was like this ultra panicked screed about how AI is going to kill everyone if we don't stop development immediately. And he has all this whole section about how he and his wife are really scared about their kids not having a future.
Kayla: There's this whole, why would you be scared of that? Their kids are signed up for cryonics.
Chris: Right? But see, that's the thing, is that AI, I'm just being a bitch, but it's still kind of salient because the reason he'd be scared for that and not scared for something like climate change is because like, climate change will still have the cryonically frozen guy. But if AI kills everyone, right, if the grey goo consumes the entire earth, then cryonics be damned. It doesn't matter.
Kayla: I guess I just kind of feel about not AI, but definitely about the grey goo and like maybe AI to some extent, the way that like climate change deniers feel about climate change in like the, I mean, that's such a big, not even the come on, it's like, it's such a big problem that it's like not even worth worrying about. I don't have any control there. Like, if the planet becomes grey goo, then the planet becomes grey goo.
Chris: Caleb, maybe you should donate to Eliezer Yudkowski's machine intelligence research.
Kayla: Choose. I would rather donate to Mitt Romney.
Chris: He does have a binder full of women.
Kayla: Exactly.
Chris: So, yeah, so like part of this article too is like he has this little anecdote where he's like, his wife texts him about how their kid lost their first tooth and how sad it was. She was sad watching this happen because it made her think of how doomed the kid is.
Kayla: Okay, see this I need to have, I need to talk to him. I need to talk to Eli Ajakowski because I'm just like, how do you make it through the fucking day without thinking about you fucked up by having kids? Like, you did a bad the article also. And I'm like, honestly, I'm a little worried when you get that anxious, which I totally understand, talking again, as a parent of a dead child. Totally get it. But when you get that level of anxious, I really worry about the mental health impact it has on you. Oh, and your ability to.
Chris: Oh God, I am super worried about the guy. Yeah, absolutely. He is very upset right now. Here's another quote from the article. Make it explicit international diplomacy that preventing AI extinction scenarios is considered a priority above everything else, including preventing a full nuclear exchange. And that allied nuclear countries are willing to run some risk of nuclear exchange if that's what it takes to reduce the risk of large AI training runs.
Kayla: What is nuclear exchange?
Chris: Nuclear exchange is what we talked about at the top of this episode where, like, we nuke each other and, like, almost everyone dies. So because he thinks that AI is everyone dies, he's saying that nuclear exchange is preferable if it prevents everyone dies. It's preferable to have almost everyone dies.
Kayla: See, I watched BBC's threads, kind of nuts, and I think that the grey goo scenario is better than that.
Chris: I think so, too.
Kayla: Not that I'm not. I'm not advocating for. I'm not advocating for anything here. I'm just simply saying, are you on.
Chris: The side of the nanobots?
Kayla: I'm on the side of the.
Chris: Oh, my God. Are you native nanobots? Are you nanobots, man?
Kayla: I'm nanobots, Mandy. I don't know. I don't even know what I'm saying anymore.
Chris: When he says international diplomacy, he basically means everybody needs to agree to have this not happen. And if there's a rogue actor, that's like, I'm going to make some. Even if it's a state entity, that's like, yeah, we're doing AI research. The other nations in this accord should be willing to use nuclear weapons and even be retaliated against. If Russia did it, if everybody was like, let's agree to stop doing this, and Russia was like, fuck you, we're doing it, then everybody should be willing to nuke Russia and be nuked.
Kayla: I can't follow you all the way down.
Chris: That's good, Kayla.
Kayla: Yeah. I'm not going to lie. I'm scared of AI.
Chris: Sure.
Kayla: I do not think it's wrong to consider AI an existential threat.
Chris: I agree. I agree.
Kayla: And potentially a very bad one.
Chris: The danger is not nearly clear in presence enough to warrant comparison to nuclear exchange.
Kayla: I think as long as we're having the experience of, like, hey, this AI company just revealed that all of their AI was actually, like, a warehouse full of people india googling for you, we can calm down a little. And also, like, we know what. Yeah, this is what you said. We know what nukes would do to us. We don't know what AI will do.
Chris: Right. It's an unknown quantity that a lot of these people think of as known.
Kayla: And I think you can convince yourself of how it's actually going. This is the problem with Roko's basilisk, is that you can convince yourself that you know absolutely what's going to happen.
Chris: We are more rational than everyone else. We have thought this through and it's a logical inevitability. If you just simply apply utilitarianism and the decision theory and you've read the sequences, this is gonna happen. Like, you can definitely convince you. That's why people got so scared of it. It's not because it was a scary thought. It was because it was like, it felt like certainty.
Kayla: Right. And I, and I have been in that. I've, I've been there.
Chris: Yeah.
Kayla: And I get it.
Chris: And like, I don't disagree with Eliezer's the rest. You know, there's part of the article where it's like, we have to, like, stop this now and like, we have to like really take it seriously and like, we don't have any idea what we're making right now. And like, we, it's not that we shouldn't make it like 50 years from now once we've like, done the work to understand what the fuck is going on, but we should stop it right now.
Kayla: I'm also don't disagree with any of that.
Chris: But then when he goes to like, and we should nuke people, like, that's like, whoa.
Kayla: Yeah. And I think it's also like, it's not bad to have in this community to have the voices that are like, let's find some breaks when so many voices in this community are like.
Chris: But like, when the, let's put a.
Kayla: Fucking brick on the pedal.
Chris: Oh, yeah, absolutely. And I, yeah, that's, I think that there's a lot of value in the sort of what's called, you know, the doomer community.
Kayla: Right.
Chris: But like, actually I shouldn't even say that because I think doomers, the proper definition of Doomer is like, you think everything's fucked and nothing matters, which is as much of.
Kayla: I know it's gonna happen.
Chris: Paralyzed. Yeah, exactly. Like, I know this is gonna happen, so it doesn't matter.
Kayla: Right. I know it's gonna happen. I know this is gonna happen. So therefore Roko's basket is gonna happen.
Chris: Right. So that's just as bad. But like, for people that are saying let's pump the brakes. Yeah. I definitely think that's super valuable. I think it's counterproductive though. When you have somebody, then you have a pump the brakes guy also saying these crazy things, then it's easier to go like, okay, the doomers are stupid. That's obviously, we can just go full forward because look at the guy saying we shouldn't. He's a nut job.
Kayla: Is there.
Chris: Right. Like, reading this article made me for a second go like, oh, I guess we're probably fine. This is nuts.
Kayla: How does Eliezer make his money?
Chris: I believe he is. He still runs Miri, the Machine intelligence research institute. Okay, so it's a nonprofit that is like, again, it's, their whole thing is like AI alignment. We want to make safe AI. We want to make sure that people that are doing AI do it safely.
Kayla: And this could be a bias, but I feel like that at least gives me a sense of whether or not this is somebody who is profiting off or grifting from fear mongering versus somebody who is a true believer and not profiting off of that necessarily. You can make the argument that, well, if he's saying that, oh, we're all going to die from AI, then people are more likely to go to donate to his thing. But I think that there's way more money these days in being on the pro AI style.
Chris: Way more money.
Kayla: So if you were simply in it for the grift, and again, this could be bias, whatever, but I just, that's a data point. You're more likely to be in it for the Griffith.
Chris: Yeah. You're gonna be Marc Andreessen, right? Not Eliezer Yudkowski. Like, he says crazy things, but I do think that he is being authentic when he in his beliefs. I'll give you a little bit of context, though. This isn't the first doomsday he's been wrong about.
Kayla: Oh, no. Was he a 2012 guide? Did he think the end in 2012?
Chris: No, no. He only does science based doomsdays. So in 1990s, a young Yudkowski predicted that nanotech would destroy the world by 2010. It didn't, which it didn't. Apparently, he abandoned that idea at some point because he later predicted his development team. So the team he was working with at the time would build a super intelligent AI by 2008.
Kayla: This is not.
Chris: It also didn't happen.
Kayla: This is not, this is now going exactly against what I was just saying because this is the pattern of somebody. This is like those cult leaders that keep changing the date of the end.
Chris: Of the world, right?
Kayla: They're like, oh, yeah, it's totally end on this date versus, oh, that date passed. So it's totally going to end on this date.
Chris: So the build a super intelligent AI date wasn't a doomsday prediction at the time. This is when he was like, rah. I think we should build AI's. And then at some point, I don't exactly know when he went from, like, we should build AI's and it's cool and we'll align it. Okay. To, like, we need to really hardcore pump the brakes. At some point, he made that switch. So that wasn't like a. A doomsday prediction, but the first one was. And that was like, even weirder because I'm like, really? We're gonna have nanobots by 2010 that are gonna destroy everything. Dude. Come on, man.
Kayla: The great goo really got some people.
Chris: I know, but let's bring it back. The takeaway here is that the rationalist community really cares a lot about existential risks, and they tend to care about the singularity related ones much more. And we've talked about singularity a little bit. I feel like it's time. I'm kind of obligated to mention singularitarianism here and kind of explain it a little bit because there's a huge overlap between the rationalist community and the singletarian community, largely because of Mister Yudkowski. He can definitely be considered a singletarian because he thinks that we're moving towards this future with a super God like AI. Kayla, I know you know all about the technological singularity because I was totally in the cult after reading Ray Kurzweil's book the singularity is near, which was written in 2005, so we can extrapolate his definition of near.
Chris: It has to be at least 19 years.
Kayla: I think you literally said to me, like, have you heard the good news?
Chris: Did I rinse?
Kayla: No, you didn't. No, you didn't.
Chris: Is it bad that I thought that might be something that I could have said?
Kayla: No, but it was. You did recommend this book to me with. With the fervor of a religious man. Well, and it wasn't wrong. I mean, maybe it was. I don't know. I like it.
Chris: Singularity is. Near is like a burn salve if you're, like, scared about existential threats and AI, because it's like the whole book is basically like, don't worry, this is going to be awesome. It doesn't say, like, there won't be existential threats. You know, it is honest about, like, we're just going to face different ones, right? Like, it's still. There's still going to be challenges. But if you're worried about Terminator two, Judgment Day, that's unlikely to happen. AI's are going to be good for us, and also we might become immortal. So if you're worried about dying and worried about specifically dying from robots, it's like, oh, man, this guy actually says that it's going to be a utopia, and that's kind of like where singletarianism was born. That's a bible ish for the idea of the singularity.
Chris: But, yeah, so we read that a while ago for our audience. Singularity. Why is it called that? It refers to the idea that technological progress accelerates, so eventually it'll accelerate to the point where knowing what was going on this morning is irrelevant to this afternoon, which I know, like, kind of already feels that way. But a big part of this is because the singularity will allegedly be the result of an intelligence explosion, which is what will happen when we make an AI smarter than us. And then it wants to make an AI smarter than it, which wants to make an even smarter AI, and so on, until you wind up with an AI of God like powers that brings about transformative change to humanity, which, as you mentioned, absolutely is a millenarian ethos.
Chris: And if transformative to humanity sounds familiar, that's because that's a transhumanist goal. So most singulitarians are sort of like, by definition, also transhumanists. And Eliezer is definitely one of them, as someone who's very concerned with future super intelligences. And since the rest of the less rank community more or less, either by just dissociation or by adulation, most of the less wrong community also believes in the technological singularity as well. Okay, so I want to mention the rationalist diaspora one more time here, because throughout these episodes, I've been talking about less wrong and stuff that's happened there. And I want to emphasize that most of these events are, well, in the past, like, a decade in some cases. Eliezer himself hasn't posted on less strong in quite some time. And the site activity is way down from its peak.
Chris: It's definitely still active, but a lot of folks have moved on to other homes on the Internet. And I think we mentioned this last episode. But one of the most prominent posters on Lessrong, a man by the name of Scott Alexander, created a sort of less wrong 2.0 with a slightly different flavor called Slate Star Codex, which, again, it's just like another online forum. And as near as I can tell, this is where, like, the largest chunk of the rationalist diaspora currently lives online. And now I found myself with only one more thing to talk about. And I don't know why this is coming last, because it's one of the weirdest and like most important bits about less wrong and rationalists. Bayes theorem.
Kayla: Bayes theorem.
Chris: Bayes theorem you might remember this from the definition where the guy said, like, oh, it's like a pragmatism philosophy that's concerned with cognitive biases and Bayes theorems, which, like, at the time, was totally contextless. What the fuck is he talking about?
Kayla: Yeah, yeah.
Chris: So Bayes theorem is kind of two different things. The first thing that Bayes theorem is just a theorem in mathematics that deals with how probability works. I won't go into the details. It's just something in math.
Kayla: Sorry. Something in math is. That's just like how I approached all math classes. Like, that's what. That's what trying to learn math in school felt like to me. Was somebody going like, oh, it's something in math here. It's just something in math. It was that vague. I hated it.
Chris: I mean, that's what it is. It's basically that it's just something in math. The second thing that Bayes theorem is this sort of, like, magical word of invocation that rationalists use, and it contains several ideas, a few of which are, like, even maybe sort of related to the actual math, but kind of not really. When rationalists say Bayes or Bayesianism, and they say those things a lot, what they mean is a worldview where you acknowledge you are acting on uncertainty and that your actions and decisions can only ever have some probability assigned. Nothing is ever really 100% guaranteed truth. And as a part of this worldview, you should also have an initial probability of something being true. Bayesians refer to this as your quote unquote priors. There's some more jargon for you, by the way.
Kayla: I like that one, though, which you.
Chris: Then update based on what actually happens.
Kayla: This is good. I like it.
Chris: I like it, too. I'm honestly a big fan of this sort of worldview, and I definitely would say I even share it. I think it would benefit a lot of people if they were able to think in a more probabilistic manner. And we'll talk about that in a second. I just want to help clarify an example of all this prior stuff I was talking about. Say you have a coin and you want to know if it's weighted or not. You should flip the coin a whole bunch of times to find out. Right? Yes. But a Bayesian would also say that you can and kind of should assign some chance ahead of time based on things like, well, how does the coin look? Does it look normal, or does it look like it has a little bump in it?
Chris: Did I get this coin from a bank or from Bob's discounted weighted coin emporium. Did Bob himself tell me it was weighted? All of those things are your priors, right? So, based on those priors, you might say, like, I got it from Bob's weighted coin emporium. So it's probably weighted. Or maybe you got it from the bank. So you're like, okay, it's probably just like a regular coin. It's 50. Those are your priors. Then you flip the coin and update your priors based on the results. Maybe you flip Bob's weighted coin a thousand times, and it comes up heads. 497 entails 503 times, and you're like, oh, shit. Actually, that's not weighted. You update your priors based on the results.
Kayla: This is like how to think like a robot, but the right way.
Chris: And that's why they talk about it, because they're constantly trying to solve that problem. But it's all just kind of a convoluted way for less wrongers to say that they think it's a good idea to update your beliefs, which, yeah, I think there's also a dash of extraordinary claims require extraordinary evidence in there, right? That's kind of what it means. If your prior, quote unquote, is extraordinary, then you need to say, okay, well, it's probably a low chance for this happening based on my prior knowledge of what could happen.
Kayla: Right.
Chris: I also definitely detect a hint of trust network in there, too. Right? Like, updating your priors is kind of like what we do when, like, somebody we respect says something dumb or quotes something from a site that we don't like or whatever we have to like. Okay, well, you know, before, my probability of this person saying something true was 70, and now it's like 65.
Kayla: Right.
Chris: Maybe that also updated my priors about the stupid website. Maybe it's not so stupid. I don't know. And I think everyone would benefit from checking their hindsight bias, too, and be more comfortable using uncertainty in their reasoning. Just because you gambled and won a lot of money doesn't mean that was a wise investment. And just because you invested in an index fund that went down this month doesn't mean that it was a bad investment. So I think it's a good way of thinking about things overall. It's just that rationalists use the word Bayesian in this really bizarre in group signaling sort of way, which has, like, little bearing on the actual mathematical Bayes theorem. But it's really important to them. They say it all the time. They consider themselves. Some even think that their community should be called Bayesianism instead of rationalism.
Chris: Like, that's how into this they are. Which is so weird, because, again, it's just about, like, I know that I'm acting on uncertainty and I should update my beliefs. That's, like, tangentially related to the actual Bayes theorem. But, you know, Yudkowski extolled Bayesianism in the scripture. I mean, sequences, it became a thing with a capital t. All right, Kayla, it's time. I know were kind of holding off undoing criteria since.
Kayla: Oh, I was gonna ask. I was gonna ask if we could do criteria.
Chris: Yeah, were kind of holding off because were like, let's wait till we've covered transhumanism. But, like, as we pull on that thread, I think we're getting to a point where we should stop and reflect. And also, like, less wrong is a nicely contained community.
Kayla: Yeah, we don't have to do transhumanism. We can just do these things.
Chris: I mean, I think we still should.
Kayla: Not right now, though.
Chris: Not right now. I think we still should at sort of, like, the end of this. But I think we can also stop here and be like, okay, what about lesser arm?
Kayla: Stop here and extol judgment upon a group, which I have been dying to do. No, I haven't been dying to do it. But we haven't done it in a while.
Chris: We haven't done the criteria in a while. I've definitely judged groups. I do that every day. All right, let's start. Charismatic leader.
Kayla: Check, check, check.
Chris: I mean, you might disagree with how charismatic the guy is, but people in.
Kayla: The community, I mean, he is a hero.
Chris: He is a hero. Self proclaimed hero. People in the community do appeal to authority there a lot. So I'd say this is definitely present and fairly high expected harm.
Kayla: The basilisk. It hurt people.
Chris: The basilisk did hurt people. I think that Eliezer's particular brand of dumerism is harmful too, because it kind of discredits ideas that might actually be good, like AI safety.
Kayla: I think that if Roko's basilisk hadn't been so impactful that it tore the community apart, I could be like, maybe that was just a one off. But it seems, like, kind of fundamental to the group. So I'm just gonna. I'm gonna say. And, yeah, Elliot, with the things that he's talking about now and, like, having time, news articles that are like, we're all gonna die, feels like we can say the harm is high. Yeah, expected harm is high.
Chris: There's also, like, you know, Sneer club, which is, like a community for, like, people who were in less wrong. That kind of, like, hate it now, right? That implies some harm there, too. So I'd say it's high ish.
Kayla: They're just haters.
Chris: It's not, like, extremely high.
Kayla: It's not heaven's gate.
Chris: It's not like heaven's gate. It's not like Jonestown, but it's high ish. Niche and society.
Kayla: It is niche, isn't it?
Chris: Yeah. I think it depends on what you consider, like, society in this case.
Kayla: We live in a society. We live in a society, and I don't want to anymore.
Chris: I think that if we're saying, like, all of America, I think it's definitely niche. I think when we're saying Silicon Valley, it's not niche.
Kayla: Like, if I went to a party with a bunch of tech bros. From Silicon Valley and I talked about less wrong, they would know what I was talking about.
Chris: Absolutely.
Kayla: I never want to go.
Chris: And especially I've heard that there's, like, that people will go to effective altruist gatherings, and you're gonna be a little bit lost if you don't know all of the joke from les fraud.
Kayla: Oh, my God, I'm so glad I live here. Even Hollywood's not this bad. And, like, I was gonna say this before, Hollywood has so much fucking jargon that it can be hard to have conversations with industry people, because just the nature of being on set, you have to have jargon, and there's so much jargon. There's not this much jargon.
Chris: It's very jargonized. I would also say that if you're saying influence on society, it's pretty high, because the people that this is popular with are extremely powerful, potentially powerful folks in Silicon Valley. They're the rich, they're the elite. They're the ones that are making your iPhone, and it's apps. So.
Kayla: But is it these people, or isn't it the opposite people? Is it less wrong that's these powerful people? Or is it the we're not scared of AI people that are the powerful people?
Chris: Look, Eliezer Utkowski himself is very scared of AI. That doesn't mean that everybody in the rationalist community is. There's definitely some that aren't. There's some people that are like, I'm about all about alignment and safety, and some people are more like, I'm accelerationist, and I just think that we should pedal to the metal, and then there's people in between. Okay, so I bring the influence thing up as just sort of like, well, is it niche if, like, the influential people have this in their brain all the time? I don't know. I kind of feel like I'm leaning away from niche. Yeah. I don't know. Based on the fact that it's ubiquitous within Silicon Valley and the influence that Silicon Valley has on the rest of culture, I'm thinking not niche, actually.
Kayla: I hate it. I don't know. This exists in a gray, and I don't operate well in that.
Chris: Yeah, I'm sorry that everything's gray.
Kayla: Grey goo.
Chris: Dogma.
Kayla: Seems pretty dogmatic to me, actually. You tell me.
Chris: I don't know.
Kayla: You tell me.
Chris: Yes and no. Right? This is part of the problem I've been having this whole time, is, like, these guys seem very reasonable and they're fighting bias and they're engaging in good faith.
Kayla: And if part of the whole thing is update your body of knowledge, that's not dogmatic.
Chris: But I've also seen. Yeah, but then I've seen people be kind of referring to their, like, before. Before I discovered the rationalist community, I was blah, blah, thought wrongly and was dumb. But now that I've discovered it, everything. And some of that is like, oh, that's great. You found something that worked for you really well. Some of it's like, there's a before and after where I was, like, dumb before and I'm smart now. So that feels a little dogmatic.
Kayla: Does that feel dogmatic?
Chris: Yeah. Everybody else is wrong except me because I'm the rational one.
Kayla: Yeah. Yeah. Okay. I think it's. I I think it's high.
Chris: I think it's high ish.
Kayla: But, like, I also high in a weird way. Yeah.
Chris: Ritual high.
Kayla: Yeah, yeah.
Chris: Read the sequences and you'll know. Safe or unsafe exit. I did not get the impression that.
Kayla: Like, seems safe to me.
Chris: Bailing from the community was gonna get you ostracized or anything. In fact, there's a whole diaspora of rationalists that seem to be fine with the fact that they're a diaspora.
Kayla: And you can be involved in this without having to sever all of your ties with everyone around you. So even if you leave, less wrong. It's not like QAnon, where the philosophy slowly forces you to weed out everyone in your life irl, and you're stuck with only the Internet. I don't know.
Chris: But at the same time. At the same time. Now me think, like, you know, if you get to the point where, like, all you can talk about at parties is AI.
Kayla: Right. But that's different than the explicit things in QAnon of, like, how can you associate with those baby killers type stuff?
Chris: And. Yeah. And it's not. It's not like we're talking about exit here. Not like we already talked about the. In jargon.
Kayla: Right.
Chris: This is like, whether exiting is. Okay.
Kayla: This is what I'm saying. Like, you. You're not going to leave. Less wrong. And not have a community existing. If you had a community going in, you have a community going out.
Chris: Yeah, yeah. Percent of life consumed. I don't really know how to answer this one. I didn't run across anything where people were like, holy shit, I spend all my time here. That being said, though, like, there's some people that are, like, really prolific posters where I'm like, how do you have time to do anything else?
Kayla: Well, it seems like this is all that people can, like, fucking talk about in some of these circles.
Chris: Yeah. I mean, this has the potential to be pretty high.
Kayla: I'm gonna say at least medium. Medium to the best of our knowledge.
Chris: Especially if you're one of the, like, I can only talk about AI at parties, people.
Kayla: Right.
Chris: All right, then finally, chain of recruits.
Kayla: You tell me.
Chris: I don't think anybody's recruiting anybody else here. I don't get the sense that people are like, come join this community. We're gonna love bomb you.
Kayla: I feel like it's this. It's another. There's that similarity with empty spaces, again, is that it's just like something that you happen upon or come upon or seek out versus it is knocking on your doorstep.
Chris: Yeah.
Kayla: Knocking on your doorstep. Can you knock on someone's doorstep? Why would you do that?
Chris: Yeah, we're getting to the end here, folks, so our brains are even more wrong than normal. Okay, so let me go through the criteria here. We have charismatic leader, very high expected harm, somewhat high niche in society. Not really, if you count the influence. Dogma, high ish, depending ritual, very high exit safe. It can consume your life. Chain of recruits is low. What do you feel like that is.
Kayla: I think I have a problem.
Chris: Are you confused now? Because that's the whole thing.
Kayla: I think I have a general problem.
Chris: Though, is that we shouldn't be doing this and our whole premise is flawed.
Kayla: I feel like I don't think I can call anything a cult. What the. Who the fuck am I? What are you talking about?
Chris: We probably should have done that before episode like 100 or whatever.
Kayla: I don't think it's a cult.
Chris: You don't think it's a cult. Why? There's a lot of high scores there.
Kayla: It feels that way. I think it scored low on some of the really important cult ones. I think that chain of recruits and safer, unsafe exit are really big for me. I think if there's not an unsafe exit, that's like a weighted answer. But the other things are so high. What did we call? I mean, Q and a. We called a cult, obviously. Empty spaces. Did we?
Chris: I don't remember.
Kayla: I'm just thinking of other Internet based things. Did we call Cicada? We called Cicada a beneficial cult.
Chris: I mean, we already answered this. We did this episode. We did, yeah.
Kayla: What did we say in Roku? What did we say in less room.
Chris: Before we said it was a cult. I can tell you what I think.
Kayla: Yeah. But now I have.
Chris: We also have more criteria now and more knowledge because this is all new stuff we're talking about. Do you wanna know what I think?
Kayla: I would love to.
Chris: I think it's a cult, but who cares?
Kayla: That's kind of what I feel like.
Chris: I think that it's, like, a lot of this. Okay, so first of all, these criteria are bullshit, and we're bullshit, and we're just doing this for fun. Like, the criteria are a way to kind of, like, talk about and think about and summarize all of the things that we've discussed on the episode. It's not a. Like, it is a cult. It is not a cult because cults aren't real. That's not a real thing.
Kayla: Right.
Chris: It's just a thing to, like, kind of fun talk about, like, very real stuff. So I think under the, like. Yeah, we're going to call it something thought experiment, hypothetical. Yeah, it's a cult in the real world. Like, I don't know. Maybe it is, maybe it isn't. Who cares? Like, the important stuff is that, like, it's a diverse community in terms of, like, the good and the bad.
Kayla: I think a religious studies anthropologist could write a really compelling, like, paper on this community.
Chris: Yeah. And I. And I like a lot, again, it's.
Kayla: A new religious movement or a high control group or whatever.
Chris: Sure, sure. And I like a lot of the stuff that they talk about.
Kayla: Yeah. It's a cult, though. Like, I think a lot I think is a cult. It's the same way that I think, like, because in my head I went, like, well, I think singularity is a cult. Yeah.
Chris: Yeah.
Kayla: It's totally.
Chris: It's a cult. Doesn't mean it's bad.
Kayla: Yeah.
Chris: It just. It's got some cult like stuff going on.
Kayla: Yeah, it's a cult. It's. It's probably not as bad as some of the other ones. I think that you can get out of this one a little easier than you can get out of something like QAnon, for sure. But it is a cult. It is a cult.
Chris: I agree. And that's good because we agree with ourselves. So at least we're consistent, self contained.
Kayla: We're not dogmatic at all.
Chris: The sources are all in the show notes. We're kind of just doing that now rather than yapping about them on here. I don't want to listen.
Kayla: Don't you want to hear me say Wikipedia every episode?
Chris: Do you know how many tabs I have open that are just Wikipedia, Kayla?
Kayla: I do.
Chris: Oh, my God. I have so many less wrong tabs open, too. Holy shit. Next time on culture. Just weird. We take notice of the fact that we are deep into an interconnected rabbit hole, and we talk to an expert about this whole transhuman, extropion, singletarian rationalism, yada, yada, rabbit hole that we've been down.
Kayla: Thank God. Tell me what to think. Someone tell me what to think.
Chris: I know. Just. Just tell me what's going on. Tell me what to think. Please. I want to join your cult. Whatever it is, I'll join your cult. You tell me what to think. I have to worry about anymore. And with that, this is Kayla, this.
Kayla: Is Chris, and this has been coulter.
Chris: Just please don't make me have to think about it.