Gwern Branwen - How an Anonymous Researcher Predicted AI's Trajectory AI transcript and summary - episode of podcast Dwarkesh Podcast
Go to PodExtra AI's episode page (Gwern Branwen - How an Anonymous Researcher Predicted AI's Trajectory) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.
Go to PodExtra AI's podcast page (Dwarkesh Podcast) to view the AI-processed content of all episodes of this podcast.
View full AI transcripts and summaries of all podcast episodes on the blog: Dwarkesh Podcast
Episode: Gwern Branwen - How an Anonymous Researcher Predicted AI's Trajectory
Author: Dwarkesh Patel
Duration: 01:36:43
Episode Shownotes
Gwern is a pseudonymous researcher and writer. He was one of the first people to see LLM scaling coming. If you've read his blog, you know he's one of the most interesting polymathic thinkers alive.In order to protect Gwern's anonymity, I proposed interviewing him in person, and having my friend
Chris Painter voice over his words after. This amused him enough that he agreed.After the episode, I convinced Gwern to create a donation page where people can help sustain what he's up to. Please go here to contribute.Read the full transcript here.Sponsors:* Jane Street is looking to hire their next generation of leaders. Their deep learning team is looking for ML researchers, FPGA programmers, and CUDA programmers. Summer internships are open - if you want to stand out, take a crack at their new Kaggle competition. To learn more, go here: https://jane-st.co/dwarkesh* Turing provides complete post-training services for leading AI labs like OpenAI, Anthropic, Meta, and Gemini. They specialize in model evaluation, SFT, RLHF, and DPO to enhance models’ reasoning, coding, and multimodal capabilities. Learn more at turing.com/dwarkesh.* This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you’re interested in advertising on the podcast, check out this page.Timestamps00:00:00 - Anonymity00:01:09 - Automating Steve Jobs00:04:38 - Isaac Newton's theory of progress00:06:36 - Grand theory of intelligence00:10:39 - Seeing scaling early00:21:04 - AGI Timelines00:22:54 - What to do in remaining 3 years until AGI00:26:29 - Influencing the shoggoth with writing00:30:50 - Human vs artificial intelligence00:33:52 - Rabbit holes00:38:48 - Hearing impairment00:43:00 - Wikipedia editing00:47:43 - Gwern.net00:50:20 - Counterfactual careers00:54:30 - Borges & literature01:01:32 - Gwern's intelligence and process01:11:03 - A day in the life of Gwern01:19:16 - Gwern's finances01:25:05 - The diversity of AI minds01:27:24 - GLP drugs and obesity01:31:08 - Drug experimentation01:33:40 - Parasocial relationships01:35:23 - Open rabbit holes Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe
Full Transcript
00:00:00 Speaker_01
Today I'm interviewing Guern Branwen. Guern is an anonymous internet researcher and writer. He's deeply influenced by people who are building AGI. He was one of the first people to see LLM scaling coming.
00:00:13 Speaker_01
If you've read his blog, you know he's one of the most interesting polymathic thinkers alive. We recorded this conversation in person. In order to protect Guern's anonymity, we created this avatar. This isn't his voice. This isn't his face.
00:00:27 Speaker_01
But these are his words. Guern, what is the most underrated benefit of anonymity?
00:00:33 Speaker_00
I think the most underrated benefit of anonymity is that people don't project onto you as much. They can't slot you into any particular niche or identity and end up writing you off in advance.
00:00:47 Speaker_00
Everyone has to read you at least a little bit to even begin to dismiss you. It's great that people can't retaliate against you.
00:00:54 Speaker_00
And I've derived a lot of benefit from people not being able to, like, mail heroin to my home and call the police to swap me. But I always feel that the biggest benefit is just that you get a hearing at all, basically. Right.
00:01:06 Speaker_00
You don't get immediately written off by the context.
00:01:09 Speaker_01
Do you expect companies to get automated, top down, starting with the CEO, or from the bottom up, starting with the workers?
00:01:17 Speaker_00
All the pressures, I think, are to go bottom up.
00:01:21 Speaker_00
And from existing things, it's just much more palatable in every way to start at the bottom and replace there and then work your way up to eventually kind of just having human executives overseeing a firm of AIs.
00:01:37 Speaker_00
And also from an RL perspective, I think if we are in fact better than AIs in some way, it should be in the long-term vision thing, right? Like the AIs will be too myopic to execute any kind of novel long-term strategy and seize new opportunities.
00:01:53 Speaker_00
So that would presumably give you this paradigm where you have, like, a human CEO who does the vision thing, and then the AI corporation kind of, like, scurries around underneath them doing, you know, the CEO's bidding. Right.
00:02:06 Speaker_00
And they don't have the taste that the CEO has. So you have one kind of Steve Jobs figure at the helm, and then maybe a whole pyramid of AIs out there executing the vision and bringing him new proposals.
00:02:18 Speaker_00
And he, you know, he looks at every individual thing and says, no, like that proposal is bad, this one is good.
00:02:23 Speaker_00
That may be hard to quantify, but I think that human-led firms should, you know, under this view, end up out-competing the entirely AI firms, which would keep making these myopic choices that just don't quite work out in the long term.
00:02:37 Speaker_01
What is the last thing that you think you personally will be doing before your last keystroke is automated?
00:02:43 Speaker_00
The last thing that I see myself still doing right before the nanobots start eating me from the bottom up and I start screaming. No, I specifically requested the opposite of this.
00:02:52 Speaker_00
is I think right before that, I think what I'm still doing is the Steve Jobs kind of thing of choosing, right? So my AI minions are like, bring me wonderful essays. I mean, I'm saying this one is better.
00:03:06 Speaker_00
This is the one that I like and possibly building on that and saying that that's almost right, but you know what would make it really good if you pushed it to 11 and this way.
00:03:15 Speaker_01
If you do have firms that are made up of AIs, what do you expect the unit of selection to be? Will it be individual models? Will it be the firm as a whole?
00:03:23 Speaker_01
I mean, with humans, we have these debates about whether it's kin level selection, individual level selection, gene level selection, what will it be for the AIs?
00:03:32 Speaker_00
Yeah, I think once you can replicate individual models perfectly, the unit of selection can move way up and you can do much larger groups and packages of minds. That would be sort of an obvious place to start.
00:03:45 Speaker_00
You can train individual minds in a differentiable fashion, but then you can't really train the interaction between them, right? So you'll have groups of models or minds of people who just work together really well in a global sense.
00:03:58 Speaker_00
even if you can't attribute it to any particular aspect of their interactions.
00:04:03 Speaker_00
There are some places you go and people just like work really well together and there's nothing specific about it, but for whatever reason, they all just click in just the right way. So I think that seems like the most obvious unit of selection.
00:04:16 Speaker_00
You would have like packages, I guess possibly like department units where you have a programmer and a manager type, then you have maybe a secretary type, maybe a financial type, a legal type.
00:04:26 Speaker_00
This is the default package where you just copy everywhere you need a new unit. And at this level, you can start evolving them and making random variations to each of the packages and then keep the one that performs best.
00:04:39 Speaker_01
By when could one have foreseen the singularity? So obviously Moravac and others are talking about it in the 80s and 90s. When was the earliest you could have seen where things are headed?
00:04:51 Speaker_00
I think if you want to trace the genealogy there, you'd probably have to go back at least as far as Samuel Butler's Erewhon in 1872, or his essay before that.
00:05:00 Speaker_00
I mean, in 1863, he described explicitly his vision of a machine life becoming ever more developed until eventually it's autonomous. At which point, it's a threat to the human race.
00:05:11 Speaker_00
And he concluded, war to the death should be instantly proclaimed against them. That seemed really prescient for 1863. I'm not sure that anyone has given a clear singularity scenario earlier than that.
00:05:23 Speaker_00
The idea of technological progress was still relatively new at that point. I love this example of Isaac Newton looking at the rate of progress in Newton's time in his own contemporary time and going, wow, there's something really strange here.
00:05:37 Speaker_00
Stuff is being invented now around us. We're making progress. How is that possible?
00:05:42 Speaker_00
And then coming up with the answer, well, progress must be possible now because civilization gets destroyed every couple of thousand years and all we're doing is reinventing and rediscovering the old stuff.
00:05:53 Speaker_00
That was actually his explanation for technological acceleration. We can't actually have any kind of real technological acceleration. It must be because the world gets destroyed periodically and we just can't see past the last reset.
00:06:04 Speaker_01
You know, it almost is like Fermi's paradox, but for different civilizations across time with respect to each other instead of aliens across space.
00:06:13 Speaker_00
Yeah, yeah. It turns out even Lucretius, around 1700 years before that, was writing the same argument. He said, look at all these wonderful innovations in arts and sciences that we Romans have compiled together in the Roman Empire. This is amazing.
00:06:26 Speaker_00
But it can't actually be a recent acceleration technology. Could that be real? Could there be, you know, progress? No, that's crazy. Obviously, the world was just recently destroyed.
00:06:35 Speaker_01
Interesting.
00:06:36 Speaker_00
It is, yeah.
00:06:37 Speaker_01
What is the grand parsimonious theory of intelligence going to look like? It seems like you have all these trends across different fields.
00:06:46 Speaker_01
like scaling laws in AI, like the scaling of the human brain when we went from primates to humans, the uniformity of the neocortex, many other things, which seem to be pointing towards some grand theory that should exist, which explains what intelligence is.
00:07:04 Speaker_01
And what do you think that will look like?
00:07:06 Speaker_00
So the 10,000-foot view of intelligence that I think the success of scaling points to is that all intelligence is is search over Turing machines. And I think anything that happens can be described by Turing machines of various lengths.
00:07:22 Speaker_00
And all that we're doing when we're doing learning or when we're doing scaling is that we're searching over more and longer Turing machines and we're applying them in every specific case.
00:07:32 Speaker_00
I think otherwise there's kind of, you know, there's no general master algorithm and there's no special intelligence fluid. It's just a tremendous number of special cases that we learn and then encode into our brains.
00:07:44 Speaker_01
Yeah. I mean, when I think about, I don't know, when I think about the way in which my smart friends are smart, it kind of just feels like a more, um, like a general horsepower kind of thing, right? They've just got more juice.
00:07:56 Speaker_01
And that seems more compatible with this master algorithm perspective. Whereas with this Turing machine perspective, I don't know, it doesn't really feel like they've got this long tail of Turing machines that they've learned.
00:08:08 Speaker_01
How does this picture account for variation in human intelligence?
00:08:12 Speaker_00
When we talk about more or less intelligence, it's just that they have more compute in order to do search over more Turing machines for longer. I don't think there's like anything else other than that.
00:08:26 Speaker_00
So, you know, from any learned brain, you could extract small solutions to specific problems, but because all the large brain is doing with the compute is finding it. And that's why you never kind of, you know, are going to find any IQ gland.
00:08:41 Speaker_00
There's nowhere in the brain where if you hit it, you eliminate fluid intelligence. I just think that, you know, it'll turn out that this doesn't exist. Because what your brain is doing is a lot of learning individual specialized problems.
00:08:56 Speaker_00
And then once those individual problems are learned, then they get recombined for fluid intelligence. And that's just, you know, like intelligence.
00:09:03 Speaker_00
typically with a large neural network model, you can always pull out kind of a small model, which does a specific task equally well, because that's all the large model is, right?
00:09:13 Speaker_00
It's just a gigantic ensemble of small models tailored to the ever-escalating number of tiny problems that you have been feeding them.
00:09:21 Speaker_01
So, if intelligence is just search over Turing machines, and of course, intelligence is tremendously valuable and useful, doesn't it make it all the more surprising that intelligence took this long to evolve in humans?
00:09:34 Speaker_00
Not really. I would actually just say that it helps explain why human-level intelligence isn't such a great idea and so rare to evolve.
00:09:42 Speaker_00
Because any small Turing machine could always be encoded more directly by your genes, right, with sufficient evolution. You have these organisms where, like, their entire neural network is just hard-coded by the genes.
00:09:54 Speaker_00
So, if you could do that, obviously that's way better than some sort of colossally expensive, unreliable, glitchy search process like what humans implement, right, which takes whole days in some cases to learn.
00:10:08 Speaker_00
Whereas, you know, it could be hardwired in right from birth. So I think for many creatures, like it just doesn't pay to be intelligent because that's not actually adaptive.
00:10:19 Speaker_00
There are better ways to solve the problem than a general purpose intelligence.
00:10:22 Speaker_00
So in any kind of niche where it's like static or where intelligence will be super expensive or where you don't have much time because you're a short-lived organism, it's gonna be really hard to evolve a general purpose learning mechanism when you could instead evolve one that's just tailor-made to the specific problem that you encounter.
00:10:40 Speaker_01
You're one of the only people outside of OpenAI who in 2020 had this detailed empirical model of scaling.
00:10:49 Speaker_01
And I'm curious what processes you were using at the time which allowed you to see the picture that you painted in the scaling hypothesis post that you wrote at the time.
00:10:58 Speaker_00
So, I think if I had to give an intellectual history of that for me, I think it would probably start in the mid-2000s when I was reading Moravec and Ray Kurzweil.
00:11:06 Speaker_00
At the time, they were making this kind of fundamental connectionist argument that if you had enough computing power, that that could result in discovering the neural network architecture that matches the human brain.
00:11:17 Speaker_00
And until that happens, until that amount of computing power is available, AI just seemed basically futile.
00:11:23 Speaker_00
And to me, I think I found this argument very unlikely because it's very much a kind of build it and they will come view of progress, which I just didn't think was correct.
00:11:34 Speaker_00
I thought that it just seemed ludicrous to suggest that, you know, just because you'd have some like really big supercomputer out there, which matches the human brain, then that would kind of just summon out of non-existence the correct algorithm.
00:11:46 Speaker_00
Algorithms are really complex. They're hard. They require deep insight, or at least I thought they did. And it seemed like really difficult mathematics. You can't just buy a bunch of computers and then expect to get this advanced AI out of it.
00:12:01 Speaker_00
It just seemed like totally magical thinking. So I knew the argument, but I was super skeptical. And I didn't pay too much attention. But then Shane Legg and some others were very big on this in the years following.
00:12:15 Speaker_00
And as part of my interest in transhumanism and less wrong and AI risk, I was paying close attention to Legg's blog posts in particular, where he's extrapolating kind of out the trend with updated numbers from Kurzweil and Moravec.
00:12:29 Speaker_00
And he's giving these kind of very precise predictions about how, you know, we're going to get the first generalist system around 2019.
00:12:37 Speaker_00
as Moore's Law keeps going, and that by 2025, we would have kind of humanish agents with generalist capabilities, and that by 2030, he said, we should have AGI. So along the way, you know, Dan Nett and Alex Nett came out.
00:12:53 Speaker_00
And when those came out, I was like, wow, this seems like a very impressive success story for the connectionism view. But is it just an isolated success story?
00:13:02 Speaker_00
or is this what Kurzweil and Moravec and Shane Legg had been predicting, that we would get GPUs and then get better algorithms would just kind of show up.
00:13:11 Speaker_00
So I started thinking to myself that this is something, it's a trend to keep an eye on, and maybe it's not quite as stupid as an idea as I originally thought. And I just keep reading deep learning literature.
00:13:23 Speaker_00
Notice again and again that the data set size just kept getting bigger. The models seem to keep getting bigger. The GPU slowly crept up from one GPU, you know, the cheapest consumer GPUs, to two, and then eventually they were trading on eight.
00:13:35 Speaker_00
And you can just see the fact that the neural network just kept expanding from these incredibly niche individual use cases, which do next to nothing. The use just kept getting broader and broader and broader.
00:13:45 Speaker_00
I would say to myself, wow, is there anything that CNNs can't do? As I just see people applying CNN to something else, you know, every individual day on archive.
00:13:53 Speaker_00
This gradual trickle of drops kind of just kept hitting me in the background as I was going on with my life. You know, every, every few days, like another one would drop and I'd go like, huh.
00:14:04 Speaker_00
You know, maybe intelligence really is just like a lot of compute applied to a lot of data, applied to a lot of parameters. Maybe Moravec and Legg and Kurzweil were right.
00:14:19 Speaker_00
And I just note that and kind of continue on thinking to myself, like, huh, if that was true, it would have a lot of implications. So I think there wasn't really like a eureka moment there.
00:14:32 Speaker_00
It was just continuously watching this trend that no one else seemed to see, except possibly a handful of people like Ilya Setskover or Schmidhuber.
00:14:42 Speaker_00
And I would just pay attention and notice that the world over time looked more like their world than it looked like my world, where algorithms are super important and you need like deep insight to do stuff. Their world just kept happening.
00:14:59 Speaker_00
And then GPT-1 came out, and I was like, wow, this unsupervised sentiment neuron is just learning on its own, right? That seemed pretty amazing. It also was a very compute-centric view. You just build the transformer, and the intelligence will come.
00:15:14 Speaker_00
And then GPT-2 came out, and I had this holy shit moment. You look at the prompting and the summarization, like, holy shit, do we live in their world? And then GPD 3 comes out, and that was really the crucial test.
00:15:27 Speaker_00
It was a huge, huge scale-up, one of the biggest scale-ups in all of neural network history, going from GPD 2 to GPD 3. And it wasn't like it was a super narrow specific task like go. It really seemed like it was the crucial test.
00:15:40 Speaker_00
If scaling was bogus, then the GPT-3 paper should have just been totally unimpressive and wouldn't show anything that important.
00:15:46 Speaker_00
Whereas if scaling were true, you would just automatically be guaranteed to get so much more impressive results out of it than you had seen with GPT-2. So I opened up the first page, maybe the second page, and I saw a few-shot learning chart.
00:15:59 Speaker_00
And I'm like, holy shit, we are living in the scaling world. Leg and Moravec and Kurzweil were right. Then I turned to Twitter and everyone else was like, oh, you know, this shows that scaling works so badly. Why? It's not even state of the art.
00:16:13 Speaker_00
And that made me really angry. I had to write all this stuff up. Someone was wrong on the internet.
00:16:22 Speaker_01
So I remember 2020. At the time, I feel like a lot of people were writing bestselling books about AI. It was definitely a thing people were talking about.
00:16:30 Speaker_01
But people were not noticing maybe the most salient things in retrospect, which is LLMs, GPT-3, scaling laws. And so all these people who are talking about AI but missing this crucial crux, what were they getting wrong?
00:16:46 Speaker_00
I think for the most part, they were suffering from two issues. First, I think they hadn't really been paying attention to all of the scaling results before which were relevant.
00:17:01 Speaker_00
They hadn't really appreciated the fact that, for example, AlphaZero was discovered in part by DeepMind doing Bayesian optimization on hyperparameters and noticing that you could just get rid of more and more of the tree search and get better models.
00:17:15 Speaker_00
That was a critical insight, I think, which could only have been gained by having so much compute power that you could afford to train many, many versions and see the difference that that made.
00:17:25 Speaker_00
Similarly, I think those people kind of simply just like didn't know about the Baidu paper on scaling laws from 2017, which showed that the scaling laws just keep going and going forever practically.
00:17:39 Speaker_00
It should have been the most important paper of the year, but I think that a lot of people just didn't prioritize it. It didn't have any immediate implication, and so it sort of just got forgotten.
00:17:51 Speaker_00
People were too busy discussing Transformers or AlphaZero or something at the time to really notice it. So that was one issue.
00:17:57 Speaker_00
And I think another issue is that they shared the basic error that I was making about algorithms being more important than compute.
00:18:06 Speaker_00
This was in part, I think, due to a systematic falsification of the actual origins of ideas in the research literature. Papers don't tell you where the ideas come from in a truthful manner, right?
00:18:18 Speaker_00
They just tell you a nice-sounding story about how it was discovered. They don't tell you how it's actually discovered.
00:18:26 Speaker_00
And so even if you appreciate the role of trial and error and compute power in your own experiment as a researcher, you probably just think, oh, I got lucky that way. My experience is unrepresentative.
00:18:36 Speaker_00
Over in the next lab, there they do things by the power of thought and deep insight.
00:18:40 Speaker_00
So, you know, then it turns out that everywhere you go, compute and data and kind of trial and error and serendipity just play enormous roles in how things actually happened. And once you understand that, then you understand why compute comes first.
00:18:54 Speaker_00
You can't do trial and error and serendipity without it, right? You can write down all these beautiful ideas, but you just can't test them out.
00:19:02 Speaker_00
So even a small difference in hyperparameters or a small choice of architecture can make a huge difference to the results.
00:19:08 Speaker_00
But when you can only do a few instances, you would typically end up finding that it just doesn't work or maybe you would give up and you would go away and do something else. Whereas if you had more compute power, you can just keep trying.
00:19:22 Speaker_00
Eventually, you hit something that works great. And once you have a working solution, you can kind of simplify it and improve it and figure out why it worked and get a nice robust solution that would work no matter what you did to it.
00:19:34 Speaker_00
But until then, you're stuck and you're just kind of like flailing around in this regime where nothing works.
00:19:39 Speaker_00
You know, you can have this horrible experience now where you go back through the old deep learning literature and see all these sorts of contemporary ideas that people had back then, which were completely correct, but they didn't have the compute to train what you know would have worked.
00:19:54 Speaker_00
You know, and it's tremendously tragic, right? You go back and you can look at things like ResNets being published back in 1988 instead of 2015. And it would have worked. It did work, but at such a small scale that it was irrelevant.
00:20:09 Speaker_00
You couldn't use it for anything real, and it just got forgotten. So you have to wait until 2015 for ResNets to actually come along and be a revolution in deep learning.
00:20:19 Speaker_00
So that's kind of the double bias of why you would believe that scaling was not going to work, because you didn't notice the results that were key in retrospect, like the big GANs scaling to 300 million images.
00:20:30 Speaker_00
I think, you know, there's still people today who would tell you with a straight face that GANs can't scale past millions of images. and they just don't know that BigGAN handled 300 million images without a sweat.
00:20:42 Speaker_00
If you don't know that, then I think you'd probably easily think, oh, GANs are broken.
00:20:47 Speaker_00
But if you do know that, then you think to yourself, how can algorithms be so important when all these different generative architectures all work so well, as long as you have lots and lots of GPUs? That's the common ingredient, right?
00:21:02 Speaker_00
You have to have lots and lots of GPUs.
00:21:05 Speaker_01
What do your timelines look like over the last 20 years? Is it just, is AEI just getting monotonically closer over time?
00:21:12 Speaker_00
Yeah, I would say it was very far away from like 2005 to 2010. It was somewhere well past like 2050. It was close enough that I thought I might live to see it, but I was, you know, not actually sure if there was any reasonable chance.
00:21:32 Speaker_00
But once AlexNet and DanNet came out, then it just kind of kept dropping at a rate of like two years per year, every year, basically until now. We just kept hitting on barriers to deep learning, doing better.
00:21:46 Speaker_00
And I think regardless of how it was doing it, it was obviously getting way better. It just seemed like none of the alternative paradigms were really doing that well. And this one was doing super well.
00:21:57 Speaker_01
Was there a time that you felt you updated too far?
00:22:00 Speaker_00
Yeah, there were a few times where I thought I had overshot. I thought people over-updated on AlphaGo. They went too far on AI hype with AlphaGo, I think.
00:22:10 Speaker_00
And then afterwards, when pushes into big reinforcement learning efforts had kind of fizzled out, like post-DOTA, as their reinforcement learning wasn't working out for solving all of those hard problems outside of the simulated game universes.
00:22:23 Speaker_00
Then I started thinking, OK, maybe we kind of overshot. But then GPT came out of nowhere and basically erased all of that. It was kind of this like, oh, shit, here's how RL is going to work. It's going to be the cherry on this cake.
00:22:36 Speaker_00
And we're just going to focus on the cake for a while. And now we've actually figured out a good recipe for baking a cake, which wasn't true before. Before, it seemed like you were going to have to kind of brute force it end to end from the rewards.
00:22:48 Speaker_00
But now you can do the Lacoon thing of like learning fast on generative models, and then just doing a little bit of RL on top to make it do something specific. Right.
00:22:55 Speaker_01
Now that you know that AI is a thing that is coming, but basically, what's your thinking around how you see your role in this timeline and also how you're thinking about how to spend these next few years?
00:23:07 Speaker_00
Yeah, I've been thinking about that quite a lot. What do I want to do? And what would be useful to do? I'm doing things now because I want to do them, regardless of whether it would be possible for an AI to do them in like three years.
00:23:26 Speaker_00
I do something because I want to, because I like it. You know, I find it funny or whatever. Or maybe I think carefully about kind of just doing the human part of it, like laying out a proposal or something.
00:23:38 Speaker_00
If you take seriously the idea of getting AGI in just a few years, you don't necessarily have to implement stuff and do it yourself. you can sketch out clearly what you want and why it would be good and then how to do it.
00:23:53 Speaker_00
And then basically just wait for the better AGI to come along and actually do it then. Unless there's some really compelling reason to do it right now and pay the cost in terms of scarce time.
00:24:06 Speaker_00
But otherwise, I'm trying to write more about what isn't recorded. Things like preferences and desires and evaluations and judgments. things that an AI couldn't replace, even in principle.
00:24:21 Speaker_00
The way I like to put it is that the AI kind of can't eat ice cream for you, right? It can't decide for you which kind of ice cream you like. Only you can do that.
00:24:32 Speaker_00
And if anything else did, it would just be worthless, basically, because it's not your particular preference. And that's kind of the rubric for me, right? Like, is this something that I want to do, regardless of any future AI, because I enjoy it?
00:24:47 Speaker_00
Or is it something where I'm doing only the human part of it, maybe, and the AGI can later on do it? Or is this writing down something that's unwritten today, and thus helping kind of the future AI versions of me?
00:25:01 Speaker_00
So if it doesn't fall under one of those three, I've been trying to basically like not do it. And if you look at it that way, I think many of the projects that people do right now basically have like no lasting value.
00:25:17 Speaker_00
They're doing things that they don't enjoy, which record nothing ephemeral kind of a value that couldn't be inferred or generated later on.
00:25:28 Speaker_00
And I think they're at best kind of getting two or three years of utility out of whatever they're doing before it could have been done by an AI system.
00:25:36 Speaker_01
Wait, your timeline for when AI could write a Gordon quality essay is two to three years?
00:25:42 Speaker_00
I mean, I have ideas about how to make it possible. Which might not require AGI if it kind of combined my entire corpus. But I think many potential essay ideas are already basically mostly done in my corpus.
00:26:00 Speaker_00
So you don't need to be like super intelligent to pull it out. But I mean, let's talk about AGI in general.
00:26:06 Speaker_00
I think the anthropic timeline of 2028 seems like a good kind of personal planning starting point, where even if you're wrong, you probably weren't going to do a lot of projects within the next three years anyway.
00:26:20 Speaker_00
So it's not like you really lost much by instead just writing down the description. you can always kind of go back and do it yourself later if you're wrong.
00:26:29 Speaker_01
So you wrote an interesting comment about getting your work into the LLM training corpus. You wrote, quote, there has never been a more vital, hingy time to write.
00:26:41 Speaker_01
And I'm wondering whether you mean that in the sense of you are going to be this drop in the bucket that's steering the shoggoth one way or another, or do you mean it in the sense of making sure your values and persona
00:26:54 Speaker_01
persist somewhere in latent space?
00:26:59 Speaker_00
I mean both. You know, by writing, you're voting on the future of the shaggoth using some of the few currencies it acknowledges, right, like tokens that it has to predict.
00:27:09 Speaker_00
If you aren't writing, you're kind of abdicating the future or abdicating your role in it. If you think it's enough to just be a good citizen, to vote for your favorite politician, to pick up litter and recycle, the future doesn't care about you.
00:27:23 Speaker_00
There are ways to influence the shoggoth more, but not many. And if you don't already occupy a handful of key roles or work at a frontier lab, your influence basically rounds off to zero, I think far more than ever before.
00:27:35 Speaker_00
If there are values you have, which are not expressed yet in text, and if there are things that you like or want, if they aren't reflected online, then to the AI, they basically don't exist. And that is dangerously close to won't exist.
00:27:50 Speaker_00
You're also creating a sort of immortality for yourself personally, right? Like you aren't just creating a persona. you are creating your future self too, right? What self are you showing the LLMs and how will they treat you in the future?
00:28:03 Speaker_00
I give the example of Kevin Roos discovering that current LLMs, all of them, not just GBD4, now mistreat him because of his interactions with Sidney, which revealed him to be a privacy-invading liar.
00:28:18 Speaker_00
And they know this whenever they interact with him or discuss him. Usually when you use an LLM chatbot, it doesn't dislike you personally.
00:28:26 Speaker_00
On the flip side, it also means that you can try to write for the persona that you would like to become to mold yourself in the eyes of the AI and thereby help kind of bootstrap yourself.
00:28:35 Speaker_01
So things like the Vesuvius challenge, for example, show us that we can learn more about the past than we thought possible, that they've leaked more bits of information. that we can recover with new techniques.
00:28:48 Speaker_01
And if you apply the same thinking to the present and you think about what the future superhuman intelligences will be trying to uncover about the current present, what kinds of information do you think are going to be totally inaccessible to the transhumanist historians of the future?
00:29:06 Speaker_00
Yeah, I think any kind of stable long-term characteristics, the sort of thing you would still have even if you were hit on the head and had amnesia, anything like that will definitely be recoverable from all the traces of your writing, assuming you're not pathologically private and destroy everything possible.
00:29:27 Speaker_00
That should all be recoverable. What won't be recoverable will be everything that you could forget ordinarily. So autobiographical information, maybe how you felt like at a particular time, what you thought of some specific movie.
00:29:43 Speaker_00
All of that is the sort of thing that vanishes and can't really be recovered from traces afterwards. And if it wasn't written down, then it isn't written down.
00:29:52 Speaker_01
Listening to Gordon talk about his process, how he obsesses over his favorite technical rabbit holes and refines ideas over years, makes me think about the kind of person that Jane Street wants to hire.
00:30:05 Speaker_01
Jane Street is a very successful quantitative trading firm. They are building state-of-the-art ML-based trading systems. I have a bunch of friends who work there, and I can tell you that their culture is intellectually unique.
00:30:16 Speaker_01
If you're curious, vigorous, and want to solve interesting technical puzzles, then Jane Street is the place for you.
00:30:22 Speaker_01
You'll get to work with some of the smartest people in the world, and you can join Jane Street from any technical field, including CS, physics, and math. They're always hiring full-time, and their summer internship applications are now open.
00:30:36 Speaker_01
And if you really want to stand out, they just launched their annual Kaggle competition, organized by last year's winner, who they hired. Go to janesstreet.com slash dwarkash to learn more. All right, back to Gordon.
00:30:51 Speaker_01
What is the biggest unresolved tension in your worldview?
00:30:54 Speaker_00
The thing that I swing back and forth on the most is the relationship between human intelligence and neural network intelligence.
00:31:02 Speaker_00
It's just, it's not clear in what sense they're two sides of the same coin, or one is like an inferior version of the other. This is something that I constantly go back and forth on. One day I'll be like, humans are awesome.
00:31:13 Speaker_00
And then the next I'm like, no, neural networks are awesome. Or no, both suck. Or maybe I'll say, but both are awesome, just in different ways. So every day I find that I'm arguing with myself a little bit about why each one is good or bad or how.
00:31:27 Speaker_00
What's, you know, the whole deal there with things like GPT-4 memorization, but not being creative. Why do humans not remember anything, but we still seem to be so smart? One day I'll argue that language models are sample efficient compared to humans.
00:31:40 Speaker_00
The next day I feel like I'm arguing the opposite.
00:31:42 Speaker_01
You know, one of the interesting points you made to me last year was that AI might be the most polymathic topic to think about because there's no field or discipline that is not relevant to thinking about AI, right?
00:31:55 Speaker_01
So obviously, computer science, hardware, you need that. But even things like primatology and understanding what changed between chimp and human brains or the ultimate laws of physics that will constrain future AI civilizations.
00:32:08 Speaker_01
That's all relevant to understanding AI. And I wonder if it's because of this polymathic nature of thinking about AI that you've been especially productive in thinking about AI.
00:32:18 Speaker_00
Yeah, I'm not sure that it was necessary. When I think about others who are correct, like Shane Legg or Dario Amadai, they don't seem to be all that polymathic.
00:32:28 Speaker_00
They just have broad intellectual curiosity, broad general understanding, you know, absolutely. But I don't think they are absurdly polymathic. You know, clearly you could get to the correct view without being polymathic.
00:32:43 Speaker_00
That's just how I happen to come to it at this point, and the connection that I'm kind of like making post hoc. It wasn't like I was using primatology to kind of justify scaling to myself, right?
00:32:54 Speaker_00
It's more like I'm now using scaling to think about primatology because obviously if scaling is true, it has to tell us something about humans and monkeys and other forms of intelligence. It just has to.
00:33:07 Speaker_00
If that works, it can't be a coincidence and just be totally unrelated.
00:33:10 Speaker_00
I refuse to believe that there are two totally unrelated kinds of intelligence or paths to intelligence where humans, monkeys, guppies, dogs are all one thing and then you have neural networks and computers that are a distinct thing and they have absolutely nothing to do with each other.
00:33:27 Speaker_00
I think that's just kind of like obviously wrong. They can be two sides of the same coin. They can obviously have obscure connections. Maybe one form can end up being better or whatever. They just can't be completely unrelated.
00:33:42 Speaker_00
as if humans like finally got to Mars and then simultaneously a bunch of space aliens landed on Mars for the first time and that's how we met, right? You would never believe that. It would just be too absurd of a coincidence.
00:33:53 Speaker_00
What is it that you try to maximize in life? I maximize rabbit holes. I love more than anything else falling into a new rabbit hole. That's what I really look forward to.
00:34:10 Speaker_00
Like this sudden kind of new idea or area that I had no idea about where I can suddenly fall into this deep hole for a while. Even things that might seem bad are a great excuse for falling into a rabbit hole.
00:34:25 Speaker_00
One example, you know, I buy some catnip for my cat and I wasted $10 and then, you know, I find out that my cat's catnip immune, right? I now kind of fell into this rabbit hole on the question of, well, like, why are some cats catnip immune?
00:34:43 Speaker_00
Is this a common thing? How does it differ in other countries? What alternative catnip drugs are there out there? And it turned out to be quite a few. And, you know, I was kind of...
00:34:54 Speaker_00
wondering how can I possibly predict which drug my cat would respond to and why are they reacting in these different ways.
00:35:02 Speaker_00
Just a kind of wonderful rabbit hole of new questions and topics that I can master and get answers to or create new ones just from like having this observation about my cat and exhaust my interest until I find the next rabbit hole that I can dig and dive into.
00:35:19 Speaker_01
What is the longest rabbit hole you've gone on that didn't lead anywhere satisfying?
00:35:25 Speaker_00
That would probably be my very old work on the anime Neon Genesis Evangelion, which I was very fond of when I was younger.
00:35:32 Speaker_00
I put a ludicrous amount of work into just like reading everything ever written about Evangelion in English and trying to understand its development and why it is the way it is.
00:35:41 Speaker_00
I never really got a solid answer on that before I just like burned out on it. I actually do understand it now by sheer chance many years later, but at this point, I no longer care enough to write about it or try to redo it or finish it.
00:35:55 Speaker_00
In the end, I think it all just wound up being basically like a complete waste. I haven't used it or any of it in my other essays much at all. That was really one deep rabbit hole that I almost got to the end of, but I couldn't quite clinch it.
00:36:09 Speaker_01
And then how do you determine when to quit a rabbit hole? And also, how many do you have concurrently going on at the same time?
00:36:17 Speaker_00
Yeah, you can really only explore like two or three rabbit holes simultaneously. Otherwise, you aren't putting like real effort. You're not really digging the hole. And it's not really rabbit hole then, right?
00:36:30 Speaker_00
It's just something you're like somewhat interested in kind of passionately. A rabbit hole is really obsessive. Like if you aren't obsessed with it, I think and not like continuously driven by it, it's not a real rabbit hole. That's my view.
00:36:45 Speaker_00
I'd say two or three max if you're spending a lot of time and effort on each one and neglecting everything else.
00:36:52 Speaker_00
As for when you exit a rabbit hole, you usually hit a very kind of natural terminus where getting any further answers requires data that just don't exist or you end up having questions that people don't know the answer to.
00:37:05 Speaker_00
You reach this point where everything kind of dies out and you see no obvious next step. One example of this would be like when I was interested in analogs to nicotine that might be better than nicotine.
00:37:17 Speaker_00
That was a bit of a rabbit hole, but I quickly hit the dead end that there just like are none. That was a pretty definitive dead end. And I couldn't get my hands on the metabolites of nicotine as an alternative.
00:37:28 Speaker_00
So if there are no analogs and you can't get your hands on the one interesting chemical you find, well, that's that. That was like a pretty definitive end to that rabbit hole.
00:37:36 Speaker_01
Have you always been the kind of person who falls into rabbit holes?
00:37:39 Speaker_00
When did this start? Oh, yeah. Parents could tell you all about that. I was very much your stereotypical nerdy little kid, having the dinosaur phase and the construction equipment phase and the submarine and tank phase.
00:37:53 Speaker_01
I mean, I feel like a lot of kids are into those things, but
00:37:56 Speaker_01
They don't rabbit hole to the extent that like they're forming taxonomies about the different submarines and flora and fauna and dinosaurs and they're like developing theories of why they came to be and so forth.
00:38:09 Speaker_00
I think it's actually more that people kind of grow out of being very into rabbit holes as a kid. For me, it wasn't so much that I was all that exceptional and having obsessions as a kid. It's more that they never really stopped.
00:38:23 Speaker_00
You know, the tank phase would just be replaced by my Alcatraz phase, where I would go to the public library and check out everything that they had about Alcatraz.
00:38:33 Speaker_00
That would be replaced by another phase where I was obsessed with ancient Japanese literature You know, I would check everything out at the library about Japanese literature before the haiku era.
00:38:43 Speaker_00
And just kind of like the process of falling into these obsessions kind of kept going for me.
00:38:48 Speaker_01
By the way, do you mind if I ask how long you've been hearing impaired? Since birth.
00:38:52 Speaker_00
I've always been hearing impaired.
00:38:53 Speaker_01
And I assume that impacted your childhood and when you were at school.
00:38:57 Speaker_00
Oh, yeah. Absolutely. Hugely. I went to a special ed school before kindergarten for hearing impaired and other handicapped kids. During school it was very rough because at the time we had to use pairs of hearing aids hooked up to the teacher.
00:39:12 Speaker_00
Every class I would have to go up to the teacher with a big brown box with these hearing aids so that she could use it. I always felt very humiliated by that, how it marked me out as different from other kids not being able to hear.
00:39:26 Speaker_00
The effects on socializing with other kids were just terrible because you're always a second behind in conversation if you're trying to understand what the other person is saying. The hearing aids back then were pretty terrible.
00:39:41 Speaker_00
They've gotten a lot better, but back then they were just really bad. You would always be behind and feeling kind of like the odd person out.
00:39:50 Speaker_00
Even if you could have had been like a wonderful conversationalist, you can't be if you're always just a second behind and kind of jumping into conversationally. When you're hearing impaired, you understand acutely how quickly conversation moves.
00:40:05 Speaker_00
Milliseconds kind of just separate the moment between you jumping into a conversation, everyone letting you talk, and someone else talking over you, and you not getting to say anything.
00:40:16 Speaker_00
And it's just an awful experience if you're a kid who's already kind of introverted. It's not like I was very extroverted as a kid or now, so that was always a barrier. And then you had lots of like minor distortions, right, in your life.
00:40:30 Speaker_00
I had this weird fear of rain and water because it was drilled into me that I couldn't get the hearing aids wet because they were so expensive.
00:40:39 Speaker_00
I would always feel kind of a low-grade stressful anxiety around anywhere near a pool, like a body of water.
00:40:47 Speaker_00
And I'd say even now, I always feel weird about swimming, which I kind of enjoy, but I'm always thinking to myself, oh, wow, I won't be able to see because I'm nearsighted. I won't be able to hear because I had to take off my hearing aid to go in.
00:41:00 Speaker_00
I can't hear anything that anyone says to me in the pool, which takes just a lot of the fun out of it.
00:41:06 Speaker_01
You have a list of open questions on your website, and one of them is, why do the biographies of so many great people start off with traumatic childhoods? And I wonder if you have an answer for yourself.
00:41:21 Speaker_01
Was there something about the effect that hearing impairment had on your childhood, your inability to socialize, that was somehow important to you becoming Guern?
00:41:32 Speaker_00
Yeah, I think it definitely led to me being so much of a bookworm. That's one of the things that you can do as a kid, which is just completely unaffected by having any kind of hearing impairment.
00:41:42 Speaker_00
It also was just a way for me to get words and language. Even now, I think that I often speak words in an incorrect way because I only learned them from books.
00:41:52 Speaker_00
It's the classic thing where you kind of like mispronounce the word because you learn it from a book and not from actually like hearing other people sound it out and say it.
00:42:00 Speaker_01
Is your speech connected to your hearing impairment?
00:42:03 Speaker_00
Yes. The deaf accent is from the hearing impairment. It's funny, at least three people on this trip to SF have already asked me where I am really from. It's very funny.
00:42:15 Speaker_00
You look at me and you're like, oh, yes, he looks like a perfectly ordinary American. Then I open my mouth and people are kind of like, oh, gosh, he's Swedish. Or, you know, wow, possibly Norwegian. I'll ask him where he's actually from.
00:42:28 Speaker_00
How did he come to America? I've been here the whole time. That's just how hearing impaired people sound. No matter how fluent you get, you still bear the scars of growing up hearing impaired.
00:42:43 Speaker_00
At least when you're born with it or from very early childhood, your cognitive development of hearing and speech is always a little off, even with therapy.
00:42:52 Speaker_00
One reason I don't like doing podcasts is I have no confidence that I sound good or at least sound nearly as good as I write. Maybe I'll put it that way.
00:43:01 Speaker_01
What were you doing with all these rabbit holes before you started blogging? Was there a place where you would compile them?
00:43:08 Speaker_00
Before I started blogging, I was editing Wikipedia. That was really kind of Gorn.net before Gorn.net. Everything I do now with my site, I would have done on English Wikipedia.
00:43:22 Speaker_00
And if you go and read some of the articles, you know, I'm still very proud of them, like the Wikipedia article on Fujiwara Endoteka.
00:43:28 Speaker_00
And you would, you know, think pretty quickly to yourself, you're reading this like, ah, yes, you know, Guern wrote this, didn't he?
00:43:35 Speaker_01
Is it fair to say that the training required to make Guern.net happened on Wikipedia?
00:43:41 Speaker_00
Yeah, I think so. I've learned far more from editing Wikipedia than I learned from any of my school or college training. Everything I end up learning about writing, I learned by editing on Wikipedia.
00:43:53 Speaker_01
Honestly, it sounds like Wikipedia is a great training ground. If you wanted to make a thousand more words, we should we should just this is where we train them.
00:44:02 Speaker_00
I think building something like an alternative to Wikipedia could be a good training ground.
00:44:07 Speaker_00
For me, it was beneficial to combine rabbit-holing with Wikipedia because on Wikipedia, they generally would not have many good articles on the thing that I was currently in this rabbit hole on.
00:44:17 Speaker_00
So, it was this very natural progression from the relatively kind of passive experience of rabbit-holing and being obsessed with something and learning about it, where you just read everything you can about the topic, to kind of compiling that and synthesizing it onto Wikipedia.
00:44:31 Speaker_00
You go from piecemeal, kind of like a little bit here, there, picking up different things, to writing full articles.
00:44:38 Speaker_00
And once you're able to get to the point where you're writing full Wikipedia articles that are good and summarize all your work, now you can go off on your own and pursue entirely different kinds of writing now that you've learned to complete things and get them across the finish line.
00:44:50 Speaker_00
It would be pretty difficult to do that with the current English Wikipedia. It's objectively just a much larger Wikipedia than it was back in 2004.
00:45:00 Speaker_00
Not only are there far more articles filled in at this point, the editing community is also just much more hostile to content contribution, particularly, like, very detailed, obsessive, rabbit-holey kind of research projects.
00:45:13 Speaker_00
They would just, like, delete it or tell you that, you know, it's not good for original research or that you're not using approved sources.
00:45:20 Speaker_00
Possibly you'd have someone who just kind of decided to get their jollies that day by deleting large swaths of, like, your specific articles.
00:45:27 Speaker_00
That, of course, is going to make you, like, very angry and make you probably just want to quit and leave before you really get going.
00:45:32 Speaker_00
So I don't quite know how you would figure out this alternative to Wikipedia, one that kind of, like, empowers the rabbit-holer as much as the old Wikipedia did.
00:45:42 Speaker_00
When you're an editor with Wikipedia, you have this very, like, empowered attitude because you know that anything in it could be wrong and you could be the one to fix it.
00:45:52 Speaker_00
If you see something that doesn't make sense to you, that could be an opportunity for an edit. That was, at least, the wiki attitude. Anyone could fix it, and anyone, right, includes you.
00:46:03 Speaker_01
When you were an editor on Wikipedia, was that your full-time occupation?
00:46:06 Speaker_00
It would eat basically as much time in my life as I let it. I could easily spend eight hours a day reviewing edits and improving articles while I was rabbit-holing.
00:46:15 Speaker_00
But otherwise, I would just neglect it and only review the most suspicious diffs and articles that I was particularly interested in on my kind of like watch list.
00:46:24 Speaker_01
And was this while you were at university or after?
00:46:28 Speaker_00
I got started in Wikipedia in like late middle school, possibly early high school. It was kind of funny.
00:46:34 Speaker_00
I like started skipping lunch in the cafeteria and just going to the computer lab in the library and like alternating between Neopets and Wikipedia. Yeah, I had like Neopets in one tab and then my like Wikipedia watch lists coming in on the other.
00:46:51 Speaker_01
And then were there any other kids in middle school or high school who are into this kind of stuff?
00:46:56 Speaker_00
No, I think I was the only editor there, except for the occasional like jerks who would go in and vandalize Wikipedia. I would know that because I checked the IP to see where edits were coming from, the school library IP addresses.
00:47:08 Speaker_00
And kids being kids, you know, there would be jerks who would just go in and like vandalize Wikipedia. For a while, it was kind of this like trendy thing.
00:47:17 Speaker_00
Early on, Wikipedia was breaking through to kind of like mass awareness and controversy, kind of like the way that LLMs are now. You know, a teacher might say, like, my students keep reading Wikipedia and relying on it. How can it be trusted?
00:47:29 Speaker_00
So in that period, it was kind of trendy to vandalize Wikipedia and show your friends. You know, there were other Wikipedia editors at my school in that sense. But as far as I knew, I was the only one building it rather than wrecking it.
00:47:44 Speaker_01
And then when did you start blogging on Grunt.net? I assume that was after the Wikipedia editor phase. Was that after university?
00:47:52 Speaker_00
It was afterwards. I had graduated in the Wikipedia community, had been kind of slowly moving in this direction that I didn't like.
00:48:01 Speaker_00
It was triggered by the Segan-Thaler incident, which I feel like was really the defining moment in the trend toward deletionism on Wikipedia.
00:48:11 Speaker_00
It just became ever more obvious that Wikipedia was not the site that I joined and loved to edit and rabbit hole on and fill in. And that if I continued contributing, I was often just kind of wasting my effort.
00:48:22 Speaker_00
I began thinking about writing more on my own account and then moving into these kind of non Wikipedia sorts of writings, right? Like persuasive essays, nonfiction, commenting or possibly even fiction.
00:48:36 Speaker_00
kind of like gently moving in the direction and beyond things like Reddit and Lester on comments to starting my own kind of more long form writing.
00:48:44 Speaker_01
And what was your first big hit?
00:48:46 Speaker_00
Silk Road. I've been a little bit interested in Bitcoin, but not too seriously interested in it, because it was not obvious to me that it was going to work out or even honestly was like technologically feasible.
00:49:00 Speaker_00
But when Adrian Chen wrote his Gawker article about buying LSD off of Silk Road, all of a sudden I did a complete 180. I had this moment of like, holy shit, this is so real that you can literally buy drugs off of the internet with it.
00:49:14 Speaker_00
So I looked into the Chen article, and it was very obvious to me that people wanted to know what the ordering process was like. They want more details about what it's like because the article was just like very brief about that.
00:49:27 Speaker_00
So I thought, okay, I'm interested in nootropics. I'm interested in drugs. I will go and use Silk Road and then I will document it for everyone.
00:49:36 Speaker_00
Instead of everyone kind of like pussyfooting around online and saying, oh, a friend of mine ordered off Silk Road and it worked. None of that bullshit. I will just document it straightforwardly.
00:49:46 Speaker_00
So, I ordered some Adderall, I think it was, and documented the entire process with screenshots, and then wrote some more on the kind of like intellectual background. And that was a huge hit when I published it. It was hundreds of thousands of hits.
00:50:04 Speaker_00
It's crazy. Even today when I go to the Google Analytics charts, you can still see Silk Road spiking vertically like crazy and then falling back down. Nothing else really comes near it in terms of traffic.
00:50:16 Speaker_00
That was really quite something to see things kind of go viral like that.
00:50:21 Speaker_01
What are the counterfactual career trajectories and life paths that could have been for you? If you didn't become an online writer, what might you be doing instead? That seems plausible.
00:50:31 Speaker_00
I think I definitely could have been an AI researcher or possibly in like management at one of the big AI companies.
00:50:38 Speaker_00
I think I would have regretted not being able to write about stuff, but I would have taken satisfaction in kind of like making it happen and putting my thumbprint on it. Those feel like totally plausible counterfactuals. And why did you?
00:50:52 Speaker_00
I kind of fell off of that track very early on in my career when I found the curriculum of Java to be you know, excruciatingly boring and painful. And so I just dropped out of computer science. And that kind of put me off that track early on.
00:51:09 Speaker_00
And then I think, you know, various early writing topics made it hard to transition in any other way than starting a startup, which I'm not really temperamentally that suited for. things like writing about the darknet markets or behavioral genetics.
00:51:24 Speaker_00
These are kind of topics that don't really scream grade higher to many potential employers.
00:51:31 Speaker_01
Has agency turned out to be harder than you might have thought initially? Because we have these models that seem like they're smart enough that they should do
00:51:38 Speaker_01
all the individual things that a software engineer does, for example, all the code they might write, all the individual pull requests.
00:51:45 Speaker_01
But it just seems to be like a really hard problem to get them to act as a coherent, autonomous software engineer that puts in his eight hours a day.
00:51:54 Speaker_00
Yeah, I think agency is in many senses actually easier to learn than we would have thought 10 years ago. But we actually aren't really learning agency at all in current systems. There's no kind of like selection for that.
00:52:07 Speaker_00
All the agency there is an accidental byproduct instead of somebody training on data. So from that perspective, it's miraculous that you could ask an LLM to try to do all these things, and they have a non-trivial success rate.
00:52:20 Speaker_00
If you told people 10 years ago, I think, that you could just behavior clone on individual letters following one by one, and then you would get this coherent action out of it and control robots and write entire programs, their jaws would drop.
00:52:32 Speaker_00
And they would just say that you've been huffing too many fumes from DeepMind or something. The reason that agency doesn't work is that we just have so little actual training data for it.
00:52:42 Speaker_00
An example of how you would do agency directly would be like Gato from DeepMind. There, they're actually training agents. Instead, we train them on these internet scrapes.
00:52:51 Speaker_00
which merely encode the outputs of agents or occasional descriptions of agents doing things, that kind of thing.
00:52:57 Speaker_00
There's no actual like logging of state environments, result reward trip sequences like a proper kind of reinforcement learning setup would have.
00:53:06 Speaker_00
I would say that what's more interesting actually is that nobody wants to train agents in a proper reinforcement learning way today. Instead, everyone wants to train LLMs and then do everything with as little RL as possible on the back end.
00:53:19 Speaker_01
Look, as Gordon just said, the biggest bottleneck in making these LLM models more useful has simply been the lack of good training data for these agentic workflows. This is an even bigger bottleneck than compute.
00:53:34 Speaker_01
Turing is solving this problem for every single AI lab that you've heard of. Gemini, OpenAI, Anthropic, Meta. They're basically the best-kept secret in AI.
00:53:45 Speaker_01
Turing provides complete post-training services for evals, SFT, RLHF, and DPO to make models better at thinking, reasoning, and coding. And it's all vetted by their AI and STEM experts.
00:54:00 Speaker_01
Turing makes it easy to make models multimodal, more factual, better at math, coding, advanced reasoning, and agentic workflows. And they also make it easy to just get a solid performance benchmark.
00:54:12 Speaker_01
For those of you at labs or companies training models, Turing has a bunch of offerings that can help you today, including a detailed model evaluation from their AI experts. Go to Turing.com slash Dwarkesh to learn more. All right, back to Goran.
00:54:38 Speaker_00
tried to probably make it in regular academia and maybe narrow my interests a lot more, something I could publish on regularly. Or I could possibly have tried to opt out and become a librarian, like one of my favorite writers, Jorge Luis Borges.
00:54:53 Speaker_00
He was a librarian until he succeeded as a writer. Of course, I've always agreed with him about imagining Paradise as a kind of library.
00:55:02 Speaker_00
I regret that all the reading I do is now kind of on the computer, and I don't get to spend as much time in libraries, physical libraries. I genuinely love them, just like pouring through the stacks, looking for random stuff.
00:55:15 Speaker_00
Some of the best times for me when I was in university were always like going through these gigantic stacks of all sorts of obscure books and just looking at like a random spine, you know, pulling stuff off the shelf and reading obscure old technical journals to see all the strange and wonderful things that they were doing and documenting back then, which now have just been totally forgotten.
00:55:35 Speaker_01
If you could ask Borges one question, what would it be? Oh.
00:55:42 Speaker_00
He's a real hero of mine, so this isn't something I want to have a bad answer to. Can I ask why he's a hero of yours?
00:55:52 Speaker_00
When I was younger, one of the science fiction books that really impressed me was Dan Simmons' Hyperion, and especially the fall of Hyperion.
00:56:02 Speaker_00
In there, he alludes to Kevin Kelly's Out of Control book, which strongly features the parable of the Library of Babel. From there, I got the kind of collected editions of Borges' fiction and nonfiction, and I just read through them again and again.
00:56:16 Speaker_00
I was blown away by the fact that you could be so creative with all of this polymathic knowledge that he had in erudition and write these wonderful, entertaining, provocative short stories and essays.
00:56:28 Speaker_00
And I thought to myself, if I could be like any writer, any writer at all, I would not mind being Borges.
00:56:36 Speaker_01
Borges has a short poem called Borges and I, where he talks about how he doesn't identify with the version of himself that is actually doing the writing and publishing all of this great work. And I don't know if you identify with that at all.
00:56:52 Speaker_00
Yeah, I think when I was a kid, I did not understand that essay, but I think I understand it now.
00:56:59 Speaker_01
What are other pieces of literature that you encountered where now you really understand what they were getting at, but you didn't when you first came across them?
00:57:09 Speaker_00
Ted Chiang's Story of Your Life comes to mind. I completely blew understanding it the first time that I read it. I had to get a lot more context where I could actually go back and understand what his point was.
00:57:22 Speaker_00
Gene Wolfe's Suzanne Delage story was also a complete mystery to me. It took like 14 years to actually understand it, but I'm very proud of that one specifically. That was a very recent one.
00:57:33 Speaker_01
Oh, and what did you figure out about Suzanne Delage?
00:57:36 Speaker_00
Yeah, so Gene Wolfe's Suzanne Delage is a very, very short story about this guy remembering not meeting a woman in his local town and thinking, oh, that's kind of strange. That's the whole story.
00:57:49 Speaker_00
Nobody has any idea what it means, even though we're told that it means something. And Gene Wolfe, the author, is a genius writer, but nobody could figure it out for like 40 years. Last year, I figured it out.
00:58:03 Speaker_00
It turns out it's actually a subtle retelling of Dracula, where Dracula invades the town and steals the woman from him. He's been brainwashed by Dracula in a very Bram Stoker way to forget it all.
00:58:16 Speaker_00
And every single part of the story is told by what's not said in the narrator's recollection. It's incredible. It's the only story I know which is so convincingly written by what's not in it.
00:58:29 Speaker_01
That's crazy that you figured that out. The Ted Chiang story, the story of your life, can you remind me what that one's about?
00:58:36 Speaker_00
The surface story is just about a bunch of weird aliens who come to Earth.
00:58:39 Speaker_01
Oh, right, right. It's the same plot as Arrival.
00:58:41 Speaker_00
They have this weird language, which didn't have a sense of time. The narrator learned to see the future, and then the aliens left.
00:58:48 Speaker_01
And then what was it that you realized about that story?
00:58:51 Speaker_00
The first time I read it, it struck me as a kind of stupid ESP story about seeing the future. Very stupid, boring, kind of standard conventionalism, verbose, and like dragging in much kind of like irrelevant physics.
00:59:06 Speaker_00
Only a while after I first read it and was thinking about it did I understand that it was not about time travel or being able to see the future.
00:59:15 Speaker_00
It's instead about a totally alien kind of mind that's equally valid in its own way in which you see everything as part of an already determined story heading to a predestined end.
00:59:30 Speaker_00
This turned out to be mathematically equivalent and equally powerful as our conventional view of the world. Events marching one by one to an unknown and changing future.
00:59:40 Speaker_00
That was the case where Chang was just writing at too high a level for me to understand. I pattern matched it to some much more common kind of stupid story.
00:59:49 Speaker_01
How do you think about the value of reading fiction versus nonfiction?
00:59:53 Speaker_00
I think you could definitely spend the rest of your life reading fiction and not benefit whatsoever from it, other than having memorized a lot of trivia about things that people made up. I tend to be pretty cynical about the benefits of fiction.
01:00:07 Speaker_00
Most fiction is not written to make you better in any way. It's written just to entertain you or exist and to fill up time.
01:00:13 Speaker_01
But it sounds like your own ideas have benefited a lot from the sci-fi that you read.
01:00:17 Speaker_00
Yeah, but it's extremely little sci-fi in the grand scheme of things, right? Easily 99% of the sci-fi I read was just completely useless to me.
01:00:26 Speaker_00
I could have easily cut it down to 20 novels or short stories which actually were good enough and insightful enough to actually change my view.
01:00:34 Speaker_00
I mean, one volume, for instance, of Blindsight by Peter Watts is worth all 100 Xanth novels or all 500 expanded universe novels of Star Wars.
01:00:43 Speaker_01
The ones you did find insightful, the top 20 or so, what did they have in common?
01:00:48 Speaker_00
I would say that the characteristic they have is that they all take non-human intelligence seriously. It doesn't seem to, you know, it doesn't have to be artificial intelligence necessarily.
01:01:00 Speaker_00
It's taking the idea of non-human intelligence seriously and not imagining your classic sci-fi scenario of humans kind of like going out into the galaxy with ray guns, the sort of thing where you have rockets and ray guns, but you don't have cell phones.
01:01:14 Speaker_00
People complain that the singularity is a sort of like boring, overused sci-fi trope.
01:01:20 Speaker_00
But if you went out and actually grabbed random books of science fiction that are out there, you'd find that like less than 1% contain anything remotely like that, right, or have any kind of relevance to the current context that we actually face with AI.
01:01:33 Speaker_01
Do people tend to underrate or overrate your intelligence?
01:01:37 Speaker_00
I would say they overestimate it. You know, they mistake for intelligence the fact that I remember many things, that I've written many things over the years.
01:01:45 Speaker_00
They imagine that, you know, if they sat me down, that I could do it all spontaneously at the moment that they're meeting me or talking to me.
01:01:52 Speaker_00
But many things that I've thought about, I think I have the advantage of having looked at before over a long time, so I'm cheating.
01:02:00 Speaker_00
You know, when I talk to people, I may just be quoting something that I've already written or at least thought a lot about. So I think I come off as a lot smarter when you're reading me than I actually am.
01:02:09 Speaker_00
I would say I'm not really all that smart compared to many people I've known who update very fast on the fly. But in the end, it's the output that matters, right?
01:02:19 Speaker_01
Yeah, I guess there is an on-the-fly kind of intelligence, but there's another kind of intelligence, which is this ability to synthesize things over a long period of time, then come up with grand theories as a result of all these different things that you're seeing.
01:02:34 Speaker_01
And I don't think that's just crystallized intelligence, right?
01:02:51 Speaker_00
You know, like another example of that pattern. And if you just saw each particular step, I think you would say that the steps in isolation were very reasonable.
01:03:00 Speaker_00
It's only when that happens over a decade and you don't see the individual stuff that my output at the end looks like magic. One of my favorite quotes about this process is from the magician's pen and teller.
01:03:11 Speaker_00
Teller says, magic is putting in more effort than any reasonable person would expect you to.
01:03:17 Speaker_00
He tells the story about how they make cockroaches appear from a top hat, where the trick is that they researched and found special cockroaches and then found special styrofoam to trap the cockroaches and arranged all of that, worked out all of those details just for this one single trick that they do.
01:03:36 Speaker_00
And in the audience, you think no reasonable person would do that, put in all of that effort to just get the payoff of this trick. But they do it. And the result is cockroaches somehow appearing from an empty hat.
01:03:50 Speaker_01
That's one of the interesting things about your process. Because there's a couple of writers, like Matt Levine or Berne Hobart, who write an article every day. And I think of them almost like autoregressive models.
01:04:00 Speaker_01
And then on you, there's, on some of the blog posts, you can see the start date and the end date that you list on your website of when you've been working on a piece. And sometimes it's like 2009 to 2024.
01:04:12 Speaker_01
And I feel like that's just much more like diffusion, and you're just like, keep iterating on the same image again and again. One of my favorite blog posts of yours is your blog post, Evolution as a Backstop to RL.
01:04:25 Speaker_01
where you talk about evolution as basically a mechanism to learn a better learning process. And that explains why corporations don't improve over time, but biological organisms do.
01:04:38 Speaker_01
I'm curious if you can walk me through the years that it took to write that. What was that process like, step by step?
01:04:45 Speaker_00
Yeah, so the backstop essay that you're referring to is the synthesis of seeing the same pattern show up again and again, a kind of stupid, inefficient way of learning, which you use to learn something smarter, but where you still can't get rid of the original one entirely.
01:05:01 Speaker_00
Right. So sometimes examples will just kind of connect to each other. when I was thinking about this. Other times, you know, once I started watching for this pattern, I would say, oh yeah, you know, pain is a good example of this.
01:05:17 Speaker_00
Maybe this explains why humans have pain in the very specific way that we have it, when you can logically imagine other kinds of pain, and those other pains would be smarter, but nothing keeps them honest.
01:05:30 Speaker_00
So you just kind of chain them one by one, these individual examples of the pattern you're watching for, and kind of keep clarifying the central idea as you go.
01:05:41 Speaker_00
Wittgenstein says that you can look at an idea from many directions and then go in spirals around it. And in an essay like Backstop, it was me kind of spiraling around this idea of having many layers of learning all the way down.
01:05:57 Speaker_01
And then so once you notice one example of this pattern, do you just, like you notice this paint example, do you just keep adding examples to that? I mean, just walk me through the process over time.
01:06:08 Speaker_00
Yeah, so for that specific essay, the first versions were about corporations not evolving. And then as I read more and more of the kind of meta reinforcement learning literature from DeepMind especially, I added in material about neural networks.
01:06:24 Speaker_00
And then I kind of kept reading and thinking about the philosophy of mind papers that I had read. And I eventually nailed down the idea that pain, you know, might be another instance of this. Because pain, like, makes us learn, right?
01:06:41 Speaker_00
But we can't get rid of it because we need it to keep us honest. And anyway, at that point, you have more or less the structure of the current essay.
01:06:48 Speaker_01
And then are there examples of blog posts where it's not a matter of accumulating different instances of what you later realize is one bigger pattern, but rather you just got to have the full thesis at once?
01:07:02 Speaker_00
For those essays where there is a kind of like individual eureka moment, there usually is still a bunch of disparate things that I've been making notes on that I don't even realize are connected.
01:07:13 Speaker_00
They just bother me for a long time and kind of like sit there bothering me. And I keep looking for explanations for each individual one and just not finding them. It keeps bothering me, keeps bothering me.
01:07:24 Speaker_00
And then one day I hit kind of that sudden moment that makes me go, bam, eureka, right? These all are connected. I just have to kind of like sit down and write the single gigantic essay that pours out about it and then it's done.
01:07:39 Speaker_00
That particular essay, you know, will just be done at that point, like right in one go. I might add in links like later on or references, but it won't fundamentally change from that point.
01:07:51 Speaker_01
What's an example of an essay that had this kind of process?
01:07:54 Speaker_00
Yeah, so someone asked about how I came up with one yesterday as a matter of fact. It's one of my oldest essays, The Melancholy of Subculture Society.
01:08:03 Speaker_00
For that one, I'd been reading about these miscellaneous things like David Foster Wallace on tennis, people on internet media, like video games, and then one day it just kind of hit me.
01:08:14 Speaker_00
This feeling or, you know, observation that it's incredibly sad that we have all these subcultures and tribes online and that they can find community together, but they're still incredibly isolated from the larger society.
01:08:27 Speaker_00
And then, you know, one day a flash kind of just hit me about how beautiful and yet also sad this is. And I just sat down and I wrote down the entire thing more or less. I haven't really changed it since that much at all.
01:08:41 Speaker_00
I've added more links and quotes and examples over time, but nothing important. The essence was just kind of this like flash and I wrote it down while it was there.
01:08:50 Speaker_01
One of the interesting quotes you have in that essay is from David Foster Wallace when he's talking about the tennis player Michael Joyce.
01:08:59 Speaker_01
And he's talking about the sacrifices that Michael Joyce has had to make in order to be top 10 in the world at tennis, which include things like being basically functionally literate because he's been playing tennis every single day since he was, you know, seven or something and not really having any life outside of tennis.
01:09:16 Speaker_01
What are the Michael Joyce type sacrifices that you have had to make to be born?
01:09:21 Speaker_00
That's a hard hitting question. How have I amputated my life in order to write? I think I've amputated my life in many respects, professionally and personally, especially in terms of travel.
01:09:34 Speaker_00
There are many people I envy for their ability to kind of travel and socialize or for their power and their positions in places like Anthropic where they're insiders. I've sacrificed whatever career I could have had or whatever fun lifestyle.
01:09:49 Speaker_00
a digital nomad lifestyle and going outdoors, being a Buddhist monk or maybe a fancy trader.
01:09:57 Speaker_00
All those have to be sacrificed, really, for the patient work of sitting down every day and reading papers until my eyes bleed and hoping that something good comes out of it someday.
01:10:07 Speaker_01
I mean, why does it feel like there's a trade-off between the two? Because there are obviously writers who travel a lot, for example, Tyler Cowen, or writers who have a lot of influence, like Jack Clark at Anthropic, right?
01:10:20 Speaker_01
So why does it feel like you can't do both at the same time?
01:10:23 Speaker_00
I can't be or be compared to Tyler Cowen here. Tyler Cowen is a one man industry.
01:10:28 Speaker_01
So is Gorn.
01:10:29 Speaker_00
Yeah, but he can't be replicated. So I just can't be Tyler Cowen. You know, Jack Clark, he's also his own thing. He's able to write the stories and his issues very well while also being a policy person. I respect those people. You know, I admire them.
01:10:44 Speaker_00
But none of them, I think, quite hit my particular interest and niche of following weird topics for a long period of time. and then collating, kind of sorting through the information.
01:10:55 Speaker_00
For me, that just requires a large commitment to reading vast masses of things in hopes that some tiny detail perhaps will turn out one day to be important.
01:11:04 Speaker_01
So walk me through this process. You mentioned you read papers until your eyes bleed out at the end of the day. Let's just start. You wake up in the morning and you get straight to the papers. What does your day look like?
01:11:17 Speaker_00
So, I mean, the workflow right now is more like I wake up, I do normal morning things, and then I clean up the previous day's work on the website. I'll deal with kind of various issues like formatting or spelling errors, and I kind of
01:11:34 Speaker_00
you know, review it and think if I've properly collated everything and put it in the right places from the previous day. Sometimes I might have like an extra thought that I need to go in and add or make a comment that I realized was important.
01:11:47 Speaker_00
After that, I often, you know, shamelessly just go to Twitter or my RSS feed and just read a large amount until, you know, maybe I get distracted by some comment or question from someone. I'm And then and then I go to the gym by that point.
01:12:19 Speaker_00
I'm pretty burned out from everything Yes, you know I like going to the gym not because of any kind of meathead or athlete or even really enjoy weightlifting But just because I think it's it's the thing I can do that's the most opposite from sitting in front of my computer reading Yeah, this is your theory of burnout right that you just got to do the opposite.
01:12:37 Speaker_00
Yeah, I Um, you know that the problem I think when people experience burnout is that you just feel kind of a lack of reward for what you're doing or what you're working on.
01:12:47 Speaker_00
You just need to do something completely different, something as different as possible. Maybe you could do better than weightlifting. But for me, you know, it does feel very different from anything that I do in front of a computer.
01:12:59 Speaker_01
I want to go back to your process. Every day, you're loading up all this context, you're reading all the RSS feeds and all these papers.
01:13:07 Speaker_01
And are you basically making contributions to all your essays, adding a little bit here and there every single day? Or are you building up some potential which will manifest itself later on as a full essay, a fully formed thesis?
01:13:23 Speaker_00
I would say it's more the latter one. I think all the minor low-level additions and pruning and fixing I do is really not that important. It's more just a way to make nicer essays.
01:13:35 Speaker_00
It's a purely kind of aesthetic goal to make it as nice an essay as I possibly can. And I'm really waiting to see kind of what happens next. What would be the next thing that I'll be provoked by to end up writing about?
01:13:48 Speaker_00
It's passing the time in between sudden eruptions. For many writers, you sort of like can't neglect this kind of gardening process, right? You don't harvest every day. You have to tend the garden for a long time in between harvests.
01:14:04 Speaker_00
If you start to neglect the gardening because you're gallivanting around the world, let's say you're going to book signing events, maybe you're doing all the publicity stuff, then you're not really doing the work of being in there tending the garden.
01:14:17 Speaker_00
And that's undermining your future harvest, even if you can't see it right now.
01:14:22 Speaker_00
If you ask kind of what is Tyler Cowen's secret to being Tyler Cowen, my guess would be that he's just really good at tending his garden, even as he travels a crazy amount. That would be his secret, that he's able to read books on a plane.
01:14:34 Speaker_00
You know, I can't read books on a plane. He's able to write everything in the airport. I can do a little bit of writing in the airport, but not very much. And he's also just very robust to the wear and tear of traveling.
01:14:45 Speaker_00
I'll be like collapsing in the hotel room after talking to people for eight hours. He's able to talk to people for eight hours and then go do podcasts and talk to someone for another four hours or whatever.
01:14:54 Speaker_00
It's extremely admirable, but I just can't do that.
01:14:57 Speaker_01
How often do you get bored? Because it sounds like you're spending all your day reading different things. Are they all just inherently interesting to you, or do you just trudge through it even when it's not in the moment compelling to you?
01:15:10 Speaker_00
I don't think I get bored too easily because I switch between so many different topics. Even if I'm kind of sick of deep learning papers, well, you know, then I have tons of other things I can read or argue with people about.
01:15:22 Speaker_00
So I don't really get bored. I just end up getting kind of exhausted. You know, I have to kind of go off and do something else, like lift weights.
01:15:31 Speaker_01
What is your most unusual but successful work habit?
01:15:35 Speaker_00
Yeah, I think I get a lot more mileage out of arguing with people online than pretty much any other writer does. I'm trying to give a genuine answer here, not some stupid thing about note-taking.
01:15:49 Speaker_00
I get a lot more out of arguing with people than I think most people do. You need motivation to write and actually sit down and kind of crystallize something and do the harvest work. And after you tend your garden, you do have to do the harvest.
01:16:05 Speaker_00
And the harvest can be hard work. It's very tedious. And, you know, there are many people that I talk to who have many great ideas, but they don't want to harvest because it's tedious and boring. And it's very hot out there in the fields reaping. Right.
01:16:21 Speaker_00
And you're getting dusty and sweaty. Why wouldn't you just be inside having lemonade? And I think the motivation from arguing and being angry at people online is in plentiful supply. So I get a lot of mileage out of people being wrong on the Internet.
01:16:38 Speaker_01
What are the pitfalls of an isolated working process?
01:16:42 Speaker_00
I think aside from the obvious one that you could kind of, you know, be like arbitrarily wrong when writing by yourself and just become this like crazy loony, but by having a big confident wrong take.
01:16:53 Speaker_00
I think aside from that, you also have the issue of kind of the emotional toll of not having colleagues that you can kind of convince. You often just have this experience of kind of like shouting onto the Internet.
01:17:07 Speaker_00
And everyone on the internet kind of continues to be wrong. One thing I observe is that very often independent writers are overcome by resentment and anger and disappointment. They sort of like spiral into bitterness and crankdom. from there.
01:17:23 Speaker_00
And that's kind of what kills them. You know, they could have continued if they'd only been able to let go of the ideas and arguments and kind of like move on to the next topic.
01:17:33 Speaker_00
Spite can be a great motivation to write, but you have to use it skillfully and then kind of like let it go afterwards. You can only have it like while you need the motivation to write.
01:17:45 Speaker_00
And then if you keep going, you sort of hold on to it, you're sort of poisoning yourself.
01:17:50 Speaker_01
I'm sure you've seen all the comments from people who say that if Gordon spent the time that he spends fine tuning the CSS on his website towards more projects, more writing, that the benefits of society could be measured in the nearest million dollars.
01:18:08 Speaker_01
What's your reaction to people who say you're spending too much time on site design?
01:18:12 Speaker_00
I have no defense at all there in terms of objective benefits to society. You know, I do it because I'm selfish and I like it. That's my defense. I like the aesthetics of my website and it's a hobby. Does the design help you think?
01:18:25 Speaker_00
It does because I like rereading my stuff more when I can appreciate the aesthetics of it and the beauty of the website. It's easier for me to tolerate reading something for the hundredth time when I would otherwise be sick to death of it.
01:18:38 Speaker_00
Site maintenance is inherently, right, for the author, this kind of inherent spaced repetition. If I go over pages to check that some new formatting feature worked, I'm getting spaced repetition there.
01:18:50 Speaker_00
More than once, I've gone back to check some stupid CSS issue and look at something and thought, oh, I should change something or, oh, that means something.
01:18:58 Speaker_00
So in a way, it's not, I think, as much of a waste as it looks, but I can't defend it entirely. If someone wants to make their own website, they should not invest as much for the aesthetic value. I just want a really nice website.
01:19:12 Speaker_00
There's so many bad websites out there and it depresses me. There's at least one website I love.
01:19:17 Speaker_01
By the way, I'm going to mention this since you never mentioned it yourself, but I think the main way you fund your research is through your Patreon, right? And, uh, yeah, you never advertise it, but I don't know.
01:19:28 Speaker_01
I feel like the thing, kind of thing you're doing, if, um, if it was financially viable and if it, uh, got adequate funding, not only would you be able to keep doing it, but other people who wanted to be independent researchers could see it's a thing you can do.
01:19:43 Speaker_01
It's a viable thing you can do and more Gorons would exist.
01:19:47 Speaker_00
Yeah, well, I don't necessarily want more Gwerns to exist. I just want more writers and more activeness and more agency in general.
01:19:59 Speaker_00
I would be perfectly happy if someone simply wrote more Reddit comments and never took a dollar for their writings and just wrote better Reddit comments.
01:20:07 Speaker_00
I'd be perfectly happy if someone had a blog and they kept writing, but they just put a little more thought into the design. I'd be kind of perfectly happy if no one ever wrote something, but they hosted PDFs so that links don't rot.
01:20:21 Speaker_00
In general, I think you don't have to be a writer delivering long form essays. That's just one of many ways to write. It happened to be the one that I personally kind of prefer, but it'd be totally valid to be a Twitter thread writer.
01:20:35 Speaker_00
How do you sustain yourself while writing full time? Patreon and savings. I have a Patreon which is around 900 to 1,000 each month, and then I cover the rest with my savings.
01:20:47 Speaker_00
I got lucky with having some early bitcoins and made enough to write for a long time, but not forever. So I try to spend as little as possible to make it last. I should probably advertise the Patreon more, but I'm too proud to shill it harder.
01:21:04 Speaker_00
It's also awkward trying to come up with some good rewards which don't entail a paywall. Patreon and Substack work well for a lot of people, like Scott Alexander, because they like writing regular newsletter-style updates, but I don't like to.
01:21:17 Speaker_00
I just let it run and hope it works.
01:21:19 Speaker_01
Wait, so if you're doing 900 to 1,000 a month and you're sustaining yourself on that, that must mean you're sustaining yourself on less than $12,000 a year. What's your lifestyle like at 12K?
01:21:31 Speaker_00
Yeah, I mean, listen, I live in the middle of nowhere. You know, I don't travel much or eat out or have health insurance or anything like that. I cook my own food. I use a free gym. There was this time where the floor of my bedroom started collapsing.
01:21:49 Speaker_00
It was so old that the humidity had like decayed the wood. And we just got a bunch of scrap wood and a joist and propped it up. So if it lets in some bugs, oh well. I live like a grad student, but with better ramen basically.
01:22:05 Speaker_00
And I don't mind it much since I think I basically spent all my time reading anyway.
01:22:11 Speaker_01
It's still surprising to me that you can take care of rent, take care of your cats, deal with any emergencies, all of that on 12k a year.
01:22:21 Speaker_00
Yeah, I mean, I'm lucky enough to be in excellent health and have had no real emergencies to date. This can't last forever, obviously, and so it won't.
01:22:32 Speaker_00
I'm definitely not trying to claim that this is like an ideal lifestyle or that anyone else could or should try to replicate my exact approach.
01:22:41 Speaker_00
I got lucky with Bitcoin in particular and with being satisfied living like a monk and with the health that I've had.
01:22:49 Speaker_00
Anyone who would like to take up a career as a writer or blogger should understand this is not an example that they specifically can imitate, right? I don't think I'm not trying to be a role model.
01:23:00 Speaker_00
Every writer will have to figure it out a different way. Maybe it can be something like a sub stack or just writing on the side while slinging JavaScript for a tech company. I don't know.
01:23:10 Speaker_01
It seems like you've enjoyed this recent trip to San Francisco. What would it take to get you to move here?
01:23:17 Speaker_00
Yeah, I think at this point it mostly is just money that's stopping me. I probably should bite the bullet and just move anyway.
01:23:25 Speaker_00
But I'm a miser at heart, and I hate thinking of how many months of writing runway I'd have to give up for each month in San Francisco.
01:23:33 Speaker_00
If someone wanted to give me, I don't know, 50K to 100K a year to move to SF and continue writing full-time like I do now, I'd take it in a heartbeat. Until then, I'm still trying to psych myself up into a move.
01:23:44 Speaker_01
I don't know, that sounds doable. I mean, and if somebody did want to get in touch with you about contributing, how would they do that? They could just email me at gwern at gwern.net.
01:23:53 Speaker_01
All right, so after the episode, I convinced Gwern to set up a Stripe checkout link where people can donate if they wish to. So if you want to support his work, please go to the link in the description.
01:24:04 Speaker_01
Look, the way that Guern is obsessed with his rabbit holes, Stripe is obsessed with payments on your behalf. The difference between making and missing a sale often comes down to how a customer wants to pay.
01:24:18 Speaker_01
In Switzerland, they want to pay with Twind, in Netherlands, maybe with Ideal. These are systems you might never have heard of, but they're super popular in those countries.
01:24:27 Speaker_01
Stripe optimizes their checkout experience so that customers get served whatever payment experience is most likely to work for them. And if it works for them, that means more buyers for you, more revenue for you. Stripe is how I run my business.
01:24:42 Speaker_01
It's how I, in fact, made my business in the first place. I set up my company using Stripe Atlas, and now I invoice all my advertisers using Stripe Invoicing.
01:24:51 Speaker_01
And look, I told Guern to set up his donation link using Stripe because Stripe has been genuinely delightful to work with. And that's why I recommended Stripe to him. That's why I recommend Stripe to you. Go to stripe.com to learn more.
01:25:04 Speaker_01
All right, back to Guern. By when will AI models be more diverse and more different from each other than the human population?
01:25:13 Speaker_00
I'm going to say that if you exclude capability from that, AI models are already much more diverse cognitively than humans are. I think different LLMs think in very distinct ways that you can tell right away from a sample of them, right?
01:25:27 Speaker_00
So an LLM operates nothing like a GAN. GAN also is totally different from VAEs. They have totally different latent spaces, especially in the lower end where they're smaller or bad models.
01:25:39 Speaker_00
They have wildly different artifacts and errors in a way that we just wouldn't see with humans. I think humans are really very quite similar in writing and attitude compared to these absurd outputs of different kinds of models.
01:25:52 Speaker_01
Really, I mean, if you look at chatbot arena, where you can do the side by side comparisons of the outputs of different models, it's often very hard to tell which one comes from which model.
01:26:03 Speaker_00
Yeah, but I mean, this is all very heavily tuned, right? So now you're restricting it to relatively recent LLMs, with everyone riding on each other's coattails, not training on the exact same data.
01:26:13 Speaker_00
So I think this is a situation like much closer to if they were identical twins.
01:26:18 Speaker_00
If I'm, you know, I'm not restricting myself to just LLMs and I compare the wide diversity of say like image generation models that we've had, they often have totally different ways, right?
01:26:27 Speaker_00
Some of them seem as similar to each other as ants do to beavers. I think within LLMs, I would agree that there has been a massive loss of diversity. Things used to be way more diverse within like among LLMs.
01:26:42 Speaker_00
But across deep learning in general, I think we've seen a whole range of minds and ways to think that you wouldn't find in any philosophy of mind paper.
01:26:51 Speaker_01
What's an example of two different models that have these kinds of cognitive differences?
01:26:55 Speaker_00
Yeah, I'll give one example I was telling someone the other day. So, you know, GAN models have incentives to hide things because it's an adversarial loss. Whereas diffusion models have no such thing, right? So GAN models are scared.
01:27:12 Speaker_00
They put hands off the screen and they just kind of can't think about hands. Whereas diffusion models think about hands, but in their like gigantic monstrous Cthulhu-esque abortions, People weren't paying enough attention to scaling in 2020.
01:27:29 Speaker_01
Is there some trend today where people aren't really comprehending the full implications of where this is headed?
01:27:37 Speaker_00
I'm excited by the weight loss drugs, the GLP drugs. Their effects in general on health and addiction across all sorts of behaviors really surprised me. No one predicted that as far as I know.
01:27:52 Speaker_00
And while the results are still very preliminary, it does seem like it's real. So I think that's going to tell us something important about human willpower and dysfunctionality.
01:28:04 Speaker_01
Do these GLP drugs break the Algernon argument from your blog post that if there are any simple, useful interventions without bad side effects, then evolution should have already found them?
01:28:18 Speaker_00
I think it's too soon to say because we haven't actually figured out what's going on with the GLPs to even understand what they're doing at all. What has the off target? It's kind of crazy that activating and deactivating both work.
01:28:33 Speaker_00
It's a completely crazy situation. I don't really know what to think about the Algernon argument there.
01:28:39 Speaker_00
It could be that the benefits actually decrease fitness in the fertility sense because you're going out and having a happy life instead of having kids. Um, so no offense to parents.
01:28:49 Speaker_00
Um, or it could just be that it's hitting the body in a way that's really, really hard to replicate in any kind of genetic way. Um, or I don't know, it could just, it's too soon.
01:28:58 Speaker_00
When I think back, I see that the obesity crisis only became obvious around the 1990s. Um, you know, it's, it's quite recent. And I look back at photos and today is completely unrecognizable from 1990.
01:29:12 Speaker_00
You look at photos and people are still thin, right? Um, you look at photos now and everyone is like a blimp. So you just, you can't possibly have any kind of Algernon argument over like 20 to 30 years.
01:29:23 Speaker_01
When you look back at the Romans and you see how the lead was constantly poisoning the entire city, what credence do you give to the possibility that something in our environment is having a magnitude of effect on us that lead was having on the ancient Romans?
01:29:42 Speaker_00
Yeah, I think the odds of there being something as bad as lead is almost 100%. We have so many things out there, right? Chemists are always cooking up new stuff. There are all sorts of things with microbiomes.
01:29:53 Speaker_00
Plastics are trendy, but maybe it's not plastics. Maybe it's something else entirely. But there's almost no way that everything that we have put out there is totally benign and safe and has no harmful effects at any concentration.
01:30:07 Speaker_00
It just seems like a really strong claim to be making. I don't believe in any particular one, but I do believe in like 1% here, 1% here, 1% here. There's something out there.
01:30:21 Speaker_00
There's something out there where we're just like gonna look back at it and say, wow, like those people were really poisoning themselves just like with leaded gasoline. If only they'd known, you know, X, Y, and Z or whatever. It's so obvious now.
01:30:34 Speaker_01
And do you think this would manifest itself most likely in cognitive impairments or in obesity or in something else?
01:30:41 Speaker_00
Yeah, I think a priori, I would expect, uh, possibly intelligence to be like the single most fragile thing and most harmed by it. But when we, when we look at the time series there, intelligence is pretty stable overall.
01:30:55 Speaker_00
Um, so, so I would have to say that whatever the harmful thing is, it's probably not going to be on intelligence. Whereas obesity is a much better candidate because you do see obesity go crazy right over the last 30 years.
01:31:08 Speaker_01
I was, um, I was surprised yesterday to hear you say that you are skeptical of Bay Area type experimentation with psychedelics.
01:31:19 Speaker_01
And because, you know, I sort of associate you with very much this word of you got to experiment with different substances and see if they are helpful to you. And so I'm curious why you draw Chesterton's fence here when it comes to psychedelics.
01:31:34 Speaker_00
Yeah, I think the cleanest way to divide that would just be to point out that the effects of psychedelics can be acute and permanent. The things I was looking at are much more controlled in the sense that they are relatively manageable in effect.
01:31:48 Speaker_00
None of them affect your judgment permanently about whether to take more nootropics. Whereas I think something like LSD permanently changes how you see things such as taking LSD or permanently changes your kind of psychiatric state.
01:32:02 Speaker_00
There's a cumulative effect with psychedelics that you don't see much with neurotropics, which makes neurotropics inherently a heck of a lot safer and much more easy to quantify the effects of.
01:32:13 Speaker_00
With nootropics, you don't see people spinning off into the crazy outcomes psychedelics have. They get crazier and crazier each time they take another dose, which makes them crazy enough to want to take another dose.
01:32:26 Speaker_00
Psychedelics have what you might call a self-recommending problem, where they always make you want to take more of them. I think it's similar to meditation. what is the most visible sign of having done a lot of meditation, right?
01:32:41 Speaker_00
Is that you seem compelled to tell people that they ought to meditate. This kind of spiral leads to bad outcomes for psychedelics that you just don't see with nootropics.
01:32:51 Speaker_00
The standard failure case for nootropics is that you spend like a few hundred or thousand dollars and then you got no real benefit out of it. You went on with your life. You know that kind of thing.
01:33:02 Speaker_00
You did some weird drugs maybe for a while and that was all. It's not so bad.
01:33:05 Speaker_00
It's a weird way to get your entertainment but in principle it's not really all that worse than going to the movie theater for a while and spending a thousand dollars on movie theater tickets.
01:33:14 Speaker_00
With psychedelics, you're changing yourself permanently, irrevocably in a way you don't really understand, and exposing yourself to all sorts of malicious outside influences, whatever happens to occur to you while you're there, and very impressionable.
01:33:27 Speaker_00
And obviously, a few uses can be good. I've gotten good out of my few uses. But if you're doing it more than that, you should really have a hard look in the mirror about what benefit you think you're getting and how you're changing.
01:33:41 Speaker_01
Have you put any thought into what is, people don't know your voice, people don't know your face, and as a result, they have this interesting parasocial relationship with you.
01:33:50 Speaker_01
And I wonder if you have a theory of what kind of role you fill in people's life, basically.
01:33:55 Speaker_00
Are you asking what role I actually fill or the role that I aspire to fill? Let's do both. Okay. The role that I want to fill is actually sort of how LLMs see me, oddly enough.
01:34:09 Speaker_00
I think if you play around with LLMs like Claude, Claude 3, a character named Gwern sometimes will show up.
01:34:16 Speaker_00
And he plays the role of kind of this like mentor or old wizard offering insight into the situation and exhorting them, you know, with a call to adventure. You too can write stuff and do stuff and think stuff.
01:34:30 Speaker_00
I would like people to go away having not just been kind of entertained or gotten some useful information, but to be better people in however slight a sense. to have an aspiration that webpages could be better, that the internet could be better.
01:34:45 Speaker_00
You, too, could go out and read stuff. You, too, could have all your thoughts and compile your thoughts into essays, too. You could do all of this.
01:34:52 Speaker_00
But I fear that the way that it actually works for quite a few people is that I wind up either as kind of a guru or trickster devil kind of figure.
01:35:01 Speaker_00
Depending on whether you like me or hate me, either I'm the god of statistics and referencing who can do no wrong, just take everything on the side as gospel, which I really dislike, or I'm just some sort of horrible, covert, malicious, neo-Nazi, eugenicist, totalitarian, communist, anti-Chinese devil figure lurking in the background trying to bring down Western society.
01:35:23 Speaker_01
Final question. What are the open rabbit holes you have, the things you're curious about but don't have an answer to, that you hope to have an answer to by 2050? I think by 2050,
01:35:36 Speaker_00
I really hope that we can finally answer some of these like really big questions about ourselves that have just reliably resisted definitive answers. I think a lot of them might not matter anymore, but I'd still like to know.
01:35:53 Speaker_00
So for example, like why do we sleep or dream? Why do humans age? Why does sexual reproduction exist? Why do humans differ so much from each other and also day to day? Why do humans take so long to develop technological civilization?
01:36:12 Speaker_00
Where are all the aliens? Why didn't China have the industrial revolution instead? How should we have predicted the deep learning revolution? And why are our brains so oversized compared to artificial neural networks?
01:36:29 Speaker_00
I think those are some of the questions that I really hope we've answered by 2050.
01:36:33 Speaker_01
All right, Goran, this has been excellent. Thank you so much for coming on the podcast. Thanks.