#447 – Cursor Team: Future of Programming with AI AI transcript and summary - episode of podcast Lex Fridman Podcast

Go to PodExtra AI's episode page (#447 – Cursor Team: Future of Programming with AI) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.

Go to PodExtra AI's podcast page (Lex Fridman Podcast) to view the AI-processed content of all episodes of this podcast.

Lex Fridman Podcast episodes list: view full AI transcripts and summaries of this podcast on the blog

Episode: #447 – Cursor Team: Future of Programming with AI

Author: Lex Fridman
Duration: 02:37:38

Episode Shownotes

Aman Sanger, Arvid Lunnemark, Michael Truell, and Sualeh Asif are creators of Cursor, a popular code editor that specializes in AI-assisted programming. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep447-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/cursor-team-transcript CONTACT LEX:

Feedback - give feedback to Lex: https://lexfridman.com/survey AMA - submit questions, videos or call-in: https://lexfridman.com/ama Hiring - join our team: https://lexfridman.com/hiring Other - other ways to get in touch: https://lexfridman.com/contact EPISODE LINKS: Cursor Website: https://cursor.com Cursor on X: https://x.com/cursor_ai Anysphere Website: https://anysphere.inc/ Aman's X: https://x.com/amanrsanger Aman's Website: https://amansanger.com/ Arvid's X: https://x.com/ArVID220u Arvid's Website: https://arvid.xyz/ Michael's Website: https://mntruell.com/ Michael's LinkedIn: https://bit.ly/3zIDkPN Sualeh's X: https://x.com/sualehasif996 Sualeh's Website: https://sualehasif.me/ SPONSORS: To support this podcast, check out our sponsors & get discounts: Encord: AI tooling for annotation & data management. Go to https://encord.com/lex MasterClass: Online classes from world-class experts. Go to https://masterclass.com/lexpod Shopify: Sell stuff online. Go to https://shopify.com/lex NetSuite: Business management software. Go to http://netsuite.com/lex AG1: All-in-one daily nutrition drinks. Go to https://drinkag1.com/lex OUTLINE: (00:00) - Introduction (09:25) - Code editor basics (11:35) - GitHub Copilot (18:53) - Cursor (25:20) - Cursor Tab (31:35) - Code diff (39:46) - ML details (45:20) - GPT vs Claude (51:54) - Prompt engineering (59:20) - AI agents (1:13:18) - Running code in background (1:17:57) - Debugging (1:23:25) - Dangerous code (1:34:35) - Branching file systems (1:37:47) - Scaling challenges (1:51:58) - Context (1:57:05) - OpenAI o1 (2:08:27) - Synthetic data (2:12:14) - RLHF vs RLAIF (2:14:01) - Fields Medal for AI (2:16:43) - Scaling laws (2:25:32) - The future of programming PODCAST LINKS: - Podcast Website: https://lexfridman.com/podcast - Apple Podcasts: https://apple.co/2lwqZIr - Spotify: https://spoti.fi/2nEwCF8 - RSS: https://lexfridman.com/feed/podcast/ - Podcast Playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4 - Clips Channel: https://www.youtube.com/lexclips

Summary

In this episode of the Lex Fridman Podcast, AI researcher Lex Fridman hosts the founders of Cursor, an AI-assisted code editor, to discuss AI's transformative impact on programming. As a modified version of VS Code, Cursor seeks to overcome the limitations of current tools like GitHub Copilot by innovating and enhancing coding productivity through features like Cursor Tab and code diffs. The conversation explores the challenges and strategies in AI tool development, including prompt engineering, speculative edits, and cache optimization. Broader implications for programming's future are considered, such as synthetic data, reinforcement learning, and potential breakthroughs in theorem proving.

Go to PodExtra AI's episode page (#447 – Cursor Team: Future of Programming with AI) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.

Full Transcript

00:00:00 Speaker_05
The following is a conversation with the founding members of the Cursor team, Michael Truel, Swale Asif, Arvid Lundmark, and Aman Sanger. Cursor is a code editor based on VS Code that has a lot of powerful features for AI-assisted coding.

00:00:18 Speaker_05
It has captivated the attention and excitement of the programming and AI communities. So I thought this is an excellent opportunity to dive deep into the role of AI in programming.

00:00:30 Speaker_05
This is a super technical conversation that is bigger than just about one code editor. It's about the future of programming, and in general, the future of human-AI collaboration in designing and engineering complicated and powerful systems.

00:00:47 Speaker_05
And now, a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast.

00:00:54 Speaker_05
We got Encored for unifying your machine learning stack, Masterclass for learning, Shopify for selling stuff online, NetSuite for your business, and AG1 for your health. Choose wisely, my friends.

00:01:07 Speaker_05
Also, if you want to get in touch with me for whatever reason, or take a survey, or submit questions for an AMA, all of that would be great. Go to LexFreeman.com contact. And now, on to the full ad reads.

00:01:21 Speaker_05
I try to make them interesting, but if you skip them, please still check out our sponsors. I enjoy their stuff. Maybe you will, too.

00:01:28 Speaker_05
This episode is brought to you by Encord, a platform that provides data-focused AI tooling for data annotation, curation, management, and for model evaluation.

00:01:39 Speaker_05
One of the things I love about these guys is they have a great blog that describes cleanly, I mean, it's technical, but it's not too technical, but it's sufficiently technical to where it's actually describing ideas, not BS.

00:01:52 Speaker_05
blog posts on sort of the state-of-the-art, like the OpenAI 01 model that was just released. So sometimes they integrate it into why this is a part of Encore, why this makes sense, and sometimes not. And so I love that.

00:02:07 Speaker_05
I recommend their blog just in general. That said, when they are looking at state-of-the-art models, they are always looking for ways to integrate it into their platform. Basically, it's a place to organize your data, and data is everything.

00:02:19 Speaker_05
This was true before the popularity and the explosion of attention methods of transformers, and it is still very much true now. Sort of the non-synthetic, the human-generated data is extremely important.

00:02:34 Speaker_05
How you generate that data, how you organize that data, how you leverage it, how you train on it, how you fine-tune on it, the pre-training, the post-training, all of it, the whole thing. Data is extremely, extremely important.

00:02:45 Speaker_05
And so Encord takes data very seriously. Anyway, go try out Encord to create, annotate, and manage your AI data at Encord.com slash Lex. That's Encord.com slash Lex.

00:02:58 Speaker_05
This episode is also brought to you by Masterclass, where you can watch over 200 classes from the best people in the world in their respective disciplines. Carlos Santana on guitar, for example. I loved that one. There's a few guitar ones.

00:03:11 Speaker_05
Tom Morello, too. Great, great, great stuff. But Carlos Santana, his instrumental Europa. I haven't quite tried to play that, but it's on my to-do list.

00:03:23 Speaker_05
It's sort of one of those things you know for sure this is the thing I will play because it's too beautiful. It's too soulful. It feels like once you play, you understand something about the guitar that you didn't before. It's not blues.

00:03:38 Speaker_05
It's not, I don't know what it is. It's some kind of dreamlike teleportation into a psychedelic world. where the tone is warmer than anything else I've ever heard and still the guitar can cry. I don't know. I love it. He's a genius.

00:03:59 Speaker_05
So it's such a gift that you can get a genius like that. to teach us about his secrets. Get unlimited access to every Masterclass and get an additional 15% off an annual membership at masterclass.com slash lexpod. That's masterclass.com slash lexpod.

00:04:21 Speaker_05
This episode is also brought to you by Shopify, a platform designed for anyone to sell anywhere with a great looking online store, or simple looking online store, like the one I put together at lexfriedman.com slash store.

00:04:35 Speaker_05
I have a few shirts on there in case you're interested. And speaking of shirts, I'm reminded of thrift stores, which I very much loved for a long time. I still love thrift stores.

00:04:48 Speaker_05
or a nice place to get stuff like, I don't know, kitchen stuff and clothing. And the kind of clothing you get at thrift stores is actually pretty interesting because there's shirts there that are just unlike anything else you would get anywhere else.

00:05:03 Speaker_05
So if you're sort of selective, and creative minded, there's a lot of interesting fashion that's there. And in terms of t-shirts, there's just like hilarious t-shirts.

00:05:15 Speaker_05
T-shirts that are very far away from the kind of trajectories you have taken in life, or are not, but you just haven't thought about it. Like a band that you love, but you never would have thought to wear their t-shirt.

00:05:26 Speaker_05
Anyway, a little bit, I think of Shopify as the internet's thrift store. Of course, you can do super classy, you can do super fancy, or you can do super thrift. All of it is possible.

00:05:38 Speaker_05
Sign up for a $1 per month trial period at shopify.comslashlex, that's all lowercase. Go to shopify.comslashlex to take your business to the next level today.

00:05:49 Speaker_05
This episode is also brought to you by NetSuite, an all-in-one cloud business management system. Sometimes I think that NetSuite is supporting this podcast because they're trolling me.

00:06:01 Speaker_05
They're saying, hey Lex, aren't you doing a little too much talking? Maybe you should be building more. I agree with you, NetSuite. I agree with you. And so every time I do an ad read for NetSuite, it is a chance for me to confront my Jungian shadow.

00:06:19 Speaker_05
Some of the demons emerge from the subconscious and ask questions that I don't have answers to.

00:06:27 Speaker_05
Questions about one's mortality and that life is short and that one of the most fulfilling things in life is to have a family and kids and all of these things I would very much like to have.

00:06:37 Speaker_05
And also the reality that I love programming and I love building. I love creating cool things that people can use and share and that would make their life better. All of that.

00:06:49 Speaker_05
Of course I also love listening to podcasts and I kind of think of this podcast as me listening to a podcast where I can also maybe participate by asking questions.

00:06:59 Speaker_05
So all these things that you love, but you ask the hard question of like, okay, well, life is slipping away, it's short. It really, really is short. What do you wanna do with the rest of the minutes and the hours that make up your life?

00:07:13 Speaker_05
Yeah, so thank you for the existential crisis, Nasweet. I appreciate it. If you're running a business, if you have taken the leap into the unknown and started a company, then you should be using the right tools to manage that company.

00:07:27 Speaker_05
In fact, over 37,000 companies have upgraded to NetSuite. Take advantage of NetSuite's flexible financing plan at netsuite.com slash Lex. That's netsuite.com slash Lex. This episode is also brought to you by the delicious, the delicious AG1.

00:07:43 Speaker_05
It's an all-in-one daily drink to support better health and peak performance. It's basically a super awesome multivitamin that makes me feel like I have my life together, even when everything else feels like it's falling apart. At least I have AG1.

00:07:57 Speaker_05
At least I have that nutritional foundation to my life.

00:08:00 Speaker_05
So all the fasting I'm doing, all the carnivore diets, all the physical endurance events and the mental madness of staying up all night or just the stress of certain things I'm going through, all of that, AG1 is there. At least I have the vitamins.

00:08:19 Speaker_05
I also sometimes wonder, they used to be called Athletic Greens, and now they're called AG1. I always wonder, is AG2 coming? Like, why is it just one? It's an interesting branding decision. Like, AG1.

00:08:32 Speaker_05
Me, as an OCD kind of programmer type, it's like, okay, is this a versioning thing? Okay, is this like AG 0.1 alpha? When's the final release? Anyway, the thing I like to say and to consume is AG1.

00:08:51 Speaker_05
They'll give you one month supply of fish oil when you sign up at drinkag1.com slash lex. This is the Lex Friedman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Michael, Swale, Arvid, and Aman.

00:09:25 Speaker_05
All right, this is awesome. We have Michael, Aman, Swale, Arvid here from the Cursor team. First up, big ridiculous question. What's the point of a code editor?

00:09:36 Speaker_02
So the code editor is largely the place where you build software. And today, or for a long time, that's meant the place where you text edit a formal programming language.

00:09:47 Speaker_02
And for people who aren't programmers, the way to think of a code editor is like a really souped-up word processor for programmers, where the reason it's souped up is code has a lot of structure.

00:09:57 Speaker_02
And so the quote-unquote word processor, the code editor, can actually do a lot for you that word processors in the writing space haven't been able to do for people editing text there.

00:10:08 Speaker_02
And so that's everything from giving you visual differentiation of the actual tokens in the code so you can scan it quickly, to letting you navigate around the code base, sort of like you're navigating around the internet with hyperlinks, you're going to definitions of things you're using, to error checking to catch rudimentary bugs.

00:10:28 Speaker_02
And so traditionally, that's what a code editor has meant. And I think that what a code editor is, is going to change a lot over the next 10 years, as what it means to build software maybe starts to look a bit different.

00:10:43 Speaker_02
I think also a code editor should just be fun.

00:10:46 Speaker_03
Yes, that is very important. That is very important. And it's actually sort of an underrated aspect of how we decide what to build.

00:10:53 Speaker_03
Like a lot of the things that we build and then we try them out, we do an experiment and then we actually throw them out because they're not fun. And so a big part of being fun is like being fast a lot of the time. Fast is fun.

00:11:09 Speaker_05
Yeah, that should be a t-shirt.

00:11:14 Speaker_02
Like fundamentally, I think one of the things that draws a lot of people to building stuff on computers is this like insane iteration speed where, you know, in other disciplines you might be sort of gatecapped by resources or the ability, even the ability, you know, to get a large group together and coding is this like amazing thing where it's you and the computer and that alone, you can build really cool stuff really quickly.

00:11:36 Speaker_05
So for people who don't know, Cursor is this super cool new editor that's a fork of VS Code. It would be interesting to get your kind of explanation of your own journey of editors. I think all of you were big fans of VS Code with Copilot.

00:11:54 Speaker_05
How did you arrive to VS Code and how did that lead to your journey with Cursor?

00:11:59 Speaker_01
Yeah, so, I think a lot of us, well, all of us were originally Vim users.

00:12:06 Speaker_04
Pure Vim.

00:12:07 Speaker_01
Pure Vim, yeah. No NeoVim, just pure Vim and a terminal. And at least for myself, it was around the time that CodePilot came out, so 2021. that I really wanted to try it.

00:12:21 Speaker_01
So I went into VS Code, the only platform, the only coder in which it was available. And even though I really enjoyed using Vim, just the experience of Copilot with VS Code was more than good enough to convince me to switch.

00:12:37 Speaker_01
And so that kind of was the default until we started working on Cursor.

00:12:41 Speaker_05
And maybe we should explain what Copilot does. It's like a really nice auto-complete. It suggests, as you start writing a thing, it suggests one or two or three lines how to complete the thing.

00:12:52 Speaker_05
And there's a fun experience in that, you know like when you have a close friendship and your friend completes your sentences? Like when it's done well, there's an intimate feeling.

00:13:02 Speaker_05
There's probably a better word than intimate, but there's a cool feeling of like, holy shit, it gets me. And then there's an unpleasant feeling when it doesn't get you.

00:13:14 Speaker_05
And so there's that kind of friction, but I would say for a lot of people, the feeling that it gets me overpowers that it doesn't.

00:13:21 Speaker_03
And I think actually one of the underrated aspects of GitHub Copilot is that even when it's wrong, it's like a little bit annoying, but it's not that bad because you just type another character and then maybe then it gets you or you type another character and then it gets you.

00:13:34 Speaker_03
So even when it's wrong, it's not that bad.

00:13:35 Speaker_04
Yeah, you can sort of iterate and fix it. I mean, the other underrated part of Copilot for me sort of was just the first real AI product. So the first language model consumer product

00:13:47 Speaker_05
So Copilot was kind of like the first killer app for LLMs. Yeah. And like the beta was out in 2021. Right.

00:13:56 Speaker_02
Okay. So what's the origin story of Cursor? So around 2020, the scaling loss papers came out from OpenAI.

00:14:05 Speaker_02
And that was a moment where this looked like clear, predictable progress for the field, where even if we didn't have any more ideas, it looked like you could make these models a lot better if you had more compute and more data.

00:14:16 Speaker_05
By the way, we'll probably talk for three to four hours on the topic of scaling laws.

00:14:22 Speaker_05
Just to summarize, it's a paper and a set of papers and a set of ideas that say bigger might be better for model size and data size in the realm of machine learning.

00:14:32 Speaker_04
It's bigger and better, but predictably better. Okay.

00:14:35 Speaker_05
There's another topic of conversation.

00:14:36 Speaker_02
Yeah. So around that time for some of us, there were like a lot of conceptual conversations about what's this going to look like.

00:14:43 Speaker_02
What's the story going to be for all these different knowledge worker fields about how they're going to be made better by this technology getting better.

00:14:51 Speaker_02
And then I think there were a couple of moments where like the theoretical gains predicted in that paper started to feel really concrete and it started to feel like a moment where you could actually go and not do a PhD if you wanted to work on, do useful work in AI.

00:15:06 Speaker_02
Actually felt like now there was this whole set of systems one could build that were really useful.

00:15:11 Speaker_02
And I think that the first moment we already talked about a little bit, which was playing with the early beta of Copilot, like that was awesome and magical. I think that the next big moment where everything kind of clicked together

00:15:21 Speaker_02
was actually getting early access to GPT-4. So it was sort of end of 2022 was when we were tinkering with that model. And the step-up in capabilities felt enormous. And previous to that, we had been working on a couple of different projects. We had been

00:15:37 Speaker_02
because of Copilot, because of ScalingOz, because of our prior interest in the technology, we had been tinkering around with tools for programmers, but things that are like very specific.

00:15:46 Speaker_02
So, you know, we were building tools for financial professionals who have to work within a Jupyter Notebook or like, you know, playing around with, can you do static analysis with these models?

00:15:55 Speaker_02
And then the step up in GPT-4 felt like, look, that really made concrete the theoretical gains that we had predicted before. felt like you could build a lot more just immediately at that point in time.

00:16:07 Speaker_02
And also, if we were being consistent, it really felt like this wasn't just going to be a point solution thing. This was going to be all of programming was going to flow through these models.

00:16:17 Speaker_02
It felt like that demanded a different type of programming environment, a different type of programming. And so we set off to build that sort of larger vision around that.

00:16:26 Speaker_04
There's one that I distinctly remember. So my roommate is an IML gold winner, and there's a competition in the US called the Putnam, which is sort of the IML for college people, and it's this math competition. He's exceptionally good.

00:16:40 Speaker_04
So Shengtong and Aman, I remember it's sort of June of 2022. had this bet on whether 2024, June or July, you were going to win a gold medal in the IMO with models.

00:16:57 Speaker_05
IMO is the International Math Olympiad.

00:16:59 Speaker_04
Yeah, I was International Math Olympiad. And so Arvid and I are both sort of, you know, also competed in it. So it was sort of personal. And I remember thinking, Matt, this is just, this is not going to happen.

00:17:13 Speaker_04
This was like, even though I sort of believed in progress, I thought, you know, I'm a girl just, like Aman is just delusional. And to be honest, I mean, I was, to be clear, very wrong, but that was maybe the most prescient bet in the group.

00:17:31 Speaker_05
So the new results from DeepMind, it turned out that you were correct.

00:17:36 Speaker_03
That's what the... Well, it was technically not.

00:17:38 Speaker_02
Technically incorrect, but one point away. Amon was very enthusiastic about this stuff. And before, Amon had this like scaling loss t-shirt that he would walk around with, where it had the like... charts and like the formulas on it.

00:17:51 Speaker_05
So you like felt the AGI or you felt the scaling laws?

00:17:55 Speaker_01
Yeah, I distinctly remember there's this one conversation I had with Michael where before I hadn't thought super deeply and critically about scaling laws. And he kind of posed the question, why isn't scaling all you need?

00:18:08 Speaker_01
Or why isn't scaling going to result in massive gains in progress? and I think I went through the stages of grief. There is anger, denial, and then finally at the end, just thinking about it, acceptance.

00:18:22 Speaker_01
And I think I've been quite hopeful and optimistic about progress since. I think one thing I'll caveat is I think it also depends on which domains you're going to see progress.

00:18:35 Speaker_01
Math is a great domain, especially for theorem proving, because you get this fantastic signal of actually verifying if the thing was correct. And so this means something like RL can work really, really well.

00:18:47 Speaker_01
And I think you could have systems that are perhaps very superhuman at math and still not technically have AGI.

00:18:54 Speaker_05
Okay, so can we take it all the way to Cursor? And what is Cursor? It's a fork of VS Code. And VS Code is one of the most popular editors for a long time. Like everybody fell in love with it. Everybody loved Vim. I left Emacs for it. Sorry.

00:19:13 Speaker_05
So unified in some fundamental way, the developer community. And then you look at the space of things, you look at the scaling laws, AI is becoming amazing.

00:19:24 Speaker_05
and you decided, okay, it's not enough to just write an extension for your VS Code, because there's a lot of limitations to that.

00:19:31 Speaker_05
If AI's gonna keep getting better and better and better, we need to really rethink how the AI's gonna be part of the editing process. And so you decided to fork VS Code and start to build a lot of the amazing features we'll be able to talk about.

00:19:48 Speaker_05
But what was that decision like? Because there's a lot of extensions, including Copilot, of VS Code that are doing sort of AI type stuff. What was the decision like to just fork VS Code?

00:19:59 Speaker_02
So the decision to do an editor seemed kind of self-evident to us for at least what we wanted to do and achieve.

00:20:06 Speaker_02
Because when we started working on the editor, the idea was these models are going to get much better, their capabilities are going to improve, and it's going to entirely change how you build software.

00:20:14 Speaker_02
Both in a, you will have big productivity gains, but also radical in how like the act of building software is going to change a lot.

00:20:20 Speaker_02
And so you're very limited in the control you have over a code editor if you're a plugin to an existing coding environment. And we didn't want to get locked in by those limitations. We wanted to be able to just build the most useful stuff.

00:20:34 Speaker_05
Okay, well then the natural question is, you know, VS Code is kind of with Copilot a competitor. So how do you win? Is it basically just the speed and the quality of the features?

00:20:46 Speaker_01
Yeah, I mean, I think this is a space that is quite interesting, perhaps quite unique, where if you look at previous tech waves, maybe there's kind of one major thing that happened and it unlocked a new wave of companies.

00:21:00 Speaker_01
But every single year, every single model capability, or jump you get in model capabilities, you now unlock this new wave of features, things that are possible, especially in programming.

00:21:13 Speaker_01
And so I think in AI programming, being even just a few months ahead, let alone a year ahead, makes your product much, much, much more useful. I think the cursor a year from now will need to make the cursor of today look obsolete.

00:21:28 Speaker_01
And I think, you know, Microsoft has done a number of like fantastic things, but I don't think they're in a great place to really keep innovating and pushing on this in the way that a startup can. Just rapidly implementing features.

00:21:42 Speaker_01
And kind of doing the research experimentation necessary to really push the ceiling.

00:21:50 Speaker_04
I don't know if I think of it in terms of features as I think of it in terms of capabilities for programmers.

00:21:56 Speaker_04
It's that as the new O1 model came out, and I'm sure there are going to be more models of different types, like longer context and maybe faster. There's all these

00:22:08 Speaker_04
crazy ideas that you can try, and hopefully 10% of the crazy ideas will make it into something kind of cool and useful. And we want people to have that sooner. To rephrase, it's like an underrated fact is we're making it for ourself.

00:22:25 Speaker_04
When we started Cursor, you really felt this frustration that, you know, models, you could see models getting better. But the Cobalt experience had not changed. It was like, man, these guys, the ceiling is getting higher.

00:22:39 Speaker_04
Why are they not making new things? They should be making new things. Where's all the alpha features? There were no alpha features. It was like...

00:22:49 Speaker_04
I'm sure it was selling well, I'm sure it was a great business, but it didn't feel, I'm one of these people that really want to try and use new things, and it was just, there's no new thing for like a very long while.

00:23:01 Speaker_05
Yeah, it's interesting. I don't know how you put that into words, but when you compare Cursor with Copilot, Copilot pretty quickly became, started to feel stale for some reason.

00:23:12 Speaker_03
Yeah, I think one thing that I think helps us is that we're sort of doing it all in one, where we're developing the UX and the way you interact with the model at the same time as we're developing how we actually make the model give better answers.

00:23:29 Speaker_03
how you build up the prompt, or how do you find the context, and for a cursor tab, how do you train the model? So I think that helps us to have all of it, sort of like the same people working on the entire experience end-to-end.

00:23:43 Speaker_04
Yeah, it's like the person making the UI and the person training the model like sit to like 18 feet away.

00:23:50 Speaker_01
Often the same person even.

00:23:52 Speaker_04
Yeah, often even the same person. You can create things that are sort of not possible if you're not talking, you're not experimenting.

00:24:00 Speaker_05
And you're using, like you said, Cursor to write Cursor. Of course. Oh, yeah. Well, let's talk about some of these features. Let's talk about the all-knowing, the all-powerful Praise B to the tab. You know, autocomplete on steroids. basically.

00:24:17 Speaker_05
So how does tab work?

00:24:18 Speaker_02
What is tab? To highlight and summarize at a high level, I'd say that there are two things that Cursor is pretty good at right now. There are other things that it does, but two things that it helps programmers with.

00:24:31 Speaker_02
One is this idea of looking over your shoulder and being like a really fast colleague who can kind of jump ahead of you and type and figure out what you're gonna do next.

00:24:41 Speaker_02
And that was the original idea behind, that was kind of the kernel of the idea behind a good autocomplete was predicting what you're gonna do next.

00:24:49 Speaker_02
But you can make that concept even more ambitious by not just predicting the characters after your cursor, but actually predicting the next entire change you're gonna make, the next diff, the next place you're gonna jump to.

00:25:01 Speaker_02
And the second thing Cursor is pretty good at right now, too, is helping you sometimes jump ahead of the AI and tell it what to do and go from instructions to code.

00:25:13 Speaker_02
And on both of those, we've done a lot of work on making the editing experience for those things ergonomic and also making those things smart and fast.

00:25:20 Speaker_04
One of the things we really wanted was we wanted the model to be able to edit code for us. That was kind of a wish, and we had multiple attempts at it before we had a good model that could edit code for you.

00:25:34 Speaker_04
Then after we had a good model, I think there'd been a lot of effort to make the inference fast for having a good experience. And we've been starting to incorporate, I mean, Michael sort of mentioned this, like, ability to jump to different places.

00:25:52 Speaker_04
And that jump to different places, I think, came from a feeling of, you know, once you accept an edit, it's like, man, it should be just really obvious where to go next.

00:26:04 Speaker_04
It's like, I'd made this change, the model should just know that, like, the next place to go to is, like, 18 lines down. Like, if you're a WIM user, you could press 1-8-JJ or whatever. But like, why am I doing this? The model should just know it.

00:26:21 Speaker_04
So the idea was, you just press tab, it would go 18 lines down and then show you the next edit and you would press tab. So as long as you could keep pressing tab. And so the internal competition was, how many tabs can we make someone press?

00:26:34 Speaker_04
Once you have the idea, more sort of Abstractly, the thing to think about is sort of like, how are the edits sort of zero entropy?

00:26:45 Speaker_04
So once you've sort of expressed your intent and the edit is, there's no new bits of information to finish your thought, but you still have to type some characters to make the computer understand what you're actually thinking.

00:26:59 Speaker_04
Then maybe the model should just sort of read your mind and all the zero entropy bits should just be tabbed away.

00:27:07 Speaker_01
There's this interesting thing where if you look at language model loss on different domains, I believe the bits per byte, which is kind of character normalized loss for code is lower than language, which means in general, there are a lot of tokens in code that are super predictable.

00:27:24 Speaker_01
A lot of characters that are super predictable. And this is, I think, even magnified when you're not just trying to auto-complete code, but predicting what the user's going to do next in their editing of existing code.

00:27:37 Speaker_01
And so, you know, the goal of CursorTap is let's eliminate all the low-entropy actions you take inside of the editor. When the intent is effectively determined, let's just jump you forward in time, skip you forward.

00:27:48 Speaker_05
Well, what's the intuition and what's the technical details of how to do next cursor prediction? That's not so intuitive, I think, to people.

00:27:57 Speaker_01
Yeah. I think I can speak to a few of the details on how to make these things work. They're incredibly low latency, so you need to train small models on this task. In particular, they're incredibly pre-fill token hungry.

00:28:14 Speaker_01
What that means is they have these really, really long prompts where they see a lot of your code, and they're not actually generating that many tokens. And so the perfect fit for that is using a sparse model, meaning an MOE model.

00:28:27 Speaker_01
So that was kind of one breakthrough we made that substantially improved its performance at longer context. The other being, a variant of speculative decoding that we kind of built out called speculative edits.

00:28:39 Speaker_01
These are two, I think, important pieces of what make it quite high quality and very fast.

00:28:46 Speaker_05
Okay, so MOE, mixture of experts, the input is huge, the output is small.

00:28:51 Speaker_01
Yeah.

00:28:51 Speaker_05
Okay, so what else can you say about how to make, does caching play a role in this particular?

00:28:57 Speaker_01
Oh, caching plays a huge role. Because you're dealing with this many input tokens, if every single keystroke that you're typing in a given line, you had to rerun the model on all of those tokens passed in,

00:29:10 Speaker_01
you're just going to, one, significantly degrade latency. Two, you're going to kill your GPUs with load. So you need to design the actual prompts used for the models such that they're caching aware.

00:29:23 Speaker_01
And then, yeah, you need to reuse the KV cache across requests just so that you're spending less work, less compute.

00:29:31 Speaker_05
again, what are the things that tab is supposed to be able to do kind of in the near term? Just to like sort of linger on that. Generate code, like fill empty space, also edit code across multiple lines.

00:29:47 Speaker_05
And then jump to different locations inside the same file.

00:29:50 Speaker_04
And then like launch. Hopefully jump to different files also. So if you make an edit in one file and Maybe you have to go to another file to finish your thought. It should go to the second file also.

00:30:01 Speaker_03
The full generalization is like next action prediction. Sometimes you need to run a command in the terminal and it should be able to suggest the command based on the code that you wrote too. Or sometimes you actually need to

00:30:19 Speaker_03
Like it suggests something, but it's hard for you to know if it's correct because you actually need some more information to learn. Like you need to know the type to be able to verify that it's correct.

00:30:29 Speaker_03
And so maybe it should actually take you to a place that's like the definition of something and then take you back so that you have all the requisite knowledge to be able to accept the next completion.

00:30:39 Speaker_05
So providing the human the knowledge. Yes. Right, can you integrate, like, I just got to know a guy named Prime Jen, who I believe has an SS, you can order coffee via SSH.

00:30:55 Speaker_04
Oh yeah. Oh, we did that. We did that.

00:30:57 Speaker_05
So can that, also the model do that? Feed you and provide you with caffeine? Okay, so that's the general framework.

00:31:05 Speaker_02
Yeah, and the magic moment would be if, Programming is this weird discipline where sometimes the next five minutes, not always, but sometimes the next five minutes, what you're going to do is actually predictable from the stuff you've done recently.

00:31:20 Speaker_02
And so can you get to a world where that next five minutes either happens by you disengaging and it taking you through, or maybe a little bit more of just you seeing next step, what it's going to do. And you're like, okay, that's good. That's good.

00:31:31 Speaker_02
That's good. That's good. And you can just sort of tap, tap, tap through these big changes.

00:31:35 Speaker_05
As we're talking about this, I should mention that one of the really cool and noticeable things about Cursor is that there's this whole diff interface situation going on.

00:31:44 Speaker_05
So like the model suggests with the red and the green of like, here's how we're going to modify the code. And in the chat window, you can apply and it shows you the diff and you can accept the diff. So maybe can you speak to whatever direction of that?

00:31:59 Speaker_04
We'll probably have like four or five different kinds of diffs. So we have optimized the diff for the autocomplete. So that has a different diff interface than when you're reviewing larger blocks of code.

00:32:14 Speaker_04
And then we're trying to optimize another diff thing for when you're doing multiple different files. And sort of at a high level, the difference is for when you're doing autocomplete, it should be really, really fast to read.

00:32:30 Speaker_04
Actually, it should be really fast to read in all situations, but in autocomplete, it's sort of, you're really like your eyes focused in one area. You can't be in too many, like humans can't look in too many different places.

00:32:41 Speaker_05
So you're talking about on the interface side?

00:32:43 Speaker_04
On the interface side. So it currently has this box on the side. So we have the current box. And if it tries to delete code in some place and tries to add other code, it tries to show you a box on the side.

00:32:55 Speaker_04
You can maybe show it if we pull it up on cursor.com.

00:32:57 Speaker_01
This is what we're talking about.

00:32:59 Speaker_04
So that box, it was like three or four different attempts at trying to make this thing work. Where first the attempt was like this blue crossed out line.

00:33:11 Speaker_04
So before it was a box on the side, it used to show you the code to delete by showing you like Google Docs style, you would see like a line through it, then you would see the new code. And that was super distracting.

00:33:26 Speaker_04
And then we tried many different, you know, there was sort of deletions, there was trying to do red highlight. Then the next iteration of it, which is sort of funny, you would hold on Mac the option button.

00:33:40 Speaker_04
So it would sort of highlight a region of code to show you that there might be something coming. So maybe in this example, like the input and the value would all get blue. And the blue would to highlight that the AI had a suggestion for you.

00:33:57 Speaker_04
So instead of directly showing you the thing, it would show you that the AI, it would just hint that the AI had a suggestion. And if you really wanted to see it, you would hold the option button, and then you would see the new suggestion.

00:34:08 Speaker_04
Then if you release the option button, you would then see your original code.

00:34:14 Speaker_05
So that's, by the way, that's pretty nice, but you have to know to hold the option button.

00:34:18 Speaker_04
Yeah.

00:34:18 Speaker_05
By the way, I'm not a Mac user, but I got it. It's a button, I guess, you people have.

00:34:25 Speaker_04
It's, you know, it's, again, it's just, it's just not intuitive. I think that's the, that's the key thing.

00:34:30 Speaker_01
And there's a chance this, this is also not the final version of it.

00:34:33 Speaker_03
I am personally very excited for, um, making a lot of improvements in this area. We often talk about it as the verification problem, where these diffs are great for small edits.

00:34:48 Speaker_03
For large edits, or when it's multiple files or something, it's actually a little bit prohibitive to review these diffs. And so there are a couple of different ideas here. One idea that we have is, okay, parts of the diffs are important.

00:35:07 Speaker_03
They have a lot of information. And then parts of the diff are just very low entropy. They're the same thing over and over again. And so maybe you can highlight the important pieces and then gray out the not so important pieces.

00:35:21 Speaker_03
Or maybe you can have a model that looks at the diff and sees, oh, there's a likely bug here. I will mark this with a little red squiggly and say, you should probably review this part of the diff. And ideas in that vein, I think, are exciting.

00:35:37 Speaker_05
Yeah, that's a really fascinating space of like UX design engineering. So you're basically trying to guide the human programmer through all the things they need to read and nothing more, like optimally.

00:35:51 Speaker_03
Yeah, and you want an intelligent model to do it. Like currently, diff algorithms are, they're like, They're just normal algorithms. There is no intelligence.

00:36:04 Speaker_03
There's intelligence that went into designing the algorithm, but then you don't care if it's about this thing or this thing, as you want a model to do this.

00:36:13 Speaker_04
So I think the general question is, Matt, these models are going to get much smarter. As the models get much smarter, the changes they will be able to propose are much bigger.

00:36:26 Speaker_04
So as the changes gets bigger and bigger and bigger, the humans have to do more and more and more verification work. It gets more and more and more hard. You need to help them out. I don't want to spend all my time reviewing code.

00:36:41 Speaker_05
Can you say a little more across multiple files, Div?

00:36:46 Speaker_01
Yeah, I mean, so GitHub tries to solve this, right, with code review. When you're doing code review, you're reviewing multiple diffs across multiple files. But like Arvid said earlier, I think you can do much better than code review.

00:37:01 Speaker_01
You know, code review kind of sucks. Like you spend a lot of time trying to grok this code that's often quite unfamiliar to you. And it often doesn't even actually catch that many bugs.

00:37:14 Speaker_01
And I think you can significantly improve that review experience using language models, for example, using the kinds of tricks that Arp had described, maybe pointing you towards the regions that actually matter.

00:37:27 Speaker_01
I think also if the code is produced by these language models, and it's not produced by someone else, like the code review experience is designed for both the reviewer and the person that produced the code.

00:37:42 Speaker_01
In the case where the person that produced the code is a language model. you don't have to care that much about their experience.

00:37:48 Speaker_01
And you can design the entire thing around the reviewer such that the reviewer's job is as fun, as easy, as productive as possible. And I think that feels like the issue with just kind of naively trying to make these things look like code review.

00:38:05 Speaker_01
I think you can be a lot more creative and push the boundary in what's possible.

00:38:09 Speaker_03
And just one idea there is I think ordering matters. Generally, when you review a PR, you have this list of files and you're reviewing them from top to bottom.

00:38:18 Speaker_03
But actually, you actually want to understand this part first, because that came logically first. And then you want to understand the next part. And you don't want to have to figure out that yourself. You want a model to guide you through the thing.

00:38:32 Speaker_05
And is the step of creation going to be more and more in natural language is the goal versus with actual writing?

00:38:38 Speaker_03
I think sometimes. I don't think it's going to be the case that all of programming will be natural language. And the reason for that is, you know, if I'm pair programming with Swalla and Swalla is at the computer and the keyboard,

00:38:51 Speaker_03
And sometimes, if I'm driving, I want to say to Swalla, hey, implement this function, and that works. And then sometimes it's just so annoying to explain to Swalla what I want him to do.

00:39:03 Speaker_03
And so I actually take over the keyboard and I show him, I write part of the example, and then it makes sense. And that's the easiest way to communicate. And so I think that's also the case for AI.

00:39:16 Speaker_03
Sometimes the easiest way to communicate with AI will be to show an example and then it goes and does the thing everywhere else.

00:39:21 Speaker_03
Or sometimes if you're making a website, for example, the easiest way to show to the AI what you want is not to tell it what to do, but drag things around or draw things.

00:39:33 Speaker_03
Yeah, and maybe eventually we will get to brain-machine interfaces or whatever and can understand what you're thinking. And so I think natural language will have a place. I think it will definitely not be the way most people program most of the time.

00:39:46 Speaker_05
I'm really feeling the AGI with this editor. It feels like there's a lot of machine learning going on underneath. Tell me about some of the ML stuff that makes it all work.

00:39:57 Speaker_01
Well, Cursor really works via this ensemble of custom models that we've trained alongside the Frontier models that are fantastic at the reasoning intense things.

00:40:07 Speaker_01
And so Cursor tab, for example, is a great example of where you can specialize this model to be even better than even Frontier models if you look at evals on the task we set it at.

00:40:16 Speaker_01
The other domain, which it's kind of surprising that it requires custom models, but it's kind of necessary and works quite well, is in apply.

00:40:25 Speaker_01
So I think these models are like the frontier models are quite good at sketching out plans for code and generating like rough sketches of like the change, but actually Creating diffs is quite hard for frontier models, for when you're training models.

00:40:44 Speaker_01
You try to do this with Sonnet, with O1, any frontier model, and it really messes up stupid things like counting line numbers, especially in super, super large files.

00:40:56 Speaker_01
And so what we've done to alleviate this is we let the model kind of sketch out this rough code block that indicates what the change will be. And we train a model to then apply that change to the file.

00:41:08 Speaker_05
And we should say that apply is the model looks at your code. It gives you a really damn good suggestion of what new things to do. And the seemingly for humans trivial step of combining the two, you're saying is not so trivial.

00:41:25 Speaker_04
Contrary to popular perception, it is not a deterministic algorithm.

00:41:29 Speaker_01
Yeah. I think like you see shallow copies of apply elsewhere, and it just breaks like most of the time because you think you can kind of try to do some deterministic matching and then it fails, you know, at least 40% of the time.

00:41:44 Speaker_01
And that just results in a terrible product experience. I think in general, this regime of you are going to get smarter and smarter models.

00:41:55 Speaker_01
And like, so one other thing that apply lets you do is it lets you use fewer tokens with the most intelligent models. This is both expensive in terms of latency for generating all these tokens and cost.

00:42:10 Speaker_01
So you can give this very, very rough sketch and then have your model models go and implement it because it's a much easier task to implement this very, very sketched out code.

00:42:20 Speaker_01
And I think that this regime will continue where you can use smarter and smarter models to do the planning. And then maybe the implementation details can be handled by the less intelligent ones.

00:42:29 Speaker_01
Perhaps you'll have, you know, maybe a one, maybe it'll be even more capable models given an even higher level plan that is kind of recursively Applied by sonnet and then the apply model.

00:42:43 Speaker_04
Maybe we should we should talk about how to how to make it fast Yeah, I feel like yes is always an interesting detail.

00:42:47 Speaker_01
That's good.

00:42:49 Speaker_05
Yeah, how do you make it fast?

00:42:51 Speaker_01
Yeah, so one big component of making it fast is speculative edits. So speculative edits are a variant of speculative decoding, and maybe it'd be helpful to briefly describe speculative decoding.

00:43:04 Speaker_01
With speculative decoding, what you do is you can kind of take advantage of the fact that most of the time, and I'll add the caveat that it would be when you're memory bound in language model generation, if you

00:43:19 Speaker_01
process multiple tokens at once, it is faster than generating one token at a time. So this is like the same reason why if you look at tokens per second with prompt tokens versus generated tokens, it's much, much faster for prompt tokens.

00:43:35 Speaker_01
So what we do is instead of using what speculative decoding normally does, which is using a really small model to predict these draft tokens that your larger model will then go in and verify.

00:43:48 Speaker_01
With code edits, we have a very strong prior of what the existing code will look like. And that prior is literally the same exact code. So what you can do is you can just feed chunks of the original code back into the model.

00:44:00 Speaker_01
And then the model will just pretty much agree most of the time that, OK, I'm just going to spit this code back out. And so you can process all of those lines in parallel. And you just do this with sufficiently many chunks.

00:44:11 Speaker_01
And then eventually, you'll reach a point of disagreement. where the model will now predict text that is different from the ground truth original code.

00:44:19 Speaker_01
It'll generate those tokens and then we kind of will decide after enough tokens match the original code to restart speculating in chunks of code. What this actually ends up looking like is just a much faster version of normal editing code.

00:44:35 Speaker_01
So it's just like, it looks like a much faster version of the model rewriting all the code. So just, we can use the same exact interface, that we use for diffs, but it will just stream down a lot faster.

00:44:48 Speaker_04
And then the advantage is that while it's streaming, you can just also start reviewing the code before it's done. So there's no big loading screen. So maybe that is part of the advantage.

00:45:02 Speaker_05
So the human can start reading before the thing is done.

00:45:05 Speaker_04
I think the interesting riff here is something like, I feel like speculation is a fairly common idea nowadays.

00:45:11 Speaker_04
It's like not only in language models, I mean, there's obviously speculation in CPUs and there's speculation for databases and speculation all over the place.

00:45:21 Speaker_05
Let me ask the sort of the ridiculous question of which LLM is better at coding. GPT, Claude, who wins in the context of programming?

00:45:30 Speaker_05
And I'm sure the answer is much more nuanced because it sounds like every single part of this involves a different model.

00:45:38 Speaker_01
Yeah, I think there's no model that Greta dominates others, meaning it is better in all categories that we think matter. The categories being speed,

00:45:53 Speaker_01
ability to edit code, ability to process lots of code, long context, you know, a couple of other things and kind of coding capabilities. The one that I'd say right now is just kind of net best is Sonnet. I think this is a consensus opinion.

00:46:07 Speaker_01
Our one's really interesting, and it's really good at reasoning. So if you give it really hard programming interview style problems, or leetcode problems, it can do quite quite well on them.

00:46:20 Speaker_01
But it doesn't feel like it kind of understands your rough intent as well as Sonnet does. Like, if you look at a lot of the other frontier models, one qualm I have is it feels like they're not necessarily over I'm not saying they train on benchmarks.

00:46:38 Speaker_01
But they perform really well in benchmarks, relative to kind of everything that's kind of in the middle.

00:46:44 Speaker_01
So if you try it in all these benchmarks, and things that are in the distribution of the benchmarks they're evaluated on, you know, they'll do really well.

00:46:49 Speaker_01
But when you push them a little bit outside of that, Sonnet's I think the one that that kind of does best at kind of maintaining that same capability, like you kind of have the same capability in the benchmark as when you try to instruct it to do anything with coding.

00:47:03 Speaker_05
What, another ridiculous question, is the difference between the normal programming experience versus what benchmarks represent? Like where do benchmarks fall short, do you think, when we're evaluating these models?

00:47:15 Speaker_04
By the way, that's like a really, really hard, it's like critically important detail, like how different like benchmarks are versus like real coding, where real coding, it's not interview style coding, it's you're doing these,

00:47:32 Speaker_04
You know, humans are saying like half broken English sometimes. And sometimes you're saying like, oh, do what I did before. Sometimes you're saying, uh. you know, go add this thing and then do this other thing for me and then make this UI element.

00:47:48 Speaker_04
And then, you know, it's just like a lot of things are sort of context dependent.

00:47:54 Speaker_04
You really want to like understand the human and then do what the human wants, as opposed to sort of this, maybe the way to put it is sort of abstractly is the interview problems are very well specified.

00:48:08 Speaker_04
they lean a lot on specification, while the human stuff is less specified. Yeah.

00:48:16 Speaker_02
I think that this benchmark question is both complicated by what Sohla just mentioned, and then also to

00:48:24 Speaker_02
What Aman was getting into is that even if you like, you know, there's this problem of like the skew between what can you actually model in a benchmark versus real programming.

00:48:32 Speaker_02
And that can be sometimes hard to encapsulate because it's like real programming is like very messy and sometimes things aren't super well specified what's correct or what isn't.

00:48:40 Speaker_02
But then it's also doubly hard because of this public benchmark problem.

00:48:44 Speaker_02
And that's both because public benchmarks are sometimes kind of hill-climbed on, but then it's really, really hard to also get the data from the public benchmarks out of the models.

00:48:54 Speaker_02
And so, for instance, one of the most popular agent benchmarks, SuiBench, is really, really contaminated in the training data of these foundation models.

00:49:05 Speaker_02
And so if you ask these foundation models to do a suite bench problem, but you actually don't give them the context of a code base, they can like hallucinate the right file pass, they can hallucinate the right function names.

00:49:15 Speaker_02
And so it's also just the public aspect of these things is tricky.

00:49:19 Speaker_01
Yeah, like in that case, it could be trained on the literal issues or pull requests themselves. And maybe the labs will start to do a better job, or they've already done a good job at decontaminating those things.

00:49:31 Speaker_01
But they're not going to emit the actual training data of the repository itself. Like these are all like some of the most popular Python repositories, like SymPy is one example.

00:49:41 Speaker_01
I don't think they're going to handicap their models on SymPy and all these popular Python repositories in order to get true evaluation scores in these benchmarks.

00:49:50 Speaker_02
I think that given the dearths in benchmarks, there have been a few interesting crutches that places that build systems with these models or build these models actually use to get a sense of, are they going in the right direction or not?

00:50:03 Speaker_02
And in a lot of places, people will actually just have humans play with the things and give qualitative feedback on these. like one or two of the foundation model companies, they have people who, that's a big part of their role.

00:50:15 Speaker_02
And, you know, internally we also, you know, qualitatively assess these models and actually lean on that a lot in addition to like private evals that we have. It's like the vibe. The vibe, yeah. It's like the vibe.

00:50:26 Speaker_05
The vibe benchmark, human benchmark. Yeah. You pull in the humans to do a vibe check. Yeah. Okay. I mean, that's kind of what I do, like just like reading online forums and Reddit and X, just like,

00:50:40 Speaker_05
I don't know how to properly load in people's opinions, because they'll say things like, I feel like Claude or GPT's gotten dumber or something.

00:50:51 Speaker_05
They'll say, I feel like, and then I sometimes feel like that too, but I wonder if it's the model's problem or mine.

00:51:00 Speaker_01
With Claw, there's an interesting take I heard where I think AWS has different chips. And I suspect they have slightly different numerics than NVIDIA GPUs.

00:51:13 Speaker_01
And someone speculated that Claw's degraded performance had to do with maybe using the quantized version that existed on AWS Bedrock versus whatever was running on Anthropx GPUs.

00:51:27 Speaker_05
I interview a bunch of people that have conspiracy theories, so I'm glad you spoke to this conspiracy theory.

00:51:32 Speaker_04
Well, it's not like conspiracy theory as much.

00:51:35 Speaker_04
They're just, they're like, they're, you know, humans, humans are humans and there's these details and, you know, you're doing like these queasy amount of flops and, you know, chips are messy and man, you can just have bugs.

00:51:48 Speaker_04
Like bugs are, it's hard to overstate how hard bugs are to avoid.

00:51:55 Speaker_05
What's the role of a good prompt in all of this? You mentioned that benchmarks have really structured, well-formulated prompts. What should a human be doing to maximize success?

00:52:11 Speaker_05
And what's the importance of what the humans, you wrote a blog post on, you called it prompt design.

00:52:17 Speaker_03
Yeah, I think it depends on which model you're using. And all of them are slightly different, and they respond differently to different prompts.

00:52:26 Speaker_03
But I think the original GPT-4 and the original sort of pre-double models last year, they were quite sensitive to the prompts. And they also had a very small context window.

00:52:40 Speaker_03
And so we have all of these pieces of information around the code base that would maybe be relevant in the prompt. Like you have the docs, you have the files that you add, you have the conversation history.

00:52:51 Speaker_03
And then there's a problem like how do you decide what you actually put in the prompt and when you have a limited space. And even for today's models, even when you have long context, filling out the entire context window means that it's slower.

00:53:04 Speaker_03
It means that sometimes the model actually gets confused and some models get more confused than others. others. And we have this one system internally that we call preempt, which helps us with that a little bit.

00:53:16 Speaker_03
And I think it was built for the era before where we had 8,000 token context windows.

00:53:27 Speaker_03
And it's a little bit similar to when you're making a website, you wanted to work on mobile, you wanted to work on a desktop screen, and you have this dynamic information, which you don't have, for example, if you're designing a print magazine, you know exactly where you can put stuff.

00:53:48 Speaker_03
But when you have a website or when you have a prompt, you have these inputs, and then you need to format them to always work. Even if the input is really big, then you might have to cut something down.

00:53:58 Speaker_03
And so the idea was, okay, let's take some inspiration. What's the best way to design websites?

00:54:03 Speaker_03
Well, the thing that we really like is React and the declarative approach where you use JSX in JavaScript, and then you declare, this is what I want, and I think this has higher priority, or this has higher z-index than something else.

00:54:23 Speaker_03
Then you have this rendering engine. In web design, it's like Chrome. And in our case, it's a preempt renderer, which then fits everything onto the page. And as you declare it, it will decide what you want, and then it figures out what you want.

00:54:38 Speaker_03
And so we have found that to be quite helpful. And I think the role of it has sort of shifted over time, where initially it was to fit to these small context windows. Now it's really useful because it helps us with

00:54:52 Speaker_03
splitting up the data that goes into the prompt and the actual rendering of it. And so it's easier to debug because you can change the rendering of the prompt and then try it on old prompts because you have the raw data that went into their prompt.

00:55:06 Speaker_03
And then you can see, did my change actually improve it for like this entire eval set.

00:55:11 Speaker_05
So do you literally prompt with JSX?

00:55:14 Speaker_03
Yes, yes. So it kind of looks like React. There are components, like we have one component that's a file component and it takes in like the cursor, like usually there's like one line where the cursor is in your file.

00:55:27 Speaker_03
And that's like probably the most important line because that's the one you're looking at. And so then you can give priorities. So like that line has the highest priority and then you subtract one for every line that is farther away.

00:55:37 Speaker_03
And then eventually when it's rendered, it'd figure out how many lines can actually fit and it centers around that thing.

00:55:43 Speaker_01
That's amazing. And you can do other fancy things where if you have lots of code blocks from the entire code base, you could use retrieval and things like embedding and re-ranking scores to add priorities for each of these components.

00:55:57 Speaker_05
So should humans, when they ask questions, also try to use something like that? Would it be beneficial to write JSX in the problem? Or the whole idea is it should be loose and messy?

00:56:09 Speaker_03
I think our goal is kind of that you should just do whatever is the most natural thing for you. And then we, our job is to figure out how do we actually retrieve the relative thing so that your thing actually makes sense.

00:56:23 Speaker_05
Well, this is sort of the discussion I had with Arvind of Perplexity, is like his whole idea is like, you should let the person be as lazy as he wants. But like, yeah, that's a beautiful thing.

00:56:36 Speaker_05
But I feel like you're allowed to ask more of programmers, right? So like if you say, just do what you want, I mean, humans are lazy. There's a kind of tension between just being lazy versus like provide more as,

00:56:51 Speaker_05
be prompted, almost like the system pressuring you or inspiring you to be articulate, not in terms of the grammar of the sentences, but in terms of the depth of thoughts that you convey inside the prompts.

00:57:05 Speaker_01
I think even as the system gets closer to some level of perfection, Often when you ask the model for something, you just are not, not enough intent is conveyed to know what to do. And there are like a few ways to resolve that intent.

00:57:20 Speaker_01
One is the simple thing of having model just ask you, I'm not sure how to do these parts based on your query. Could you clarify that? I think the other could be maybe

00:57:34 Speaker_01
if you, there are five or six possible generations given the uncertainty present in your query so far, why don't we just actually show you all of those and let you pick them?

00:57:43 Speaker_05
How hard is it for the model to choose to talk back? Sort of versus, it's hard, it's sort of like how to deal with the uncertainty. Do I choose to ask for more information to reduce the ambiguity?

00:58:02 Speaker_04
So, I mean, one of the things we do is, it's like a recent addition, is try to suggest files that you can add. So, and while you're typing, one can guess what the uncertainty is and maybe suggest that like, you know, maybe you're writing your API,

00:58:23 Speaker_04
And we can guess using the commits that you've made previously in the same file that the client and the server is super useful.

00:58:35 Speaker_04
And there's like a hard technical problem of how do you resolve it across all commits, which files are the most important given your current prompt. And we're still sort of initial versions rolled out, and I'm sure we can make it much more accurate.

00:58:53 Speaker_04
It's very experimental. But then the idea is we show you, like, do you just want to add this file, this file, this file also to tell, you know, the model to edit those files for you?

00:59:03 Speaker_04
Because if maybe you're making the API, like, you should also edit the client and the server that is using the API and the other one resolving the API.

00:59:10 Speaker_04
So that'll be kind of cool as both there's the phase where you're writing the prompt and there's before you even click enter, maybe we can help resolve some of the uncertainty.

00:59:20 Speaker_05
To what degree do you use agentic approaches? How useful are agents?

00:59:25 Speaker_03
We think agents are really, really cool.

00:59:28 Speaker_03
Like, I think agents is like, it's like, it resembles sort of like a human, it's sort of like the thing, like you can kind of feel that it, like you're getting closer to AGI because you see a demo where it acts as a human would, and it's really, really cool.

00:59:46 Speaker_03
I think, Agents are not yet super useful for many things. I think we're getting close to where they will actually be useful. And so I think there are certain types of tasks where having an agent would be really nice. I would love to have an agent.

01:00:07 Speaker_03
For example, we have a bug where you sometimes can't Command-C and Command-V inside our chat input box. And that's a task that's super well specified. I just want to say in two sentences, this does not work, please fix it.

01:00:20 Speaker_03
And then I would love to have an agent that just goes off, does it, and then a day later I come back and I review the thing.

01:00:29 Speaker_05
You mean it goes, finds the right file?

01:00:31 Speaker_03
Yeah, it finds the right files, it tries to reproduce the bug, it fixes the bug, and then it verifies that it's correct. And this could be a process that takes a long time. And so I think I would love to have that.

01:00:43 Speaker_03
And then I think a lot of programming, there is often this belief that agents will take over all of programming.

01:00:52 Speaker_03
I don't think we think that that's the case, because a lot of programming, a lot of the value is in iterating, or you don't actually want to specify something upfront, because you don't really know what you want until you've seen an initial version, and then you want to iterate on that, and then you provide more information.

01:01:09 Speaker_03
And so for a lot of programming, I think you actually want a system that's instant, that gives you an initial version instantly back, and then you can iterate super, super quickly.

01:01:18 Speaker_05
What about something like that Rethink ML Replicant Agent that does also like setting up the development environment, installing software packages, configuring everything, configuring the databases, and actually deploying the app.

01:01:32 Speaker_05
Is that also in the set of things you dream about?

01:01:36 Speaker_03
I think so. I think that would be really cool. For certain types of programming, it would be really cool.

01:01:41 Speaker_05
Is that within scope of Cursor?

01:01:44 Speaker_03
Yeah, we aren't actively working on it right now, but it's definitely like we want to make the programmer's life easier and more fun.

01:01:54 Speaker_03
And some things are just really tedious and you need to go through a bunch of steps and you want to delegate that to an agent. And then some things, you can actually have an agent in the background while you're working.

01:02:04 Speaker_03
Like, let's say you have a PR that's both backend and frontend, and you're working in the frontend, and then you can have a background agent that does some work and figure out kind of what you're doing.

01:02:13 Speaker_03
And then when you get to the backend part of your PR, then you have some initial piece of code that you can iterate on. And so that would also be really cool.

01:02:25 Speaker_05
One of the things we already talked about is speed. But I wonder if we can just linger on that some more in the various places that the technical details involved in making this thing really fast.

01:02:38 Speaker_05
So every single aspect of a cursor, most aspects of cursor feel really fast. Like I mentioned, the apply is probably the slowest thing. And for me from, I'm sorry, the pain.

01:02:48 Speaker_03
It's a pain, it's a pain that we're feeling and we're working on fixing it.

01:02:53 Speaker_05
Yeah, I mean, it says something that feels, I don't know what it is, like one second or two seconds. That feels slow, that actually shows that everything else is just really, really fast.

01:03:06 Speaker_05
So is there some technical details about how to make some of these models, how to make the chat fast, how to make the diffs fast? Is there something that just jumps to mind?

01:03:15 Speaker_01
Yeah, I mean, so we can go over a lot of the strategies that we use. One interesting thing is cache warming. And so what you can do is if as the user is typing, you can have, you're probably going to use some piece of context.

01:03:31 Speaker_01
And you can know that before the user is done typing. So, you know, as we discussed before, Reusing the KVCache results in lower latency, lower costs, cross requests.

01:03:42 Speaker_01
So as the user starts typing, you can immediately warm the cache with like, let's say the current file contents. And then when they press enter, there's very few tokens it actually has to pre-fill and compute before starting the generation.

01:03:55 Speaker_01
This will significantly lower TTFD.

01:03:57 Speaker_05
Can you explain how KVCache works?

01:03:59 Speaker_01
Yeah, so the way transformers work... I like it. I mean, one of the mechanisms that allow transformers to not just independently, like the mechanism that allows transformers to not just independently look at each token, but see previous tokens

01:04:16 Speaker_01
are the keys and values to attention.

01:04:18 Speaker_01
And generally, the way attention works is you have your current token, some query, and then you've all the keys and values of all your previous tokens, which are some kind of representation that the model stores internally of all the previous tokens in the prompt.

01:04:34 Speaker_01
And like by default, when you're doing a chat, the model has to for every single token, do this forward pass through the entire model. That's a lot of matrix multiplies that happen. And that is really, really slow.

01:04:50 Speaker_01
Instead, if you have already done that, and you stored the keys and values, and you keep that in the GPU,

01:04:56 Speaker_01
Then when I'm, let's say I have sorted for the last N tokens, if I now want to compute the output token for the N plus one token, I don't need to pass those first N tokens through the entire model because I already have all those keys and values.

01:05:12 Speaker_01
And so you just need to do the forward pass through that last token.

01:05:15 Speaker_01
And then when you're doing attention, you're reusing those keys and values that have been computed, which is the only kind of sequential part or sequentially dependent part of the transformer.

01:05:26 Speaker_05
Is there higher level caching, like caching of the prompts or that kind of stuff that can help?

01:05:33 Speaker_01
Yeah, there's other types of caching you can kind of do. One interesting thing that you can do for CursorTab is you can basically predict ahead as if the user would have accepted the suggestion and then trigger another request.

01:05:53 Speaker_01
And so then you've cached, you've done the speculative, it's a mix of speculation and caching, right? Because you're speculating what would happen if they accepted it. And then you have this value that is cached, this suggestion.

01:06:04 Speaker_01
And then when they press tab, the next one would be waiting for them immediately. It's a kind of clever heuristic slash trick. that uses a higher level caching and can give the, it feels fast, despite there not actually being any changes in the model.

01:06:20 Speaker_04
And if you can make the KV cache smaller, one of the advantages you get is like, maybe you can speculate even more. Maybe you can guess, here's the 10 things that that could be useful.

01:06:30 Speaker_04
Like, you predict the next 10 and then it's possible the user hits one of the 10. It's a much higher chance than the user hits the exact one that you showed them. Maybe they type in another character and we sort of hit something else in the cache.

01:06:45 Speaker_04
So there's all these tricks where... The general phenomena here is... I think it's also super useful for RL. maybe a single sample from the model isn't very good.

01:06:59 Speaker_04
But if you predict like 10 different things, it turns out that one of the 10, that's right, is the probability is much higher. There's these passive key curves. And part of what RL does is

01:07:15 Speaker_04
You can exploit this passive k-phenomena to make many different predictions. One way to think about this, the model sort of knows internally, has some uncertainty over which of the k-things is correct, or which of the k-things does the human want.

01:07:31 Speaker_04
When we RL our cursor tab model, one of the things we're doing is we're predicting Which of the hundred different suggestions the model produces is more amenable for humans? Which of them do humans more like than other things?

01:07:50 Speaker_04
Maybe there's something where the model can predict very far ahead versus a little bit and maybe somewhere in the middle.

01:07:59 Speaker_04
And then you can give a reward to the things that humans would like more, and punish the things that it won't like, and then train the model to output the suggestions that humans would like more.

01:08:07 Speaker_04
You have these RL loops that are very useful, that exploit these Basset k-curves. Oman maybe can go into even more detail.

01:08:15 Speaker_01
Yeah, it's a little it is a little different than speed. But I mean, like, technically, you tie it back in because you can get away with the smaller model if you are all your smaller model, and it gets the same performances as the bigger one.

01:08:29 Speaker_01
That's like, and while I was mentioning stuff about about reducing the size of your KB cache. There are other techniques there as well that are really helpful for speed.

01:08:39 Speaker_01
So kind of back in the day, like all the way two years ago, people mainly use multi-head attention. And I think there's been a migration towards more efficient attention schemes, like group query or multi-query attention.

01:08:55 Speaker_01
And this is really helpful for then with larger batch sizes, being able to generate the tokens much faster. the interesting thing here is this now has no effect on that time to first token pre fill speed.

01:09:11 Speaker_01
The thing this matters for is now generating tokens. And why is that? Because when you're generating tokens, instead of being bottlenecked by doing these super parallelizable matrix multiplies across all your tokens.

01:09:25 Speaker_01
You're bottlenecked by how quickly, it's for long context with large batch sizes, by how quickly you can read those cache keys and values. And so then that's memory bandwidth and how can we make this faster?

01:09:38 Speaker_01
We can try to compress the size of these keys and values. So multi-query attention is the most aggressive of these. where normally with multi-head attention you have some number of quote-unquote attention heads and some number of kind of query heads.

01:09:55 Speaker_01
Multi-query just preserves the query heads, gets rid of all the key value heads. So there's only one kind of key value head and there's all the remaining query heads. With group query, you instead preserve all the query heads.

01:10:13 Speaker_01
And then your keys and values are kind of, there are fewer heads for the keys and values, but you're not reducing it to just one. But anyways, like the whole point here is you're just reducing the size of your KB cache.

01:10:26 Speaker_03
And then there is MLA.

01:10:28 Speaker_01
Yeah, multilatent. That's a little more complicated. And the way that this works is it kind of turns the entirety of your keys and values across all your heads into this kind of one latent vector that is then kind of expanded inference time.

01:10:46 Speaker_04
But MLA is from this company called DeepSeek. It's quite an interesting algorithm. Maybe the key idea is sort of in both MQA and in other places, what you're doing is you're sort of reducing the number of KV heads. The advantage you get from that is

01:11:07 Speaker_04
is, you know, there's less of them, but maybe the theory is that you actually want a lot of different, like you want each of the keys and values to actually be different. So one way to reduce the size is you keep one big shared vector for

01:11:25 Speaker_04
for all the keys and values, and then you have smaller vectors for every single token, so that you can store only the smaller thing, as some sort of low-rank reduction.

01:11:36 Speaker_04
And the low-rank reduction, at the end of the time, when you eventually want to compute the final thing, remember that you're memory-bound, which means that you still have some compute left that you can use for these things. So if you can expand the

01:11:50 Speaker_04
the latent vector back out. And somehow this is far more efficient because you're reducing, for example, maybe you're reducing 32 or something, the size of the vector that you're keeping.

01:12:04 Speaker_01
Yeah, there's perhaps some richness in having a separate set of keys and values and query that pairwise match up versus compressing that all into one. In that interaction, at least.

01:12:17 Speaker_05
Okay, and all of that is dealing with being memory-bound. Yeah. what, I mean, ultimately, how does that map to the user experience?

01:12:27 Speaker_01
Trying to get the... Yeah, the two things that it maps to is you can now make your cache a lot larger, because you've less space allocated for the KV cache, you can maybe cache a lot more aggressively and a lot more things.

01:12:38 Speaker_01
So you get more cache hits, which are helpful for reducing the time to first token for the reasons that were kind of described earlier. And then the second being when you

01:12:48 Speaker_01
start doing inference with more and more requests and larger and larger batch sizes, you don't see much of a slowdown as it's generating the tokens, the speed of that.

01:12:58 Speaker_04
would it also allow you to make your prompt bigger for certain?

01:13:00 Speaker_01
Yeah, yeah. So like the basic, the size of your KVCache is both the size of all your prompts multiplied by the number of prompts being processed in parallel. So you could increase either of those dimensions, right?

01:13:12 Speaker_01
The batch size or the size of your prompts without degrading the latency of generating tokens.

01:13:18 Speaker_05
Arvid, you wrote a blog post, Shadow of a Workspace, iterating on code in the background. So what's going on?

01:13:24 Speaker_03
So to be clear, we want there to be a lot of stuff happening in the background, and we're experimenting with a lot of things.

01:13:32 Speaker_03
Right now, we don't have much of that happening, other than the cache warming or figuring out the right context that goes into your command key prompts, for example.

01:13:42 Speaker_03
But the idea is, if you can actually spend computation in the background, then you can help help the user maybe at a slightly longer time horizon than just predicting the next few lines that you're going to make.

01:13:57 Speaker_03
But actually, in the next 10 minutes, what are you going to make? And by doing it in the background, you can spend more computation doing that.

01:14:05 Speaker_03
And so the idea of the shadow workspace that we implemented, and we use it internally for experiments, is that to actually get advantage of doing stuff in the background, you want some kind of feedback signal to give back to the model.

01:14:21 Speaker_03
Because otherwise, you can get higher performance by just letting the model think for longer. And so like O1 is a good example of that. But another way you can improve performance is by letting the model iterate and get feedback.

01:14:34 Speaker_03
And so one very important piece of feedback when you're a programmer is the language server, which is this thing. It exists for most different languages, and there's like a separate language server per language.

01:14:49 Speaker_03
And it can tell you, you know, you're using the wrong type here, and then gives you an error. Or it can allow you to go to definition and sort of understands the structure of your code.

01:14:58 Speaker_03
So language servers are extensions developed by, like there's a TypeScript language server developed by the TypeScript people, a Rust language server developed by the Rust people, and then they all interface over the language server protocol to VS Code.

01:15:09 Speaker_03
So that VS Code doesn't need to have all of the different languages built into VS Code, but rather you can use the existing compiler infrastructure.

01:15:17 Speaker_05
For linting purposes?

01:15:19 Speaker_03
It's for linting, it's for going to definition, and for like seeing the right types that you're using.

01:15:25 Speaker_05
So it's doing like type checking also?

01:15:27 Speaker_03
Yes, type checking and going to references. And that's like, when you're working in a big project, you kind of need that. If you don't have that, it's like really hard to code in a big project.

01:15:39 Speaker_05
Can you say again how that's being used inside Cursor, the language server protocol communication thing?

01:15:46 Speaker_03
So it's being used in Cursor to show to the programmer, just like in VS Code. But then the idea is you want to show that same information to the models, the IAM models.

01:15:56 Speaker_03
And you want to do that in a way that doesn't affect the user, because you want to do it in background. And so the idea behind this shadow workspace was, okay, like one way we can do this is we spawn a separate window of cursor that's hidden.

01:16:12 Speaker_03
So you can set this flag and an electron is hidden. There is a window, but you don't actually see it.

01:16:17 Speaker_03
Inside of this window, the AI agents can modify code however they want, as long as they don't save it because it's still the same folder, and then can get feedback from the linters and go to definition and iterate on their code.

01:16:30 Speaker_05
So like literally run everything in the background, like as if, right?

01:16:34 Speaker_03
Yeah.

01:16:35 Speaker_05
Maybe even run the code?

01:16:36 Speaker_03
So that's the eventual version. Okay. That's what you want. And a lot of the blog post is actually about how do you make that happen? Because it's a little bit tricky.

01:16:47 Speaker_03
You want it to be on the user's machine so that it exactly mirrors the user's environment. And then on Linux, you can do this cool thing where you can actually mirror the file system and have the

01:16:59 Speaker_03
AI makes changes to the files and it thinks that it's operating on the file level, but actually that's stored in memory and you can create this kernel extension to make it work.

01:17:13 Speaker_03
Whereas on Mac and Windows, it's a little bit more difficult, but it's a fun technical problem. So that's why

01:17:23 Speaker_01
One maybe hacky, but interesting idea that I like is holding a lock on saving. And so basically you can then have the language model kind of hold the lock on saving to disk.

01:17:33 Speaker_01
And then instead of you operating in the ground truth version of the files that are saved to disk, you actually are operating in what was the shadow workspace before and these unsaved things that only exist in memory that you still get linter errors for.

01:17:44 Speaker_01
code in and then when you try to maybe run code it's just like there's a small warning that there's a lock and then you kind of will take back the lock from the language server if you're trying to do things concurrently or from the shadow workspace if you're trying to do things concurrently that's such an exciting future by the way it's a bit of a tangent but like to allow a model to change files

01:18:04 Speaker_05
It's scary for people, but like, it's really cool to be able to just like let the agent do a set of tasks and you come back the next day and kind of observe like it's a colleague or something like that.

01:18:18 Speaker_01
Yeah. And I think there may be different versions of like run ability where

01:18:22 Speaker_01
For the simple things where you're doing things in the span of a few minutes on behalf of the user as they're programming, it makes sense to make something work locally in their machine.

01:18:31 Speaker_01
I think for the more aggressive things where you're making larger changes that take longer periods of time, you'll probably want to do this in some sandbox remote environment. And that's another incredibly tricky problem of how do you

01:18:44 Speaker_01
exactly reproduce, or mostly reproduce, to the point of it being effectively equivalent for running code, the user's environment with this remote sandbox.

01:18:53 Speaker_04
I'm curious what kind of agency you want for coding. Do you want them to find bugs? Do you want them to implement new features? What agency do you want?

01:19:03 Speaker_05
So by the way, when I think about agents, I don't think just about coding. I think, so for this particular podcast, there's video editing, and a lot of, if you look in Adobe, there's code behind. It's very poorly documented code, but you can

01:19:19 Speaker_05
interact with Premiere, for example, using code. And basically all the uploading, everything I do on YouTube, everything as you could probably imagine, I do all of that through code. And including translation, overdubbing, all of this.

01:19:33 Speaker_05
So I envision all those kinds of tasks. So automating many of the tasks that don't have to do directly with the editing. So that, okay, that's what I was thinking about.

01:19:43 Speaker_05
But in terms of coding, I would be fundamentally thinking about bug finding, like many levels of kind of bug finding, and also bug finding like logical bugs, not logical, like spiritual bugs or something.

01:20:00 Speaker_05
One's like sort of big directions of implementation, that kind of stuff.

01:20:05 Speaker_04
Let's opine on bug finding.

01:20:06 Speaker_01
Yeah. I mean, it's really interesting that these models are so bad at bug finding when just naively prompted to find a bug. They're incredibly poorly calibrated.

01:20:17 Speaker_03
Even the smartest models.

01:20:19 Speaker_01
Exactly. Even O1. How do you explain that?

01:20:21 Speaker_05
Is there a good intuition?

01:20:25 Speaker_01
I think these models are really strong reflection of the pre training distribution. And, you know, I do think they generalize as the loss gets lower and lower.

01:20:34 Speaker_01
But I don't think the loss in the scale is quite the loss is low enough, such that they're like really fully generalizing and code like the things that we use these things for the frontier models.

01:20:46 Speaker_01
that they're quite good at are really code generation and question answering.

01:20:51 Speaker_01
And these things exist in massive quantities in pre-training with all of the code on GitHub on the scale of many, many trillions of tokens and questions and answers on things like Stack Overflow and maybe GitHub issues.

01:21:05 Speaker_01
And so when you try to push into these things that really don't exist, very much online, like, for example, the cursor tab objective of predicting the next edit, given the edits done so far, the brittleness kind of shows.

01:21:20 Speaker_01
And then bug detection is another great example where there aren't really that many examples of like actually detecting real bugs and then proposing fixes. And the models just really struggle at it.

01:21:32 Speaker_01
But I think it's a question of transferring the model.

01:21:35 Speaker_01
In the same way that you get this fantastic transfer from pre-trained models just on code in general to the cursor tab objective, you'll see a very, very similar thing with generalized models that are really good at code to bug detection.

01:21:48 Speaker_01
It just takes a little bit of nudging in that direction.

01:21:51 Speaker_04
To be clear, I think they sort of understand code really well. While they're being pre-trained, the representation that's being built up, almost certainly, somewhere in the stream, the model knows that maybe there's something sketchy going on.

01:22:08 Speaker_04
It sort of has some sketchiness, but actually eliciting the sketchiness to Part of it is that humans are really calibrated on which bugs are really important. It's not just actually saying there's something sketchy. It's like, is this sketchy trivial?

01:22:26 Speaker_04
Is this sketchy like you're going to take the server down? Part of it is maybe the cultural knowledge of Why is a staff engineer a staff engineer?

01:22:35 Speaker_04
A staff engineer is good because they know that three years ago, someone wrote a really sketchy piece of code that took the server down. And as opposed to maybe just like, you know, you just this thing is like an experiment. So a few bugs are fine.

01:22:53 Speaker_04
You're just trying to experiment and get the feel of the thing. And so if the model gets really annoying when you're writing an experiment, that's really bad. But if you're writing something for super production, you're writing a database, right?

01:23:04 Speaker_04
You're writing code in Postgres or Linux or whatever, like you're Linus Torvalds. It's sort of unacceptable to have even an edge case. And just having the calibration of

01:23:15 Speaker_01
how paranoid is the user? But even then, if you're putting in a maximum paranoia, it still just doesn't quite get it.

01:23:23 Speaker_04
Yeah, yeah, yeah.

01:23:25 Speaker_05
I mean, but this is hard for humans, too, to understand which line of code is important and which is not.

01:23:31 Speaker_05
I think one of your principles on a website says if a code can do a lot of damage, one should add a comment that say this line of code is dangerous.

01:23:43 Speaker_03
And all caps, repeat it 10 times.

01:23:47 Speaker_05
No, you say like for every single line of code inside the function, you have to, and that's quite profound.

01:23:54 Speaker_05
That says something about human beings because the engineers move on, even the same person might just forget how it can sync the Titanic, a single function.

01:24:04 Speaker_05
Like you don't, you might not intuit that quite clearly by looking at the single piece of code.

01:24:09 Speaker_03
Yeah, and I think that one is also partially also for today's AI models, where if you actually write dangerous, dangerous, dangerous in every single line, the models will pay more attention to that and will be more likely to find bugs in that region.

01:24:26 Speaker_05
That's actually just straight up a really good practice of labeling code of how much damage this can do.

01:24:34 Speaker_03
Yeah, I mean, it's controversial. Some people think it's ugly.

01:24:38 Speaker_04
Well, I actually think it's like, in fact, I actually think this is one of the things I learned from Arvid is, you know, like I sort of aesthetically, I don't like it, but I think there's certainly something where like it's useful for the models and humans just forget a lot.

01:24:54 Speaker_04
And it's really easy to make a small mistake and cause like bring down, you know, like just bring down the server and like, like, of course, we like test a lot and whatever. But there's always these things that you have to be very careful.

01:25:08 Speaker_01
Yeah, like with just normal doc strings, I think people will often just skim it when making a change and think, Oh, I know how to do this. And you kind of really need to point it out to them so that that doesn't slip through.

01:25:22 Speaker_05
Yeah, you have to be reminded that you can do a lot of damage. That's like, we don't really think about that. You think about, okay, how do I figure out how this works so I can improve it? You don't think about the other direction.

01:25:35 Speaker_03
Until we have formal verification for everything, then you can do whatever you want and you know for certain that you have not introduced a bug if the proof pass.

01:25:45 Speaker_01
But concretely, what do you think that future would look like?

01:25:48 Speaker_03
I think people will just not write tests anymore and the model will suggest, like you write a function, the model will suggest a spec and you review the spec.

01:26:00 Speaker_03
And in the meantime, a smart reasoning model computes a proof that the implementation follows the spec. And I think that happens for most functions.

01:26:10 Speaker_02
Don't you think this gets at a little bit some of the stuff you were talking about earlier with the difficulty of specifying intent for what you want with software?

01:26:18 Speaker_02
Where sometimes it might be because the intent is really hard to specify, it's also then going to be really hard to prove that it's actually matching whatever your intent is.

01:26:24 Speaker_03
Like you think that spec is hard to generate?

01:26:28 Speaker_02
Yeah, or just like, for a given spec, maybe you can, I think there is a question of like, can you actually do the formal verification? Like, that's like, is that possible? I think that there's like more to dig into there.

01:26:41 Speaker_03
But then also, even if you have the spec, But how do you have the spec?

01:26:46 Speaker_02
Is the spec written in natural language?

01:26:47 Speaker_03
Yeah, how do you map the spec? The spec would be formal.

01:26:51 Speaker_02
But how easy would that be to draw? So then I think that you care about things that are not going to be easily well specified in the spec language. I see, I see, yeah. Maybe an argument against formal verification is all you need.

01:27:02 Speaker_01
Yeah, the worry is there's this massive document. Replacing something like unit tests, sure.

01:27:07 Speaker_03
Yeah, yeah. I think you can probably also evolve the spec languages to capture some of the things that they don't really capture right now. But I don't know, I think it's very exciting.

01:27:21 Speaker_05
And you're speaking not just about like single functions, you're speaking about entire code bases.

01:27:26 Speaker_03
I think entire code bases is harder, but that is what I would love to have. And I think it should be possible. And because you can even, there's like a lot of work recently where you can prove formally verified down to the hardware.

01:27:40 Speaker_03
So you formally verify the C code, and then you formally verify through the GCC compiler, and then through the Verilog down to the hardware. And that's an incredibly big system, but it actually works.

01:27:53 Speaker_03
And I think big codebases are sort of similar in that they're a multi-layered system. And if you can decompose it and formally verify each part, then I think it should be possible.

01:28:02 Speaker_03
I think the specification problem is a real problem, but how do you handle side effects?

01:28:07 Speaker_01
Or how do you handle, I guess, external dependencies, like calling the Stripe API?

01:28:12 Speaker_04
Maybe Stripe would write a spec for their API.

01:28:15 Speaker_01
But like, you can't do this for everything. Like, can you do this for everything you use? Like, how do you, how do you do it for, if there's a language, like maybe, maybe like people will use language models as primitives in the programs they write.

01:28:25 Speaker_01
And there's like a dependence on it. And like, how, how do you now include that?

01:28:29 Speaker_03
I think you might be able to prove, prove that still.

01:28:32 Speaker_01
Prove what about language models?

01:28:33 Speaker_03
I think it feels possible that you could actually prove that a language model is aligned, for example. Or like you can prove that it actually gives the right answer. That's the dream.

01:28:47 Speaker_05
Yeah, that is. I mean, if it's possible, that's your, I have a dream speech. If it's possible, that will certainly help with, you know, making sure your code doesn't have bugs and making sure AI doesn't destroy all of human civilization.

01:29:01 Speaker_05
So the full spectrum of AI safety to just bug finding. So you said the models struggle with bug finding. What's the hope?

01:29:10 Speaker_04
You know, my hope initially is, and I can let Michael chime in too, but it was like, it should first help with the stupid bugs. It should very quickly catch the stupid bugs.

01:29:23 Speaker_04
Off by one error is like, sometimes you write something in a comment and do it the other way. It's very common. I do this. I write less than in a comment and I maybe write greater than. Something like that. And the model is like, it looks sketchy.

01:29:36 Speaker_04
Are you sure you want to do that? But eventually it should be able to catch harder bugs too.

01:29:42 Speaker_02
Yeah.

01:29:42 Speaker_02
And I think that it's also important to note that this is having good bug-finding models feels necessary to get to the highest reaches of having AI do more and more programming for you, where you're going to, you know, if the AI is building more and more of the system for you, you need to not just generate but also verify.

01:30:00 Speaker_02
And without that, some of the problems that we've talked about before with programming with these models will just become untenable.

01:30:08 Speaker_02
So it's not just for humans, like you write a bug, I write a bug, find the bug for me, but it's also being able to verify the AI's code and check it is really important.

01:30:18 Speaker_03
Yeah, and then how do you actually do this? Like, we've had a lot of contentious dinner discussions of how do you actually train a bug model?

01:30:24 Speaker_03
But one very popular idea is, you know, it's kind of potentially easier to introduce a bug than actually finding the bug. And so you can train a model to introduce bugs in existing code.

01:30:36 Speaker_03
And then you can train a reverse bug model then that can find bugs using this synthetic data. So that's like one example, but yeah, there are lots of ideas for how to change this.

01:30:49 Speaker_02
You can also do a bunch of work, not even at the model level, of taking the biggest models and then maybe giving them access to a lot of information that's not just the code.

01:30:58 Speaker_02
It's kind of a hard problem to stare at a file and be like, where's the bug? And that's hard for humans often, right? And so often you have to run the code and being able to see things like traces and step through a debugger.

01:31:09 Speaker_02
There's a whole other direction where it kind of tends toward that. And it could also be that there are kind of two different product form factors here.

01:31:14 Speaker_02
It could be that you have a really specialty model that's quite fast, that's kind of running in the background and trying to spot bugs.

01:31:20 Speaker_02
And it might be that sometimes, sort of to Arvid's earlier example about some nefarious input box bug, it might be that sometimes you want There's, you know, there's a bug. You're not just like checking hypothesis free. You're like, this is a problem.

01:31:31 Speaker_02
I really want to solve it. And you zap that with tons and tons and tons of compute. And you're willing to put in like $50 to solve that bug or something even more.

01:31:39 Speaker_05
Have you thought about integrating money into this whole thing? Like I would pay probably a large amount of money for if you found a bug or even generated code that I really appreciated.

01:31:47 Speaker_05
Like I had a moment a few days ago when I started using cursor or generated a perfect. like perfect three functions for interacting with the YouTube API, to update captions, and for localization in different languages.

01:32:08 Speaker_05
The API documentation is not very good, and the code across, like if I Googled it for a while, I couldn't find exactly, there's a lot of confusing information, and Cursor generated it perfectly.

01:32:19 Speaker_05
I just sat back, I read the code, I was like, this is correct, I tested it, it's correct. I was like, I want a tip. on a button that goes, here's $5.

01:32:30 Speaker_05
One that's really good just to support the company and support what the interface is, and the other is that probably sends a strong signal, like, good job. There's a much stronger signal than just accepting the code.

01:32:43 Speaker_05
You just actually send a strong good job. That, and for bug finding, obviously, there's a lot of people, that would pay a huge amount of money for a bug, like a bug bounty thing. Right? Is that, do you guys think about that?

01:33:00 Speaker_03
Yeah, it's a controversial idea inside the company. I think it sort of depends on how much you believe in humanity, almost, you know?

01:33:08 Speaker_03
Like, I think it would be really cool if, like, you spend nothing to try to find a bug, and if it doesn't find a bug, you spend $0. And then if it does find a bug, and you click accept, then it also shows, like, in parentheses, like, $1.

01:33:24 Speaker_03
And so you spend $1 to accept the bug. And then, of course, there is a worry like, okay, we spent a lot of computation, maybe people will just copy-paste. I think that's a worry.

01:33:34 Speaker_03
And then there is also the worry that introducing money into the product makes it kind of... you know, like it doesn't feel as fun anymore. Like you have to like think about money and you, all you want to think about is like the code.

01:33:47 Speaker_03
And so maybe it actually makes more sense to separate it out. And like you pay some fee like every month and then you get all of these things for free.

01:33:55 Speaker_05
But there could be a tipping component, which is not like it costs.

01:33:58 Speaker_03
It still has that like dollar symbol. I think it's fine. But I also see the point where like maybe you don't want to introduce it.

01:34:06 Speaker_01
Yeah, I was gonna say the moment that feels like people do this is when they share it, when they have this fantastic example, they just kind of share it with their friends.

01:34:13 Speaker_02
There is also a potential world where there's a technical solution to this, like, honor system problem too, where if we can get to a place where we understand the output of the system more, I mean, to the stuff we were talking about with, like, you know, error checking with the LSP and then also running the code, but if you could get to a place where you could actually somehow verify, oh, I have fixed the bug, maybe then the bounty system doesn't need to rely on the honor system too.

01:34:35 Speaker_05
How much interaction is there between the terminal and the code? How much information is gained if you run the code in the terminal? Can you do a loop where it runs the code and suggests how to change the code if the code in runtime gives an error?

01:34:54 Speaker_05
Is right now they're separate worlds completely? I know you can do Control-K inside the terminal to help you write the code,

01:35:01 Speaker_01
You can use terminal context as well, inside of JackmanK, kind of everything. We don't have the looping part yet, though we suspect something like this could make a lot of sense.

01:35:13 Speaker_01
There's a question of whether it happens in the foreground too, or if it happens in the background, like what we've been discussing.

01:35:19 Speaker_05
Sure, the background is pretty cool. I could be running the code in different ways. Plus there's a database side to this, which, how do you protect it from not modifying the database? But okay.

01:35:29 Speaker_04
I mean, there's certainly cool solutions there. There's this new API that is being developed for... It's not in AWS, but, you know, it certainly is. I think it's in PlanetScale. I don't know if PlanetScale was the first one to add it.

01:35:45 Speaker_04
It's this ability to sort of add branches to a database, which is...

01:35:50 Speaker_04
Like if you're working on a feature and you want to test against a broad database, but you don't actually want to test against a broad database, you could sort of add a branch to the database.

01:35:58 Speaker_04
And the way to do that is to add a branch to the write-ahead log. And there's obviously a lot of technical complexity in doing it correctly. I guess database companies need new things to do. They have good databases now.

01:36:13 Speaker_04
And I think like TurboBuffer, which is one of the databases we use, is going to add maybe branching to the Red Hat log. And so maybe the AI agents will use branching.

01:36:29 Speaker_04
They'll like test against some branch and it's sort of gonna be a requirement for the database to like support branching or something.

01:36:36 Speaker_01
It'd be really interesting if you could branch a file system, right?

01:36:40 Speaker_04
Yeah. I feel like everything needs branching. Yeah.

01:36:43 Speaker_05
It's like the problem with the multiverse, right? Like if you branch on everything, that's like a lot.

01:36:50 Speaker_04
I mean, there's obviously these like super clever algorithms to make sure that you don't actually use a lot of space or CPU or whatever.

01:36:58 Speaker_05
Okay, this is a good place to ask about infrastructure. So you guys mostly use AWS. What are some interesting details? What are some interesting challenges? Why'd you choose AWS? Why is AWS still winning? Hashtag.

01:37:11 Speaker_03
AWS is just really, really good. It's really good. Whenever you use an AWS product, you just know that it's going to work. It might be absolute hell to go through the steps to set it up.

01:37:28 Speaker_05
Why is the interface so horrible?

01:37:30 Speaker_03
The content's so good. It's just so good. It doesn't need to be like... It's the nature of winning.

01:37:35 Speaker_04
I think it's exactly. It's just nature of winning. Yeah.

01:37:38 Speaker_03
Yeah. But AWS, you can always trust, like, it will always work. And if there is a problem, it's probably your problem. Yeah.

01:37:47 Speaker_05
Okay. Is there some interesting, like, challenges to... You guys are a pretty new startup to get scaling to, like, to so many people and...

01:37:55 Speaker_02
I think that it has been an interesting journey adding each extra zero to the request per second. You run into all of these with the general components you're using for caching and databases run into issues as you make things bigger and bigger.

01:38:10 Speaker_02
And now we're at the scale where we get int overflows on our tables and things like that.

01:38:14 Speaker_02
And then also there have been some custom systems that we've built, like, for instance, our retrieval system for computing a semantic index of your codebase and answering questions about a codebase that have continually, I feel like, been one of the trickier things to scale.

01:38:30 Speaker_04
I have a few friends who are super senior engineers, and one of their lines is like, it's very hard to predict where systems will break when you scale them.

01:38:39 Speaker_04
You can try to predict in advance, but there's always something weird that's going to happen when you add this extra zero. You thought you thought through everything, but you didn't actually think through everything.

01:38:52 Speaker_04
But I think for that particular system, we've

01:38:58 Speaker_04
So for concrete details, the thing we do is obviously we upload, we chunk up all of your code, and then we send out the code for embedding, and we embed the code, and then we store the embeddings in a database, but we don't actually store any of the code.

01:39:18 Speaker_04
And then there's reasons around making sure that We don't introduce client bugs because we're very, very paranoid about client bugs. We store much of the details on the server, like everything is sort of encrypted.

01:39:34 Speaker_04
So one of the technical challenges is always making sure that the local index, the local codebase state is the same as the state that is on the server. And the way technically we ended up doing that is, for every single file, you can keep this hash.

01:39:52 Speaker_04
And then for every folder, you can keep a hash, which is the hash of all of its children, and you can recursively do that until the top. And why do something complicated? One thing you could do is you could keep a hash for every file.

01:40:07 Speaker_04
Then every minute you could try to download the hashes that are on the server, figure out what are the files that don't exist on the server.

01:40:13 Speaker_04
Maybe you just created a new file, maybe you just deleted a file, maybe you checked out a new branch, and try to reconcile the state between the client and the server.

01:40:23 Speaker_04
But that introduces absolutely ginormous network overhead, both on the client side. I mean, nobody really wants us to hammer their Wi-Fi all the time if you're using Cursor. But also, I mean, it would introduce ginormous overhead in the database.

01:40:39 Speaker_04
I mean, it would sort of be reading tens of terabytes database sort of approaching like 20 terabytes or something database like every second that's just kind of crazy you definitely don't want to do that

01:40:56 Speaker_04
So what you do, you just try to reconcile the single hash, which is at the root of the project. And then if something mismatches, then you go, you find where all the things disagree.

01:41:05 Speaker_04
Maybe you look at the children and see if the hashes match, and if the hashes don't match, go look at their children and so on. But you only do that in the scenario where things don't match. And for most people, most of the time, the hashes match.

01:41:16 Speaker_05
So it's a kind of like hierarchical reconciliation. Yeah, something like that.

01:41:21 Speaker_01
Yeah. It's called the Merkel tree. Yeah.

01:41:23 Speaker_05
Merkel. Yeah. I mean, so yeah, this is cool to see that you kind of have to think through all these problems.

01:41:28 Speaker_04
And I mean, the point of, like, the reason it's gotten hard is just because. Like the number of people using it and

01:41:34 Speaker_04
you know, some of your customers have really, really large codebases to the point where, you know, we originally reordered our codebase, which is big, but I mean, it's just not the size of some company that's been there for 20 years and sort of has a ginormous number of files.

01:41:51 Speaker_04
And you sort of want to scale that across programmers. There's all these details where, like, building a simple thing is easy, but scaling it to a lot of people, like a lot of companies is obviously a difficult problem.

01:42:02 Speaker_04
Which is sort of, you know, independent of actually, so that's, there's part of this scaling our current solution is also, you know, coming up with new ideas that obviously we're working on.

01:42:11 Speaker_04
But then, but then scaling all of that in the last few weeks, months.

01:42:14 Speaker_01
Yeah. And there are a lot of clever things, like additional things that go into this indexing system. For example, the bottleneck in terms of costs is not storing things in the vector database or the database, it's actually embedding the code.

01:42:27 Speaker_01
And you don't want to re-embed the code base for every single person in a company that is using the same exact code, except for maybe they're in a different branch with a few different files, or they've made a few local changes.

01:42:38 Speaker_01
And so because, again, embeddings are the bottleneck, you can do one clever trick and not have to worry about the complexity of dealing with branches in the other databases where you just

01:42:48 Speaker_01
have some cache on the actual vectors computed from the hash of a given chunk. And so this means that when the nth person at a company goes and invents their code base, it's really, really fast.

01:43:03 Speaker_01
And you do all this without actually storing any code on our servers at all. No code data is stored. We just store the vectors in the vector database and the vector cache.

01:43:11 Speaker_05
What's the biggest gains at this time you get from indexing the code base. Just out of curiosity, like what... what benefit do users have?

01:43:22 Speaker_05
It seems like longer term, there'll be more and more benefit, but in the short term, just asking questions of the code base, what's the usefulness of that?

01:43:31 Speaker_03
I think the most obvious one is just you want to find out where something is happening in your large code base. And you sort of have a fuzzy memory of, okay, I want to find the place where we do X.

01:43:46 Speaker_03
but you don't exactly know what to search for in a normal text search. And so you ask a chat, you hit command enter to ask with the codebase chat, and then very often it finds the right place that you were thinking of.

01:43:58 Speaker_01
I think, like you mentioned, in the future, I think this is only going to get more and more powerful where we're working a lot on improving the quality of our retrieval.

01:44:08 Speaker_01
And I think the ceiling for that is really, really much higher than people give it credit for.

01:44:12 Speaker_05
One question that's good to ask here, have you considered and why haven't you much done sort of local stuff to where you can do the, I mean, it seems like everything we've just discussed is exceptionally difficult to do.

01:44:23 Speaker_05
To go to the cloud, you have to think about all these things with the caching and the, you know, large code base with a large number of programmers are using the same code base. You have to figure out the puzzle of that.

01:44:35 Speaker_05
A lot of it, you know, most software just does stuff, this heavy computational stuff locally. Have you considered doing sort of embeddings locally?

01:44:45 Speaker_03
Yeah, we thought about it, and I think it would be cool to do it locally. I think it's just really hard.

01:44:51 Speaker_03
And one thing to keep in mind is that, you know, some of our users use the latest MacBook Pro, but most of our users, like more than 80% of our users, are on Windows machines, and many of them are not very powerful. And so,

01:45:07 Speaker_03
local models really only works on the latest computers. And it's also a big overhead to build that in. And so even if we would like to do that, it's currently not something that we are able to focus on.

01:45:20 Speaker_03
And I think there are some people that do that, and I think that's great. But Especially as models get bigger and bigger and you want to do fancier things with like bigger models, it becomes even harder to do it locally.

01:45:34 Speaker_00
Yeah.

01:45:34 Speaker_04
And it's not a problem of like weaker computers. It's just that, for example, if you're some big company, you have big company codebase, it's just really hard to process big company codebase, even on the beefiest MacBook Pros.

01:45:48 Speaker_04
So even if it's not even a matter of like, if you're just like, a student or something. I think if you're the best programmer at a big company, you're still going to have a horrible experience if you do everything locally.

01:46:02 Speaker_04
You could do Edge and scrape by, but again, it wouldn't be fun anymore.

01:46:07 Speaker_01
you know, like an approximate nearest neighbors and this massive code base is going to just eat up your memory and your CPU.

01:46:14 Speaker_01
And that's, and that's just that, like, let's talk about, like, also the modeling side, where, as Arvid said, there are these massive headwinds against local models, where one, things that seem to move towards MOEs,

01:46:28 Speaker_01
One benefit is maybe they're more memory bandwidth bound, which plays in favor of local versus using GPUs or using NVIDIA GPUs. But the downside is these models are just bigger in total.

01:46:42 Speaker_01
And they're going to need to fit often not even on a single node, but multiple nodes. There's no way that's going to fit inside of even really good MacBooks. And I think especially for coding,

01:46:55 Speaker_01
It's not a question as much of like, does it clear some bar of like the models good enough to do these things and then like we're satisfied, which may be the case for other other problems and maybe where local models shine.

01:47:08 Speaker_01
But people are always going to want the best, the most intelligent, the most capable things. And that's going to be really, really hard to run for almost all people locally.

01:47:17 Speaker_04
Don't you want the most capable model? You want Sonnet?

01:47:22 Speaker_05
And also with O1. I like how you're pitching me. Would you be satisfied with an inferior model? Listen, yes, I'm one of those, but there's some people that like to do stuff locally, especially.

01:47:36 Speaker_05
There's a whole, obviously, open source movement that kind of resists, and it's good that they exist, actually, because you want to resist the power centers that are growing

01:47:46 Speaker_03
There's actually an alternative to local models that I am particularly fond of. I think it's still very much in the research stage, but you could imagine to do homomorphic encryption for language model inference.

01:48:00 Speaker_03
So you encrypt your input on your local machine, then you send that up, and then the server can use loss of computation. They can run models that you cannot run locally on this encrypted data, but they cannot see what the data is.

01:48:14 Speaker_03
And then they send back the answer and you decrypt the answer and only you can see the answer. So I think that's still very much research and all of it is about trying to make the overhead lower because right now the overhead is really big.

01:48:29 Speaker_03
But if you can make that happen, I think that would be really, really cool. And I think it would be really, really impactful.

01:48:36 Speaker_03
Because I think one thing that's actually kind of worrisome is that as these models get better and better, they're going to become more and more economically useful.

01:48:44 Speaker_03
And so more and more of the world's information and data will flow through, you know, one or two centralized actors.

01:48:52 Speaker_03
And then there are worries about, you know, there can be traditional hacker attempts, but it also creates this kind of scary part where

01:49:02 Speaker_03
if all of the world's information is flowing through one node in plain text, you can have surveillance in very bad ways. And sometimes that will happen for, you know, initially will be like good reasons.

01:49:16 Speaker_03
Like people will want to try to protect against like bad actors using AI models in bad ways. And then you will add in some surveillance code and then someone else will come in and, you know, you're in a slippery slope and then you start

01:49:29 Speaker_03
doing bad things with a lot of the world's data. And so I'm very hopeful that we can solve homomorphic encryption for language modeling.

01:49:38 Speaker_05
Yeah, doing privacy preserving machine learning. But I would say that's the challenge we have with all software these days. It's like,

01:49:47 Speaker_05
There's so many features that can be provided from the cloud and all of us increasingly rely on it and make our life awesome, but there's downsides. And that's why you rely on really good security to protect from basic attacks.

01:49:57 Speaker_05
But there's also only a small set of companies that are controlling that data. you know, and they obviously have leverage and they could be infiltrated in all kinds of ways. That's the world we live in.

01:50:10 Speaker_04
Yeah, I mean, the thing I'm just actually quite worried about is sort of the world where, I mean, so Entropic has this responsible scaling policy, and so we're on like the low ASLs, which is the Entropic Security Law or whatever, of like, of the models, but as we get to like, quote unquote, ASL three, ASL four, whatever models, which are sort of very powerful,

01:50:34 Speaker_04
For mostly reasonable security reasons, you would want to monitor all the prompts. But I think that's sort of reasonable and understandable where everyone is coming from.

01:50:44 Speaker_04
But, man, it'd be really horrible if all the world's information is sort of monitored that heavily. It's way too centralized. It's like this really fine line you're walking where On the one side, you don't want the models to go rogue.

01:51:01 Speaker_04
On the other side, like, man, it's humans. I don't know if I trust all the wireless information to pass through three model providers.

01:51:09 Speaker_01
Yeah. Why do you think it's different than cloud providers?

01:51:13 Speaker_03
Because I think a lot of this data would never have gone to the cloud providers in the first place. This is often like you want to give more data to the EIA models.

01:51:28 Speaker_03
You want to give personal data that you would never have put online in the first place to these companies or to these models.

01:51:37 Speaker_03
And it also centralizes control where right now for cloud, you can often use your own encryption keys and like AWS can't really do much. But here it's just centralized actors that see the exact plaintext of everything.

01:51:59 Speaker_05
On the topic of context, that's actually been a friction for me. When I'm writing code in Python, there's a bunch of stuff imported. You could probably intuit the kind of stuff I would like to include in the context.

01:52:13 Speaker_05
How hard is it to auto-figure out the context?

01:52:17 Speaker_02
It's tricky. I think we can do a lot better at computing the context automatically in the future. One thing that's important to note is there are trade-offs with including automatic context.

01:52:30 Speaker_02
So the more context you include for these models, first of all, the slower they are, and the more expensive those requests are, which means you can then do less model calls and do less fancy stuff in the background.

01:52:42 Speaker_02
Also, for a lot of these models, they get confused if you have a lot of information in the prompt. So the bar for accuracy and for relevance of the context you include should be quite high.

01:52:55 Speaker_02
Already we do some automatic context in some places within the product. It's definitely something we want to get a lot better at.

01:53:01 Speaker_02
And I think that there are a lot of cool ideas to try there, both on the learning better retrieval systems, like better embedding models, better re-rankers.

01:53:14 Speaker_02
I think that there are also cool academic ideas, stuff we've tried out internally, but also the field is grappling with writ large, about can you get language models to a place where you can actually just have the model itself, like understand a new corpus of information,

01:53:29 Speaker_02
And the most popular talked about version of this is, can you make the context windows infinite? Then if you make the context windows infinite, can you make the model actually pay attention to the infinite context?

01:53:38 Speaker_02
And then after you can make it pay attention to the infinite context, to make it somewhat feasible to actually do it, can you then do caching for that infinite context? You don't have to recompute that all the time.

01:53:47 Speaker_02
But there are other cool ideas that are being tried that are a little bit more analogous to fine tuning of actually learning this information and the weights of the model.

01:53:55 Speaker_02
And it might be that you actually get sort of a qualitatively different type of understanding if you do it more at the weight level than if you do it at the in-context learning level.

01:54:04 Speaker_02
I think the jury's still a little bit out on how this is all going to work in the end.

01:54:09 Speaker_02
But in the interim, us as a company, we are really excited about better retrieval systems and picking the parts of the code base that are most relevant to what you're doing. We could do that a lot better.

01:54:18 Speaker_01
Like one interesting proof of concept for the learning this knowledge directly in the weights is with VS Code. So we're in a VS Code fork and VS Code, the code is all public. So these models in pre-training have seen all the code.

01:54:34 Speaker_01
They've probably also seen questions and answers about it, and then they've been fine-tuned and RLA-chefed to be able to answer questions about code in general.

01:54:42 Speaker_01
So when you ask it a question about VS Code, sometimes it'll hallucinate, but sometimes it actually does a pretty good job at answering the question. And I think like, this is just by it happens to be okay.

01:54:55 Speaker_01
But what if you could actually like specifically, train or post train a model such that it really was built to understand this code base? It's an open research question, one that we're quite interested in.

01:55:07 Speaker_01
And then there's also uncertainty of like, do you want the model to be the thing that end to end is doing everything, i.e. it's doing the retrieval and its internals, and then kind of answering the question, creating the code?

01:55:18 Speaker_01
Or do you want to separate the retrieval from the frontier model, where maybe, you know, you'll get some really capable models that are much better than like the best open source ones in a handful of months.

01:55:30 Speaker_01
And then you'll want to separately train a really good open source model to be the retriever, to be the thing that feeds in the context to these larger models.

01:55:40 Speaker_05
Can you speak a little more to the post-training model to understand the code base? What do you mean by that? Is this a synthetic data direction?

01:55:48 Speaker_01
Is this... Yeah, I mean, there are many possible ways you could try doing it. There's certainly no shortage of ideas. It's just a question of going in and trying all of them and being empirical about which one works best.

01:56:02 Speaker_01
One very naive thing is to try to replicate what's done with VS Code and these frontier models.

01:56:09 Speaker_01
So let's continue pre-training, some kind of continued pre-training that includes general code data, but also throws in a lot of the data of some particular repository that you care about.

01:56:19 Speaker_01
And then in post training, meaning in let's just start with instruction fine tuning, you have like a normal instruction fine tuning data set about code, but you throw in a lot of questions about code in that repository.

01:56:33 Speaker_01
So you could either get ground truth ones, which might be difficult, or you could do what you kind of hinted at or suggested using synthetic data, ie, kind of having the model ask questions about various pieces of the code.

01:56:49 Speaker_01
So you kind of take the pieces of the code, then prompt the model or have a model propose a question for that piece of code, and then add those as instruction finds you new data points.

01:56:58 Speaker_01
And then in theory, this might unlock the model's ability to answer questions about that code base.

01:57:05 Speaker_05
Let me ask you about OpenAI 01. What do you think is the role of that kind of test-time compute system in programming?

01:57:13 Speaker_01
I think test-time compute is really, really interesting.

01:57:17 Speaker_01
So there's been the pre-training regime, which will kind of, as you scale up the amount of data and the size of your model, get you better and better performance, both on loss and then on downstream benchmarks, and just general performance when we use it for coding or other tasks.

01:57:35 Speaker_01
We're starting to hit a bit of a data wall, meaning it's going to be hard to continue scaling up this regime.

01:57:42 Speaker_01
And so scaling up test time compute is an interesting way of now, you know, increasing the number of inference time flops that we use, but still getting like, like, yeah, as you increase the number of flops use inference time, getting corresponding improvements in the performance of these models

01:57:59 Speaker_01
Traditionally, we just had to literally train a bigger model that always used that many more flops. But now we could perhaps use the same size model and run it for longer to be able to get an answer at the quality of a much larger model.

01:58:13 Speaker_01
And so the really interesting thing I like about this is there are some problems that perhaps require hundred trillion parameter model intelligence trained on 100 trillion tokens. But that's like maybe 1%, maybe like point 1% of all queries.

01:58:29 Speaker_01
So are you going to spend all of this effort, all this compute training model

01:58:34 Speaker_01
that costs that much and then run it so infrequently, it feels completely wasteful when instead you get the model that can, that you train the model that's capable of doing the 99.9% of queries, then you have a way of inference time running it longer for those few people that really, really want max intelligence.

01:58:54 Speaker_05
How do you figure out which problem requires what level of intelligence? Is that possible to dynamically figure out when to use GPT-4, when to use a small model, and when you need the O1?

01:59:09 Speaker_01
I mean, yeah, that's an open research problem, certainly. I don't think anyone's actually cracked this model routing problem quite well. We'd like to. We have initial implementations of this for things, for something like CursorTab.

01:59:24 Speaker_01
But at the level of going between 4.0 Sonnet to O1, It's a bit trickier. There's also a question of what level of intelligence do you need to determine if the thing is too hard for the four level model. Maybe you need the O1 level model.

01:59:44 Speaker_01
It's really unclear.

01:59:45 Speaker_05
But you mentioned there's a pre-training process, then there's post-training, and then there's test time compute. Is that fair to sort of separate? Where's the biggest gains?

01:59:57 Speaker_01
Well, it's weird, because like test time compute, there's like a whole training strategy needed to get test time to compute to work.

02:00:04 Speaker_01
And the really, the other really weird thing about this is no one like outside of the big labs, and maybe even just open AI, no one really knows how it works. Like there have been some really interesting papers that

02:00:17 Speaker_01
show hints of what they might be doing. And so perhaps they're doing something with tree search using process reward models. But yeah, I just I think the issue is, we don't quite know exactly what it looks like.

02:00:31 Speaker_01
So it would be hard to kind of comment on like where it fits in, I would put it in post training, but maybe like the compute spent for this kind of for getting test time compute to work for a model is going to dwarf pre training eventually.

02:00:45 Speaker_05
So we don't even know if O1 is using just like chain of thought RL. We don't know how they're using any of these. I don't know anything.

02:00:53 Speaker_01
It's fun to speculate.

02:00:56 Speaker_05
Like if you were to build a competing model, what would you do?

02:01:01 Speaker_01
Yeah. So one thing to do would be, I think you probably need to train a process reward model, which is so maybe we can get into reward models and outcome reward models versus process reward models.

02:01:12 Speaker_01
Outcome reward models are the kind of traditional reward models that people are trained for these for for language models, language modeling. And it's just looking at the final thing.

02:01:21 Speaker_01
So if you're doing some math problem, let's look at that final thing you've done, everything, and let's assign a grade to it, how likely we think, like what's the reward for this outcome.

02:01:32 Speaker_01
Process reward models instead try to grade the chain of thought.

02:01:35 Speaker_01
And so OpenAI had some preliminary paper on this, I think last summer, where they use human labelers to get this pretty large, several hundred thousand dataset of grading chains of thought.

02:01:50 Speaker_01
Ultimately, it feels like I haven't seen anything interesting in the ways that people use process reward models outside of just using it as a means of affecting how we choose between a bunch of samples.

02:02:02 Speaker_01
So like what people do in all these papers is they sample a bunch of outputs from the language model and then use the process reward models to grade all those generations alongside maybe some other heuristics and then use that to choose the best answer.

02:02:18 Speaker_01
The really interesting thing that people think might work and people want to work is tree search with these process reward models.

02:02:25 Speaker_01
Because if you really can grade every single step of the chain of thought, then you can kind of branch out and, you know, explore multiple paths of this chain of thought, and then use these process reward models to evaluate how good is this branch that you're taking.

02:02:40 Speaker_05
Yeah, when the quality of the branch is somehow strongly correlated with the quality of the outcome at the very end. So you have a good model of knowing which branch to take. So not just in the short term, but in the long term.

02:02:52 Speaker_01
And like the interesting work that I think has been done is figuring out how to properly train the process, or the interesting work that has been open sourced and people I think talk about is how to train the process reward models, maybe in a more automated way.

02:03:07 Speaker_01
I could be wrong here, could not be mentioning something because I haven't seen anything super, that seems to work really well for using the process reward models creatively to do tree searching code.

02:03:19 Speaker_05
This is kind of an AI safety, maybe a bit of a philosophy question. So OpenAI says that they're hiding the chain of thought from the user. And they've said that that was a difficult decision to make.

02:03:29 Speaker_05
They, instead of showing the chain of thought, they're asking the model to summarize the chain of thought.

02:03:35 Speaker_05
They're also in the background saying they're going to monitor the chain of thought to make sure the model is not trying to manipulate the user, which is a fascinating possibility.

02:03:44 Speaker_02
But anyway, what do you think about hiding the chain of thought? One consideration for OpenAI, and this is completely speculative, could be that they want to make it hard for people to distill these capabilities out of their model.

02:03:56 Speaker_02
it might actually be easier if you had access to that hidden chain of thought to replicate the technology. Because that's pretty important data, like seeing the steps that the model took to get to the final result.

02:04:05 Speaker_02
So you could probably train on that also.

02:04:08 Speaker_02
And there was sort of a mirror situation with this, with some of the large language model providers, and also this is speculation, but some of these APIs used to offer easy access to log probabilities for all the tokens that they're generating, and also log probabilities for the prompt tokens.

02:04:26 Speaker_02
And then some of these APIs took those away. And again, complete speculation, but...

02:04:32 Speaker_02
One of the thoughts is that the reason those were taken away is if you have access log probabilities, similar to this hidden train of thought, that can give you even more information to try and distill these capabilities out of the APIs, out of these biggest models into models you control.

02:04:46 Speaker_02
As an asterisk on also the previous discussion about us integrating O1, I think that we're still learning how to use this model. So we made O1 available in Cursor because when we got the model, we were really interested in trying it out.

02:05:02 Speaker_02
I think a lot of programmers are going to be interested in trying it out. 01 is not part of the default cursor experience in any way.

02:05:12 Speaker_02
And we still haven't found a way to get integrated into the editor in a way that we reach for sort of every hour, maybe even every day. And so I think the jury's still out on how to use the model.

02:05:27 Speaker_02
And we haven't seen examples yet of people releasing things where it seems really clear like, oh, that's now the use case.

02:05:36 Speaker_02
The obvious one to return to is maybe this can make it easier for you to have these background things running, to have these models in loops, to have these models be agentic. But we're still discovering.

02:05:48 Speaker_04
To be clear, we have ideas. We just need to try and get something incredibly useful before we put it out there.

02:05:55 Speaker_01
But it has these significant limitations. Even barring capabilities, it does not stream. And that means it's really, really painful to use for things where you want to supervise the output.

02:06:09 Speaker_01
And instead, you're just waiting for the wall of text to show up. Also, it does feel like the early innings of test, time, compute, and search, where it's just very, very much a V0. And there's so many things that like don't feel quite right.

02:06:25 Speaker_01
And I suspect in parallel to people increasing the amount of pre-training data and the size of the models and pre-training and finding tricks there, you'll now have this other thread of getting search to work better and better.

02:06:40 Speaker_05
So let me ask you about strawberry tomorrow eyes. So it looks like GitHub Copilot might be integrating O1 in some kind of way. And I think some of the comments are saying, does this mean cursor is done? I think I saw one comment saying that.

02:07:01 Speaker_03
It's time to shut down Cursor. Time to shut down Cursor, thank you.

02:07:05 Speaker_02
So is it time to shut down Cursor? I think this space is a little bit different from past software spaces over the 2010s, where I think that the ceiling here is really, really, really incredibly high.

02:07:17 Speaker_02
And so I think that the best product in three to four years will just be so much more useful than the best product today. And you can wax poetic about moats this and brand that, and this is our advantage.

02:07:31 Speaker_02
But I think in the end, just if you don't have, if you stop innovating on the product, you will lose. And that's also great for startups.

02:07:39 Speaker_02
That's great for people trying to enter this market because it means you have an opportunity to win against people who have lots of users already. by just building something better.

02:07:50 Speaker_02
And so I think, yeah, over the next few years, it's just about building the best product, building the best system, and that both comes down to the modeling engine side of things, and it also comes down to the editing experience.

02:08:04 Speaker_01
I think most of the additional value from Cursor versus everything else out there is not just integrating the new model fast like 0.1.

02:08:12 Speaker_01
It comes from all of the depth that goes into these custom models that you don't realize are working for you in every facet of the product, as well as the really thoughtful UX with every single feature.

02:08:28 Speaker_05
All right, from that profound answer, let's descend back down to the technical. You mentioned you have a taxonomy of synthetic data.

02:08:34 Speaker_01
Oh yeah.

02:08:36 Speaker_05
Can you please explain?

02:08:36 Speaker_01
Yeah, I think there are three main kinds of synthetic data. The first is, so what is synthetic data first? So there's normal data, like non-synthetic data, which is just data that's naturally created, i.e.

02:08:51 Speaker_01
usually it'll be from humans having done things. So from some human process, you get this data. Synthetic data, the first one would be distillation. So having a language model, kind of output tokens or probability distributions over tokens.

02:09:08 Speaker_01
And then you can train some less capable model on this. This approach is not going to get you a net more capable model than the original one that has produced the tokens.

02:09:19 Speaker_01
But it's really useful for if there's some capability you want to elicit from some really expensive, high latency model, you can then distill that down into some smaller task-specific model.

02:09:31 Speaker_01
The second kind is when one direction of the problem is easier than the reverse. And so a great example of this is bug detection, like we mentioned earlier, where it's a lot easier to introduce reasonable-looking bugs

02:09:49 Speaker_01
than it is to actually detect them. And this is probably the case for humans too. And so what you can do is you can get a model that's not training that much data, that's not that smart to introduce a bunch of bugs in code.

02:10:02 Speaker_01
And then you can use that to then train, use a synthetic data to train a model that can be really good at detecting bugs. The last category, I think is, I guess, the main one that it feels like the big labs are doing for synthetic data, which is,

02:10:18 Speaker_01
producing text with language models that can then be verified easily.

02:10:23 Speaker_01
So like, you know, extreme example of this is if you have a verification system that can detect if language is Shakespeare level, and then you have a bunch of monkeys typing in typewriters, like you can eventually get enough training data to train a Shakespeare level language model.

02:10:39 Speaker_01
And I mean, this is the case, like very much the case for math, where verification is actually really, really easy for formal. formal languages.

02:10:49 Speaker_01
And then what you can do is you can have an OK model, generate a ton of rollouts, and then choose the ones that you know have actually proved the ground truth theorems and train that further.

02:11:00 Speaker_01
There are similar things you can do for code with LeetCode-like problems, where if you have some set of tests that you know correspond to, if something passes these tests, it has actually solved the problem.

02:11:11 Speaker_01
You could do the same thing where you verify that it's passed the test and then train the model and the outputs that have passed the tests. I think it's gonna be a little tricky getting this to work in all domains or just in general.

02:11:24 Speaker_01
Like having the perfect verifier feels really, really hard to do with just like open-ended miscellaneous tasks. You get the model or more like long horizon tasks, even in coding.

02:11:35 Speaker_05
That's because you're not as optimistic as Arvid, but yeah. So yeah, so that third category requires having a verifier.

02:11:42 Speaker_01
Yeah. Verification, it feels like it's best when you know for a fact that it's correct. And then it wouldn't be using a language model to verify, it would be using tests or formal systems. Or running the thing too.

02:11:57 Speaker_02
Doing the human form of verification where you just do manual quality control.

02:12:00 Speaker_01
Yeah.

02:12:01 Speaker_02
But the language model version of that where it's running the thing and it actually understands the output. Yeah, no, that's true. For somewhere between.

02:12:08 Speaker_01
Yeah, I think that's the category that is most likely to result in massive gains.

02:12:14 Speaker_05
What about RL with feedback side, RLHF versus RLAIF? What's the role of that in getting better performance on the models?

02:12:26 Speaker_01
Yeah, so RLHF is when the reward model you use is trained from some labels you've collected from humans giving feedback. I think this works if you have the ability to get a ton of human feedback for this kind of task that you care about.

02:12:47 Speaker_01
RLAIF is interesting because you're kind of depending on, like this is actually kind of going to, it's depending on the constraint that verification is actually a decent bit easier than generation.

02:13:03 Speaker_01
Because it feels like, okay, like, what are you doing? Are you using this language model to look at the language model outputs and then prove the language model? But no, it actually may work.

02:13:11 Speaker_01
If the language model has a much easier time verifying some solution than it does generating it, then you actually could perhaps get this kind of recursive loop. I don't think it's going to look exactly like that.

02:13:23 Speaker_01
The other the other thing you could do is we kind of do is like a little bit of a mix of RLAIF and RLHF, where usually the model is actually quite correct.

02:13:33 Speaker_01
And this is in the case of cursor tab at picking between like two possible generations of what is what is what is the better one. And then it just needs like a hand a little bit of human nudging with only like on the on the order of 50 100

02:13:48 Speaker_01
examples to like kind of align that prior the model has with exactly with what you want. It looks different than I think normal RLHF where you're usually training these reward models on tons of examples.

02:14:02 Speaker_05
What's your intuition when you compare generation and verification or generation and ranking? Is ranking way easier than generation?

02:14:12 Speaker_01
My intuition would just say, yeah, it should be. Like this is kind of going back to, like if you believe P does not equal NP, then there's this massive class of problems that are much, much easier to verify given a proof than actually proving it.

02:14:30 Speaker_05
I wonder if the same thing will prove P not equal to NP or P equal to NP.

02:14:35 Speaker_03
That would be really cool.

02:14:37 Speaker_05
That'd be a whatever Fields Medal by AI. Who gets the credit? Another open philosophical question.

02:14:47 Speaker_04
I'm actually surprisingly curious what a good bet for when AI will get the Fields Medal will be.

02:14:57 Speaker_05
Isn't this Amon's specialty?

02:14:59 Speaker_04
I don't know what Amon's bet here is.

02:15:01 Speaker_05
Oh, sorry, Nobel Prize or Fields Medal first?

02:15:04 Speaker_04
Fields Medal. Well, Fields Medal level. Fields Medal comes first, I think.

02:15:07 Speaker_05
Fields Medal comes first. Well, you would say that, of course.

02:15:10 Speaker_03
But it's also this, like, isolated system you can verify and... Yeah, sure.

02:15:15 Speaker_04
I don't even know if I don't need to do.

02:15:16 Speaker_01
I feel like I have much more to do there. It felt like the path to get to IMO was a little bit more clear because it already could get a few IMO problems.

02:15:24 Speaker_01
And there are a bunch of like, there's a bunch of low hanging fruit given the literature at the time of like what what tactics people could take. I think I'm one much less versed in the space of theorem proving now.

02:15:34 Speaker_01
And two, yeah, less intuition about how close we are to solving these really, really hard open problems.

02:15:41 Speaker_05
So you think it'll be fields matter first? It won't be like in physics or in... Oh, 100%.

02:15:48 Speaker_04
I think that's probably more likely. Like, it's probably much more likely that it'll get in. Yeah, yeah, yeah.

02:15:53 Speaker_04
Well, I think it goes to like, I don't know, like BSD, which is a Burtzman-Ritter-Dyer conjecture, or like Riemann hypothesis, or any one of these like hard math problems, which is actually really hard.

02:16:04 Speaker_04
It's sort of unclear what the path to get even a solution looks like. Like, we don't even know what a path looks like, let alone

02:16:12 Speaker_03
Yeah, this is like an isolated system and you can actually you have a good reward system and It feels like it's easier to train for that.

02:16:22 Speaker_01
I think we might get feels metal before AGI.

02:16:25 Speaker_04
I think I mean, yeah, I'd be very happy Very happy, but I don't know if I I think 2028 2030 Feels metal feels metal. All right

02:16:39 Speaker_05
It feels like forever from now, given how fast things have been going. Speaking of how fast things have been going, let's talk about scaling laws. So for people who don't know, maybe it's good to talk about this whole idea of scaling laws.

02:16:56 Speaker_05
What are they, where do you think stand, and where do you think things are going?

02:16:59 Speaker_01
I think it was interesting, the original scaling laws paper by OpenAI was slightly wrong, because I think of some issues they did with learning rate schedules. And then Chinchilla showed a more correct version.

02:17:12 Speaker_01
And then from then people have again kind of deviated from doing the compute optimal thing because people start now optimizing more so for making the thing work really well, given an inference budget.

02:17:26 Speaker_01
And I think there are a lot more dimensions to these curves than what we originally used of just compute number of parameters and data. like inference compute is the obvious one. I think context length is another obvious one.

02:17:41 Speaker_01
So if you care, like, let's say you care about the two things of inference compute and then context window, maybe the thing you want to train is some kind of SSM because they're much, much cheaper and faster at super, super long context.

02:17:55 Speaker_01
And even if maybe it is 10X worth scaling properties during training, meaning you've spent 10X more compute to train the thing to get the same level of capabilities,

02:18:04 Speaker_01
it's worth it because you care most about that inference budget for really long context windows. So it'll be interesting to see how people kind of play with all these dimensions.

02:18:13 Speaker_05
So yeah, I mean, you speak to the multiple dimensions, obviously.

02:18:16 Speaker_05
The original conception was just looking at the variables of the size of the model as measured by parameters and the size of the data as measured by the number of tokens and looking at the ratio of the two.

02:18:26 Speaker_00
Yeah.

02:18:26 Speaker_05
And it's kind of a compelling notion that there is a number. or at least a minimum, and it seems like one was emerging. Do you still believe that there is a kind of bigger is better?

02:18:42 Speaker_01
I mean, I think bigger is certainly better for just raw performance and raw intelligence. I think that the path that people might take is I'm particularly bullish on distillation.

02:18:54 Speaker_01
And like, yeah, how many knobs can you turn to if we spend like a ton, ton of money on training, like get the most capable, cheap model?

02:19:04 Speaker_01
right, like really, really caring as much as you can, because like the naive version of caring as much as you can about inference time compute is what people have already done with like the llama models are just over training the shit out of seven B models on way, way, way more tokens than is essential optimal.

02:19:20 Speaker_01
But if you really care about it, maybe the thing to do is what gamma did, which is, let's just not let's not just train on tokens, let's literally train on minimizing the KL divergence with the distribution of gamma 2070, right?

02:19:34 Speaker_01
So knowledge distillation there. And you're spending the compute of literally training this 27 billion model, billion parameter model on all these tokens just to get out this, I don't know, smaller model.

02:19:46 Speaker_05
And the distillation gives you just a faster model, smaller means faster.

02:19:50 Speaker_01
Yeah, distillation in theory is, I think getting out more signal from the data that you're training on.

02:19:57 Speaker_01
And it's like another, it's perhaps another way of getting over, not like completely over, but like partially helping with the data wall where like you only have so much data to train on.

02:20:05 Speaker_01
Let's like train this really, really big model on all these tokens and we'll distill it into this smaller one. And maybe we can get more signal per token for this much smaller model than we would have originally if we trained it.

02:20:17 Speaker_05
So if I gave you $10 trillion, how would you spend it? I mean, you can't buy an island or whatever. How would you allocate it? in terms of improving the big model versus maybe paying for HF in the RLHF or?

02:20:37 Speaker_01
Yeah, I think there's a lot of these secrets and details about training these large models that I just don't know and are only privy to the large labs.

02:20:46 Speaker_01
And the issue is I would waste a lot of that money if I even attempted this because I wouldn't know those things.

02:20:52 Speaker_01
suspending a lot of disbelief and assuming like you had the know-how and operate, or if you're saying like you have to operate with like the limited information you have now.

02:21:04 Speaker_05
No, no, actually I would say you swoop in and you get all the information, all the little heuristics, all the little parameters, all the parameters that define how the thing is trained.

02:21:16 Speaker_05
if we look in how to invest money for the next five years in terms of maximizing what you called raw intelligence?

02:21:23 Speaker_04
I mean, isn't the answer like really simple? You just try to get as much compute as possible.

02:21:28 Speaker_04
Like at the end of the day, all you need to buy is the GPUs and then the researchers can find all the, they can sort of, you can tune whether you want to pre-train a big model or a small model.

02:21:41 Speaker_01
Well, this gets into the question of like, are you really limited by compute and money or are you limited by these other things?

02:21:48 Speaker_04
I'm more privy to Arvid's belief that we're sort of ideal limited.

02:21:54 Speaker_03
But if you have a lot of compute, you can run a lot of experiments.

02:21:59 Speaker_05
So you would run a lot of experiments versus like use that compute to train a gigantic model?

02:22:05 Speaker_03
I would, but I do believe that we are limited in terms of ideas that we have.

02:22:11 Speaker_01
I think, yeah, because even with all this compute and like, you know, all the data you could collect in the world, I think you really are ultimately limited by not even ideas, but just like really good engineering.

02:22:25 Speaker_01
Even with all the capital in the world, would you really be able to assemble... There aren't that many people in the world who really make the difference here.

02:22:33 Speaker_01
And there's so much work that goes into research that is just pure, really, really hard engineering work. As a very

02:22:44 Speaker_01
hand-wavy example, if you look at the original Transformer paper, how much work was joining together a lot of these really interesting concepts embedded in the literature versus then going in and writing all the code, like maybe the CUDA kernels, maybe whatever else.

02:22:58 Speaker_01
I don't know if it ran on GPUs or TPUs originally, such that it actually saturated the GPU performance. Getting Gnome to go in and do all this code. Gnome is probably one of the best engineers in the world.

02:23:10 Speaker_01
or maybe going a step further, like the next generation of models, having these things, like getting model parallelism to work and scaling it on like, you know, thousands of, or maybe tens of thousands of like V100s, which I think GBDE3 may have been.

02:23:23 Speaker_01
There's just so much engineering effort that has to go into all of these things to make it work. If you really brought that cost down to

02:23:33 Speaker_01
like, you know, maybe not zero, but just made it 10x easier, made it super easy for someone with really fantastic ideas to immediately get to the version of like the new architecture they dreamed up that is like getting 50, 40% utilization on the GPUs.

02:23:49 Speaker_01
I think that would just speed up research by a ton.

02:23:54 Speaker_04
I mean, I think if you see a clear path to improvement, you should always sort of take the low-hanging fruit first, right?

02:23:59 Speaker_04
And I think probably OpenAI and all the other labs that did the right thing to pick off the low-hanging fruit, where the low-hanging fruit is like sort of

02:24:09 Speaker_04
You could scale up to a GPT 4.25 scale and you just keep scaling and things keep getting better. There's no point of experimenting with new ideas when everything is working.

02:24:26 Speaker_04
and you should sort of bang on it and try to try to get as much as much juice out of the possible and then maybe maybe when you really need new ideas for I think I think if you're if you're spending 10 trillion dollars probably want to spend some so you know then actually like re-evaluate your ideas like probably your idea limited at that point.

02:24:41 Speaker_01
I think all of us believe new ideas are probably needed to get you know all the way there to AGI and of us also probably believe there exist ways of testing out those ideas at smaller scales and being fairly confident that they'll play out.

02:25:02 Speaker_01
It's just quite difficult for the labs in their current position to dedicate their very limited research and engineering talent to exploring all these other ideas when there's this core thing that will probably improve performance for some decent amount of time.

02:25:22 Speaker_05
Yeah, but also these big labs like winning. So they're just going wild. Okay. So how, a big question looking out into the future, you're now at the center of the programming world.

02:25:38 Speaker_05
How do you think programming, the nature of programming changes in the next, few months, and the next year, and the next two years, next five years, ten years.

02:25:47 Speaker_02
I think we're really excited about a future where the programmer's in the driver's seat for a long time.

02:25:54 Speaker_02
And you've heard us talk about this a little bit, but one that emphasizes speed and agency for the programmer and control, the ability to modify anything you want to modify, the ability to iterate really fast on what you're building, and

02:26:09 Speaker_02
This is a little different, I think, than where some people are jumping to in this space, where I think one idea that's captivated people is, can you talk to your computer?

02:26:24 Speaker_02
Can you have it build software for you as if you're talking to an engineering department or an engineer over Slack? And can it just be this sort of isolated text box? And part of the reason we're not excited about that

02:26:37 Speaker_02
is some of the stuff we've talked about with latency. But then a big reason we're not excited about that is because that comes with giving up a lot of control. It's much harder to be really specific when you're talking in the text box.

02:26:48 Speaker_02
And if you're necessarily just going to communicate with a thing like you would be communicating with an engineering department, you're actually abdicating tons of really important decisions to this bot.

02:26:59 Speaker_02
And this kind of gets at fundamentally what engineering is.

02:27:05 Speaker_02
I think that some people who are a little bit more removed from engineering might think of it as, you know, the spec is completely written out and then the engineers just come and they just implement.

02:27:14 Speaker_02
And it's just about making the thing happen in code and making the thing exist. But I think a lot of the best engineering, the engineering we enjoy,

02:27:24 Speaker_02
involves tons of tiny micro decisions about what exactly you're building and about really hard trade-offs between speed and cost and all the other things involved in a system.

02:27:34 Speaker_02
And we want, as long as humans are actually the ones designing the software and the ones specifying what they want to be built, and it's not just like company run by all AIs, we think you'll really want the human in a driver's seat.

02:27:51 Speaker_02
dictating these decisions. And so there's the jury's still out on kind of what that looks like.

02:27:57 Speaker_02
I think that, you know, one weird idea for what that could look like is it could look like you kind of, you can control the level of abstraction you view a code base at.

02:28:06 Speaker_02
And you can point at specific parts of a code base that may like, maybe you digest a code base by looking at it in the form of pseudocode.

02:28:15 Speaker_02
And you can actually edit that pseudocode too, and then have changes get made down at the sort of formal programming level. And you keep the, like, you know, you can gesture at any piece of logic in your software component of programming.

02:28:30 Speaker_02
You keep the inflow text editing component of programming. You keep the control of, you can even go down into the code. You can go at higher levels of abstraction while also giving you these big productivity gains.

02:28:40 Speaker_05
It'd be nice if you can go up and down the abstraction stack. Yeah.

02:28:45 Speaker_02
And there are a lot of details to figure out there that's sort of like a fuzzy idea, time will tell if it actually works. But these principles of control and speed and the human in the driver's seat, we think are really important.

02:28:55 Speaker_02
We think for some things, like Arvid mentioned before, for some styles of programming, you can kind of hand it off chatbot style, you know, if you have a bug that's really well specified, but that's not most of programming.

02:29:05 Speaker_02
And that's also not most of the programming we think a lot of people value.

02:29:10 Speaker_05
What about like the fundamental skill of programming? There's a lot of people like,

02:29:15 Speaker_05
young people right now kind of scared, like thinking, because they like love programming, but they're scared about like, will I be able to have a future if I pursue this career path?

02:29:26 Speaker_05
Do you think the very skill of programming will change fundamentally?

02:29:30 Speaker_02
I actually think this is a really, really exciting time to be building software. We remember what programming was like in 2013, 2012, whatever it was. And there was just so much more cruft and boilerplate. looking up something really gnarly.

02:29:51 Speaker_02
And yeah, that stuff still exists. It's definitely not at zero. But programming today is way more fun than back then. We're really getting down to the delight concentration.

02:30:03 Speaker_02
And all the things that really draw people to programming, like for instance, this element of being able to build things really fast, and speed, and also individual control, all those are just being turned up a ton.

02:30:15 Speaker_02
And so I think it's just gonna be, I think it's gonna be a really, really fun time for people who build software. I think that the skills will probably change too.

02:30:22 Speaker_02
I think that people's taste in creative ideas will be magnified and it will be less about, maybe less a little bit about boilerplate text editing, maybe even a little bit less about carefulness, which I think is really important today.

02:30:36 Speaker_02
If you're a programmer, I think it'll be a lot more fun. What do you guys think?

02:30:41 Speaker_03
I agree. I'm very excited to be able to change. One thing that happened recently was we wanted to do a relatively big migration to our codebase.

02:30:52 Speaker_03
We were using async local storage in Node.js, which is known to be not very performant, and we wanted to migrate to a context object. And this is a big migration and affects the entire codebase.

02:31:04 Speaker_03
And Swale and I spent, I don't know, five days working through this, even with today's AI tools. And I am really excited for a future where I can just show a couple of examples and then the AI applies that to all of the locations.

02:31:20 Speaker_03
And then it highlights, oh, this is a new example, like what should I do? And then I show exactly what to do there. And then that can be done in like 10 minutes. And then you can iterate much, much faster than you can

02:31:32 Speaker_03
then you don't have to think as much upfront and stand at the blackboard and think exactly, like, how are we going to do this? Because the cost is so high.

02:31:40 Speaker_03
But you can just try something first and you realize, oh, this is not actually exactly what I want. And then you can change it instantly again after. And so, yeah, I think being a programmer in the future is going to be a lot of fun.

02:31:52 Speaker_01
Yeah, I really like that point about it feels like a lot of the time with programming, there two ways you can go about it.

02:31:59 Speaker_01
One is you think really hard, carefully up front about the best possible way to do it, and then you spend your limited time of engineering to actually implement it.

02:32:10 Speaker_01
But I much prefer just getting in the code and taking a crack at it, seeing how it lays out, and then iterating really quickly on that. That feels more fun.

02:32:22 Speaker_05
Yeah, just speaking to generate the boilerplate is great. So you just focus on the difficult design, nuanced, difficult design decisions. Migration, I feel like this is a cool one.

02:32:34 Speaker_05
It seems like large language models are able to basically translate from one program language to another or translate, migrate in the general sense of what migrate is. But that's in the current moment.

02:32:47 Speaker_05
So I mean, the fear has to do with, okay, as these models get better and better, then you're doing less and less creative decisions.

02:32:53 Speaker_05
And is it going to kind of move to a place where it's, you're operating in the design space of natural language, where natural language is the main programming language. And I guess I could ask that by way of advice.

02:33:05 Speaker_05
Like if somebody is interested in programming now, what do you think they should learn? Like today, you guys started in some Java and, I forget the, oh, some PHP. Objective C. Objective C, there you go.

02:33:21 Speaker_05
I mean, in the end, we all know JavaScript is going to win. And not TypeScript. It's going to be like vanilla JavaScript. It's going to eat the world and maybe a little bit of PHP.

02:33:34 Speaker_05
And I mean, it also brings up the question of like, I think Don Knuth has this idea that some percent of the population is geeks. And like there's a particular kind of psychology in mind required for programming.

02:33:48 Speaker_05
And it feels like more and more that expands the kind of person that should be able to, can do great programming might expand.

02:33:58 Speaker_01
I think different people do programming for different reasons, but I think the true, maybe like the best programmers are the ones that really love

02:34:11 Speaker_01
just like absolutely love programming, for example, their folks in our team, who literally, when they're, they get back from work, they go, and then they boot up cursor, and then they start coding on their side projects for the entire night, and they stay up till 3am doing that.

02:34:31 Speaker_01
And when they're sad, they said, I just really need to code. And I think like, You know, there's that level of programmer where like this obsession and love of programming I think makes really the best programmers.

02:34:49 Speaker_01
And I think these types of people will really get into the details of how things work.

02:34:55 Speaker_05
I guess the question I'm asking, that exact program, let's think about that person. When the super tab, the super awesome praise be the tab succeeds and you keep pressing tab,

02:35:08 Speaker_04
That person on the team loves to curse the tab more than anybody else.

02:35:12 Speaker_03
Yeah, and it's also not just like... Like, pressing tab is like the... Just pressing tab, that's like the easy way to say it, and the catchphrase, you know?

02:35:20 Speaker_03
But what you're actually doing when you're pressing tab is that you're injecting intent all the time while you're doing it. Sometimes you're rejecting it, sometimes you're typing a few more characters. And that's the way that you're...

02:35:36 Speaker_03
you're sort of shaping the things that's being created. And I think programming will change a lot to just what is it that you want to make.

02:35:43 Speaker_04
It's sort of higher bandwidth. The communication to the computer just becomes higher and higher bandwidth as opposed to like just typing is much lower bandwidth than communicating intent.

02:35:54 Speaker_05
I mean, this goes to your manifesto titled Engineering Genius. We are an applied research lab building extraordinary productive human AI systems. So speaking to this like hybrid element.

02:36:08 Speaker_05
To start, we're building the engineer of the future, a human AI programmer that's an order of magnitude more effective than any one engineer. This hybrid engineer will have effortless control over their code base and no low entropy keystrokes.

02:36:23 Speaker_05
They will iterate at the speed of their judgment, even in the most complex systems, using a combination of AI and human ingenuity. They will outsmart and out-engineer the best pure AI systems. We are a group of researchers and engineers.

02:36:38 Speaker_05
We build software and models to invent at the edge of what's useful and what's possible. Our work has already improved the lives of hundreds of thousands of programmers. And on the way to that, we'll at least make programming more fun.

02:36:51 Speaker_05
So thank you for talking today. Thank you.

02:36:53 Speaker_02
Thanks for having us. Thank you. Thank you.

02:36:56 Speaker_05
Thanks for listening to this conversation with Michael, Swale, Arvid, and Aman. To support this podcast, please check out our sponsors in the description.

02:37:04 Speaker_05
And now, let me leave you with a random, funny, and perhaps profound programming quote I saw on Reddit. Nothing is as permanent as a temporary solution that works. Thank you for listening, and hope to see you next time.