Nvidia Part III: The Dawn of the AI Era (2022-2023) AI transcript and summary - episode of podcast Acquired
Go to PodExtra AI's episode page (Nvidia Part III: The Dawn of the AI Era (2022-2023)) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.
Go to PodExtra AI's podcast page (Acquired) to view the AI-processed content of all episodes of this podcast.
Acquired episodes list: view full AI transcripts and summaries of this podcast on the blog
Episode: Nvidia Part III: The Dawn of the AI Era (2022-2023)
Author: Ben Gilbert and David Rosenthal
Duration: 02:52:53
Episode Shownotes
It’s a(nother) new era for Nvidia.We thought we’d closed the Acquired book on Nvidia back in April 2022. The story was all wrapped up: Jensen & crew had set out on an amazing journey to accelerate the world’s computing workloads. Along the way they’d discovered a wondrous opportunity (machine learning
powered social media feed recommendations). They forged incredible Power in the CUDA platform, and used it to triumph over seemingly insurmountable adversity — the stock market penalty-box.But, it turned out that was only the precursor to an even wilder journey. Over the past 18 months Nvidia has weathered one of the steepest stock crashes in history ($500B+ market cap wiped away peak-to-trough!). And, it has of course also experienced an even more fantastical rise — becoming the platform that’s powering the emergence of perhaps a new form of intelligence itself… and in the process becoming a trillion-dollar company.Today we tell another chapter in the amazing Nvidia saga: the dawn of the AI era. Tune in!Links:Asianometry on AI HardwareEpisode sourcesCarve Outs:AliasMoanaSponsors:ServiceNow: https://bit.ly/acqsnaiagentsHuntress: https://bit.ly/acqhuntressVanta: https://bit.ly/acquiredvantaMore Acquired!:Get email updates with hints on next episode and follow-ups from recent episodesJoin the SlackSubscribe to ACQ2Merch Store!Note: Acquired hosts and guests may hold assets discussed in this episode. This podcast is not investment advice, and is intended for informational and entertainment purposes only. You should do your own research and make your own independent decisions when considering any financial transactions.
Full Transcript
00:00:00 Speaker_04
You like my Bucks t-shirt?
00:00:01 Speaker_01
I love your Bucks t-shirt.
00:00:03 Speaker_04
I went for the first time, what, two weeks ago, when I was down for meeting at Benchmark, and the nostalgia in there is just unbelievable.
00:00:10 Speaker_01
I can't believe you hadn't been before. I know Jensen is a Denny's guy, but I feel like he would meet us at Bucks if we asked him.
00:00:17 Speaker_04
Or at the very least, we should figure out some NVIDIA memorabilia to get on the wall at Bucks.
00:00:22 Speaker_01
Totally.
00:00:23 Speaker_04
Fit right in. All right, let's do it.
00:00:25 Speaker_01
Let's do it.
00:00:26 Speaker_02
Who got the truth? Is it you, is it you, is it you who got the truth now? Is it you, is it you, is it you? Sit me down, say it straight, another story on the way
00:00:43 Speaker_04
Welcome to Season 13, Episode 3 of Acquired, the podcast about great technology companies and the stories and playbooks behind them. I'm Ben Gilbert. I'm David Rosenthal. And we are your hosts.
00:00:55 Speaker_04
Today, we tell a story that we thought we had already finished—NVIDIA. But the last 18 months have been so insane, listeners, that it warranted an entire episode on its own.
00:01:06 Speaker_04
So today is a part three for us with NVIDIA telling the story of the AI revolution, how we got here, and why it's happening now, starting all the way down at the level of atoms and silicon.
00:01:18 Speaker_04
So here's something crazy that I did a transcript search on to see if it was true. In our April 2022 episodes, we never once said the word generative. That is how fast things have changed. Unbelievable. Totally crazy.
00:01:32 Speaker_04
And the timing of all of this AI stuff in the world is unbelievably coincidental and very favorable. So recall back to 18 months ago.
00:01:42 Speaker_04
Throughout 2022, we all watched financial markets from public equities to early-stage startups to real estate just fall off a cliff. due to rapid rise in interest rates. The crypto and Web3 bubble burst, banks fail.
00:01:58 Speaker_04
It seemed like the whole tech economy, and potentially a lot with it, was heading into a long winter.
00:02:04 Speaker_01
Including NVIDIA.
00:02:06 Speaker_04
Including NVIDIA, who had that massive inventory write-off for what they thought was overordering.
00:02:12 Speaker_01
Yep. Wow how things have changed.
00:02:16 Speaker_04
But by the fall of 2022, right when everything looked the absolute bleakest, a breakthrough technology finally became useful after years in research labs.
00:02:26 Speaker_04
Large Language Models, or LLMs, built on the innovative transformer machine learning mechanism burst onto the scene
00:02:34 Speaker_04
First, with OpenAI's ChatGPT, which became the fastest app in history to 100 million active users, and then quickly followed by Microsoft, Google, and seemingly every other company. In November of 2022, AI definitely had its Netscape moment.
00:02:51 Speaker_04
And time will tell, but it may have even been its iPhone moment.
00:02:55 Speaker_01
That is definitely what Jensen believes.
00:02:58 Speaker_04
Yep. Well, today we'll explore exactly how this breakthrough came to be, the individuals behind it, and of course, why the entire thing has happened on top of NVIDIA's hardware and software.
00:03:10 Speaker_04
If you want to make sure you know every time there's a new episode, go sign up at acquired.fm slash email. You'll also get access to two things that we aren't putting anywhere else.
00:03:19 Speaker_04
One, a clue as to what the next episode will be, and two, follow-ups from previous episodes from things that we learned after release. You can come talk about this episode with us after listening at Acquire.fm slash slack.
00:03:32 Speaker_04
If you want more of David and I, check out our interview show, ACQ2.
00:03:35 Speaker_04
Our next few episodes are about AI, with CEOs leading the way in this world we are talking about today, and a great interview with Doug DiMiro, where we wanted to talk about a lot more than just Porsche with him.
00:03:48 Speaker_04
But, you know, we only had 11 hours or whatever we had in Doug's garage. So a lot of the car industry chat and learning about Doug and his journey and his business, we saved for ACQ2. So go check it out. One final announcement.
00:04:00 Speaker_04
Many of you have been wondering, and we've been getting a lot of emails, when will those hats be back in stock? Well, they're back. For a limited time, you can get an ACQ embroidered hat at acquired.fm slash store.
00:04:14 Speaker_04
Go put your order in before they go back into the Disney vault forever.
00:04:19 Speaker_01
This is great. I can finally get Jenny one of her own so she stops stealing mine.
00:04:24 Speaker_04
Yes. Well, without further ado, this show is not investment advice. David and I may have investments in the companies we discuss, and this show is for informational and entertainment purposes only. David, history and facts.
00:04:36 Speaker_01
Oh, man. Well, on the one hand, we only have 18 months to talk about.
00:04:40 Speaker_04
except that I know you're not going to start 18 months ago.
00:04:43 Speaker_01
On the other hand, we have decades and decades of foundational research to cover. So when I was starting my research, I went to the natural first place, which was our old episodes from April 2022.
00:04:54 Speaker_01
And I was listening to them and I got to the end of the second one.
00:04:58 Speaker_01
and uh man i had forgotten about this i think jensen maybe wishes we all had forgotten about this in one of nvidia's earning slides in 2021 they put up their total addressable market and they said they had a one trillion dollar tam and the way that they calculated this was that they were going to serve customers who provided a hundred trillion dollars worth of industry and they were going to capture just one percent of it and there was some stuff on the slide that was
00:05:26 Speaker_01
fairly speculative, you know, like autonomous vehicles and the omniverse and I think robotics were a big part of it.
00:05:33 Speaker_04
And the argument is basically like, well, cars plus factories plus all these things added together is a hundred trillion and we can just take 1% of that because surely their compute will amount to 1% of that, which I'm not arguing is wrong, but it is a very blunt way to analyze that market.
00:05:51 Speaker_01
Yeah, it's usually not the right way to think about starting a startup. You know, oh, if we can just get 1% of this big market, blah, blah, blah.
00:05:59 Speaker_04
It's the toppiest down way I can think of to size a market.
00:06:02 Speaker_01
So you, Ben, rightly so, called this out at the end of NVIDIA Part 2 and you're like, You know, I think to justify where Nvidia is trading at the moment, you kind of actually got to believe that all of this is going to happen and happen soon.
00:06:17 Speaker_01
Autonomous cars, robotics, everything.
00:06:20 Speaker_04
Yeah. Importantly, I felt like the way for them to become worth what they were worth at that time literally had to be to power all of this hardware in the physical world.
00:06:30 Speaker_01
Yep. I kind of can't believe that I said this because it was unintentional and uninformed, but I was kind of grasping at straws trying to play devil's advocate for you.
00:06:39 Speaker_01
And we just spent most of that whole episode talking about how machine learning powered by NVIDIA
00:06:47 Speaker_01
ended up having this incredibly valuable use case which was powering social media feed recommenders and that Facebook and Google had grown bigger than anyone ever imagined on the internet with those feed recommendations and Nvidia was powering all of it and so I just sort of idly proposed
00:07:08 Speaker_01
Well, maybe. But what if you don't actually need to believe any of that to still think that NVIDIA could be worth a trillion dollars?
00:07:17 Speaker_01
What if, maybe, just maybe, the internet and software and the digital world are going to keep growing and there will be a new foundational layer that NVIDIA can power? Is that possible? And I think we were both like, yeah, I don't know.
00:07:35 Speaker_01
Let's end the episode.
00:07:36 Speaker_04
Yeah, sure, we shrugged it off and we were like, all right, carve outs.
00:07:38 Speaker_01
But the crazy thing is that, of course, at least in this time frame, most things on Jensen's trillion dollar TAM slide have not come to pass. But that crazy question just might have come to pass.
00:07:51 Speaker_01
And from NVIDIA's revenue and earnings standpoint, definitely has. It's just wild.
00:07:57 Speaker_04
All right, so how did we get here?
00:07:59 Speaker_01
Let's rewind and tell the story. So back in 2012, there was the big bang moment of artificial intelligence, or as it was more humbly referred to back then, machine learning. And that was AlexNet. We talked a lot about this on the last episode.
00:08:17 Speaker_01
It was three researchers from the University of Toronto who submitted the AlexNet algorithm to the ImageNet Computer Science Competition.
00:08:26 Speaker_01
Now, ImageNet was a competition where you would look at a set of 14 million images that had been hand labeled with what the pictures were of, like of a strawberry or a cat or a dog or whatever.
00:08:39 Speaker_04
And David, you were telling me it's the largest ever use of Mechanical Turk up to that point was to label the ImageNet data set.
00:08:46 Speaker_01
Yeah, it's wild. I mean, until this competition and until AlexNet, there was no machine learning algorithm that could accurately label images. So thousands of people on Mechanical Turk got paid however much two bucks an hour to label these images.
00:09:01 Speaker_04
Yeah, and if I'm remembering from our episode, basically what happened is the AlexNet team did way better than anybody else had ever done. The complete step changed better.
00:09:11 Speaker_04
I think the error rate went from mislabeling images 25% of the time to suddenly only mislabeling them 15% of the time. And that was like a huge leap over the tiny incremental progress that had been made along the way.
00:09:23 Speaker_01
You were spot on.
00:09:25 Speaker_01
and the way that they did it and what completely changed the fortunes of the internet of google of facebook and certainly of nvidia was they actually used old algorithms a branch of computer science and artificial intelligence called neural networks specifically convolutional neural networks which had been around since the 60s but
00:09:49 Speaker_01
they were really computationally intensive to train. And so nobody thought it would be practical to actually train and use these things, at least not anytime soon or in our lifetimes.
00:10:01 Speaker_01
And what these guys from Toronto did is they went out probably to their local Best Buy or equivalent in Canada. They bought two GeForce GTX 580s, which were the top of the line cards at the time,
00:10:15 Speaker_01
And they wrote their algorithm, their convolutional neural network, in CUDA, in NVIDIA's software development platform for GPUs. And by God, they trained this thing on like $1,000 worth of consumer-grade hardware.
00:10:30 Speaker_04
And basically, the algorithm that other people had been trying over the years just wasn't massively parallel the way that a graphics card sort of enables.
00:10:38 Speaker_04
So if you actually can consume the full compute of a graphics card, then perhaps you could run some unique novel algorithm and do it on, you know, a fraction of the time and expense that it would take in these supercomputer laboratories.
00:10:53 Speaker_01
Yeah, everybody before was trying to run these things on CPUs. CPUs are awesome, but they only execute one instruction at a time. GPUs, on the other hand, execute hundreds or thousands of instructions at a time.
00:11:07 Speaker_01
So GPUs, NVIDIA, graphics cards, accelerated computing, what Jensen and the company likes to call this, you can really think of it like a giant Archimedes lever. Whatever advances are happening in Moore's law and the number of transistors on a chip,
00:11:23 Speaker_01
If you have an algorithm that can run in parallel, which is not all problem spaces, but many can, then you can basically lever up Moore's law by hundreds of times or thousands of times or today tens of thousands of times and execute something a lot faster than you otherwise could.
00:11:42 Speaker_04
And it's so interesting that there was this first market called graphics that was obviously parallel, where every pixel on a screen is not sequentially dependent on the pixel next to it.
00:11:54 Speaker_04
It literally can be computed independently and output to the screen. So you have however many tens of thousands or now hundreds of thousands of pixels on a screen that can all actually be done in parallel. And little did NVIDIA realize, of course, that
00:12:07 Speaker_04
AI and crypto and all this other linear algebra, matrix math based things that turned into accelerated computing, pulling things off the CPU and putting them on GPU and other parallel processors, was an entire new frontier of other applications that could use the very same technology they had pioneered for graphics.
00:12:26 Speaker_01
Yeah, it was pretty useful stuff. And this Alex Knapp moment and these three researchers from Toronto kicked off, Jensen calls it, and he's absolutely right, the Big Bang moment for AI.
00:12:39 Speaker_04
So David, the last time we told this story in full, we talked about this team from Toronto. We did not follow what this team of three went on to do afterwards.
00:12:49 Speaker_01
Yeah, so basically what we said was it turned out that a natural consequence of what these guys were doing was, oh, actually, you can use this to surface the next post in a social media feed on like an Instagram feed or the YouTube feed or something like that.
00:13:05 Speaker_01
And that unlocked billions and billions of value. And those guys and everybody else working in the field, they all got scooped up by Google and Facebook. Well, that's true.
00:13:15 Speaker_01
And then as a consequence of that, Google and Facebook started buying a lot of NVIDIA GPUs. But turns out there's also another chapter to that story that we completely skipped over. And it starts with the question you asked, Ben. Who are these people?
00:13:29 Speaker_01
Yes. So, the three people who made up the AlexNet team were, of course, Alex Krzyzewski, who was a PhD student, under his faculty advisor, the legendary computer science professor, Jeff Hinton. I have an amazing piece of trivia about Jeff Hinton.
00:13:47 Speaker_01
Do you know who his great-great-grandparents were?
00:13:53 Speaker_04
No, I have no idea.
00:13:54 Speaker_01
He is the great-great-grandson of George and Mary Boole. You know, like Boolean algebra and Boolean logic?
00:14:01 Speaker_04
This guy was born to be a computer science researcher. Oh my God. Right?
00:14:06 Speaker_01
Foundational stuff for computation and computer science.
00:14:10 Speaker_04
I also didn't know there were people named Boole, that that's where that came from. That's hilarious.
00:14:15 Speaker_01
Yeah, you know, the and or X or nor operators that comes from George and Mary wild. So he's the faculty advisor. And then there was a third person on the team. Alex's fellow PhD student in this lab won Ilya Sutskever.
00:14:33 Speaker_01
And if you know where we're going with this, you are probably jumping up and down right now in your seat. Ilya is the co-founder and current chief scientist of OpenAI. Yes. So after AlexNet, Alex, Jeff, and Ilya do the very natural thing.
00:14:50 Speaker_01
They start a company. I don't know what they were doing in the company, but it made sense to start one.
00:14:55 Speaker_04
And whatever they did, it was going to get acquired real fast.
00:14:58 Speaker_01
By Google within six months. So they get scooped up by Google. They join a bunch of other academics and researchers that Google has been monopolizing, really, in the field.
00:15:11 Speaker_01
Three specifically, Greg Corrado, Jeff Dean, and Andrew Ng, the famous Stanford professor. The three of them had just formed the Google Brain team within Google to turbocharge all of this AI work that has been unleashed by AlexNet.
00:15:28 Speaker_01
And of course, to turn it into huge amounts of profit for Google.
00:15:33 Speaker_04
Turns out, individually serving advertising that's perfectly targeted on the internet through Facebook or Google or YouTube is an enormously profitable business and one that consumes a whole lot of NVIDIA GPUs.
00:15:46 Speaker_01
Yes. So about a year later, Google also acquires DeepMind, famously. And then right around the same time, Facebook scoops up computer science professor Jan LeCun, who also is a legend in the field.
00:15:58 Speaker_01
And the two of them basically establish a duopoly on leading AI researchers. Now, at this point, nobody is mistaking what these companies and these people are doing for true human level intelligence or anything close to it.
00:16:14 Speaker_01
This is AI that is very good at narrow tasks, like we talked about social media feed recommendations. So the Google Brain team and Jeff and Alex and Ilya, one of the big projects they work on is redoing the YouTube algorithm.
00:16:30 Speaker_01
This is when YouTube goes from like money losing, you know, crazy thing that Google acquired to the just absolute juggernaut that it is today. I mean, back then in like 2013, 2014, we did our YouTube episode not that long after.
00:16:47 Speaker_01
The majority of views of YouTube videos were embeds on other webpages. This is when they build it into a social media site, they start the feed, they start autoplay. all this stuff is coming out of AI research.
00:17:00 Speaker_01
Some of the other stuff that happens at Google, famously after they acquired DeepMind, DeepMind built a bunch of algorithms to save on cooling costs.
00:17:08 Speaker_01
And Facebook, of course, they probably had the last laugh in this generation because they're using all this work and Yann LeCun is doing his thing and hiring lots of researchers there. This is just a couple of years after they acquired Instagram.
00:17:22 Speaker_01
Man, we need to like go back and redo that episode because Instagram would have been a great acquisition anyway, but it was AI-powered recommendations in the feed that made that into a $100, $200, $500 billion asset for Facebook.
00:17:40 Speaker_04
And I don't think you're exaggerating. I think that is literally what Instagram is worth to meta now. By the way, I have bought a lot of things on Instagram ads so that the targeting works.
00:17:48 Speaker_01
It absolutely does.
00:17:50 Speaker_01
There's this amazing quote from Astro Teller who ran Google X at the time and still does in a New York Times piece where he says that the gains from Google Brain during this period, I don't think this even includes DeepMind, just the gains from the Google Brain team alone in terms of profits to Google
00:18:08 Speaker_01
more than funded everything they were doing in Google X. Which has there ever been anything profitable out of Google X? Google Brain.
00:18:17 Speaker_04
Yeah, I mean, yeah.
00:18:20 Speaker_01
We'll leave it at that. So this takes us to 2015, when a few people in Silicon Valley start to realize that this Google-Facebook AI duopoly is actually a really, really big problem. And most people had no idea about this.
00:18:39 Speaker_01
This is really visionary of these two people.
00:18:42 Speaker_04
And not just a problem for, like, the other big tech companies, because you could make the argument it's a problem because, like, Siri's terrible. All the other companies that have lots of consumer touchpoints have pretty bad AI at the time.
00:18:55 Speaker_04
But the concern is for a much greater reason.
00:18:57 Speaker_01
I think there are three levels of concern here. One, obviously, is the other tech companies. Then there's the problem of startups. This is terrible for startups.
00:19:07 Speaker_01
How are you going to compete with Google and Facebook when this is the primary value driver of this generation of technology?
00:19:16 Speaker_01
There really is another lens to view what happened with Snap, what happened with Musical.ly and having to sell themselves to ByteDance and becoming TikTok and going to the Chinese.
00:19:28 Speaker_01
Maybe it was business decisions, maybe it was execution or whatever that prevented those platforms from getting to independent scale. Snap's a public company now, but like it's no Facebook.
00:19:38 Speaker_01
Maybe it was that they didn't have access to the same AI researchers that Facebook and Google had.
00:19:42 Speaker_04
Hmm. That feels like an interesting question. It's probably a couple steps too far in the conclusion, but still sort of a fun strawman to think about.
00:19:51 Speaker_01
A fun strawman. Nonetheless, this is definitely a problem. The third layer of the problem is just like, this sucks for the world that all these people are locked up in Google and Facebook.
00:20:01 Speaker_04
This is probably a good time to mention this founding of OpenAI was motivated by the desire to find AGI, or Artificial General Intelligence, first, before the big tech companies did. And DeepMind was the same thing.
00:20:13 Speaker_04
It was going to be this winding and circuitous path at the time, since really nobody knew then, or knows now, the best path to get to AGI.
00:20:21 Speaker_04
But the big idea at OpenAI's founding was whoever figures out and finds AGI first will be so big and so powerful so quickly they'll have an immense amount of control and that is best in the open.
00:20:35 Speaker_01
So these two people who are quite concerned about this convene a very fateful dinner in 2015. at of all places. Is it the Rosewood? The Rosewood Hotel on Sand Hill Road, naturally.
00:20:50 Speaker_01
It would have been way better if it were a Denny's or Bucks and Woodside or something like that.
00:20:56 Speaker_04
But it does actually just show like where the seeds of open AI come from. It is very different than this sort of organic scrappy way that the NVIDIA's of the world got started.
00:21:04 Speaker_04
You know, this is powers on high and existing money saying, no, we need to will something into existence.
00:21:09 Speaker_01
Yep. So of course, those two shadowy figures are Elon Musk and Sam Altman, who at the time was president of Y Combinator. So they get this dinner together and they invite basically all of the top AI researchers at Google and Facebook.
00:21:26 Speaker_01
And they're like, yo, what is it going to take for you to leave and to break this duopoly? And the answer from almost all of them is Nothing. You can't. Why would we ever leave? We're happy as clams here.
00:21:43 Speaker_04
We've gotten to hire the people that we want. We've built these great teams. There's a money spigot pointed at our face.
00:21:49 Speaker_01
Right. Not only are we getting paid just ungodly amounts of money. But we get to work directly with the best AI researchers in the field.
00:22:00 Speaker_01
If we were still at academic institutions, you know, say you're at the University of Washington, amazing academic institution for computer science, one of the top in the world, or the University of Toronto, where these guys came from, you're still at a fragmented market.
00:22:14 Speaker_01
If you go to Google or you go to Facebook, you're with everybody. Yep. So the answer is no from basically everybody, except there's one person who's intrigued by Elon and Sam's pitch.
00:22:29 Speaker_01
And to quote an amazing Wired article from the time by Cade Metz that we will link to in our sources, quote, the trouble was, so many of the people most qualified to solve all these AI problems were already working for Google and Facebook.
00:22:42 Speaker_01
And no one at the dinner was quite sure that these thinkers could be lured to a new startup, even if Musk and Altman were behind it. But one key player was at least open to the idea of jumping ship. And then they have a quote from that key player.
00:22:56 Speaker_01
I felt there were risks involved, but I also felt it would be a very interesting thing to try. And that key player was Ilya Sutskever. Yep.
00:23:06 Speaker_01
So after the dinner, Ilya leaves Google and signs up to become, as we said, co-founder and chief scientist of a new independent AI nonprofit research lab backed by Elon and Sam. OpenAI.
00:23:20 Speaker_04
Okay listeners, now is a great time to tell you about longtime friend of the show, ServiceNow.
00:23:26 Speaker_01
Yes, as you know, ServiceNow is the AI platform for business transformation. And they have some new news to share. ServiceNow is introducing AI agents. So only the ServiceNow platform puts AI agents to work across every corner of your business.
00:23:42 Speaker_04
Yep. And as you know from listening to us all year, ServiceNow is pretty remarkable about embracing the latest AI developments and building them into products for their customers. AI agents are the next phase of this.
00:23:54 Speaker_01
So what are AI agents? AI agents can think, learn, solve problems, and make decisions autonomously. They work on behalf of your teams, elevating their productivity and potential.
00:24:06 Speaker_01
And while you get incredible productivity enhancements, you also get to stay in full control.
00:24:11 Speaker_04
Yep. With ServiceNow, AI agents proactively solve challenges from IT to HR, customer service, software development, you name it.
00:24:19 Speaker_04
These agents collaborate, they learn from each other, and they continuously improve, handling the busy work across your business so that your teams can actually focus on what truly matters.
00:24:29 Speaker_01
Ultimately, ServiceNow and agentic AI is the way to deploy AI across every corner of your enterprise. They boost productivity for employees, enrich customer experiences, and make work better for everyone.
00:24:41 Speaker_04
Yep. So learn how you can put AI agents to work for your people by clicking the link in the show notes or going to servicenow.com slash ai-agents. Okay, so David, OpenAI is formed, it's 2015, here we are eight years later and we have ChatGPT.
00:24:59 Speaker_04
Super linear path from there to here, right? Turns out, uh, no.
00:25:05 Speaker_01
So, as we were talking about a little bit, AI at this point in time, super good for narrow use cases. Looks nothing like GPT-4 today. The capabilities that it had were pretty limited.
00:25:24 Speaker_01
And one of the big reasons was that the amount of data that you could practically train these models on was pretty limited. So the AlexNet example, you're talking about 14 million images.
00:25:38 Speaker_01
In the grand scheme of the internet, 14 million images is a drop in the bucket.
00:25:45 Speaker_04
And this was both a hardware and a software constraint. On the software side, we just didn't actually have the algorithms to sort of suppose that we could be so bold to train one single foundational model on the whole internet. Like it wasn't a thing.
00:26:00 Speaker_04
Yeah, that was a crazy idea. Right. People were excited about the concept. of language models, but we actually didn't know how we could algorithmically get it done.
00:26:11 Speaker_04
So in 2015, Andrej Karpathy, who was then at OpenAI and went on to lead AI for Tesla and is actually now back at OpenAI, writes this seminal blog post called The Unreasonable Effectiveness of Neural Networks.
00:26:28 Speaker_04
And David, I don't think we're going to go into it on this episode, but note that recurrent neural networks are a little bit of a different thing than convolutional neural networks, which was the 2012 paper.
00:26:37 Speaker_01
The state of the art had evolved.
00:26:38 Speaker_04
Yes. And right around that same time, there is also a video that hits YouTube a little bit later in 2016 that is actually on NVIDIA's channel. and it has two people in this very short 1 minute and 45 second video.
00:26:53 Speaker_04
One is a young Ilya Sutskever and two is Andrej Karpathy. And here is a quote from Andrej from that YouTube video. One algorithm I'm excited about is a language model.
00:27:06 Speaker_04
The idea that you can take a large amount of data and you feed it into the network and it figures out the pattern in how words follow each other in sentences.
00:27:14 Speaker_04
So for example, you could take a large amount of data on how people talk to each other on the internet, you can train basically a chatbot, but you can do it in a way that the computer learns how language works and how people interact.
00:27:28 Speaker_04
Eventually, we'll use that to talk to computers just like we talk to each other.
00:27:34 Speaker_01
Wow, this is 2015.
00:27:35 Speaker_04
This is two years before the Transformer, while Karpathy is at OpenAI. He both comes up with the idea or espouses the idea of a chat bot. So that sort of had already been discussed.
00:27:47 Speaker_04
But even before we had the Transformer, the method to actually pull this off, he sort of had the idea that there's an important part here. it figures out the pattern in how words follow each other in sentences.
00:28:00 Speaker_04
So there's this idea that the very structure of language and the way to interpret knowledge is actually embedded in the training data itself rather than requiring labeling.
00:28:13 Speaker_01
This is so cool. So at Spring GTC this year, Jensen did a fireside chat with Ilya, and it's amazing. You should go watch the whole thing. But in it, this question comes up.
00:28:26 Speaker_01
Jensen kind of poses as a straw man, like, hey, some people say that GPT 3, 4, chat GPT, everything going on, all these LLMs, they're just probabilistically predicting the next word in a sentence. They don't actually have knowledge.
00:28:43 Speaker_01
And Ilya has this amazing response to that. He says, okay, well, consider a detective novel. Yes.
00:28:51 Speaker_01
At the end of the novel, the detective gathers everyone together in a room and says, I am now going to tell you all the name of the person who committed the crime. And that person's name is... Blank. The more accurately
00:29:09 Speaker_01
an LLM predicts that next word, i.e., the name of the criminal, ipso facto, the greater its understanding, not only of the novel, but of all general human-level knowledge and intelligence, because you need all of your experience in the world and as a human to be able to guess who the criminal is.
00:29:32 Speaker_01
And the LLMs that are out there today GPT-3, GPT-4, LLAMA, BARD, these others, they can guess who the criminal is.
00:29:42 Speaker_04
Ooh, yeah. Put a pin in that. Understanding versus predicting. It's a hot topic du jour. So, David, is now a good time to fast forward two years to 2017 to the Transformer paper?
00:29:55 Speaker_01
Absolutely. Ben, tell us about the Transformer.
00:29:58 Speaker_04
Okay. So, Google 2017 Transformer paper. Paper comes out, it's called Attention is All You Need.
00:30:07 Speaker_01
And it's from the Google Brain team, right? Yes. That Ilya just left.
00:30:11 Speaker_04
Just left, two years before, to start OpenAI. So machine learning on natural language, just to set the table here, had long been used for things like autocorrect or foreign language translation.
00:30:24 Speaker_04
But in 2017, Google came out with this paper and discovered a new model that would change everything for these fields and unlock another one. So here is the scenario. You're translating a sentence from English to French.
00:30:36 Speaker_04
you could imagine that a way to do this would be one word at a time, in order. But for anyone who's ever traveled abroad and tried to do this, you know that words are sometimes rearranged in different languages, so that's a terrible way to do it.
00:30:50 Speaker_04
You know, United States in Spanish is Estados Unidos, so failure on the very first word in that example. So enter this concept of attention, which is a key part of this research paper.
00:31:01 Speaker_04
So this attention, this fairly magical component of the transformer paper, it literally is what it sounds like. It is a way for the model to attend to different areas of the input text at different times. You can look at a large amount of context
00:31:17 Speaker_04
while considering what word to pick next in your translation.
00:31:21 Speaker_04
So for every single word that you're about to output in French, you can look over the entire set of inputted words to figure out what words you should weight heavily in your decision for what to do next.
00:31:35 Speaker_01
This is why AI and machine learning was so narrowly applicable before. If you anthropomorphize it and you think of it like a human, it was like a human with a very, very short attention span.
00:31:47 Speaker_04
Yes. Now, here's the magical part. While it does look at the whole input text to consider what the next word should be, it doesn't mean that it throws away the notion of position entirely.
00:31:58 Speaker_04
It uses a technique called positional encoding so it doesn't forget the position of the words altogether. So it's got this cool thing where it weights the important part relevant to your particular word and it still understands position.
00:32:12 Speaker_04
So remember I said the attention mechanism looks over the entire input every time it's picking what word to output.
00:32:19 Speaker_01
That sounds very computationally hard.
00:32:21 Speaker_04
Yes. In computer science terms, this means that the attention mechanism is O of n squared.
00:32:27 Speaker_01
Oh, that's given me the heebie-jeebies back to my intro CS classes in college.
00:32:31 Speaker_04
Oh, just wait till we get through this episode. It gets deeper. So, obviously, yes, traditionally you'd say this is very, very inefficient, and it actually means that the larger your context window, aka token limit, aka prompt length, gets,
00:32:45 Speaker_04
the more computationally expensive it gets on a quadratic basis. So doubling your input means quadrupling the cost to compute an output, or tripling your input means nine times the cost. It gets real gnarly. Yeah, it gets real expensive real fast.
00:33:01 Speaker_04
But, GPUs to the rescue. The amazing news for us here is that these transformer comparisons can be done in parallel.
00:33:09 Speaker_04
So even though there are lots of them to do, if you have big GPU chips with tons of cores, you can do them all at exactly the same time.
00:33:18 Speaker_04
And previous technologies to accomplish this, like recurrent neural networks or LSTMs, long short-term memory networks, which is a type of recurrent neural network, etc.
00:33:28 Speaker_04
Those required knowing the output of each step before beginning the next one, before you picked the next word. So in other words, they were sequential since they depended on the previous word.
00:33:40 Speaker_04
Now, with transformers, even if your string of text that you're inputting is a thousand words long, it can happen just as quickly in humid, measurable time as if it were 10 words long, supposing that there were enough cores in that big GPU.
00:33:53 Speaker_04
So the big innovation here is you could now train sequence-based models in a parallel way. You couldn't train models of this size at all before, let alone cost-effectively.
00:34:04 Speaker_01
Yeah, this is huge and probably for all listeners out there starting to sound very familiar to the world that we live in today.
00:34:13 Speaker_04
Yeah, I sort of did a sleight of hand there, morphing translation to using words like context window and token length. You can kind of see where this is going.
00:34:20 Speaker_01
Yep. So this Transformer paper comes out in 2017. The significance is huge. But for whatever reason, there's a window of time where the rest of the world doesn't quite realize it. So Google obviously knows how important this is.
00:34:38 Speaker_01
And there's like a year where Google's AI work even though Ilya has left and OpenAI is a thing now, accelerates again beyond anybody else in the field.
00:34:49 Speaker_01
So this is when Google comes out with Smart Compose in Gmail, and they do that thing where they have an AI bot that'll call local businesses for you. Remember that demo from IO that they did?
00:35:01 Speaker_04
Did that ever ship?
00:35:02 Speaker_01
I don't know. Maybe it did, maybe. I mean, this is Google here. The capabilities are there. The product sense, not as much. This is when they really start investing in Waymo.
00:35:13 Speaker_01
But again, where it really manifests is just back to serving ads in search and recommending YouTube videos. They're just crushing it in this period of time. OpenAI and everyone else though, they haven't adopted transformers yet.
00:35:25 Speaker_01
They're kind of stuck in the past. And they're still doing these really researchy computer vision projects. So like this is when they build a bot to play Dota 2, Defense of the Agents 2, the video game.
00:35:37 Speaker_04
And super impressive stuff, like they beat the best Dota players in the world at Dota by literally just consuming computer vision, like consuming screenshots and inferring from there.
00:35:47 Speaker_04
And that's a really hard problem because Dota 2 is not a game where you get to see the whole board at once. So it has to do a lot of like really intelligent construction of the rest of the game based on just a single player's worth of input.
00:35:59 Speaker_04
So it's unbelievably cutting edge research.
00:36:02 Speaker_01
For the past generation. It's a faster horse, basically.
00:36:05 Speaker_04
Maybe, yeah. I mean, they were also doing stuff like Universe, which was the 3D modeled world to train self-driving cars. You don't really hear anything about that anymore, but they built this whole thing.
00:36:16 Speaker_04
I think it was using Grand Theft Auto as the environment, and then it was doing computer vision training for cars using the GTA world. I mean, it was crazy stuff, but it was kind of scattershot.
00:36:29 Speaker_01
Yeah, it was scattershot. And I guess what I'm saying is it was still in this narrow use case world. They weren't doing anything approaching GPT at this point in time. Meanwhile, Google had kind of moved on. Yep.
00:36:42 Speaker_01
Now, one thing I do want to say in defense of OpenAI and everybody else in the field at the time, they didn't just have their heads in the sand.
00:36:52 Speaker_01
To do what Transformers enabled you to do, which Ben you're going to talk about in a sec, cost a lot in computing power.
00:37:01 Speaker_01
GPUs and NVIDIA and the Transformer made it possible, but to work with the size of models you're talking about, you're talking about spending an amount of money that's certainly for a nonprofit and anybody really except Google, was untenable.
00:37:19 Speaker_04
Right. It's funny, David, you made this leap to expensive and large models. All we were doing before was merely talking about translating one sentence to another.
00:37:27 Speaker_04
The application of a transformer does not necessarily require you to go and consume the whole internet and create a foundational model. But let's talk about this. Transformers lend themselves quite well, as we now know, to a different type of task.
00:37:41 Speaker_04
So for a given input sentence, instead of translating to a target language, they can also be used as next word predictors to figure out what word should come next in a sequence.
00:37:52 Speaker_04
You could even do this idea of pre-training with some corpus of text to help the model understand how it should go about predicting that next word.
00:38:01 Speaker_04
So, backing up a little bit, let's go back to the recurrent neural networks, the state-of-the-art before transformers. Well, they had this problem.
00:38:09 Speaker_04
In addition to the fact that they were sequential rather than parallel, they also had a very short context window. So, you could do a next-word predictor But it wasn't that useful because it didn't know what you were saying more than a few words ago.
00:38:24 Speaker_04
By the time you'd get to the end of the paragraph, it would forget what was happening at the beginning. It couldn't sort of hold on to all that information at the same time.
00:38:32 Speaker_04
So this idea of a next word predictor that was pre-trained with a transformer could really start to do something pretty powerful, which is consume large amounts of text and then complete the next word based on a huge amount of context.
00:38:46 Speaker_00
Yep.
00:38:47 Speaker_04
We're starting to come up to this idea of a large language model, and we're going to flash forward here just for a moment to do some illustration, and then we'll come back to the story.
00:38:56 Speaker_04
In GPT-1, the first open AI model, this generative pre-trained transformer model, GPT, it used unsupervised pre-training, which basically meant that as it was consuming this corpus of language, it was unlabeled data.
00:39:12 Speaker_04
The model was inferring the structure and meaning of language merely by reading it, which is a very new concept in machine learning.
00:39:23 Speaker_04
The canonical wisdom is that you needed extremely structured data to train your smallish model on, because how else are you going to learn what the data actually means? This was a new thing. You can learn what the data means from the data itself.
00:39:36 Speaker_04
It's like how a child consumes the world, where only occasionally does their parent say, no, no, no, you have that wrong. That's actually the color red. But most of the time, they're just self-teaching by observing the world.
00:39:47 Speaker_01
As a parent of a two-year-old can confirm.
00:39:50 Speaker_04
And then a second thing happens after this unsupervised pre-training step, where you then have supervised fine-tuning.
00:39:58 Speaker_04
The unsupervised pre-training used a large corpus of text to learn the sort of general language, and then it was fine-tuned on labeled datasets for specific tasks that you sort of really want the model to be actually useful for.
00:40:11 Speaker_01
So to give people a sense of why we're saying that the idea of training on very, very, very large amounts of data here is crazy expensive, GPT-1 had roughly 120 million parameters that it was trained on. GPT-2 had 1.5 billion.
00:40:33 Speaker_01
GPT-3 had 175 billion, and GPT-4 OpenAI hasn't announced, but it's rumored that it has about 1.7 trillion parameters that it was trained on. This is a long way from AlexNet here.
00:40:49 Speaker_04
It's scaling like Nvidia's market cap. There is this interesting discovery, basically, that the more parameters you have, the more correctly you can predict the next word. These models were basically bad sub-10 billion parameters.
00:41:06 Speaker_04
I mean, maybe even sub-100 billion parameters. They would just hallucinate or they would be nonsensical. It's funny when you look at some of the, like, 1 billion parameter models, you're like, there is no chance that turns into anything useful ever.
00:41:16 Speaker_04
But by merely adding more training data and more parameters, It just gets way, way better. There's this weirdly emergent property where transformer based models scale really well due to the parallelism.
00:41:31 Speaker_01
So as you throw huge amounts of data at training them, you can also throw huge amounts of NVIDIA GPUs at processing that.
00:41:38 Speaker_04
Exactly. And the output sort of unexpectedly gets magically better. I mean, I know I keep saying that, but it is like
00:41:46 Speaker_04
wait, so we don't change anything about the structure, we just give it way more data and let it run these models for a long time and make the parameters of the model way bigger?
00:41:54 Speaker_04
And like, no researchers expected them to reason about the world as well as they do, but it just kind of happened as they were exploring larger and larger models.
00:42:04 Speaker_01
So in defense of OpenAI, they knew all this, but the amount of money that you would have to spend to buy GPUs or to rent GPUs in the cloud to train these models is prohibitively expensive.
00:42:19 Speaker_01
And, you know, even Google at this point in time, this is when they start building their own chips, TPUs, because they're still buying tons of hardware from NVIDIA, but they're also starting to source their own here.
00:42:31 Speaker_04
Yep. And importantly, they've at this point are getting ready to release TensorFlow to the public. So they have a framework where people can develop for stuff.
00:42:38 Speaker_04
And they're like, look, if people are developing using our software, then maybe it should run on our hardware that's optimized to work with that software.
00:42:45 Speaker_04
So they actually do have this very plausible story around why their hardware, why their software framework.
00:42:51 Speaker_04
It was kind of a surprising move when they open sourced it because people were like, gasp, you know, why is Google giving away the farm for free here? But
00:42:58 Speaker_04
This was three, four years early and a very prescient move to really get a lot of people using Google architecture compute at scale.
00:43:06 Speaker_01
Yep. All within Google Cloud.
00:43:09 Speaker_04
Yep.
00:43:10 Speaker_01
So with this, it starts to look like maybe this whole open AI boondoggle didn't actually accomplish anything and the world's AI resources are more than ever just locked back into Google.
00:43:22 Speaker_01
So in 2018, Elon gets super frustrated by all this, basically throws a hissy fit and quits and pieces out of OpenAI. There's a lot of drama around this that we're not going to cover now.
00:43:36 Speaker_01
He may or may not have given an ultimatum to the rest of the team that he would either take over and run things or leave. Who knows? It's Elon.
00:43:43 Speaker_01
But whatever happened, this turns out to be a major catalyst for the rest of the OpenAI team and truly a history-turning-on-a-knife-point moment. It was also a probably super bad decision by Elon, but again, story for another day.
00:43:59 Speaker_04
So there's this great explanation of what happened in the semaphore piece that we'll link to in our sources.
00:44:05 Speaker_04
The author says, that fall, it became even more apparent to some people at OpenAI that the costs of becoming a cutting-edge AI company were going to go up. Google Brain's transformer had blown open a new frontier where AI could improve endlessly.
00:44:18 Speaker_04
But that meant feeding endless data to train it, a costly endeavor. OpenAI made a big decision to pivot toward these transformer models.
00:44:26 Speaker_04
On March 11, 2019, OpenAI announced it was creating a for-profit entity so it could raise enough money to pay for all the compute power necessary to pursue the most ambitious AI models.
00:44:37 Speaker_04
We want to increase our ability to raise capital while still serving our mission, and no pre-existing legal structure that we know of strikes the right balance, the company wrote at the time.
00:44:46 Speaker_04
OpenAI said it was capping profits for investors, with any excess going back to the original nonprofit. Less than six months later, OpenAI took a $1 billion investment from Microsoft.
00:44:57 Speaker_01
Yeah, and I believe this is mostly, if not all, due to Sam Altman's influence and taking over here.
00:45:06 Speaker_01
You know, on the one hand, you can look at this sort of skeptically and say, okay, Sam, you took your nonprofit and you converted it into an entity worth $30 billion today.
00:45:17 Speaker_01
On the other hand, knowing this history now, this was kind of the only path they had. They had to raise money to get the computing resources to compete with Google. And Sam goes out and does these landmark deals with Microsoft.
00:45:33 Speaker_04
Yeah, truly amazing. And their opinion at the time of why they're doing this is basically this is going to be super expensive.
00:45:40 Speaker_04
We still have the same mission to ensure that artificial general intelligence benefits all of humanity, but it's going to be ludicrously expensive to get there.
00:45:47 Speaker_04
And so we need to basically be a for-profit enterprise and a going concern and have a business that funds our research eventually to pursue that mission.
00:45:56 Speaker_01
Yep, so 2019, they do the conversion to a for-profit company. Microsoft invests a billion dollars, as you say, and becomes the exclusive cloud provider for OpenAI, which is going to become highly relevant here for NVIDIA. More on that in a minute.
00:46:13 Speaker_01
June of 2020, GPT-3 comes out. In September of 2020, Microsoft licenses exclusive commercial use of the underlying model for Microsoft products. 2021, GitHub Copilot comes out. Microsoft invests another $2 billion in OpenAI.
00:46:29 Speaker_01
And then of course, this all leads to November 30th, 2022, in Jensen's words, the AI heard around the world, OpenAI comes out with chat GPT. As you said, Ben, the fastest product in history to reach a hundred million users.
00:46:45 Speaker_01
In January, 2023, this year, Microsoft invests another $10 billion in OpenAI, announces they're integrating GPT into all of their products and then in May of this year GPT-4 comes out and that basically catches us up to today.
00:47:03 Speaker_01
We eventually need to go do a whole nother episode about all the details here of OpenAI and Microsoft. But for today, the salient points are, one, thanks to all this, generative AI as a user-facing product emerges as this enormous opportunity.
00:47:22 Speaker_01
Two, to facilitate that happening, you needed enormous amounts of GPU compute, obviously benefiting NVIDIA.
00:47:32 Speaker_01
But just as important, three, it becomes obvious now that the predominant way that companies are gonna access and provide that compute is through the cloud.
00:47:45 Speaker_01
And the combination of those three things turns out to be basically the single greatest moment that could ever happen for NVIDIA.
00:47:54 Speaker_04
Yes. So you're teeing all of this up. And so far I'm thinking, so this is like the OpenAI and Microsoft episode. Like, what does this have to do with NVIDIA? And God, there's a great NVIDIA story here to be told. So let's get to the NVIDIA side of it.
00:48:11 Speaker_01
All right, listeners, our next sponsor is a new friend of the show, Huntress. Huntress is one of the fastest growing and most loved cybersecurity companies today.
00:48:21 Speaker_01
It's purpose-built for small to mid-sized businesses and provides enterprise-grade security with the technology, services, and expertise needed to protect you.
00:48:30 Speaker_04
They offer a revolutionary approach to managed cybersecurity that isn't only about tech, it's about real people providing real defense around the clock.
00:48:39 Speaker_01
So how does it work? Well, you probably already know this, but it has become pretty trivial for an entry-level hacker to buy access and data about compromised businesses.
00:48:50 Speaker_01
This means cybercriminal activity towards small and medium businesses is at an all-time high.
00:48:54 Speaker_04
So Huntress created a full managed security platform for their customers to guard from these threats.
00:49:01 Speaker_04
This includes endpoint detection and response, identity threat detection response, security awareness training, and a revolutionary security information and event management product that actually just got launched.
00:49:14 Speaker_04
Essentially, it is the full suite of great software that you need to secure your business,
00:49:19 Speaker_04
plus 24-7 monitoring by an elite team of human threat hunters in a security operations center to stop attacks that really software-only solutions could sometimes miss.
00:49:30 Speaker_04
Huntress is democratizing security, particularly cybersecurity, by taking security techniques that were historically only available to large enterprises and bringing them to businesses with as few as 10, 100, or 1,000 employees at price points that make sense for them.
00:49:45 Speaker_01
In fact, it's pretty wild. There are over 125,000 businesses now using Huntress, and they rave about it from the hilltops.
00:49:55 Speaker_01
They were voted by customers in the G2 rankings as the industry leader in endpoint detection and response for the eighth consecutive season and the industry leader in managed detection and response again this summer.
00:50:07 Speaker_04
Yep. So if you want cutting-edge cybersecurity solutions backed by a 24-7 team of experts who monitor, investigate, and respond to threats with unmatched precision, head on over to huntress.com slash acquired or click the link in the show notes.
00:50:22 Speaker_04
Our huge thanks to Huntress. Okay, so NVIDIA.
00:50:27 Speaker_01
Okay, so we just said these three things that we've painted the picture of on the first part of the episode here that A, generative AI is like possible a thing and it's now getting traction.
00:50:38 Speaker_01
B, it requires an unbelievably massive amount of GPU compute to train. And three, it looks like the predominant way that companies are going to use that compute is gonna be in the cloud. The combination of these three things
00:50:55 Speaker_01
is I think the most perfect example we've ever covered on this show of the old saying about luck being what happens when preparation meets opportunity for NVIDIA here.
00:51:04 Speaker_01
So obviously the opportunity is generative AI, but the preparation front, NVIDIA has literally just spent the past five years working insanely hard to build a new computing platform for the data center.
00:51:23 Speaker_01
a GPU-accelerated computing platform to, in their minds, replace the old CPU-led, Intel-dominated x86 architecture in the data center. And for many years, I mean, they were getting some traction, right?
00:51:41 Speaker_01
And the data center segment was growing for NVIDIA, but people were like, okay, you want this to happen, but like, why is it going to happen?
00:51:49 Speaker_04
Right. There's these little workloads here and there that will toss you Jensen that we think can be accelerated by your cool GPUs.
00:51:56 Speaker_04
And then you know crazy things like crypto happened and there was like AI researchers in academic labs that are using it as you know supercomputers. But for the longest time the data center segment of NVIDIA
00:52:08 Speaker_04
It just wasn't clear that organizations had enormous parts of their software stack that they were going to shift to GPUs. Like, why? What's driving this? And now we know what could be driving it, and that is AI.
00:52:22 Speaker_01
Not only could be, but if you look at their most recent quarter, absolutely freaking is.
00:52:29 Speaker_04
Okay, so now it begs the question, why is it driving it? And David, are you open to me giving a little computer science lecture on computer architecture? Ooh, please do. All right, I need to do my best professor impression here.
00:52:45 Speaker_01
Dude, I loved computer science in college. They were my favorite classes.
00:52:50 Speaker_04
I will say doing these episodes, this TSMC, it really does bring back the thrill of being in a CS lecture and being like, oh, that's how that works. Like, it's just really fun.
00:53:01 Speaker_04
So let's take a step back and consider the classic computer architecture, the von Neumann architecture.
00:53:09 Speaker_04
Now, the von Neumann architecture is what most computers, most CPUs, are based on today, where they can store a program in the computer's memory and run that program. You can imagine why this is the dominant architecture.
00:53:23 Speaker_04
Otherwise, we'd need a computer that is specialized for every single task.
00:53:28 Speaker_04
The key thing to know is that the memory of the computer can store two different things, the data that the program uses and the instructions of the program itself, the literal lines of code.
00:53:41 Speaker_04
And in this example we're about to paint, all of this is wildly simplified because I don't want to get into caching and speeds of memory and, you know, where memory is located, not located. So let's just keep it simple.
00:53:54 Speaker_04
So the processor in the von Neumann architecture executes this program written in assembly language, which is the language that compiles down to the bytecode that the processor itself can speak.
00:54:06 Speaker_04
So it's written in an instruction set architecture, an ISA from ARM, for example.
00:54:12 Speaker_01
Or Intel before that.
00:54:14 Speaker_04
Yes. And each line of the program is very simplistic. So we're going to consider this example where I'm going to use some assembly language pseudocode to add the numbers 2 and 3 to equal 5.
00:54:28 Speaker_01
Ben, are you about to program live on Acquired?
00:54:31 Speaker_04
Well, it's pseudo assembly language code. So the first line is we're going to load the number 2 from memory. We're going to fetch it out of memory and we're going to load it into a register on the processor.
00:54:47 Speaker_04
So now we've got the number two actually sitting right there on our CPU ready to do something with. That's line of code number one. Two, we're going to load the number three in exactly the same fashion into a second register.
00:55:00 Speaker_04
So we've got two CPU registers with two different numbers.
00:55:03 Speaker_04
The third line, we're going to perform an add operation, which performs the arithmetic to add the two registers together on the CPU and store the value in some either third register or into one of those registers.
00:55:17 Speaker_04
So that's a more complex instruction since it's arithmetic that we actually have to perform. But these are the things that CPUs are very good at doing math operations on data fetched from memory.
00:55:29 Speaker_04
And then the fourth and final line of code in our example is we are going to take that five that has just been computed and is currently held temporarily in a register on the CPU, and we're going to write that back to an address in memory.
00:55:42 Speaker_04
So the four lines of code are load, load, add, store.
00:55:45 Speaker_01
This all sounds familiar to me.
00:55:47 Speaker_04
So you can see each of those four steps is capable of performing one and only one operation at a time. And each of these happens with one cycle of the CPU. So if you've heard of gigahertz, that's the number of cycles per second.
00:56:00 Speaker_04
So a one gigahertz computer could handle the simple program that we just wrote 250 million times in a single second. But you can see something going on here. Three of our four clock cycles are taken up by loading and storing data to memory.
00:56:16 Speaker_04
Now, this is known as the von Neumann bottleneck, and it is one of the central constraints of AI, or at least it has been historically. Each step must happen in order and only one at a time.
00:56:31 Speaker_04
So in this simple example, it actually would not be helpful for us to add a bunch more memory to this computer. I can't do anything with it. It's also only incrementally helpful to increase the clock speed.
00:56:42 Speaker_04
If I double the clock speed, I can only execute the program twice as fast. If I need like a million X speed up for some AI work that I'm doing, I'm not going to get it there with just a faster clock speed. That's not going to do it.
00:56:54 Speaker_04
And it would of course be helpful to increase the speed at which I can read and write to memory, but I'm kind of bound by the laws of physics there. There's only so fast that I can transmit data over a wire
00:57:06 Speaker_04
Now, the great irony of all of this is that the bottleneck actually gets worse over time, not better, because the CPUs get faster, and the memory size increases, but the architecture is still limited, so this one pesky single channel, known as a bus, I don't actually get to enjoy the performance gains nearly as much as I should, because I'm jamming everything through that one channel, and it only gets to sort of be used one time per every clock cycle.
00:57:33 Speaker_04
So the magical unlock, of course, is to make a computer that is not a von Neumann architecture, to make programs executable in parallel and massively increase the number of processors or cores.
00:57:46 Speaker_04
And that is exactly what NVIDIA did on the hardware side, and all these AI researchers figured out how to leverage on the software side.
00:57:56 Speaker_04
But interestingly, now that we've done that, David, the constraint is not the clock speed or the number of cores anymore. For these absolutely enormous language models, it's actually the amount of on-chip memory that concerns us.
00:58:10 Speaker_01
I thought you were going. And this is why the data center and what NVIDIA has been doing is so important.
00:58:16 Speaker_04
Yes. There's this amazing video that we'll link to on the Asianometry YouTube channel that we link to also on the TSMC episode. But the constraint today is actually in how much high performance memory is available on the chip.
00:58:29 Speaker_04
These models need to be in memory all at the same time, and they take up hundreds of gigabytes. So while memory has scaled up, I mean, we're going to get flashing all the way forward, the H100s on-chip RAM is like 80 gigabytes.
00:58:44 Speaker_04
The memory hasn't scaled up nearly as fast as the models have actually scaled in size.
00:58:50 Speaker_04
The memory requirements for training AI are just obscene, so it becomes imperative to network multiple chips and multiple servers of chips and multiple racks of servers of chips together into one single computer, and I'm putting computer in air quotes there, in order to actually train these models.
00:59:09 Speaker_04
It's also worth noting, we can't make the memory chips any bigger. Due to a quirk of the extreme ultraviolet photolithography that we talked about, the EUV, on the TSMC episode, chips are already the full size of the reticle.
00:59:23 Speaker_04
It's a physics and wavelength constraint. You really can't etch chips larger without some new invention that we don't have commercially viable yet.
00:59:32 Speaker_04
So what it ends up meaning is you need huge amounts of memory, very close to the processors, all running in parallel with the fastest possible data transfer.
00:59:41 Speaker_04
And again, this is a vast oversimplification, but you kind of get the idea of why all of this becomes so important.
00:59:49 Speaker_01
Okay, so back to the data center and here's what NVIDIA is doing that I don't think anybody else out there is doing and why it's so important for them that all of this new generative AI world, this new computing era as Jensen dubs it, runs in the data center.
01:00:08 Speaker_01
So NVIDIA has done three things over the last five years. One, and probably most importantly related to what you're talking about Ben, They made one of the best acquisitions of all time back in 2020, and nobody had any idea.
01:00:24 Speaker_01
They bought a quirky little networking company out of Israel called Mellanox.
01:00:31 Speaker_04
Well, it wasn't little. They paid $7 billion for it.
01:00:33 Speaker_01
Yeah. And it was already a public company, right? It was. Yep. Yep. But it was definitely quirky. Now, what was Mellanox?
01:00:41 Speaker_01
Mellanox's primary product was something called InfiniBand, which we talked about a lot with Chase Lockmiller on our ACQ2 episode with him from Crusoe.
01:00:51 Speaker_04
And actually, InfiniBand was an open source standard or managed by a consortium. There were a bunch of players in it.
01:00:59 Speaker_04
But the traditional wisdom was, well, InfiniBand is way faster, way higher bandwidth, a much more efficient way to transfer data around a data center. At the end of the day, Ethernet is the lowest common denominator.
01:01:12 Speaker_04
And so everyone had to implement Ethernet anyway. And so most companies actually exited the market. And Mellanox was kind of the only InfiniBand spec provider left.
01:01:22 Speaker_01
Yeah. So you said, wait, what is InfiniBand? It is a competing standard to Ethernet. It is a way to move data between racks in a data center. And back in 2020, everybody was like, Ethernet's fine.
01:01:38 Speaker_01
Why do you need more bandwidth than Ethernet between racks in a data center? What could ever require 3,200 gigabits a second of bandwidth running down a wire in a data center?
01:01:51 Speaker_01
Well, it turns out if you're trying to address hundreds, maybe more than hundreds of GPUs as one single compute cluster to train a massive AI model, yeah, you want really fast data interconnects between them.
01:02:05 Speaker_04
Right. People thought, oh, sure, for supercomputers, for these academic purposes. But what the enterprise market needs in my shared cloud computing data center is Ethernet. And that's fine. And most workloads are going to happen right there on one rack.
01:02:18 Speaker_04
And maybe, maybe, maybe things will expand to multiple computers on that rack. But certainly, they won't need to network multiple racks together. And NVIDIA steps in. And you got Jensen saying, hey, dummies, the data center is the computer.
01:02:34 Speaker_04
Listen to me when I tell you the whole data center needs to be one computer. And when you start thinking that way, you start thinking, geez, we're really going to be cramming huge amounts of data through wires that are going between these racks?
01:02:50 Speaker_04
Like, how can we sort of think about them as if it's all sort of on-ship memory, or as close as we can make it to on-ship memory, even though that's in a box located three feet away?
01:03:01 Speaker_01
Yep. So that's piece number one of NVIDIA's grand data center plan over the last five years. Piece number two is in September 2022, NVIDIA makes a quite surprising announcement of a new chip.
01:03:20 Speaker_01
Not just a new chip, an entirely new class of chips that they are making called the Grace CPU processor. NVIDIA is baking a CPU. This is like heretical.
01:03:33 Speaker_04
But Jensen, I thought all computing was going to be accelerated. What are we doing here on these ARM CPUs?
01:03:40 Speaker_01
Yeah, these grace CPUs are not for putting in your laptop. They are for being the CPU component of your entire data center solution that is specifically from the ground up design to orchestrate with these massive GPU clusters.
01:04:00 Speaker_04
This is the endgame of a ballet that has been in motion for 30 years. Remember when the graphics card was subservient to the PCIe slot in Intel's motherboard?
01:04:11 Speaker_04
And then eventually, you know, we fast forward to the future, Nvidia makes these GPUs that are these beautiful standalone boxes in your data center, or perhaps these little workstations that sit next to you while you're doing graphics programming, while you're
01:04:23 Speaker_04
directly programming your GPU. And then, of course, they need some CPU to put in that, so they're using AMD or Intel, or they're licensing some CPU. And now they're saying, you know what? We're actually just going to do the CPU, too.
01:04:35 Speaker_04
So now we make a box, and it's a fully integrated NVIDIA solution with our GPUs, our CPUs, our NVLink between them, our InfiniBand to network it to other boxes, and, you know, welcome to the show.
01:04:50 Speaker_01
One more piece to talk about the third leg of the stool there strategy before we get to what it all means that I think you're about to go to spoiler alert. You say solution. I hear gross margin. The third part of it is the GPUs.
01:05:05 Speaker_01
Up until NVIDIA's current GPU generation, the hopper generation of GPUs for the data center, there was only one GPU architecture at NVIDIA. And that same architecture and those same chips from the same wafers made at TSMC
01:05:24 Speaker_01
Some of them went to consumer gaming graphics cards, and some of those dies went to A100 GPUs in the data center. It was all the same architecture. Starting in September of 2022, They broke out the two business lines into different architectures.
01:05:42 Speaker_01
So there's the Hopper architecture named after great computer scientist, Grace Hopper, I think, Rear Admiral in the US Navy, Grace Hopper. Get it? Grace, CPU, Hopper, GPU, Grace Hopper, the H100s. That was for the data centers.
01:05:59 Speaker_01
And then on the consumer side, they start a whole new architecture called Lovelace after Ada Lovelace. And that is the RTX40XX. So you buy, you know, top of the line RTX40, what have you gaming card right now.
01:06:12 Speaker_01
That is no longer the same architecture as the H100s that are powering chat GPT. It's got its own architecture. This is a really big deal because what they do with the hopper architecture is they start using what's called chip on wafer on substrate.
01:06:27 Speaker_01
C-O-W-O-S.
01:06:29 Speaker_04
COOS. When you start talking to the real semi-nerds, that's when they start busting out the COOS conversation.
01:06:34 Speaker_01
This is when a certain segment of our listeners are going to get really excited. So, essentially what this is, back to this whole concept of memory being so important for GPUs and for AI workloads.
01:06:48 Speaker_01
This is a way to stack more memory on the GPU chips themselves, essentially by going vertical in how you build the chips. This is the absolute bleeding edge technology that is coming out of TSMC.
01:07:04 Speaker_01
And by NVIDIA bifurcating their chip architectures into a gaming segment that does not have this latest CoWAS technology, this allows them to monopolize like a huge amount of TSMC's capacity to make the CoWAS chips specifically for these H100s, which allows them to have way more memory than other GPUs on the market.
01:07:29 Speaker_04
Yes. So this gets to the point of why can't they seem to make enough chips right now? Well, it's literally a TSMC capacity problem.
01:07:36 Speaker_04
So there's these two components that are extremely related that you're talking about, the co-auth chip on wafer on substrate and the high bandwidth memory.
01:07:43 Speaker_04
So there's this great post from Semianalysis where the author points out a 2.5D chip, which is basically how you assemble this co-auth stuff to get the memory really close to the processor.
01:07:55 Speaker_04
And of course, 2.5D, it is literally 3D, but 3D means something else. It's even more 3D, so they came up with this 2.5D denominator.
01:08:04 Speaker_04
Anyway, the 2.5D chip packaging technology from TSMC is where you take multiple active silicon dies, like the logic chips and the stack of high bandwidth memory, and they stack them on one piece of silicon.
01:08:18 Speaker_04
And there's more complexity here, but the important thing is COAS is the most popular technology for GPUs and AI accelerators for packaging these chips. And it's the primary method to co-package high bandwidth memory
01:08:33 Speaker_04
Again, remember, think back to the thing that's most important right now is get as much high bandwidth memory as you can closest to the CPU, next to the logic to get the most performance for trading and inference.
01:08:43 Speaker_04
So COAS represents right now, about 10 to 15% of TSMC's capacities. And many of the facilities are custom built for exactly these types of chips that they're producing.
01:08:54 Speaker_04
So when NVIDIA needs to reserve more capacity, there's a pretty good chance that they've already reserved some large part of the 10 to 15 percent of TSMC's total footprint, and TSMC needs to, like, go make more fabs in order for NVIDIA to have access to more COAS-capable capacity.
01:09:12 Speaker_01
Yeah, which as we know, it takes years for TSMC to do this.
01:09:17 Speaker_04
Yep. There are more experimental things that are happening. Like I would be remiss not to mention, there are actually experiments of doing compute in memory.
01:09:27 Speaker_04
Like as we shift away from von Neumann and sort of all bets are off now that we're open to new computing architectures, there are people exploring, well, what if we just process the data where it is in memory instead of doing the very lossy, expensive,
01:09:41 Speaker_04
energy-intensive thing of moving data over the copper wire to get it to the CPU.
01:09:46 Speaker_04
All sorts of trade-offs in there, but it is very fun to sort of dive into the academic computer science world right now, where they really are rethinking, like, what is a computer?
01:09:55 Speaker_01
So, these three things that NVIDIA has been building, the dedicated Hopper data-centered GPU architecture, the Grace CPU platform, the Mellanox-powered networking stack, they now have a full suite solution for generative AI data centers.
01:10:17 Speaker_01
And Ben, when I say solution, I hear margins.
01:10:21 Speaker_04
But let's be clear, you don't need to offer some sort of solution to get high margins if you're NVIDIA. Price is set where supply meets demand, and they're adding as much supply as they possibly can right now.
01:10:32 Speaker_04
Like, believe me, for all sorts of reasons, NVIDIA wants everyone who wants H100s to have H100s. But for now, the price is kind of like a, I'll write you a blank check and NVIDIA, you write whatever you want on the check.
01:10:45 Speaker_04
So their margins are crazy right now, just literally because there's way more demand than supply for these things.
01:10:52 Speaker_01
Yes. Okay. So let's break down what they're actually selling. So like you were saying, Ben, of course you can, and lots of people do just go buy H100s. You're like, I don't care about the grace CPU. I don't care about this Mellanox stuff.
01:11:06 Speaker_01
I'm running my own data center. I'm really good at it.
01:11:08 Speaker_04
And the people who are most likely to do this are the hyperscalers, or as NVIDIA refers to them, the CSPs, the cloud service providers.
01:11:15 Speaker_01
This is AWS, this is Azure, this is Google, this is Facebook for their internal use.
01:11:21 Speaker_04
Like NVIDIA, don't give me one of these DGX servers that you assemble, just give me the chip and I will integrate it the way that I want to integrate it.
01:11:29 Speaker_01
I am a world-class data center architect and operator. I don't want your solution. I just want your chips. So they sell a lot of those.
01:11:39 Speaker_01
Now, NVIDIA, of course, has also been seeding new cloud providers out there in the ecosystem, like our friends at Crusoe, also CoreWeave and Lambda Labs, if you've heard of them.
01:11:50 Speaker_01
These are all new GPU-dedicated clouds that NVIDIA is working closely with. So they're selling H100s and A100s before that to all these cloud providers.
01:12:00 Speaker_04
But let's say you are an arbitrary company in the Fortune 500 that is not a technology company. And my God, do you not want to miss the boat on generative AI and you've got a data center of your own? Well, NVIDIA has a DGX for you.
01:12:14 Speaker_01
Yes, they do. Full GPU-based supercomputer solution in a box that you can just plug right into your data center and it just works. There's nothing else on the market like this.
01:12:26 Speaker_04
And it all runs CUDA. It is all speaking the exact language of the entire ecosystem of developers that know exactly how to write software for this thing.
01:12:35 Speaker_01
Which means that whatever developers you already had who were working on AI or anything else, everything they were working on is just going to come right over and run within your brand new shiny AI supercomputer because it all runs CUDA.
01:12:50 Speaker_04
Amazing.
01:12:50 Speaker_01
More on CUDA in a minute, but as we said, you say solution, I hear gross margin. NVIDIA sells these DGX systems for like $150,000 to $300,000 a box. That's wild. And now with all these three new legs of the stool, Hopper, Grace, and Mellanox,
01:13:10 Speaker_01
These systems are just getting way more integrated, way more proprietary and way better. So if you want to buy a new top of the line DGX-H100 system, the price starts at $500,000 for one box.
01:13:27 Speaker_01
And if you want to buy the DGX-GH200 SuperPod, this is the AI wall that Jensen recently unveiled, the huge like room full of AI.
01:13:38 Speaker_04
And it's like 20 racks wide. Imagine an entire row at a data center.
01:13:43 Speaker_01
Yes, this is 256 Grace Hopper DGX racks all connected together in one wall. They're billing this as the first turnkey AI data center that you can just buy and can train a trillion parameter GPT-4 class model. The pricing on that is, call us.
01:14:05 Speaker_01
Of course it is. But I'm imagining like hundreds of millions of dollars. Like, I doubt it's a billion, but hundreds of millions easily.
01:14:13 Speaker_04
Wild. Well, let's talk about the H100. I've got the baseball card right here on this insane thing that they've built. So they launched it in September 2022. It's the successor to the A100. One GPU, one H100 costs $40,000.
01:14:28 Speaker_04
So that's how you get to that price point you're talking about
01:14:30 Speaker_01
That's what they're selling to Amazon and Google and Facebook.
01:14:34 Speaker_04
Right. And you mentioned that $500,000 price point. The $500,000 is the eight $40,000 H100s in a box with the gray CPU and, you know, the nice bow around it. Yep. Which do the math on that. So eight times 40,000, that's $320,000.
01:14:43 Speaker_04
So that's essentially an extra $180,000 of margin that Nvidia is getting out of selling the solution.
01:15:00 Speaker_01
It's an ARM CPU. It doesn't cost them anything to make that.
01:15:03 Speaker_04
And these $40,000 H100s have margin of their own. So like every time they bundle more, there's more margin in the fully assembled. I mean, that's literally bundle economics.
01:15:14 Speaker_04
You are entitled to margin when you bundle more things together and provide more value for customers. But just to like illustrate the way that this pricing works.
01:15:21 Speaker_04
So the reason you want an H100 is they're 30 times faster than an A100, which, mind you, is only like two and a half years older. It is nine times faster for AI training.
01:15:32 Speaker_04
The H100 is literally purpose-built for training LLMs, like the full self-driving video stuff. It's super easy to scale up. It's got 18,500 CUDA cores. Remember when we were talking about the von Neumann example earlier?
01:15:49 Speaker_04
That is one computing core that is able to handle, you know, those four assembly language instructions. This one H100, which they're calling AGPU, has 18,500 cores that are capable of running CUDA software.
01:16:05 Speaker_04
It's got 640 tensor cores, which are highly specialized for matrix multiplication. They have 80 streaming multiprocessors. So what are we up to here? Close to 20,000 unique cores on this thing. It's got meaningfully higher energy usage than the A100.
01:16:23 Speaker_04
I mean, a big takeaway here is that NVIDIA is massively increasing the power requirement every time they come out with the next generation. They're both figuring out how to push the edge of physics, but they're also constrained by physics.
01:16:35 Speaker_04
Some of this stuff is only possible with way more energy. This thing weighs 70 pounds. This is one H100.
01:16:43 Speaker_01
Jensen makes a big deal about this every keynote that he gives. Like, oh, I can't lift it.
01:16:48 Speaker_04
It's got a quarter trillion transistors across 35,000 parts. It requires robots to assemble it. Not only does it require physical robots to assemble it, it requires AI to design it. They're actually using AI to design the chips themselves now.
01:17:04 Speaker_04
I mean, they have completely reinvented the notion of what a computer is.
01:17:09 Speaker_01
Totally. And this is all part of Jensen's pitch here to customers. Yes. our solutions are very expensive. However, he uses the line that he loves, the more you buy, the more you save. If you could get your hands on some. Right.
01:17:26 Speaker_01
But what he means by that is like, okay, say you're McDonald's and you're trying to build a generative AI so that, I don't know, customers can order something. You're using it in your business.
01:17:38 Speaker_01
If you were going to try and build and run that in your existing data center infrastructure, It would take so much time and cost you so much more over the long run in compute than if you just went and bought my super pod here.
01:17:52 Speaker_01
You can plug and play and have it up and running in a month.
01:17:55 Speaker_04
Yep. And by the fact that this is all accelerated computing, the things you're doing on it, you literally wouldn't be able to do otherwise or might take you a lot more energy, a lot more time, a lot more cost.
01:18:04 Speaker_04
There is a very valid story to buying and running your workloads here or renting from any of the cloud service providers and running your workloads here is more performant because the results just happen much faster, much cheaper, or at all.
01:18:17 Speaker_01
Yep, you mentioned energy here, like this is also Jensen's argument. He's like, yes, these things take a ton of energy, but the alternative takes even more energy. So we are actually saving energy if you assume this stuff is going to happen.
01:18:33 Speaker_01
Now, there's a bit of caveat here in that it can't happen except on these types of machines, so he enabled this whole thing, but he has a point.
01:18:42 Speaker_04
Oh, I totally buy it, though. I mean, I think there's a very real case around, look, you only have to train a model once, and then you can do inference on it over and over and over again.
01:18:51 Speaker_04
I mean, the analogy I think makes a lot of sense for model training is to think about it as a form of compression. LLMs are turning the entire internet of text into a much smaller set of model weights.
01:19:06 Speaker_04
This has the benefit of storing a huge amount of usefulness in a small footprint, but also enabling a very inexpensive amount of compute, again, relatively speaking, in the inference step for every time that you need to prompt that model for an answer.
01:19:19 Speaker_04
Of course, the trade-off you're making there is once you encode all of the training data into the model, it is very expensive to redo it.
01:19:25 Speaker_04
So you better do it right the first time or figure out little ways to modify it later, which a lot of ML researchers are working on. But I always think a reasonable comparison here is to compress a zillion-layer Photoshop file.
01:19:37 Speaker_04
For anybody that's ever dealt with, oh, I've got a three-gigabyte Photoshop file. Well, that's not a thing you're going to send to a client. You're going to compress it into a JPEG and you're going to send that.
01:19:47 Speaker_04
And the JPEG is in many ways more useful as a compressed facsimile of the original layers comprising the Photoshop file. But the tradeoff is you can never get from that compressed little JPEG back to the original thing.
01:19:59 Speaker_04
So I think the analogy here is like you're saving everyone from needing to make the full PSD every time because you can just use the JPEG the vast, vast majority of the time.
01:20:09 Speaker_01
So hopefully we've now painted a relatively coherent picture of both the advances that made the generative AI opportunity possible, that it has truly become a real opportunity,
01:20:25 Speaker_01
and why NVIDIA, even above the obvious reasons, was just so well positioned here, particularly because of the data center-centric nature of these workloads, and that they had been working so hard for the past five years to fundamentally re-architect the data center.
01:20:43 Speaker_00
Yep.
01:20:44 Speaker_01
So, on top of all this, NVIDIA recently announced yet another pretty incredible piece of their cloud strategy here.
01:20:53 Speaker_01
So today, like we've been saying, if you want to use H100s and A100s, say you're an AI startup, the way you're probably going to do that is you're going to go to a cloud, either a hyperscaler or a dedicated GPU cloud like Crusoe or CoreWeaver, Lambda Labs or the like, and you're going to rent your GPUs.
01:21:14 Speaker_01
And Ben, you did some research on this. So like, what does that cost?
01:21:16 Speaker_04
Oh, I just looked at the pricing pages on public clouds today. I think Azure and AWS were where I looked. You can get access to a DGX server that's eight A100s for about $30 an hour.
01:21:27 Speaker_04
Or you can go over to AWS and get a P5.48x large instance, which is eight H100s, which I believe is an HGX server for about $100 an hour. So about three times as much. And again, when I say you can get access, I don't actually mean you can get access.
01:21:45 Speaker_04
I mean, that's the price.
01:21:46 Speaker_01
Right. If you could get access, that's what you would pay for it. Correct. Okay, that's just getting the GPUs.
01:21:54 Speaker_01
But if you buy everything we were talking about a minute ago, say your McDonald's or UPS or whoever, and you're like, you know, I really like, Jensen, I buy what you're selling. I want this whole integrated package.
01:22:06 Speaker_01
I want an AI supercomputer in a box that I can plug into my wall and have it run. But I'm all in on the cloud. I don't run my own data centers anymore. NVIDIA has now introduced DGX Cloud.
01:22:22 Speaker_04
Yeah, and of course you could rent these instances from Amazon, Microsoft, Google, Oracle, but like... You're not getting that full integrated solution. Right.
01:22:33 Speaker_04
And you're getting some integration the way that the cloud service provider wants to create the integration using their proprietary services.
01:22:41 Speaker_04
And to be honest, you might not have the right people on staff to be able to deal with this stuff in a pseudo bare metal way.
01:22:49 Speaker_04
even if it's not in your data center and you're renting it from the cloud, you might actually need, based on your workforce, to just use a web browser and just use a real nice, easy web interface to load some models in from a trusted source that you can easily pair with your data and just click run and not have to worry about any of the complexity of managing a cloud application that's in Amazon or Microsoft or something a little bit scarier and closer to the metal.
01:23:17 Speaker_01
Yep, so NVIDIA has introduced DGX Cloud, which is a virtualized DGX system that is provided to you right now via other clouds, so Azure and Oracle and Google.
01:23:31 Speaker_04
Right, the boxes are sitting in the data centers of these other CSPs.
01:23:35 Speaker_01
Right, they're sitting in the other cloud service providers. But as a customer, it looks like you have your own box that you're renting.
01:23:45 Speaker_04
You log into the DGX Cloud website through NVIDIA and it's all nice WYSIWYG stuff. There's an integration with Hugging Face where you can easily deploy models right off of Hugging Face. You can upload your data.
01:23:59 Speaker_04
Like everything is just really WYSIWYG is probably the way to describe it.
01:24:04 Speaker_01
This is unbelievable. NVIDIA launched their own cloud service through other clouds.
01:24:12 Speaker_04
And NVIDIA does have, I think, six data centers, but that, I don't believe, is what they're actually using to back DGX Cloud.
01:24:19 Speaker_01
No, so starting price for DGX Cloud is $37,000 a month, which will get you an A100 based system, not an H100 based system. So the margins on this are insane for Nvidia and their partners.
01:24:36 Speaker_01
A listener helped us out and estimated that the cost to actually build an equivalent A100 DGX system would be today something like 120k.
01:24:47 Speaker_01
Remember this is the previous generation this is not h100s and you can rent it for 37k a month so that's three month payback on the capex for this stuff for nvidia and their cloud partners together and even more for nvidia more important longer term for enterprises that buy this
01:25:08 Speaker_01
NVIDIA now has a direct sales relationship with those companies, not necessarily intermediated by sales through Azure or Google or AWS, even though the compute is sitting in their clouds.
01:25:24 Speaker_04
Which is crucially important because at this point, the CFO Colette Kress said on their last earnings call that about half of the revenue from the data center business unit is CSPs.
01:25:36 Speaker_04
And then I believe after that is the consumer internet companies and after that is enterprises. So there's a few interesting things in there.
01:25:44 Speaker_04
One of which is, oh my God, their revenue for this is concentrated among like five to eight companies with these CSPs. Two, they don't necessarily own the customer relationship. They own the developer relationship through CUDA.
01:25:58 Speaker_04
You know, they've got this unbelievable ecosystem right now of NVIDIA developers that's stronger than ever. But in terms of the actual customer, half of their revenue is intermediated by cloud providers.
01:26:08 Speaker_04
The second interesting thing about this is even today, in this AI explosion, the second biggest segment of data centers is still the consumer internet companies.
01:26:19 Speaker_04
It's still all that stuff we were talking about before of the uses of machine learning to figure out what should show up in your social media algorithms and match ads to you.
01:26:28 Speaker_04
That's actually bigger than all of the direct enterprises who are buying from NVIDIA. So the DGX Cloud Play is a way to sort of shift some of that CSP revenue into direct relationship revenue.
01:26:41 Speaker_01
So all of this brings us to 2023. In May of this year, NVIDIA reported their Q1 fiscal 24 earnings. NVIDIA is on this weird January fiscal year end thing. So Q1 24 is essentially Q1 23.
01:26:59 Speaker_01
But anyway, in which revenue was up 19% quarter over quarter to 7.2 billion, which is great because remember they had a terrible end of 2022. with the write-offs and crypto falling off a cliff and all that.
01:27:13 Speaker_04
Yes, it's amazing that in that Stratechery interview, when was that? In March of 2023, Jensen said last year was unquestionably a disappointing year. This is the year ChatGPT was released. It is wild the roller coaster this company has been on.
01:27:32 Speaker_01
The time frame is so compressed here.
01:27:35 Speaker_04
And part of that, of course, is Ethereum moving to proof of stay, the end of the crypto thing for Nvidia, which I'm sure they're actually thrilled about.
01:27:42 Speaker_04
But part of it was they also put in a ton of preorders for capacity with TSMC that then they thought they weren't going to need. So they had to write down.
01:27:52 Speaker_04
So from an accounting perspective, it looks like a big loss, like a really big blemish on their finances last year. But now, oh my god, are they glad that they reserved all that capacity.
01:28:02 Speaker_01
Yep, it's actually going to be quite valuable. So speaking of, you know, this Q1 earnings is like great, up 19% quarter over quarter. But then they dropped the bombshell due to unprecedented demand for generative AI compute in data centers.
01:28:23 Speaker_01
NVIDIA forecasts Q2 revenue of $11 billion, which would be up another 53% quarter over quarter over Q1 and 65% year over year. The stock goes nuts.
01:28:38 Speaker_04
25% in after-hours training.
01:28:40 Speaker_01
Yep.
01:28:41 Speaker_04
This is a trillion-dollar company, or at least this made them a trillion-dollar company, but a company that was previously valued at around $800 billion popped 25% after earnings.
01:28:54 Speaker_01
Well, and it's even crazier than that. Back when we did our episodes last April, NVIDIA was the eighth largest company in the world by market cap, had about a $660 billion market cap.
01:29:05 Speaker_01
That was down slightly off the highs, but that was kind of the order of magnitude back then. It crashed down below $300 billion, and then within a matter of months, it's now back up over a trillion. Just wild.
01:29:19 Speaker_01
And then all of this culminates last week at the time of this recording when NVIDIA reports Q2 Fiscal 24 earnings.
01:29:29 Speaker_01
And this earnings release, we usually don't talk about like individual earnings releases on Acquired because like in the long arc of time, who cares? This was a historic event.
01:29:39 Speaker_01
I think this was one of, if not the most incredible earnings release by any scaled public company ever. Seriously, no matter what happens going forward, last week was a historic moment.
01:29:54 Speaker_04
The thing that blows my mind the most is that their data center segment alone did $10 billion in the quarter. That's more than doubling off of the previous quarter.
01:30:07 Speaker_04
In three months, they grew from $4-ish billion to $10 billion of revenue in that segment. And revenue only happens when they deliver products to customers. This isn't pre-orders.
01:30:19 Speaker_04
This isn't clicks, this isn't wave your hands around stuff, this is we delivered stuff to customers and they paid us an additional $6 billion this quarter than they did last quarter.
01:30:30 Speaker_01
So here are the full numbers. For the quarter, total company revenue of $13.5 billion, up 88% from the previous quarter and over 100% from a year ago. And then Ben, like you said, in the data center segment, revenue of $10.3 billion.
01:30:46 Speaker_01
So $10.3 out of $13.5 for a segment that basically didn't exist five years ago for the company. That's up 141% from Q1 and 171% from a year ago. This is $10 billion. That kind of growth at this scale. I've never seen anything like it.
01:31:05 Speaker_01
No, neither has the market.
01:31:07 Speaker_04
That's right.
01:31:08 Speaker_01
And so this, this is the first time I noticed it. Jensen had talked about this in Q1 earnings, so it wasn't the first time, but he brings back the trillion dollar TAM. Not in a slide. I think this time he just talks about it.
01:31:22 Speaker_04
No, but in a new way that I think is a better way to slice it.
01:31:25 Speaker_01
This time it's different. You know, look, we'll spend a while here now talking about what we think about this, but this is very different. This time he frames NVIDIA's trillion dollar opportunity as the data center. And this is what he says.
01:31:39 Speaker_01
There is $1 trillion worth of hard assets sitting in data centers around the world right now.
01:31:47 Speaker_04
Growing at $250 billion a year.
01:31:50 Speaker_01
annual spend on data centers to update and add to that capex is 250 billion dollars a year and nvidia has certainly the most cohesive fulsome and coherent platform to be the future of what those data centers are going to look like for a large amount of compute workloads
01:32:17 Speaker_01
This is a very different story than like, oh, we're going to get 1% of this hundred trillion dollars of industry out there.
01:32:24 Speaker_04
And the thing you have to believe now, because whenever someone paints a picture, you say, OK, what do I have to believe?
01:32:28 Speaker_04
The thing you have to believe is there is real user value being created by these AI workloads and the applications that they are creating. And there's pretty good evidence.
01:32:40 Speaker_04
I mean, chat GPT made it so open AI is rumored to be doing over a billion dollar run rate now, maybe multiple single digit billions and still growing meaningfully. And so that is like the shining example.
01:32:54 Speaker_04
Again, that's the Netscape navigator here of this whole boom. But the bet, especially with all these Fortune 500s, is that there are going to be GPT like experiences in everyone's private applications in a zillion other public interfaces.
01:33:12 Speaker_04
I mean Jensen frames it as in the future every application will have a GPT front end. It will be a way that you decide that you want to interact with computers that is more natural. And I don't think he means like versus clicking buttons.
01:33:27 Speaker_04
I think he means everyone can kind of become a programmer, but the programming language is English. And so when you're sort of like, well, why is everyone spending all of this money?
01:33:37 Speaker_04
It is that the world's executives with the purchasing power to go write a $10 billion check last quarter to Nvidia for all this stuff.
01:33:46 Speaker_04
wholeheartedly believes from the data they've seen so far that this technology is going to change the world enough for them to make these huge bets. And the thing that we don't know yet is, is that true?
01:33:59 Speaker_04
Is the GPT-like experiences going to be an enduring thing for the far future or not?
01:34:05 Speaker_04
There's pretty good evidence so far that people like this stuff and that it's quite useful in transforming the way that, you know, everyone lives their lives and goes about day to day and does their jobs and goes through school and, you know, on and on and on.
01:34:18 Speaker_04
But that is the thing you have to believe. We want to thank our longtime friend of the show, Vanta, the leading trust management platform. Vanta, of course, automates your security reviews and compliance efforts.
01:34:32 Speaker_04
So frameworks like SOC 2, ISO 2701, GDPR, and HIPAA compliance and monitoring, Vanta takes care of these otherwise incredibly time and resource draining efforts for your organization and makes them fast and simple.
01:34:44 Speaker_01
Yep, Fanta is the perfect example of the quote that we talk about all the time here on Acquired, Jeff Bezos, his idea that a company should only focus on what actually makes your beer taste better, i.e.
01:34:55 Speaker_01
spend your time and resources only on what's actually going to move the needle for your product and your customers and outsource everything else that doesn't. Every company needs compliance and trust with their vendors and customers.
01:35:06 Speaker_01
It plays a major role in enabling revenue because customers and partners demand it. But yet it adds zero flavor to your actual product.
01:35:13 Speaker_04
Vanta takes care of all of it for you. No more spreadsheets, no fragmented tools, no manual reviews to cobble together your security and compliance requirements.
01:35:20 Speaker_04
It is one single software pane of glass that connects to all of your services via APIs and eliminates countless hours of work for your organization.
01:35:29 Speaker_04
There are now AI capabilities to make this even more powerful, and they even integrate with over 300 external tools. Plus, they let customers build private integrations with their internal systems.
01:35:40 Speaker_01
And perhaps most importantly, your security reviews are now real-time instead of static, so you can monitor and share with your customers and partners to give them added confidence.
01:35:48 Speaker_04
So whether you're a startup or a large enterprise, and your company is ready to automate compliance and streamline security reviews like Vanta's 7,000 customers around the globe, and go back to making your beer taste better, head on over to vanta.com slash acquired and just tell them that Ben and David sent you.
01:36:04 Speaker_04
And thanks to friend of the show, Christina, Vanta's CEO, all Acquired listeners get $1,000 of free credit. Vanta.com slash Acquired. Okay, so David, analysis. We got to talk about CUDA before we start analyzing anything else here.
01:36:21 Speaker_04
Talked about a lot of hardware so far on this episode, but there's this huge piece of the NVIDIA puzzle that we haven't talked about since part two. And CUDA, as folks know, was the initiative started in 2006.
01:36:35 Speaker_04
by Jensen and Ian Buck and a bunch of other folks on the NVIDIA team to really make a bet on scientific computing, that people could use graphics cards for more than just graphics, and they would need great software tools to help them do that.
01:36:48 Speaker_04
It also was the glimmer in Jensen's eye of, ooh, maybe I can build my own relationship with developers.
01:36:54 Speaker_04
And, you know, there can be this notion not of a Microsoft or an Intel developer who happens to be able to, you know, have a standard interface to my chip, but I can have my own developer ecosystem, which has been huge for the company.
01:37:05 Speaker_04
So, CUDA has become the foundation that everything that we've talked about, all the AI applications, are written on top of today.
01:37:16 Speaker_04
So, you know, you hear Jensen in these keynotes reference CUDA the platform, CUDA the language, and I spent some time trying to figure out, like, when I was watching developer sessions and, like, literally learning some CUDA programs, what is the right way to characterize it?
01:37:30 Speaker_01
And what is the right way to characterize it today? Because it has evolved a lot.
01:37:33 Speaker_04
Yes. So today, CUDA is, starting from the bottom and going up, a compiler, a runtime, a set of development tools like a debugger and a profiler. It is its own programming language, CUDA C++. It has industry-specific libraries.
01:37:50 Speaker_04
It works on every card that they ship and have shipped since 2006. which is a really important thing to know. And if you're a CUDA developer, your stuff works on everything, anything NVIDIA, all this unified interface.
01:38:04 Speaker_04
It has many layers of abstractions and existing libraries that are optimized. So these libraries of code that you can call to keep your development work short and simple instead of reinventing the wheel.
01:38:15 Speaker_04
So, you know, there are things that you can decide that you want to write in C++ and just rely on their compiler to make it run well on
01:38:23 Speaker_04
NVIDIA hardware for you, or you can write stuff in their native language and try to implement things yourself in CUDA C++.
01:38:30 Speaker_04
The answer is, it's incredibly flexible, it is very well supported, and there's this huge community of people that are developing with you and building stuff for you to build on top of.
01:38:44 Speaker_04
If you look at the number of CUDA developers over time, it was released in 2006, It took four years to get the first 100,000 people. Then by 2016, 13 years in, they got to a million developers. Then just two years later, they got to two million.
01:39:02 Speaker_04
So 13 years to add their first 13 million, then two years to add their second. 2022, they hit three million developers, and then just one year later, in May of 2023, CUDA has four million registered developers. So at this point,
01:39:18 Speaker_04
There's a huge moat for Nvidia. And I think when you talk to folks there, and frankly, when we did talk to folks there, they don't describe it this way. They don't think about it like, well, CUDA is our moat versus competitors.
01:39:30 Speaker_04
It's more like, well, look, we envisioned a world of accelerated computing in the future. And we thought there are way more workloads that should be parallelized and made more efficient that we want people to run on our hardware.
01:39:42 Speaker_04
And we need to make it as easy as possible for them to do that. And we're going to go to great lengths and have
01:39:49 Speaker_04
1-2,000 people that work at our company that are going to be full-time software engineers building this programming language and compiler and foundation and framework and everything on top of it to let the maximum number of people build on our stuff.
01:40:02 Speaker_04
That is how you build a developer ecosystem. It's different language, but the bottom line is they have a huge reverence for the power that it gives them at the company.
01:40:11 Speaker_01
This is something we touched on in our last episode, but has really crystallized for me in doing this one. NVIDIA thinks of themselves as, and I believe is, a platform company, especially
01:40:27 Speaker_01
this week after the blowout earnings and everything that happened this quarter and the stock and whatnot. Sort of a popular take out there that you've been seeing a lot is, oh, we've seen this movie before. This happened with Cisco.
01:40:40 Speaker_01
You could say over a longer timescale, this happened with Intel. Yeah, these hardware providers, these semiconductor companies, they're hot when they're hot and people want to spend capex and then when they're not hot, they're not hot.
01:40:55 Speaker_01
But I don't think that's quite the right way to characterize NVIDIA. They do make semiconductors and they do make data center gear, but really they are a platform company. The right analogy for NVIDIA also is Microsoft. They make the operating system.
01:41:14 Speaker_01
They make the programming environment. They make many of the applications.
01:41:18 Speaker_04
Right. Cisco doesn't really have developers. Intel never had developers. Microsoft had developers, and Intel had Microsoft, but Intel didn't have developers. NVIDIA has developers.
01:41:30 Speaker_04
I mean, they've built a new architecture that is not a von Neumann computer. They've bucked 50 years of progress, and instead every GPU has a stream processor unit
01:41:40 Speaker_04
And as you'd imagine, you need a whole new type of programming language and compiler and everything to deal with this new computing model. And that's CUDA, and it freaking works. And there's all these people that develop their livelihood in it.
01:41:53 Speaker_01
You talk to Jensen, and you talk to other people at the company, and they will tell you, we are a foundational computer science company. We're not just slinging hardware here.
01:42:02 Speaker_04
Yeah, I mean, it's interesting. They're a platform company for sure. They're also a systems company. They're effectively selling mainframes. I mean, it's not that different than IBM way back when.
01:42:11 Speaker_04
They're trying to sell you a $100 million wall that goes in your data center. And it's all fully integrated and it all just works.
01:42:20 Speaker_01
Yeah, and maybe IBM actually is a really good analogy, like old school IBM here.
01:42:24 Speaker_01
They make the underlying technology, they make the hardware, they make the silicon, they make the operating system for the silicon, they make the solutions for customers, they make everything, and they sell it as a solution.
01:42:38 Speaker_04
Yep. Okay, so a couple other things to catch us up here as we're starting analysis. One big point I want to make is, let's look at a timeline, because I didn't discover this until like two hours before we started recording.
01:42:50 Speaker_04
In March of 2019, NVIDIA announced they were acquiring Mellanox for $7 billion in cash. And I think Intel was considering the purchase and then NVIDIA came in and kind of blew them out of the water.
01:43:02 Speaker_04
And it is fair to say nobody really understood what NVIDIA was going to do there and why it was so important, but the question is why?
01:43:09 Speaker_04
Well, NVIDIA knew that these new models coming out would need to run across multiple servers, multiple racks, and they put a huge level of importance on the bandwidth between the machines. And, of course, how did they know that?
01:43:23 Speaker_04
Well, in August of 2019, NVIDIA released what was at the time the largest transformer-based language model called Megatron.
01:43:33 Speaker_04
8.3 billion parameters trained on 512 GPUs for nine days, which at the time at retail would have cost something like half a million dollars to train, which at the time was a huge amount of money to spend on model training, which is
01:43:46 Speaker_04
what only four years ago, but today that's quaint Nvidia did that because they do a huge amount of research at the company and they work with every other company doing AI research and they were like, Oh, yes, this stuff is going to work and this stuff is going to require the fastest networking available.
01:44:04 Speaker_04
And I think that has to do with why no one else saw how valuable the Mellanox technology could be. Yep. Another thing that I want to talk about for NVIDIA's business today is this notion of the data center is the computer.
01:44:18 Speaker_04
And Jensen did a great interview with Ben Thompson last year where he talks about the idea that they build their systems full stack, like their dream is that you own and operate a DGX super pod.
01:44:33 Speaker_04
And he says, we build our systems full stack, but we go to market in a disaggregated way, integrating into the compute fabric of the industry. So I think that's his sort of way of saying, Look, customers need to use us in a bunch of different ways.
01:44:48 Speaker_04
So we need to be flexible on that. But we want to build each of our components such that if you do assemble them all together, it's this unbelievable experience.
01:44:56 Speaker_04
And we'll figure out how to provide the right experience to you if you only want to use them in piecemeal ways, or you want to use us in the cloud, or the cloud providers want to use us.
01:45:05 Speaker_04
Again, it's build the product as a system, build the system full stack, but go to market in a disaggregated way.
01:45:12 Speaker_01
And I think if I remember right in that interview Ben picked up on this and was like, wait, are you building your own cloud? And Jensen was like, well, maybe we'll see.
01:45:20 Speaker_01
And of course, then they launched DGX Cloud in a, well, maybe we'll see sort of way.
01:45:25 Speaker_04
Yeah, you could imagine there are more NVIDIA data centers likely on the way that are fully owned and operated. Speaking of all of this, we got to talk some numbers on margin. This last quarter, they had a gross margin of 70%.
01:45:41 Speaker_04
And they forecasted for next quarter to have a gross margin of 72%. I mean, if you go back pre-CUDA, when they were a commoditized graphics card manufacturer, it was 24%. So they've gone 24 to 70 on gross margin.
01:45:57 Speaker_04
And with the exception of a few quarters along the way for these strange one-time events, it's basically been a linear climb quarter over quarter as they've deepened their moat and as they've deepened their differentiation in the industry.
01:46:09 Speaker_04
We're definitely at a place right now that I think is temporary due to the supply shortage of the world's enterprises, and in some cases, even governments.
01:46:19 Speaker_04
You look at the UK or some of the Middle Eastern countries, like blank check, I just need access to Nvidia hardware. That's gonna go away, but I don't think this very high, you know, 65% plus margin is gonna erode too much.
01:46:35 Speaker_01
Yes, I mean, I think two things here. One, I really do believe what we were talking about a minute ago that NVIDIA is not just a hardware company. They're not just a chips company. They are a platform company.
01:46:47 Speaker_01
And there is a lot of differentiation baked into what they do. If you want to train GPT or a GPT class model, there's one option. You're doing it on NVIDIA. There's one option.
01:46:58 Speaker_01
And yes, we should talk about, there's lots of less than GPT class stuff out there that you can do. And especially inference is more of a wide open market versus training that you can do on other platforms, but they're the best.
01:47:09 Speaker_01
And they're not just the best because of their hardware. They're not just the best because of their data center solutions. They're not just the best because of CUDA. They're the best because of all of those.
01:47:19 Speaker_01
So the other sort of illustrative thing for me that shows how wide their lead is, we haven't talked about China yet.
01:47:27 Speaker_04
The Land of Eight Hundreds.
01:47:29 Speaker_01
Yes, so what's going on? Last year, China was 25% or sales to mainland China was 25% of NVIDIA's revenue. And a lot of that is they were selling to the hyperscalers, to the cloud providers in China, Baidu, Alibaba, Tencent, others.
01:47:46 Speaker_04
And by the way, Baidu has potentially the largest model of anyone. Their GPT competitor is over a trillion parameters and may actually be larger than GPT-4.
01:47:57 Speaker_01
Wow, I didn't know that. Yep. That's wild. So then, I believe also in September of 2022, last year, the Biden administration announced pretty sweeping regulations and bans on sales of advanced computing infrastructure.
01:48:14 Speaker_04
David, their export controls don't say bans.
01:48:18 Speaker_01
I mean, yes, that's a fine line. This is pretty close to bans, what the administration introduced. As part of that, NVIDIA can no longer sell their top of the line H100s or A100s to anybody in China.
01:48:33 Speaker_01
So they created a nerfed SKU, essentially, that meets the regulations, the performance regulations, the A800 and H800s.
01:48:43 Speaker_04
which I think they basically just crank down the NVLink's data transfer speeds. So it's like buying a top-of-the-line A100, but not with as fast of data connections as you need, which basically makes it so you can't train large models.
01:48:59 Speaker_01
Or you can't train them as well or as fast as you could with the latest stuff. The incredibly telling thing is that those chips and those machines are still selling like hotcakes in China.
01:49:10 Speaker_01
They're still the best hardware and platform that you can get in China. even a crippled version. And I think that's true anywhere in the world.
01:49:19 Speaker_04
And there's been even a more recent spike of them because a lot of Chinese companies are reading the tea leaves and saying, ooh, export controls might get even more severe, so I should get them while I still can, these A800s.
01:49:31 Speaker_01
Yep. So, I mean, I can't think of a better illustration of just how wide their lead is.
01:49:37 Speaker_04
Yeah, that's a great point. Talking about the rest of NVIDIA, just for a moment, I mean, this episode is about the data center segment, but... Oh, you mean they still make gaming cards too?
01:49:46 Speaker_04
It is worth talking about this idea that Omniverse is starting to look really interesting. As of their conference six months ago, they had 700 enterprises who had signed up as customers.
01:49:56 Speaker_04
And the reason this is interesting is it could be where their two different worlds collide. 3D graphics with ray tracing, which is new and amazing, and the demos are mind-blowing, and AI.
01:50:08 Speaker_04
They have been playing in both of these markets since the workloads are both massively parallelizable. That is the sort of original reason for them to be in the AI market.
01:50:18 Speaker_04
If you recall back to way back our part one episode, the original mission of NVIDIA was to make graphics a storytelling medium.
01:50:25 Speaker_04
And then their mission has expanded as they've realized, my God, our hardware is really good at other stuff that needs to be parallelized too. But fascinatingly with Omniverse, the future
01:50:36 Speaker_04
could actually look like applications where you need both amazing graphical capability and AI capability for the same application.
01:50:45 Speaker_04
And I mean, for all the other amazing uniqueness about NVIDIA that we've been talking about and how well positioned they are, adding this on top, where they're the number one provider for graphics hardware and software and AI hardware and software.
01:51:00 Speaker_04
Oh, and by the way, there's this huge application emerging where you actually do need both, they're just going to knock it out of the park if that comes true.
01:51:09 Speaker_01
There was a super cool demo at a recent keynote, it might've been at SIGGRAPH, where NVIDIA created a game environment, you know, fully ray traced game environment, looks like a AAA game, you know, looks amazing, you know, basically distinguishable from reality, but like, you really gotta look hard to tell that this isn't real and this isn't a real human you're talking to.
01:51:32 Speaker_01
So there's a non-playable character that you're talking to, an NPC who's giving you like a mission. and they show this demo, it looks amazing. Then they're like. the script, the words that that character was saying to you were not scripted.
01:51:45 Speaker_01
That was all generated with AI, dynamically. So you're like, holy crap. You know, you think about, you play a video game, the characters are scripted.
01:51:57 Speaker_01
But in this world that you're talking about, you can have generative AI-controlled avatars that are unscripted, that have their own intelligences, and that drives the story.
01:52:08 Speaker_04
Totally. Or, you know, an airplane that's in a simulation of not just a wind tunnel, but simulating millions of hours of flying time using real-time weather that's actually going on in the world and using AI to project the weather in the future.
01:52:24 Speaker_04
So you can sort of know the real-world potential things that your aircraft could encounter all in a generated graphical AI simulation. I mean, there's gonna be a lot more of this stuff to come.
01:52:36 Speaker_00
Yep. Totally.
01:52:38 Speaker_04
Another thing to know about Nvidia that we really didn't talk about on the last episode, they're pretty employee efficient.
01:52:46 Speaker_04
They have 26,000 employees, and that sounds like a big number, but for comparison, Microsoft, whose market cap is only twice as big, has 220,000. So that is 5x the number of employees per dollar of market cap. going on over at Microsoft.
01:53:04 Speaker_04
And this is a little bit farcical since, you know, NVIDIA only recently has had such a massive market cap.
01:53:10 Speaker_01
But the scale of the platform that NVIDIA is building is on the order of magnitude of Microsoft scale.
01:53:18 Speaker_04
Right. They have 46 million dollars of market cap per employee.
01:53:22 Speaker_01
Wild.
01:53:23 Speaker_04
Crazy.
01:53:23 Speaker_01
Which I think translates into the culture there as we've gotten to know some folks there. It really is a very unique kind of culture.
01:53:31 Speaker_01
Like it is a big tech scale company, but you never hear about the same kind of silly big tech stuff that you hear at other companies at NVIDIA. As far as I know, I could be wrong on this.
01:53:44 Speaker_01
There is no like, you know, oh, work from home or return to the office policy at NVIDIA. It's like, no, it's like you do the job and, you know, nobody's forcing anybody to come into the office here and like they've accelerated their ship cycles.
01:53:58 Speaker_04
Well, I also get the sense that it's a little bit of a do your life's work or don't be here situation.
01:54:04 Speaker_04
Like Jensen is rumored to have 40 direct reports and his office is basically just an empty conference room because he's just bouncing around so much and he's on his phone and he's talking to this person and that person and like you can't manage 40 people directly if you're worrying about someone's career ambitions.
01:54:20 Speaker_01
Yep. He's talked about this. He's like, I have 40 direct reports. They are the best in the world at what they do. This is their life's work. I don't talk to them about their career ambitions. Like I don't need to like, you know, yeah.
01:54:32 Speaker_01
For like recent college grads, we do mentoring it. But if you're a senior employee, you've been here for 20 years, you're the best in the world of what you do. And we're hyper efficient. And I start my day at 5 AM, seven days a week. And you do too.
01:54:42 Speaker_04
It's crazy.
01:54:44 Speaker_01
Yeah, there's actually this amazing quote from Jensen that I heard on an interview with him that I was listening to where towards the end of the conversation, the interviewer asks him, Jensen, you and NVIDIA do these just amazing things.
01:54:58 Speaker_01
What do you do to relax? And Jensen's answer is, I'm reading, this is a quote, direct quote. I relax all the time. I enjoy relaxing at work because work is relaxing for me. Solving problems is relaxing for me. Achieving something is relaxing for me.
01:55:16 Speaker_01
And he's a hundred percent serious, like a thousand percent serious.
01:55:21 Speaker_04
How old is Jensen?
01:55:22 Speaker_01
The dude is 60 years old.
01:55:25 Speaker_04
It kind of feels like all of his peers have either decided to retire and relax or are, you know, relaxing while running their companies. I think there's another crop of people that are doing that.
01:55:35 Speaker_04
And that is just not at all interesting to him or what he's doing. And I kind of get the sense like he's got another 30 years in him and he's architected the company in such a way that that's the plan.
01:55:47 Speaker_04
I don't think there's anyone else there where they're getting ready for that person to take over. I think the company is an extension of Jensen's thoughts and will and drive and belief about the future, and that's kind of what happens.
01:56:03 Speaker_01
I don't know if there is or isn't a Jensen and Lori Hoang foundation, but if there is, he's not spending his time on it. He's not buying sports franchises.
01:56:11 Speaker_01
He's not buying mega yachts, or if he is, he isn't talking about them and he's working from them.
01:56:16 Speaker_04
Yeah. He's not buying social media platforms and newspapers.
01:56:19 Speaker_01
Yeah, totally.
01:56:20 Speaker_04
I mean, it is quite telling that when you watch one of their keynotes, it's Jensen on stage and it's some customer demos, but it's not like the Apple keynotes where Tim Cook's calling up another Apple employee. It's the Jensen show.
01:56:33 Speaker_01
Right. Nobody would accuse Tim Cook of not working hard, I don't think. But you go to those keynotes, and it's like, Tim does the welcome, and then the handoff. And, you know, a parade of other executives talk about stuff.
01:56:44 Speaker_04
Good morning.
01:56:46 Speaker_01
Tim Apple. I love it.
01:56:48 Speaker_04
Love Tim Apple.
01:56:49 Speaker_01
We gotta have Tim on the show sometime. That would be amazing.
01:56:51 Speaker_04
Yeah, text him. Text him. All right. Power?
01:56:54 Speaker_01
Let's talk power.
01:56:55 Speaker_04
All right, so for listeners who are new to the show, this is the section where we talk about what it is about the company that enables them to achieve persistent differential returns, or in other words, to be more profitable than their closest competitor and do so sustainably.
01:57:12 Speaker_04
And NVIDIA is fascinating because they sort of have a direct competitor, but that's not the most interesting form of competition for them. Disintermediation is.
01:57:22 Speaker_04
Sure, ostensibly there's NVIDIA versus AMD, but like, AMD doesn't have all this capacity reserved from TSMC, at least not for the 2.5D packaging process for the high-end GPUs. AMD doesn't have the developer ecosystem from CUDA.
01:57:38 Speaker_04
They're the closest direct comp, but it's Amazon building Tranium and Inferentia. It's if Microsoft decides to go and build their own ship as they're rumored to with AMD. It's Google and the TPU.
01:57:50 Speaker_04
Facebook developing PyTorch and then leveraging their foothold with PyTorch with the developer community to figure out how to extend underneath of PyTorch. There's a lot of competitive vectors coming at NVIDIA, but not directly.
01:58:03 Speaker_01
Not to mention all the data center hardware providers that are their direct competitors now too. Yep. Intel, etc. on down the line.
01:58:13 Speaker_04
Yep. Now all that said, they've got a lot of powers. So as we move through these one by one, I think let's just say them all and we can decide if there's something to talk about here.
01:58:23 Speaker_04
Counter positioning is the one where I actually don't think there's anything here.
01:58:27 Speaker_04
I don't think there's anything that Nvidia does where there's another company that's actively choosing not to do that because any company would want to be Nvidia right now.
01:58:37 Speaker_01
I would have agreed with you, but I actually think there is strong counter positioning in the data center world right now. NVIDIA and Jensen put a flag in the ground several years ago where they said, we are going to re-architect the data center.
01:58:52 Speaker_01
And all the existing data center hardware and compute providers had strong incentives not to do that.
01:58:59 Speaker_04
But like right now, what do you think other data center hardware providers, what are they not doing? Yeah, fair point.
01:59:05 Speaker_01
They're all trying to put GPUs in the data center too.
01:59:09 Speaker_04
Everyone's just going to chase exactly what NVIDIA is doing years behind them. That's the market right now.
01:59:14 Speaker_01
Yep. Okay. Fair enough.
01:59:16 Speaker_04
And the question is, will Nvidia be able to stay ahead in ways that matter?
01:59:22 Speaker_04
That I think is the entire analysis on the company right now, is in what ways that matter to customers at large scale in large markets, will they be able to sustainably be ahead of people that are just chasing them and trying to copy what they're doing because the margin profile is so fat and juicy that people don't want to pay it.
01:59:40 Speaker_00
Yep.
01:59:41 Speaker_04
So the second one, scale economies. This has CUDA written all over it. You can make massive fixed cost investments when you have the scale to amortize that cost across.
01:59:54 Speaker_04
And when you have 4 million developers who want to develop on your platform, you can justify whatever it is, 1,600 people who actively on LinkedIn at NVIDIA today have the word CUDA in their job title.
02:00:07 Speaker_04
I mean, I'm sure it's actually even more than that who just aren't, you know, they're saying software or something like that, but thousands of people of an investment that they don't make any money on software.
02:00:16 Speaker_04
They may, they make a de minimis amount on software, but that is amortized across the entire developer base.
02:00:23 Speaker_01
I think it's worth saying a bit more here on this too, which we also talked about in our last episode. To me, the dynamics here are a lot like Apple and iOS. Yes. Versus Android.
02:00:37 Speaker_01
Apple has thousands and thousands and thousands of developers working on iOS. Android also has thousands and thousands of developers working on it across a widespread ecosystem. But at Apple, it's all tightly controlled and it's coupled with hardware.
02:00:54 Speaker_01
At Android, it's not. And like, as a user, maybe you'll get the latest operating system update. Maybe you won't.
02:01:02 Speaker_04
I think this is exactly the right framing here, that Nvidia is the Apple of AI, and PyTorch is sort of Android because it's open source and it's got a bunch of different companies that care about it.
02:01:12 Speaker_04
OpenCL is the Android as it pertains to graphics, but it's pretty bad and pretty far behind. RockM is the CUDA competitor made by AMD for their hardware, but again, new, not a lot of adoption. They're working on it, but
02:01:27 Speaker_04
They've open sourced that because they realize they can't go directly head to head with NVIDIA. They need some different strategy. But yes, they are absolutely running the Apple playbook here.
02:01:37 Speaker_01
Yep. And I think in the current state of things, it's even more favorable to NVIDIA than iOS versus Android because NVIDIA has had first dozens and then hundreds and now thousands of engineers working on CUDA for 16 years.
02:01:55 Speaker_01
Meanwhile, the Android equivalent out there in the open source ecosystem has only just been getting going. You know, if you think about the delta of the timeline between iOS and Android, it was a year and a half, two years.
02:02:09 Speaker_01
There's probably at least 10, probably closer to 15 year lead than NVIDIA has. And so we talked to a few people about this and we're like, oh, what's going on in the open source ecosystem? Is there an Android equivalent?
02:02:22 Speaker_01
And even the most bullish people we talked to were like, oh, yeah, you know, now that Facebook has really moved PyTorch into a foundation and outside of Facebook, that means that other companies can now contribute, you know, a couple dozen engineers to work on it.
02:02:36 Speaker_01
And you're like, cool. So AMD is going to contribute a couple dozen, maybe a hundred engineers to work on PyTorch. And so will Google and so will Facebook and so will everybody else. NVIDIA has thousands of engineers working on CUDA 10 years ahead.
02:02:51 Speaker_04
I sent you this graph, David, of my estimated number of employees working on CUDA per year since inception in 2006.
02:02:58 Speaker_04
And then if you look at the area under the curve and just take the integral, it's approximately 10,000 person-years that have gone into CUDA. Like, good luck.
02:03:09 Speaker_01
Now, again, open source is a very powerful thing. The market incentives are absolutely there for this to happen.
02:03:16 Speaker_04
Right. That is the interesting point is every moat only works if the castle is sufficiently small.
02:03:23 Speaker_04
If the prize at the end of the finish line becomes sufficiently large, you're going to need a bigger moat and you need to figure out how to defend the castle harder. I'm mixing so many metaphors here, but you get the idea.
02:03:36 Speaker_01
Yeah, I love it.
02:03:37 Speaker_04
This was a perfectly fine moat when the addressable market was $100 billion. Is it at a trillion dollar market opportunity? Probably not. Basically, it means margins come down and competition gets more fierce over time.
02:03:50 Speaker_01
And I think NVIDIA totally gets this because part of this, as I was alluding to, is COVID related. But we talked way back in part one about how NVIDIA
02:04:01 Speaker_01
ended up to save the company moving to a six-month shipping cycle for their graphics cards when their competitors were on a one to two-year shipping cycle. That persisted for several years and then they relaxed back to an annual shipping cycle.
02:04:16 Speaker_01
There were annual GTCs. Since COVID, NVIDIA has re-accelerated to a six-month shipping cycle. They've been doing two GTCs a year most years since COVID, which is insane for the level of technology complexity that they're doing. Yep.
02:04:34 Speaker_01
Imagine Apple doing two WWDCs a year. Yeah. That's what's happening at NVIDIA. It's crazy. So on the one hand, that's a culture thing.
02:04:43 Speaker_01
On the other hand, that is an acknowledgement of like, we need to be pedal to the floor right now to outrun competition.
02:04:49 Speaker_04
We've built some structural ways to defend the business, but we need to continue running as fast as we've ever run to stay ahead because it's such an attractive race that we're in.
02:04:58 Speaker_01
Yep. All right. So that's scale economies. Let's move to switching costs now.
02:05:04 Speaker_04
So far, everything of consequence, especially model training, especially on LLMs, has been built on NVIDIA. And that alone is just a big pile of code and a big amount of organizational momentum.
02:05:19 Speaker_04
So switching away from that, even from the software perspective, is going to be hard. But there are companies today in 2023, both at the hyperscalers and Fortune 500 companies that own their own data centers,
02:05:34 Speaker_04
making datacenter purchase and rollout decisions that will last at least the next five years. Because these datacenter re-architectures don't happen very often.
02:05:45 Speaker_04
And so you better believe that NVIDIA is trying as hard as they can to ship as much product as they can while they have the lead in order to lock in that datacenter architecture for the next N years.
02:05:59 Speaker_01
Yeah, we talked to many people in preparation for this episode, but one of the most interesting conversations was with some of our favorite public market investors out there, the NCS Capital guys.
02:06:09 Speaker_04
Who I stole many insights from for this episode.
02:06:11 Speaker_01
Oh, they're just so great and obviously have been following NVIDIA in the space for a long time. They made the point that data center revenue and data center CapEx is some of the stickiest revenue that is known to humankind.
02:06:28 Speaker_01
Just the organizational switching costs involved in data center procurement and data center architecture standardization decisions.
02:06:36 Speaker_01
God, that's a mouthful even to say at Fortune 500 companies and the like is like, they're not changing that more than once a decade at most.
02:06:45 Speaker_04
So even if we're sort of in this bubbly moment around the excitement of generative AI before we necessarily know the full set of applications, NVIDIA is leveraging this excitement to go get some lock-in.
02:06:59 Speaker_04
I've seen some people on the internet being like, they love how supply constrained they are. I don't think so. I think they're looking for capacity in every way they can get it to exploit this opportunity while it exists.
02:07:09 Speaker_01
I completely agree with that. Yeah. I think, you know, again, we didn't talk to Colette and video CFO about this, but. I strongly suspect if I were them, I would be happy to trade some of this gross margin right now for increased throughput on sales.
02:07:22 Speaker_04
Yep. But there's only one TSMC and there's only so many fabs that they have that can do the, what do they call it? The 2.5 D architecture.
02:07:32 Speaker_01
So should we talk cornered resource?
02:07:34 Speaker_04
Yeah, this is probably the textbook cornered resource. Nvidia has access to a huge amount of capacity at TSMC that none of their competitors can get their hands on. I mean, they did luck into this cornered resource a little bit.
02:07:49 Speaker_04
They reserved all that wafer supply for a different purpose, partially crypto mining. But AMD doesn't have it.
02:07:54 Speaker_04
AMD does have a ton of capacity, it's worth saying, at TSMC for their other products, data center CPUs, which they've actually been doing very well in.
02:08:03 Speaker_04
But NVIDIA did end up with this wide open lane all to themselves on co-op's capacity at TSMC, and they got to make the most of that for as long as they have it.
02:08:12 Speaker_01
Yep. And I guess to say a little more though, It's not like, this is not a commodity, as we talked about on our TSMC episode. Although TSMC is a contract manufacturer, it is the opposite of a commodity, especially at the highest end leading edge.
02:08:29 Speaker_04
It's like an invention delivered by aliens that very few humans know how to actually do. Yes. It is worth acknowledging it's kind of a two-horse race for LLM training.
02:08:41 Speaker_04
I know we've been harping on NVIDIA, but Google TPUs are also manufactured at volume. You can just only get them through Google Cloud.
02:08:50 Speaker_04
And I think, I don't know if you have to use the TensorFlow framework, which has been waning in popularity relative to PyTorch, but it's certainly not an industry standard to use TPUs the way that it is to use NVIDIA's hardware.
02:09:03 Speaker_04
I suspect a lot of the volume of the TPUs is being used internally by Google for BARD, for doing stuff in Google search, like I know they've added a lot of the generative AI capability to search.
02:09:15 Speaker_01
Yep, totally. Two points on this. Just sticking to the scope of this business and market discussion, this is a major casualty of a strategy conflict at Google.
02:09:26 Speaker_01
Obviously, the way you want to do this is the way NVIDIA is doing this of like, your customers want to buy through the cloud, you want to be in every cloud.
02:09:34 Speaker_01
But obviously, Google is not going to be in AWS and Azure and Oracle and all the new cloud providers. They're only going to be in GCP.
02:09:43 Speaker_04
Maybe, David.
02:09:44 Speaker_01
But I was going to say, though, through the expanded lens, though, I think this makes sense for Google because their primary business is their own products.
02:09:52 Speaker_04
Right. And they run among the most profitable businesses the world has ever seen. So anything they can do to further advantage and extend that runway, they probably should do.
02:10:03 Speaker_01
Nothing has changed through all of this with respect to the fact that what the previous generation of AI enabled with machine learning with regard to social media and internet applications being the most profitable cash flow geysers known to man.
02:10:18 Speaker_01
None of that has changed. That is still true in this current world and still true for Google.
02:10:24 Speaker_04
Yep. The last one that I had highlighted is network economies. They have a large number of developers out there and a large number of customers that they can amortize these technology investments across and who all benefit from each other.
02:10:37 Speaker_04
I mean, remember, there are people building libraries on top of CUDA and you can use the building blocks that other people built to build your code.
02:10:46 Speaker_04
You can write amazing CUDA programs that just don't have that many lines of code because it's calling other preexisting stuff.
02:10:53 Speaker_04
And NVIDIA made a decision in 2006 that at the time was very costly, like big investment decision, but it looks genius in hindsight to make sure that every GPU that went out the door was fully CUDA capable.
02:11:04 Speaker_04
And today there are 500 million CUDA capable GPUs for developers to target. It's just very attractive. I'm putting this in network economies.
02:11:12 Speaker_04
I think it's probably more a scale economy than a network economy, but you could imagine a lot of people ho-humming around NVIDIA in 2006 to 2012 saying, why do I have to make it so that my software fits on this tiny little footprint and we can include CUDA, taking up a huge amount of space on this thing and make all these trade-offs in our hardware so that we can, are people going to use CUDA?
02:11:33 Speaker_04
And today it just looks so genius.
02:11:35 Speaker_01
Yeah, I mean, we've talked about this many times on the show, including with Hamilton, Helmer and Chenyi themselves.
02:11:40 Speaker_01
But for platform companies like NVIDIA clearly is, there is this special brand of power that is a combination of scale economies and network economies. And this is what you're getting at.
02:11:51 Speaker_04
Yep. They do have branding power for sure.
02:11:54 Speaker_01
Yeah. I actually think it's worth talking about this a little bit.
02:11:57 Speaker_04
This is the nobody gets fired for buying IBM. I mean, NVIDIA is the modern IBM in the AI era.
02:12:03 Speaker_01
Yeah, look, I don't feel confident enough to like pound the table on this, but given the nature of how the company started, how long they've been around, and the fact that they also have the market leading product in a totally different business in graphics, you know, which is both consumers, but also professional graphics.
02:12:24 Speaker_01
I think that probably does lend some brand power to them, especially when the CIO and the C-suite at McDonald's is making a buying decision here. Like, everybody knows NVIDIA.
02:12:37 Speaker_04
You're saying that they carried their consumer brand into their enterprise posture.
02:12:42 Speaker_01
This is way, way, way down the stack in power, but I don't think it's hurt them. They've always been known as a technology leader, and the whole world has known for decades at this point that the stuff that they can enable is magical.
02:12:57 Speaker_04
Yeah, there's a big strength leads to strength thing here, too, where I bet the revenue results from last quarter massively dwarf any brand benefit that they ever got from the consumer side.
02:13:09 Speaker_04
I think it's just the fact that like, hey, look, everyone else is buying Nvidia. I'd be an idiot not to.
02:13:13 Speaker_01
Nobody is getting fired for buying Nvidia anytime soon. Yep.
02:13:16 Speaker_04
Right. Or taking a big dependency on them or targeting that development platform. It's just the like, if you're innovating in your business, you don't want to take risk on the platform you're building on top of.
02:13:26 Speaker_04
You want to be the only risk in the value chain.
02:13:29 Speaker_01
All right. Then the last one, right, is process power.
02:13:31 Speaker_04
Yeah. And this is probably the weakest one, even though I'm sure you could make some argument that they have process power. It's just that all the other powers are so much more valuable.
02:13:40 Speaker_01
It's always so tricky to tease out. Yep. You know, I think the argument here would just be like NVIDIA's culture and their six month shipping cycle that clearly they had in the past and they didn't have for a while and now they have again.
02:13:55 Speaker_01
I don't know. I think you can make an argument here. Is it feasible? Let's do a thought exercise. Could any of their competitors really in any domain move to a six month ship cycle? That'd be really hard.
02:14:09 Speaker_03
Yeah.
02:14:09 Speaker_01
You know, could a Apple-sized company do two WWDCs a year? Like, no.
02:14:16 Speaker_04
The question is, does that actually matter? There are so many people that are using A100s right now. And in fact, most workloads can be run on A100s unless you're doing model training of GPT-4.
02:14:28 Speaker_04
I just don't know that it actually matters that much, or as much as other factors. And I'll give you an example. AMD does have 3D packaging on one of their latest GPUs.
02:14:39 Speaker_04
It's a more sophisticated way of doing real copper to real copper direct connection without a silicon interposer. I'm getting into a little bit of the details, but basically it's more sophisticated
02:14:50 Speaker_04
than the process that the H100 2.5D is using to make sure that memory is extremely close to compute. And does that matter? Not really.
02:15:01 Speaker_04
What matters is everything else that we've been talking about, and nobody's gonna make a purchase decision on this thing because it's, you know, a little bit of a better mousetrap.
02:15:09 Speaker_01
Yeah, thinking about this more, I think actually brand is a really important power for NVIDIA right now.
02:15:14 Speaker_04
Yeah, and in a strength-leads-to-strength way. So you can see why they're trying to sort of seize this moment.
02:15:19 Speaker_01
Yeah.
02:15:20 Speaker_04
Playbook?
02:15:20 Speaker_01
All right, let's move on to playbook.
02:15:23 Speaker_04
So one thing that I want to point out is Jensen keeps referring to this as the iPhone moment for AI. And when he says it, the common understanding is that he means a new mainstream method for interacting with computers.
02:15:36 Speaker_04
But there's another way to interpret it. Does this sound familiar, David, when I say a hardware company differentiated by software that then expanded into services?
02:15:47 Speaker_01
Yes, yes it does.
02:15:48 Speaker_04
It's quite tongue-in-cheek to be referring to the iPhone moment of AI when referring to oneself, NVIDIA, as the Apple.
02:15:56 Speaker_04
Because I really think that the parallels are uncanny, that they have this vertically integrated hardware and software stack provided by NVIDIA. You use their tools to develop for it.
02:16:05 Speaker_04
They've shipped the most units, so developers have a big incentive to target that market. It's the best individual buyers to target because they're the least cost sensitive and they appreciate you building the best experiences for them.
02:16:18 Speaker_04
I mean, it's the iPhone, but in many ways it's better because the target is a B2B target instead of consumers.
02:16:24 Speaker_04
The only way in which it's different is Apple has always had a market cap that sort of lagged its proven value to users, whereas Nvidia right now is exactly over their skis.
02:16:37 Speaker_01
Well, let's save that for Bull and Bear at the end.
02:16:40 Speaker_04
Great. The second one is that they've moved on from becoming a hardware company to truly being a systems company. While NVIDIA's chips are typically ahead, it really doesn't matter on a chip-to-chip comparison. That is not the playing field.
02:16:52 Speaker_04
It is all about how well multiple GPUs and multiple racks of GPUs work together as one system with all the hardware and networking and software that enables that.
02:16:59 Speaker_04
They have just entirely changed the vector of competition, which I think lots of companies can learn from.
02:17:05 Speaker_04
And my third one here is this quote that Jensen had, again, from the same Stratechery interview, which is, you build a great company by doing things that other people can't do.
02:17:15 Speaker_04
You don't build a company by fighting other people to do things that everyone can do. And I think it's so salient. It comes out in all these interesting ways, one of which is, NVIDIA never dedicated resources to building a CPU
02:17:29 Speaker_04
until there was a differentiated way and a real reason for them to build their own CPU, which is now. And the way that they're doing it, by the way, is not terribly differentiated.
02:17:38 Speaker_04
It's an off-the-shelf ARM architecture that they're putting some of their own secret sauce on, but it's not like they're doing Apple-style M3 creation of a chip from scratch.
02:17:49 Speaker_01
It's not the hero product.
02:17:51 Speaker_04
Right. There are many ways that NVIDIA sort of applies this where, I think we talked about in the last episode, if they think it's going to be a low margin opportunity, they don't go after it.
02:18:00 Speaker_04
But the nicer way to say that is, we don't want to compete for things that anybody can do. We want to do things that only we can do. Oh, and by the way, we will fully realize the value of those things when we do them.
02:18:10 Speaker_01
Yup. I think there's maybe a related playbook theme here for NVIDIA of strike when the timing is right.
02:18:18 Speaker_01
I suspect that a lot of the inner competitive drive and motivation for Jensen and the company over the past 10, 15 years here has been to really fight against Intel. Intel tried to kill them, as we talked about many times in the previous episodes.
02:18:38 Speaker_01
We talked to somebody who framed it as Intel was the country club and NVIDIA is the fight club. And back in the days, the Intel country club didn't want to let NVIDIA in. Intel controlled the motherboard.
02:18:49 Speaker_01
Intel controlled the most important chip was the CPU. Intel would integrate and commoditize all other chips into the motherboard eventually. And if they couldn't do that well, then they'd try and make the chips themselves.
02:19:01 Speaker_01
And they tried to run all these playbooks on NVIDIA and NVIDIA just barely survived. And then in the data center, Intel controlled the data center for so long.
02:19:10 Speaker_01
PCI Express, you know, that was the interconnect in the data center for so long and NVIDIA had to live in there. And I'm sure they hated every single minute of it. But they didn't turn around 10 years ago and just be like, guess what?
02:19:22 Speaker_01
We're making a CPU too. They waited until the time was right.
02:19:26 Speaker_04
It is crazy. They used to have to plug into other people's servers. And then they started making servers that plugged into other people's racks and rows and architectures. And then they started making their own entire rows and walls.
02:19:37 Speaker_04
And at some point here they're going to start running their own buildings full of servers too. And they're going to say we don't have to plug into anything.
02:19:44 Speaker_01
Yep. But I think for a lot of other leaders, it would have been hard to have the patience that they've had. Totally.
02:19:51 Speaker_04
You only get to do the stuff they're doing if you invested 10 years ahead of the industry, were wildly inventive and innovative in creating these like true breakthrough innovations, and were really, really right about huge markets.
02:20:07 Speaker_01
Yep.
02:20:08 Speaker_04
None of this stuff applies unless you're doing those three things.
02:20:11 Speaker_01
Yeah, Fortune 500 CIOs aren't making buying decisions if none of what you just said isn't true.
02:20:16 Speaker_04
Right. So there's this interesting conversation I wanted to have with you ahead of winding it up with the bull and bear case. So think back to our AWS episode. We talked a lot about how AWS is just locked in.
02:20:32 Speaker_04
The databases are a ridiculously durable advantage. Once your data has been shipped to a particular cloud, often literally in semi-trucks full of hard drives.
02:20:43 Speaker_01
Snowball, yeah.
02:20:44 Speaker_04
It's hard to move off of it. There's this sort of interesting question of will winning cloud 1.0 for all these Google, Microsoft, Amazon, will that toehold actually enable them to win in the cloud AI era?
02:21:01 Speaker_04
On the one hand, you'd think, yes, absolutely, because I want to train my AI models right next to where my data is. It's really expensive to move my data somewhere else to do that.
02:21:10 Speaker_01
Case in point, Microsoft is the exclusive cloud infrastructure provider for OpenAI, which runs, as far as we know, solely on NVIDIA infrastructure, but they buy it all through Microsoft.
02:21:22 Speaker_04
Right. On the other hand, the experience that customers are demanding is the full stack NVIDIA experience, not this, oh, you found the cheapest possible cost of goods sold way to offer me something that's like the experience that I want.
02:21:37 Speaker_04
And sometimes the cloud providers have to offer me an A100 or an H100 because my code is way too complicated to ever re-architect for whatever accelerated computing devices they're offering me that's first party and cheaper for them. I don't know.
02:21:51 Speaker_04
I just think for the first time in the last five years or so, I've sort of cocked my head a little bit at the moat of these existing cloud providers and said, huh, maybe there really is a vector to compete with them.
02:22:04 Speaker_04
And cloud is not a settled frontier.
02:22:07 Speaker_01
Yeah. Well, this is pejorative here. Cloud is a euphemism for data centers, right? There's so much more to the hyperscalers and public clouds than just data centers, right?
02:22:18 Speaker_04
But physically, they're data centers.
02:22:20 Speaker_01
Yeah, there is a mile of distance, metaphorically, between like an Equinix and AWS. Yep. their data centers. And there is a fundamental shift, at least according to Jensen, a fundamental shift that is happening in data centers.
02:22:36 Speaker_01
So I think that probably does create some shifting sands that the cloud market is going to have to navigate.
02:22:44 Speaker_04
Yep. I bet the way it plays out is that where you landed in cloud 1.0 strongly dictates where you will land in this AI cloud era.
02:22:51 Speaker_04
Because at the end of the day, if customers are demanding NVIDIA stuff, then the cloud providers have every incentive in the world to make it so that you can run your applications great in their cloud.
02:23:00 Speaker_01
but also like there's more to this too. Crusoe exists, CoreWeave exists, Lambda Labs exists. These are well-funded startups with billions of dollars that a lot of smart people think there's a major cloud-sized opportunity for. Yep.
02:23:14 Speaker_01
That would not have happened a few years ago.
02:23:16 Speaker_04
Super true. All right, let's do the bull case and bear case and bring this one home.
02:23:22 Speaker_01
Oh boy, we've been trying to delay this as long as possible. This is the crux of the question right now.
02:23:27 Speaker_04
Yeah. I mean, part of it is, is their existing moat big enough if GPUs actually become a $100 billion a year market? I mean, right now, GPUs in the data center are like a $30 billion a year market going to like a $50 billion next year.
02:23:44 Speaker_04
And like, if this actually goes the way that everyone seems to think it's going to go, There's just too many margin dollars out there for these big companies to not invest heavily. Meta threw tens of billions of dollars, making the metaverse.
02:24:00 Speaker_04
I mean, Apple's put $15 billion into, rumored, into their headset. Amazon's put tens of billions of dollars into devices, which, by all means, was a terrible investment. How is Echo paying anything back?
02:24:13 Speaker_01
Oh man, total sidebar. I'm so disappointed. I have standardized my house on the Echo ecosystem and it keeps getting dumber. How in this world of incredibly accelerating AI capabilities are my Echos getting dumber?
02:24:28 Speaker_04
Well, they need to trainium and inferentia a little bit harder.
02:24:31 Speaker_01
Ah, Jesus. Okay, rant over.
02:24:34 Speaker_04
Yeah, I mean, never doubt big tech's ability to throw tens of billions of dollars into something if the payoff could be big enough. These are ludicrously profitable monopolies, except your Amazon's not that profitable.
02:24:45 Speaker_01
AWS is.
02:24:46 Speaker_04
Yeah, but Google, Facebook, Apple, at some point here, there's a game of chicken that ends, and some of these companies go all in and say, yeah, we have smart engineers too, like, we're going to figure this out.
02:24:58 Speaker_01
Yeah, but also never underestimate the inability of big tech to execute on stuff that it thinks it can, especially with major strategy shifts.
02:25:07 Speaker_04
Yeah. Yeah. All right. So let's actually do this. Bear case. Let's start with the bear case.
02:25:13 Speaker_01
So you just illustrated, I think, bear case number one, which is Literally everybody else in the technology ecosystem is now aligned and incentivized to say, I want to take a piece of Nvidia's pie. And these companies have untold resources.
02:25:30 Speaker_04
Yep. And to put a finer point on that, let's look at PyTorch for a minute. Now that all the developers or lots of developers are using PyTorch, it does enable PyTorch to aggregate customers, which gives them the opportunity to disintermediate. Maybe.
02:25:47 Speaker_04
You've got to write a lot of new stuff underneath and ship a lot of hardware. I mean, the cloud service providers have taken some steps here.
02:25:54 Speaker_04
It was originally developed by Meta, and while it's open source, it's still hard for all these companies to invest in it if it's really sort of owned and controlled by Meta.
02:26:03 Speaker_04
So now, PyTorch has been moved out into a foundation that a lot of companies are contributing to. It is a absolute false equivalence to be like PyTorch versus NVIDIA.
02:26:14 Speaker_04
But in real Ben Thompson aggregation theory parlance, if you aggregate the customers, you have the opportunity then to take more margin, to disintermediate, to direct where that attention is going. And PyTorch has that opportunity.
02:26:28 Speaker_04
That feels like the vector that a lot of these CSPs will try and compete on and say, look, if you're building for PyTorch, it runs really well on our thing too.
02:26:37 Speaker_01
Yep, for sure. No doubt that that's going to happen. All right, so that's bear case number two kind of as part of bear case number one.
02:26:43 Speaker_04
The next one is like literally the market isn't as big as the market cap reflects. I think there's a pretty reasonable chance that there's some falter in the next 12 to 18 months where there's a crisis of confidence among investors where
02:27:01 Speaker_04
At some point, something will come out where we all observe, oh, maybe GPTs aren't as useful as we thought. Maybe people don't want chat interfaces.
02:27:09 Speaker_04
And that crisis of confidence, that mini bubble burst will trickle out to America's CIOs and CEOs, make it harder to advocate in the boardroom, to make this big fundamental purchase and re-architecture of our whole budget from this year that we agreed on that I'm trying to propose us changing.
02:27:25 Speaker_04
There's a crypto-like element to a excitement bubble bursting that will, for some companies, slow their spend. And the question is sort of like, when that happens, because it's not an if, it's a when.
02:27:38 Speaker_04
I have a hard time believing that given all the hype around everything right now,
02:27:44 Speaker_04
AI will be even more useful than everyone believes, and it will continue in a linear fashion where without any drawdowns, everyone's excitement only gets bigger from here.
02:27:57 Speaker_04
It may end up being way more useful than anyone thought, but there at some point will be some valley or trough. And it's sort of about how does NVIDIA fare during that crisis of confidence.
02:28:09 Speaker_01
It's funny, you know, again, we talked to a lot of people for this episode, including a set of some of the foremost AI researchers and practitioners out there and founders and C-suites of companies that are doing all this.
02:28:24 Speaker_01
And pretty much to a T, they all said the same thing when we asked them about this question. They all said, yeah, this is overhyped right now. Of course, obviously. But on a 10-year timescale, you haven't seen anything yet.
02:28:37 Speaker_01
The transformative change that we believe is coming, you can't even imagine.
02:28:42 Speaker_04
The most interesting thing about the overhype is that it's actually showing up in revenue. It's everyone who is buying access to all this compute believes something. And for Nvidia, because it's showing up in the form of revenue, the belief is real.
02:28:55 Speaker_04
then so they just need to make sure that they smooth the gap to customers actually realizing as much value as the CIOs of the world are currently investing ahead of.
02:29:05 Speaker_01
Yep. So I think the sub point to that that's worth a discussion right now is like, okay, generative AI. Yeah. Is it all it's cracked up to be?
02:29:15 Speaker_04
Well, David, I haven't asked you about this in like a month or so, but a month ago, you were pounding the table, insisting to me, like, I have no need for, I've never used ChatGBT. I can't find it to be useful. It's hallucinating all the time.
02:29:25 Speaker_04
I never think to use it. It's not a part of my workflow. Like, where are you at?
02:29:29 Speaker_01
Still basically there, including forcing myself to try to use it a bunch in preparation for this episode. But also as we talk to more people, I think I've realized that like, David Rosenthal's use case doesn't really matter here at all.
02:29:44 Speaker_04
Right.
02:29:45 Speaker_01
A, because as a business, we are such a hyper specialized, unique little unicorn thing where accuracy and the depth of the work and thought that we ourselves put into episodes is the paramount thing.
02:30:01 Speaker_04
Well, and we have no coworkers. There's so many things about our business that is weird. Like, we never have to prepare a brief for a meeting.
02:30:08 Speaker_01
Right, all this stuff. Anything external that we prepare is a labor of love for us. And there is nothing we prepare internal.
02:30:16 Speaker_04
I know people who use ChatGBT to set their OKRs, and I'm like, OK, what's an OKR? And they're like, I wish my life were like that, too. That's why I have ChatGBT do it.
02:30:23 Speaker_01
Right. Honestly, I think through doing this and talking to some folks and reading, I think there's a very compelling use case for it for writing code right now.
02:30:33 Speaker_01
No matter what level of software developer you are, from zero all the way up through elite software developer, you can get a lot more leverage out of this thing and GitHub Copilot. So is that valuable? Yeah, for sure that's valuable.
02:30:47 Speaker_04
Yeah, the LLMs are unbelievably good at writing and helping you write code. I'm a huge believer in that use case.
02:30:53 Speaker_01
Yeah. And then I think, you know, there's the slightly more speculative stuff, but you can actually sort of see it now of like that gaming demo that I mentioned recently from NVIDIA of like, oh,
02:31:04 Speaker_01
you're talking to a non-playable character that wasn't scripted. We did an ACQ2 episode recently with Chris Valenzuela from the CEO of Runway. That was used in everything, everywhere, all at once. And he said, that's just the tip of the iceberg.
02:31:18 Speaker_01
Like the stuff that you can do that is happening that's out there today with generative AI in these domains is astounding.
02:31:28 Speaker_04
Yeah, I think what you're saying is one could be a bear on your own experience. Every time you try to use a generative AI application, it doesn't fit into your workflow, you don't find it useful, you're not sticky.
02:31:40 Speaker_04
But on the other hand, actually what AI will be is a sum of a whole bunch of niches. There's a video game market, there's a writing market, there's a creative writing market, there's a software developer market, there's a marketing copy market.
02:31:54 Speaker_04
You know, there's a million of these things and you just may happen to not fall into one of the first few niches of it.
02:32:00 Speaker_01
Yeah, I think for me, at least again, just speaking personally, too, I had a very strong element of skepticism initially because the timing was just too perfect. You know, it's like all UVCs out there.
02:32:12 Speaker_01
You just told everybody about how crypto is the future and whatever you're talking about. And then interest rates went to, you know, 5% and your world fell off a cliff.
02:32:23 Speaker_04
Oh, the number of people who were like out raising a fund and they're like, the future is AI. Yeah, right. This is the best time ever to be investing.
02:32:33 Speaker_01
And so there was a large part of me that I was just like, come on, guys.
02:32:37 Speaker_04
Yeah, it's too perfect. You're right.
02:32:39 Speaker_01
It's too perfect. But this most recent couple months in this quarter for Nvidia has shown that, put all that aside, Fortune 500s are adopting this stuff. CIOs are adopting this stuff. NVIDIA is selling real dollars.
02:32:52 Speaker_01
And learning also about what it takes to train these models and the step scale function of knowledge and utility going from a billion parameters to 10 billion parameters to 200 to a trillion parameter models.
02:33:09 Speaker_01
Yeah, like something's going on there for sure.
02:33:11 Speaker_04
So this leads me to my next bear case, which is the models will get good enough and then they'll all be trained and then we'll shift to inference. And most of the compute load will be on inference where NVIDIA is less differentiated.
02:33:23 Speaker_04
There's a bunch of reasons I don't believe that. That is a popular narrative though. One of the big reasons I don't believe that is the transformer is not the end of the road.
02:33:31 Speaker_04
In a bunch of the research that we did, David, it's very clear that there are things beyond the transformer that are in the research phase right now, and the experiences are only going to get more magical and only going to get more efficient.
02:33:44 Speaker_04
So there's sort of a second bear case there, which is right now we threw a brute force kitchen sink at training these things, and all of that revenue accrued to NVIDIA because they're the ones that make the kitchen sinks.
02:33:58 Speaker_04
Over time, like you look at Google's Chinchilla or Lama 2, they actually use less parameters than GPT-4 and have equivalent quality. Or, you know, many other people can be the judge of that. But we're high-quality models with less parameters.
02:34:13 Speaker_04
So there is this potential bear case around future models will be more clever and not require as much compute. It's worth saying that even today, the vast majority of AI workloads don't look like LLMs, at least until very recently.
02:34:29 Speaker_04
LLMs are like the current maxima in human history of jobs to be done that require a ton of compute. And I guess the question is, will that continue?
02:34:39 Speaker_04
I mean, many other magical recent AI experiences have happened with far less expensive model training, like diffusion models and the entire genre of generative AI on images,
02:34:50 Speaker_04
which we really haven't talked about a lot on this episode because they're less compute intensive, but many tasks don't require an entire internet of training data and a trillion parameters to pull off.
02:35:00 Speaker_01
Yep. That makes sense to me. And I think there also is some merit to workloads are shifting to inference that is happening. I agree with you.
02:35:09 Speaker_01
I don't think training is going anywhere, but until recently, you know, thinking back to the Google days, training was what everybody was spending money on. That's what everybody was focused on as usage scales with this stuff.
02:35:21 Speaker_01
Then inference and inference, of course, being the compute that has to happen to get outputs out of the models after they're already trained, that becomes a bigger part of the pie. And as you say,
02:35:31 Speaker_01
The infrastructure and ecosystems around doing that is less differentiated than training.
02:35:36 Speaker_04
Yep.
02:35:36 Speaker_01
Okay, those are the bear cases. There's probably also a bear case around China, which is a legitimate one because that's gonna be a problem for lots of people.
02:35:47 Speaker_04
A large market that they won't be able to address for the foreseeable future in a meaningful way.
02:35:51 Speaker_01
And just what's going to happen generally, like obviously China is racing to develop their own homegrown ecosystems and competitors. And like, that's going to be a closed off market. So what's going to come out of there? What's going to happen?
02:36:04 Speaker_04
Yep, that's definitely one too. My last one is a bear case, but it ends up not being a bear case.
02:36:09 Speaker_04
For most companies, I would say that if they were trading at this very high multiple and they just experienced this tremendous real growth in revenue and operating profit, that that sort of spike to the system when it goes away will irreparably harm the company when things slow down.
02:36:29 Speaker_04
Stock compensation's an issue. Employee morale is an issue. Customer perception's an issue. But this is NVIDIA.
02:36:36 Speaker_01
Yeah, this is nothing new.
02:36:37 Speaker_04
The number of times that they've risen from the ashes after, you know, years-long terrible sentiment with something mind-blowingly innovative, they're probably the best positioned company or the company with the best disposition to handle that when it happens.
02:36:54 Speaker_01
Ooh, I love that. That's a great turn of phrase there. You upped your training model on language there. You should see the number of parameters. I love it.
02:37:03 Speaker_04
All right. Just to list the bulk cases. One, Jensen is right about accelerated computing. The majority of workloads right now are not accelerated. They're bound to CPUs. They could be accelerated.
02:37:14 Speaker_04
And that shifts from some crazy low number like 5 or 10 percent of workloads being accelerated today to 50 plus percent in the future. And there's way more compute happening in parallel. And that mostly accrues to NVIDIA.
02:37:26 Speaker_01
Oh, I have one nuance I want to add to that. On the surface, I think a lot of people look at that and they're like, yeah, come on.
02:37:34 Speaker_01
But I think there actually is a lot of merit to that argument in the generative AI world and everything we've talked about in this episode. I don't think Jensen and NVIDIA are saying that traditional compute is going away or can get smaller.
02:37:50 Speaker_01
I think what he's saying is that AI compute will be added on to everything and the amount of compute required for doing that will dwarf what's happening in general purpose compute.
02:38:05 Speaker_01
So like it's not that people are going to stop running SharePoint servers or that whatever products you use are going to stop using their whatever interfaces that they use.
02:38:14 Speaker_01
It's that generative AI will be added to all of those things and the use cases will pop up, which will also use traditional general purpose CPU based computing.
02:38:23 Speaker_01
But the amount of workloads that go into making those things magical is just going to be so much bigger.
02:38:29 Speaker_04
Yep. Also, just a general statement on software development, writing parallelizable code is really hard unless you have a framework to do it for you.
02:38:38 Speaker_04
Even writing code with multiple threads, like if anybody remembers a CS college in class where they had a race condition or they needed to write a semaphore, these are the hardest things to debug. And
02:38:50 Speaker_04
I would argue that a lot of things that could happen in an accelerated way aren't, just because it's harder to develop for.
02:38:57 Speaker_04
And so if we live in some future where Nvidia has reinvented the notion of a computer to shift away from von Neumann architecture into this stream processor architecture that they've developed, and they have the full stack to make it just as easy to write applications and move existing applications,
02:39:13 Speaker_04
Especially once all the hardware has been bought and paid for and sitting in data centers, there's probably a lot of workloads that actually do make sense to accelerate if it's easy enough to do so.
02:39:22 Speaker_01
Yeah, that's great. But so your point is that there's a lot of latent, accelerated, addressable computing out there that just hasn't been accelerated yet.
02:39:31 Speaker_04
Right. It's like, this workload's not that expensive, and I'm not going to pay an engineer to go re-architect the system. So it's fine how it is.
02:39:39 Speaker_01
How about that?
02:39:39 Speaker_04
I think there's a lot of that. So, bulk case one, Jensen is right about accelerated computing. Bulk case two, Jensen is right about generative AI.
02:39:48 Speaker_04
I mean, combined with accelerated computing, this will massively shift spend in the data center to NVIDIA's hardware. And as we mentioned, OpenAI is rumored to be doing over a billion dollars in recurring revenue on chat GPT.
02:40:01 Speaker_04
So I think there's, let's call it three billion, because that's the most sort of credible estimate that I've heard. And maybe that was a forecast for next year. But like, They're not the only one.
02:40:10 Speaker_04
I mean, Google with BARD, which I found tremendously useful actually preparing for this episode, is not directly monetizing that, but they're sort of retaining me as a Google search customer by doing it.
02:40:20 Speaker_04
There is a lot of real economic value even today.
02:40:24 Speaker_04
Not nearly the amount that's sort of baked into the valuation, but I suppose the bear case of this is that everything has to go right for Nvidia, but the bull case is indications are things are going right for Nvidia.
02:40:35 Speaker_00
Yep.
02:40:36 Speaker_04
Third, bulk cases. NVIDIA just moves so fast. Whatever the developments are, it's hard to believe that they're not going to find a way to be really well positioned to capture it. That's just a cultural thing.
02:40:46 Speaker_04
Four is the point that you brought up earlier, that there's a trillion dollars installed in data centers, $250 billion more being spent every year to refresh and expand capacity, and that NVIDIA could take a meaningful share of that.
02:40:58 Speaker_04
I think today, what's their annual revenue at? Like $30 billion or something?
02:41:03 Speaker_01
Well, if you run rate this current quarter, then it's like 50 50 plus. Yeah.
02:41:09 Speaker_04
So right now that puts them at like 20% of the current data center spend. You could imagine that being much higher.
02:41:18 Speaker_01
Okay. Wait, that includes the gaming revenue. It's about 40 because the data center revenue is 40 is 10. So 40 annualized.
02:41:24 Speaker_04
All right. So 15, 18%.
02:41:26 Speaker_01
Yeah.
02:41:27 Speaker_04
Woo. But you could imagine that creeping up again, if the accelerated computing and generative AI belief comes true, like they'll expand that to 50 number and they'll take a greater percent of it.
02:41:38 Speaker_00
Yep.
02:41:39 Speaker_04
An interesting way to do a sort of a check on this math is to look at what other people in the ecosystem are reporting in their numbers.
02:41:47 Speaker_04
TSMC, in their last earnings, said that AI hardware currently only represents 6% of their revenue, but all indications over there is that they expect AI revenue to grow 50% per year for the next five years. Wow.
02:42:02 Speaker_04
So we're trying to come at it from the customer workload side and say, is it useful there?
02:42:07 Speaker_04
But if you come at it from this other side of what do Nvidia suppliers forecasting and they have to put their money where their mouth is building these new wafer fabs to be able to facilitate that and packaging and all the other things that go into chips.
02:42:20 Speaker_04
So it's expensive for TSMC to be wrong.
02:42:23 Speaker_00
Yep.
02:42:24 Speaker_04
So that's another bull case. The last one that I have before leaving you with one final thought. Are you saying you have one more thing? Yes, is that NVIDIA isn't Intel. And I think that's the biggest realization that you helped me have.
02:42:38 Speaker_04
And it's not Cisco. Yeah, the comparison we were making in the last episode was wrong. They are Microsoft. They control the whole software stack, and they simultaneously can have relationships with the developer and customer ecosystems.
02:42:51 Speaker_04
And I mean, it may even be better than Microsoft because they make all the hardware too.
02:42:55 Speaker_01
Yeah, it may be old school IBM.
02:42:57 Speaker_04
Right. Imagine if IBM operated in a computing market of today's magnitude. Computing was a tiny little market back then.
02:43:04 Speaker_01
Right. I mean, it was like, I mean, it took the PC wave to disrupt IBM, which was a personal computer in today's parlance, edge computing, you know, device based computing. IBM dominated the B2B mainframe cycle of computing.
02:43:19 Speaker_01
And again, if you believe everything Jensen is saying and how he steered the company for the last five years, we are going back into a centralized data center, modern version of a mainframe dominated computing cycle.
02:43:32 Speaker_04
Yep. I suspect a lot of inference will get done on the edge. You think about the insane amount of compute that's walking around in our pockets that is not fully leveraged right now.
02:43:41 Speaker_04
There's going to be a lot of machine learning done on phones that are going to like call up to cloud-based models for the hard stuff.
02:43:47 Speaker_01
No doubt. I don't think training is happening at the edge anytime soon though.
02:43:51 Speaker_04
No, I certainly agree with that. All right, well, just like our TSMC episode, I wanted to end and leave you with a thought, David, of what it would take to compete with NVIDIA.
02:44:02 Speaker_04
Because my big takeaway from the TSMC episode was like, wow, that's a lot of things you'd have to believe about a government putting billions of dollars in and hiring all this talent. And I was like, what's the equivalent for NVIDIA?
02:44:13 Speaker_04
So here's what you would need to do to compete. Let's say you could design GPU chips that are just as good, which arguably AMD, Google, and Amazon are doing.
02:44:23 Speaker_04
You'd of course then need to build up the chip-to-chip networking capabilities like NVLink that very few have.
02:44:29 Speaker_04
And you'd of course need to build relationships with hardware assemblers like Foxconn to actually build these chips into servers like the DGX.
02:44:37 Speaker_04
And even if you did all that, you'd need to create server-to-server and rack-to-rack networking capabilities as good as Mellanox, who was the best on the market, with InfiniBand that NVIDIA now fully owns and controls, which basically nobody has.
02:44:49 Speaker_04
And even if you did all that, you'd need to go convince all the customers to buy your thing, which means it would need to be either better or cheaper or both, not just equal to NVIDIA.
02:45:01 Speaker_01
And by a wide margin, too, to this brand, you're not going to get fired for buying NVIDIA anytime soon. Like, this is the canonical, you got to be 10x better than NVIDIA on this stuff if you're going to convince a CIO.
02:45:12 Speaker_04
Yep. And even if you got the customer demand, you'd need to contract with TSMC to get the manufacturing capability of their newest cutting-edge fabs to do this 2.5D COAS lithography and packaging, which there, of course, isn't any more of.
02:45:28 Speaker_04
So, you know, good luck getting that. And even if you figured out how to do that, you'd need to build software that is as good or better than CUDA.
02:45:36 Speaker_04
And of course, that's going to take 10,000 person years, which would of course cost you not only billions and billions of dollars, but all that actual time.
02:45:45 Speaker_04
And even if you made all these investments and lined all of this up, you'd of course need to go and convince the developers to actually start using your thing instead of CUDA.
02:45:56 Speaker_04
Well, NVIDIA also wouldn't be standing still, so you'd have to do all of this in record time to catch up to them and surpass whatever additional capabilities they developed since you started this effort.
02:46:10 Speaker_04
So I think the bottom line here is nearly impossible to compete with them head on.
02:46:15 Speaker_04
And if anybody's going to unseat NVIDIA in the future of AI and accelerated computing, it's either going to be from some unknown flank attack that they don't see, or the future will turn out to just not
02:46:25 Speaker_04
be accelerated computing and AI, which seems very unlikely.
02:46:28 Speaker_01
Yeah. Well, when you put it that way, I think the conclusion that we can come to is that Marc Andreessen was right. In what years was this that we were talking about on?
02:46:42 Speaker_04
It was like 2015 or something.
02:46:44 Speaker_01
Yeah, like 2015, 2016.
02:46:46 Speaker_04
They should have put every dollar of every fund that A16Z raised into NVIDIA's market price of the stock every single day.
02:46:54 Speaker_01
Yeah, because they were seeing all of these startups doing deep learning, machine learning at the time, early AI, and they were all building on NVIDIA and they should have just said, no thank you to all of them and put it all in NVIDIA.
02:47:09 Speaker_01
Mark is right once again. Strength leads to strength. There you go.
02:47:12 Speaker_04
There it is. Well, listeners, I acknowledge that this episode generalized a lot of the details, especially for technical listeners out there, but also for the finance folks who are listening.
02:47:22 Speaker_04
Our goal was to make this more of a lasting NVIDIA part three big picture episode than sort of a how did they do last quarter and what are the implications on that of the next three quarters.
02:47:32 Speaker_04
So hopefully this holds up a little bit longer than just some current NVIDIA commentary. But thank you so much for going on the journey with us.
02:47:39 Speaker_01
Yeah, we also, as we've alluded to throughout the show, we owe a bunch of thank yous to lots of people who are so kind to help us out, including people who have way better things to do with their time. So we're very, very grateful.
02:47:50 Speaker_04
I mean, one Ian Buck from NVIDIA, who leads the data center effort and is one of the original team members that invented CUDA way back when. Really grateful to him for speaking with us to prep for this.
02:48:02 Speaker_01
Absolutely. Also, big shout out to friend and listener of the show, Jeremy from ABC Data, who prepared four PDFs for us. Completely unprompted, like an insane write-up for us about a lot of the technical detail behind this.
02:48:17 Speaker_04
Private blog posts.
02:48:19 Speaker_01
Yeah. Private blog posts. So our Acquired community is just the best. Like you guys continue to blow us away. So thank you.
02:48:27 Speaker_04
Julian, the CTO of Hugging Face, Oren Etzioni from AI2, Luis from OctoML, and of course our friends at NZS Capital. Thank you all for helping us research this.
02:48:40 Speaker_01
Indeed. All right. Carveouts. Let's shift gears. Carveouts, what you got?
02:48:46 Speaker_04
My wife and I have been on an alias binge. Oh, wow. Yeah. Jennifer Garner? Yes. I never saw it when it came out. It is like the perfect early 2000s junk food when you have one more hour at the end of the day and you're just laying on the couch.
02:49:00 Speaker_01
I never have one more hour at the end of the day. I have a two-year-old. But I really appreciate it. For 16 years from now, when she goes to college, I'll keep that on my list.
02:49:09 Speaker_04
Oh, you play games.
02:49:10 Speaker_01
Oh, that's true.
02:49:11 Speaker_04
But that's research. I'm just checking out the latest graphics technology. So my review of Alias is it's a little bit campy. They repeat themselves pretty often.
02:49:19 Speaker_04
I mean, it's weird to observe how much TV has changed between now and then, because they make very similar shows today, but they're just much more subtle, they're much darker, they leave much more sort of to the imagination.
02:49:32 Speaker_04
And in the early 2000s, everything was just so, like, explicit and on-the-nose and restated three times. I'm just glad the show doesn't have a laugh track, but it's well worth the watch.
02:49:41 Speaker_04
Sometimes you have to imagine it has a different soundtrack, because every episode has, like, a Matrix-type song to it. Bump-a-da-bump-a-da-bump-a-dump-a-dump.
02:49:49 Speaker_01
Yes, that's right. This is like the TV version of The Matrix, right?
02:49:52 Speaker_04
Yes, but it's great. I don't know. We're having a lot of fun watching it.
02:49:55 Speaker_01
My car route, related for my stage of life, also something I missed and discovered recently. We just watched our first full movie, full Disney movie, with our daughter.
02:50:09 Speaker_03
Whoa, what'd you pick?
02:50:09 Speaker_01
A major milestone. And she freaking loved it. I think we picked a great one. Moana. which neither Jenny or I had seen before. And in reading just a little bit about it afterwards, you know how super sadly Pixar kind of fell off in recent years?
02:50:28 Speaker_01
Like, such a bummer. I mean, they're still Pixar, but like, they're not Pixar.
02:50:33 Speaker_04
It's not the guaranteed hit every time that it used to be.
02:50:36 Speaker_01
yeah so moana came out in this kind of generation with tangled and some of the other stuff out of actual disney animation after the pixar acquisition that are just like these are return to form eisner era disney animated just like
02:50:52 Speaker_01
fires on all cylinders and we loved it. We watched with our brother and sister-in-law who don't have kids and are 30 somethings living in San Francisco. They loved it. Our daughter loved it. Highly recommend Moana no matter what live phase you're in.
02:51:07 Speaker_04
All right, great. Adding it to my list. And it's got the rock. How can you complain? There you go.
02:51:13 Speaker_04
Well, listeners, if you want to be notified every time we drop a new episode, and you want to make sure you don't miss it, and you want little hints to play a guessing game at our next episode, or you want follow-ups from our previous episode in case we learn from listeners, hey, here's a little piece of information that we wanted to pass along.
02:51:29 Speaker_04
We will exclusively be dropping those in the email, acquire.fm slash email.
02:51:35 Speaker_01
It was so fun. I think you're about to talk about our Slack. It was so fun watching people in Slack talk about the hints for this episode. We wrote the little teaser and I was like, oh, everybody's going to know exactly what this is.
02:51:46 Speaker_04
No one got it. I was shocked.
02:51:49 Speaker_01
Yeah, eventually somebody did, but it took a couple days.
02:51:52 Speaker_04
Yeah. Uh, we have a hat. You should buy it. And this is not a thing that we make a lot of margin on. We just are excited about more people sporting the ACQ around. So participate in the movement, show it to your friends.
02:52:04 Speaker_01
It's not our super pod, but you know. The pod is the super pod.
02:52:10 Speaker_04
If you come on Acquired LP, you can come closer to the kitchen and help us pick an episode once a season, and we'll do a Zoom call every other month or so. Acquired.fm slash LP.
02:52:20 Speaker_04
Check out ACQ2 for more Acquired content in any podcast player, and come talk about this in the Slack. Acquired.fm slash Slack. Listeners, we'll see you next time.
02:52:32 Speaker_01
We'll see you next time.
02:52:36 Speaker_02
Is it you, is it you, is it you who got the truth now?