Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) AI transcript and summary - episode of podcast Lenny's Podcast: Product | Growth | Career
Go to PodExtra AI's episode page (Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team)) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.
Go to PodExtra AI's podcast page (Lenny's Podcast: Product | Growth | Career) to view the AI-processed content of all episodes of this podcast.
Lenny's Podcast: Product | Growth | Career episodes list: view full AI transcripts and summaries of this podcast on the blog
Episode: Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team)
Author: Lenny Rachitsky
Duration: 01:28:32
Episode Shownotes
Jackson Shuttleworth is a Group PM at Duolingo, where he leads the retention team and the powerful streak feature. The streak feature, which gamifies consecutive days of learning, has been Duolingo’s most important and innovative growth lever and a key driver of their growth to a $14 billion business with
almost 600 million users. In our conversation, we dive deep into the history and lessons of this feature:• The evolution of the streak feature• Biggest insights from over 600 streak-related experiments• Biggest specific wins and misses along the way• Key principles for building effective streak mechanics• How to operate a high-velocity product team• Tips for building engaging notification systems• Much more—Brought to you by:• Pendo—The only all-in-one product experience platform for any type of application• Vanta—Automate compliance. Simplify security• Coda—The all-in-one collaborative workspace—Find the transcript at: https://www.lennysnewsletter.com/p/behind-the-product-duolingo-streaks—Where to find Jackson Shuttleworth:• LinkedIn: https://www.linkedin.com/in/jackson-shuttleworth/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Jackson’s background and an overview of Duolingo’s streak feature(06:00) The impact of streaks on Duolingo’s success(09:58) The origin and evolution of streaks(14:50) Key experiments and insights(24:38) User psychology and engagement strategies(28:36) Duolingo’s product review structure(33:07) Designing for clarity and adaptability(46:59) Developing the Streak Freeze feature(50:47) Balancing monetization and retention(54:08) Notification strategies(58:15) The Perfect Streak feature(01:00:40) Enhancing the user experience (01:04:47) Team operations and experimentation(01:18:57) Who can benefit from streaks(01:21:00) Lightning round—Referenced:• Duolingo streaks: https://duolingo.fandom.com/wiki/Streak• How to make learning as addictive as social media: https://www.ted.com/talks/luis_von_ahn_how_to_make_learning_as_addictive_as_social_media?subtitle=en• Luis von Ahn on LinkedIn: https://www.linkedin.com/in/luis-von-ahn-duolingo/• FarmVille: https://en.wikipedia.org/wiki/FarmVille• Royal Match: https://www.dreamgames.com/games/royal-match• How Duolingo reignited user growth: https://www.lennysnewsletter.com/p/how-duolingo-reignited-user-growth• You’re on fire! Or, how we brought the streak milestone to life: https://blog.duolingo.com/streak-milestone-design-animation/• Duolingo Doubles Down on Design and Animation with Acquisition of Hobbes: https://investors.duolingo.com/news-releases/news-release-details/duolingo-doubles-down-design-and-animation-acquisition-hobbes• Hobbes: https://www.hobbes.work/• Jira: https://www.atlassian.com/software/jira• Atlassian: https://www.atlassian.com/• Bing Gordon on LinkedIn: https://www.linkedin.com/in/binggordon/• Peloton: https://www.onepeloton.com/• Bluey on Disney+: https://www.disneyplus.com/series/bluey/1xy9TAOQ0M3r• Emily in Paris on Netflix: https://www.netflix.com/title/81037371• Multi-position ladders at Home Depot: https://www.homedepot.com/b/Building-Materials-Ladders-Multi-Position-Ladders/N-5yc1vZasew—Recommended books:• A Guide to Midwestern Conversation: https://www.amazon.com/Guide-Midwestern-Conversation-Taylor-Phillips/dp/1984861336• Fate Is the Hunter: A Pilot’s Memoir: https://www.amazon.com/Fate-Hunter-Ernest-K-Gann/dp/0671636030—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed. Get full access to Lenny's Newsletter at www.lennysnewsletter.com/subscribe
Full Transcript
00:00:00 Speaker_00
Duolingo is a $14 billion company. It's hitting all-time highs too. It just keeps going up. I think it doubled in value in the past six months. Streaks is the most impactful feature.
00:00:08 Speaker_01
We have right now over 9 million users with a year plus streak. If you look at the numbers, I think it's been our biggest growth lever. What Duolingo really focuses on is, you know, how do we help users build habits around language learning?
00:00:22 Speaker_01
Getting a user to come back the next day is the biggest problem to solve.
00:00:24 Speaker_00
Let's get into the mother load of learnings from the journey of Streaks. Talk about the key lessons, insights, and also wrong turns along the way.
00:00:30 Speaker_01
I'd say like test everything. We've run in the last four years, over 600 experiments on the Streaks. So every other day, we've actually set up really good infrastructure for copy testing. We used to say continue our like standard CTAs continue.
00:00:44 Speaker_01
And we changed that to commit to my goal. And it was like a massive win.
00:00:48 Speaker_00
There's so much human psychology that you all learn through all these experiments of just like how to motivate people, what motivates, what demotivates.
00:00:55 Speaker_01
Say that you played a mobile game, that you've done it for 3,000 days in a row. I don't know, maybe that hits a little bit different than you've learned Spanish for 3,000 days in a row.
00:01:07 Speaker_00
Today my guest is Jackson Shuttleworth. Jackson is a group product manager at Duolingo, leading the retention team.
00:01:14 Speaker_00
This is a different kind of episode that I'm experimenting with, where we spend the entire conversation focused on the journey and lessons of a single feature, in this case, Duolingo Streaks. Duolingo is a $14 billion business.
00:01:30 Speaker_00
Just over the past six months, they've doubled in value. They're hitting all-time highs in usage and market cap. They're also one of the very few successful and also the single biggest consumer app business in the world.
00:01:42 Speaker_00
And as you'll hear from Jackson, the streaks feature is the single most impactful feature that most contributed to this growth and success.
00:01:50 Speaker_00
In other words, you could argue this one feature created billions of dollars of value, which to me means it is worth studying in depth.
00:01:59 Speaker_00
In our conversation, Jackson shares the history of the Streaks feature, all of the biggest wins and wrong turns they've taken along the way, what he and his team have learned about what works and doesn't work with a Streaks mechanic, and also how they set up their teams to operate in a way that allows them to run over 600 experiments on this product and continue to find big wins.
00:02:18 Speaker_00
I hope to do more episodes like this on features and products that you'd love to hear more about. So leave a comment either on YouTube or on Substack and tell me which product or feature you'd love to see me cover.
00:02:29 Speaker_00
And if you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It's the best way to avoid missing future episodes and it helps the podcast tremendously. With that, I bring you Jackson Shettleworth.
00:02:44 Speaker_00
Jackson, thank you so much for being here and welcome to the podcast. Yeah, thank you. Long time listener, first time caller. I appreciate it. So this is going to be a really interesting conversation.
00:02:56 Speaker_00
I've never done an episode like this before, where we basically spent like an hour, an hour and a half going into one feature of one product. But this is a very special feature. It's a very special product.
00:03:09 Speaker_00
Have you ever spent like an hour, an hour and a half just talking about this one feature with anyone?
00:03:13 Speaker_01
Well, internally, so when we onboard new folks to the team, I actually just did this with somebody that joined the team recently where we spent, and I was like, hey, let's spend an hour just talking about the streak.
00:03:24 Speaker_01
We got through an hour and I got through about 30% of what I wanted to share. There's just so much, we'll talk about this, but there's so much that we've learned over the years. But never anything externally like this.
00:03:35 Speaker_01
I think we've shared bits and pieces of learnings, but this will be like the motherlode of learnings, hopefully. Here we go.
00:03:41 Speaker_00
Of how Duolingo's built the street. We should title this, the motherlode of learnings of Duolingo Streaks. This episode is brought to you by Pendo, the only all-in-one product experience platform for any type of application.
00:03:54 Speaker_00
Tired of bouncing around multiple tools to uncover what's really happening inside your product?
00:03:59 Speaker_00
With all the tools you need in one simple-to-use platform, Pendo makes it easy to answer critical questions about how users are engaging with your product, and then turn those insights into action.
00:04:09 Speaker_00
All so you can get your users to do what you actually want them to do. First, Pendo is built around product analytics, seeing what your users are actually doing in your apps so that you can optimize their experience.
00:04:20 Speaker_00
Next, Pendo lets you deploy in-app guides that lead users through the actions that matter most. Then, Pendo integrates user feedback so that you can capture and analyze what people actually want. And the new thing in Pendo?
00:04:33 Speaker_00
Session replays, a very cool way to visualize user sessions. I am not surprised at all that over 10,000 companies use it today.
00:04:41 Speaker_00
Visit Pendo.io slash Lenny to create your free Pendo account today and start building better experiences across every corner of your product. P.S. You want to take your product-led know-how a step further?
00:04:53 Speaker_00
Check out Pendo's lineup of free certification courses led by top product experts and designed to help you grow and advance in your career. Learn more and experience the power of the Pendo platform today at Pendo.io slash Lenny.
00:05:10 Speaker_00
This episode is brought to you by Vanta. When it comes to ensuring your company has top-notch security practices, things get complicated fast.
00:05:19 Speaker_00
Now you can assess risk, secure the trust of your customers, and automate compliance for SOC 2, ISO 27001, HIPAA, and more with a single platform, Vanta.
00:05:31 Speaker_00
Vanta's market-leading trust management platform helps you continuously monitor compliance alongside reporting and tracking risk. Plus, you can save hours by completing security questionnaires with Vanta AI.
00:05:44 Speaker_00
Join thousands of global companies that use Vanta to automate evidence collection, unify risk management, and streamline security reviews. Get $1,000 off Vanta when you go to vanta.com slash lenny. That's V-A-N-T-A dot com slash lenny.
00:06:02 Speaker_00
Okay, so for people that don't know anything about what Duolingo Streaks are, can you just first give a brief explanation of what is Duolingo Streaks, what is this feature all about?
00:06:10 Speaker_01
Well, presumably people know Duolingo is a language learning app. The Duolingo Streak tracks, currently anyway, how many days in a row you've done a lesson. So you come to Duolingo, you do a lesson.
00:06:22 Speaker_01
Your first lesson, you'll start a streak, and then every consecutive day that you come, use the app, you'll extend your streak. And I should actually put an asterisk around
00:06:30 Speaker_01
how many days in a row, because we also have built flexibility into the features. So we have these things called streak freezes. It's like insurance for your streak.
00:06:38 Speaker_01
So, you know, it's a pretty simple feature, you know, in theory, but over time we've layered on challenges and goal setting and rewards and, you know, social features, you know, a lot of our notifications are tied to the streak.
00:06:52 Speaker_01
So pretty simple feature to understand, but it's been a really rich feature for us to build on top of.
00:06:58 Speaker_00
Some people might be hearing this and be like, Duolingo streaks? What's the big deal? There's other people that are like, holy shit, I want to learn everything I can possibly learn about Duolingo streaks.
00:07:06 Speaker_00
So many companies are trying to copy what you've learned from this. Give people a sense of the impact that this one feature has had on Duolingo's success and growth.
00:07:14 Speaker_01
This is, and this is not just the, you know, subjective retention PM talking. I think this is our biggest feature with the exception of the lessons. And I think it's actually worth start that like streaks are a great engagement hack.
00:07:27 Speaker_01
I'm kind of of the opinion that any team, you know, any app out there can introduce a streak. And if you figure it out, it probably works.
00:07:33 Speaker_01
Um, you know, to retain users, but at the core, like you have to have an app that people want to use and people really like using duolingo. It's fun. It's delightful. You learn something.
00:07:41 Speaker_01
And so it allows us to layer an engagement mechanic on, on top of that, like the streak is really powerful. So it ships.
00:07:49 Speaker_01
a disgusting amount of DAUs again it is just it is it is our um one of our golden geese and and again what's cool is that like you know you look at notifications notifications for Duolingo is massive I mean you know us sending better you know whether it's copy or timing so many of our notifications work because they reference the streak because users care about the streak and so not only is it
00:08:12 Speaker_01
itself, us iterating on the streak, a huge driver of DAUs, but it's also something that enables other really high, valuable features. I was looking up some stats before I came on and it's pretty crazy.
00:08:25 Speaker_01
So we have right now over 9 million users with a year plus streak. So 9 million of our users have had uh, use Duolingo every or almost every day for, uh, you know, well over a year. Um, which, which is pretty, which is, it's, I don't know.
00:08:43 Speaker_01
It's, I like to imagine things in terms of like, you know, okay, well, like if you put all these people in a city or, you know, in a place, where would it be? And like, ah, it's like a very large city, 9 million people. So.
00:08:53 Speaker_00
for a year, have a year-long streak. That's incredible. I was just looking at the stock of Duolingo. So Duolingo is a $14 billion company at the time we're recording this. It's hitting all-time highs too. It's like, just keeps going up.
00:09:06 Speaker_00
I think it doubled in value in the past six months, something like that.
00:09:10 Speaker_00
And what I'm hearing from you is Streaks is, other than just like the core learning feature, which is just like the product of Duolingo, this is the most impactful feature in terms of growth and in that retention specifically.
00:09:23 Speaker_01
I mean, if you look at the numbers, I think pretty objectively has been our biggest growth lover for driving DAUs. And also say a lot of it's just related to how we think about growth at Duolingo. And a lot of what we try to do is organic growth.
00:09:37 Speaker_01
We think about growth just as much as bringing new users onto the platform is not losing them. If you're just bringing people onto the platform, then they churn. that's not going to be sustainable.
00:09:46 Speaker_01
And so as much as we can do to keep our users coming back and actually retaining as users, it's going to make it, it's going to give us a much easier base to continue growing DAUs off of.
00:09:57 Speaker_00
Perfect. Okay. Talk about how this feature originally came to be. What was the original version of it? What was the original insight that led to V1?
00:10:04 Speaker_01
Yeah, so the oldest Streaks are as old as Duolingo itself. When we launched Duolingo, we launched with a Streak. And I say we, I was like just graduating undergrad when Duolingo launched. So this is well before my time.
00:10:18 Speaker_01
But we launched with a Streak feature The, you know, the initially how the, how the feature worked was you'd come to Duolingo and you'd set a goal for yourself. And it was an XP based goal.
00:10:29 Speaker_01
So Duolingo, a little bit of like, you know, nomenclature, we have a experience point space system that drives a lot of our features in the app.
00:10:36 Speaker_01
And the way you'd set it as you'd say, you know, based on what your language learning goal was, you know, maybe you have a 10 experience point goal versus a 50 experience point goal.
00:10:44 Speaker_01
And so extending your streak would be, Hey, can you hit 50 experience points if that's what you set your goal to be?
00:10:51 Speaker_01
And this worked well, I think this also speaks to like how Duolingo initially grew, you know, we were, Luis launched it with a TED talk, you know, we probably had a more tech forward user base, you know, initially, and so this whole idea of like an experience point space streak system made sense.
00:11:05 Speaker_01
But what that also meant is that you could have a user come and use the app, do multiple lessons a day, and maybe they just set too hard of a goal for themselves, and then lose their streak, which you don't need to be an expert in streaks to understand that's probably not good.
00:11:19 Speaker_01
The nice thing with how we initially set it up, though, is it really did connect with what your goal was. So if you were serious, let's track how good you are at being a serious language learner.
00:11:28 Speaker_01
But I'd say one of the most impactful experiments we ran was about, this is actually as I was joining, or just after I joined, we had run this experiment, was to move it from a XP-based streak to just do one lesson a day and you'd extend your streak.
00:11:45 Speaker_01
And the risk that you can sort of imagine is, well, then users kind of care less about it because it's not connected with their goal. And we saw none of that. I mean, this was a huge driver of DAUs.
00:11:55 Speaker_01
Just making it easier to extend your streak, but I think really importantly, is still meaningful, right? The unit of use, as you're thinking about building a streak, I think it's really important to think about what the unit of use of your app is.
00:12:09 Speaker_01
The unit of use for Duolingo is like doing a lesson.
00:12:11 Speaker_01
And so if what we care about is users coming back every day and doing a lesson, because it shows that they're actually engaging with the app, then it doesn't hurt us to make our streak focus on just do one unit versus multiple.
00:12:23 Speaker_01
And so that was like probably the big sea change experiment that we ran, you know, at the time was moving from an XP-based streak to a one lesson streak. It's also simple. And I think that's like one of the things to think about with streaks.
00:12:33 Speaker_01
It's always easy as a PM to like, over, you know, have a million goals or objectives for what you want your feature to be and potentially build a more complex feature. You know, one less in the streak. It's just easy for more users to understand.
00:12:45 Speaker_00
Yeah, that's exactly what I was going to say. And just for folks that aren't super familiar with XP, it's basically experience points and you get them from like doing things. It's like an
00:12:53 Speaker_01
Yeah, you do stuff on Duolingo and then based on what you did and how well you did it, we give you experience points. This actually drives a number of our features in our app. So leaderboards is the big one.
00:13:04 Speaker_01
We have a leaderboard system where you try over the course of the week, you battle with 29 other people and you'll want to win. That's all driven by XP. So we do have other features in the app that really benefit from this XP system.
00:13:16 Speaker_01
The streak is just no longer one of them.
00:13:18 Speaker_00
Awesome. Okay. So I want to talk about the journey from that point to what it is today, but a quick tangent. I saw Luis tweet just this week. Someone asked him, how do you decide whether to optimize for learning or engagement? And he's like, no question.
00:13:33 Speaker_00
Everything we do is focused on engagement because you won't learn anything if you're not coming back to the app.
00:13:39 Speaker_01
As an engagement PM, that was like the coolest thing he could have ever said. Well, again, very much subjectively, I guess, as an engagement PM. I'm sure the learning folks at Duolingo will cringe when they hear it.
00:13:53 Speaker_01
I see myself as a learning PM as much as an engagement PM because the easiest way not to learn on Duolingo is not to come back the next day. And so if we don't make the app retentive, you will have no opportunity to engage with our learning features.
00:14:08 Speaker_01
I do think that there is a long tail of learning where if you start to dumb down, and honestly, this is something we wrestle with the streak as well. You start to dumb down the experience and your users aren't actually learning.
00:14:18 Speaker_01
They're not going to care. I mean, streaks work best when they're sitting on top of an app that users care about. But yeah, I mean, if you can't come, if you don't come learn on Duolingo, if you don't come back to Duolingo, then you're not learning.
00:14:33 Speaker_01
So we track a lot of this with You know, the work that we do, we make sure that as we're making changes to the Streak, we're not hurting the learning experience, and we don't have a ton of interaction with it.
00:14:44 Speaker_01
So we're constantly thinking about this, but thank you, Luis, for saying that.
00:14:48 Speaker_00
It makes sense to me. Okay, so let's get into the mother load of learnings from the journey of Streaks. So the first version went from XP to one lesson. Talk about the key lessons, insights, and also wrong turns along the way to what we see today.
00:15:03 Speaker_01
Duolingo has a strong test it philosophy. We're willing to test a lot of different... Honestly, we'd much rather test it than debate it for days and days.
00:15:16 Speaker_01
So we actually followed up this experiment with, and this is a little bit later, with, hey, what if we make it even easier to extend your streak? And so we actually tested
00:15:26 Speaker_01
hey, if you do one exercise, just one exercise in a lesson will extend your streak. And a lot of the insight was good.
00:15:35 Speaker_01
You look at the funnel, hey, there's a lot of users who are starting but not finishing lessons, they're not extending your streak, the loss aversion doesn't kick in, they don't come back. So this kind of followed that train of thought.
00:15:45 Speaker_01
What we realized when we ran this experiment is, you know, DAUs moved not one bit.
00:15:52 Speaker_01
And what we were doing by, and this, if we go back to like unit of measure, you know, we had dumbed down the unit of, and nobody thinks about, I just want to come, you do one question on Duolingo. Nobody thinks about that.
00:16:05 Speaker_01
So we had more, like a less clear unit of measure that we were basing our streak around. And the users that we were capturing with our streak, You know, you come, you do one or two questions on Duolingo, then you leave.
00:16:17 Speaker_01
We're like the least engaged users imaginable. And so, you know, I think that's something also to think about as you're building your streak is like, what is the user that you're solving for?
00:16:24 Speaker_01
So not only what is the kind of habit that you're building for, but like, what is the level of commitment? And that was an example where we over indexed on a type of user who we honestly just weren't going to keep.
00:16:36 Speaker_01
That was a very easy shutdown decision.
00:16:39 Speaker_00
That's an awesome story. Just to comment on that real quick, just like you went, like, let's just go to the extreme and make it just like Streaks. Yeah, get everyone going Streaks forever.
00:16:47 Speaker_00
And then I love that it turned out and it's not bringing the users that you want. And it's dumbing down experience. It makes me think of FarmVille where you have to go and like harvest your crops every whatever hour and just like that lasts for a bit.
00:17:00 Speaker_00
And then eventually people are like, what the hell am I doing with my life?
00:17:03 Speaker_01
Yeah, exactly. No, we, I mean, and we test every, so we, I was looking at the numbers as well. We, we've run in the last four years, over 600 experiments on the street. So every other day, effective hundred.
00:17:15 Speaker_01
we're running an experiment and they range from big like this, like changing how the mechanic works to like, let's swap a string with another string and see if that copy is better, you know, for, for users.
00:17:26 Speaker_01
So, so we're constantly testing on, on everything. I do think that like, I'd be more careful running that experiment now. Like at some point your streak gets big enough that Again, I got 9 million users on the streak. I got to be really careful.
00:17:39 Speaker_01
Those are our best retaining users. You got to be careful. But in the early days of the streak, I'd say test everything. Don't get super caught up in, it has to be like this. Just test a bunch of stuff and see what speaks most to users.
00:17:53 Speaker_01
Because I think, again, you will constantly be surprised by the insights that you get from whether you, we shut down about half of our experiments. So half of our experiments lose, we still learn a ton by virtue of running them.
00:18:06 Speaker_01
So super, super valuable.
00:18:08 Speaker_00
That's actually a good success rate. A lot of companies have only 20% experiments be positive. What's your policy on is if it's neutral, do you ship it or do you kill it?
00:18:18 Speaker_01
It really depends. If we're adding something and it's neutral, we tend to shut it down because it's just more cognitive load.
00:18:27 Speaker_01
It's something that we're going to have to start building around, a new UI element that we have to figure out how it fits into our system. I'd say when we do ship a neutral experiment is
00:18:35 Speaker_01
Something that we have real conviction around, okay, yeah, maybe this was neutral, but it's gonna give us a new platform to then build on top of. So that experience might be neutral, but now we can build these DAU positive experiments.
00:18:50 Speaker_01
And my general take there is though, in that case, build that in as part of your V1 so that you make sure that at least your hypothesis around this roadmap has play, should probably be the case. But in general,
00:19:05 Speaker_01
shutting down a neutral experiment so you don't introduce more complexity to the app is the way we tend to go.
00:19:10 Speaker_00
Makes sense. All right. What else? What else have you learned along the journey?
00:19:13 Speaker_01
Well, I mean, maybe I can talk about a few different ways that we structure, you know, a few different themes that I think we lean on. So the first is focusing on the zero to seven day user experience.
00:19:25 Speaker_01
And I would say this is, if you look at the kind of, you know, whatever breakdown of experiments are, we run definitely more than average number of experiments on getting users to go from a zero to seven day streak.
00:19:37 Speaker_01
And a lot of this is because we've looked at the data for our retention curves. And what we found is that once you get to seven days, loss aversion kicks in and you retain. So going from a one to a two-day streak, huge jump in retention.
00:19:50 Speaker_01
Two to three-day streak, slightly less, but still huge. And it's up until day seven. Once you hit day seven, it flattens out.
00:19:56 Speaker_01
And it's not to say that if you have a 30-day streak, you're way more attentive than day seven, but not in the order of magnitude that it is from day one through seven.
00:20:06 Speaker_01
We do a ton of work to get users to that point where loss aversion kicks in and then they don't want to, they don't want to leave the app.
00:20:15 Speaker_01
One of the fun ones that we did, and it's honestly as much about process as it is about the feature itself, was we have a feature called Streak Goal.
00:20:23 Speaker_01
And it is, again, so much of this stuff seems so obvious in retrospect, but, you know, it was really novel at the time. You know, we had this idea of like, hey, maybe we'll just goal users to hitting a certain streak length.
00:20:36 Speaker_01
As you can imagine, this is pretty powerful user psychology. And we started with the simplest version of this. And this is how Duolingo does a lot of our testing.
00:20:44 Speaker_01
Rather than design the big complex feature for V1, just do the simplest encapsulation of what that feature can be, see if it has legs, and then just add to it iteratively over time. This is partially how we get to 600 plus experiments on Streaks.
00:20:59 Speaker_01
They're not all big ones. But we started, and it was funny, we actually took a learning from our modernization team.
00:21:04 Speaker_01
So one of the strings that they had, the pieces of copy that they had worked really, really well was, I think it was your 5.6 times more likely to finish the course if you subscribe to Plus. Now it's super, our subscription.
00:21:19 Speaker_01
And it was a really good hook that if you really cared, you'd sign up. And so we had a similar thought where it's like, oh, let's just tell you how much more likely you are to finish the course if you get a 30-day streak.
00:21:34 Speaker_01
And so we started with that, and I think it was like, you're seven times more likely to finish the course if you have a 30-day streak. And just that message when you started your streak, us telling you that was awesome. Huge win.
00:21:46 Speaker_01
You know, indicating an outcome and like Duolingo doesn't have, we don't, but we have a gem economy, but we don't actually have like, you know, it's just, it's, it's all you learning, but being able to actually talk about it in terms of the outcomes that a user would think about in this case, trying to finish the course.
00:22:01 Speaker_01
was a huge win. So this is where we started and we're like, ah, goal setting. All right, we should go much harder on this. And now let's just beat the heck out of it. So we followed that up with another experiment where we tested different lengths.
00:22:14 Speaker_01
So we'll test 14 days and 50 days. And we found that they were all good, but they appealed to different users. And so we started to realize, all right, well, we probably need to be more thoughtful about who we give these different options to.
00:22:30 Speaker_01
And so then we followed that experiment up with alright let's start with thirty days and then we're gonna let you opt out.
00:22:36 Speaker_01
You know what will say no i don't think i can hit it and i think it was in that if you say no we'll hit you with an easier goal we just wanted to get you to commit to a goal and this is a fascinating one because.
00:22:48 Speaker_01
It was a good win to give users that easier goal to try to capture them before they said no. But it was almost just as big a win to add that opt out button.
00:22:57 Speaker_01
So we tested separately and I'm a huge fan of testing like way too many arms for an experiment just to like be able to isolate your hypotheses. But we captured just what happens if we add an opt-out button.
00:23:09 Speaker_01
And adding an opt-out button, and you would think as a PM, oh, I'm like, now users might not engage with my feature. That's a bad thing. But it was a huge win to let them do that. And the learning here was that this intentionality of saying, no,
00:23:22 Speaker_01
No, previously it was just a continue button, but now it's like, no, I want to hit 30 days. And having that be an intentional decision for them, yes or no, even though again, this had no impact or no impact past this screen.
00:23:36 Speaker_01
Everything that I'm talking about now was just that screen that day and then was all thrown out. So that optionality was a huge insight.
00:23:48 Speaker_01
And so because of that, we built a goal setting feature where you could choose between different goals, giving users that optionality was likewise a huge win. I'll say one final learning on this.
00:24:00 Speaker_01
Um, again, you talk about like friction and good and bad friction.
00:24:04 Speaker_01
We thought once we built, uh, you know, a goal picker screen where you could pick between different streak lengths, we, we were like, oh, well, let's like recommend that users do a hard, you know, a harder goal.
00:24:15 Speaker_01
I'm thinking that, okay, well, a harder goal is going to be better retention and, and we'll pre-select the harder goal for them. And this, based on all the learnings that I just shared with you, you probably imagine lost pretty significantly.
00:24:25 Speaker_01
We realized that like, yes, we could speed users through the screen more by virtue of picking a goal for them.
00:24:30 Speaker_01
But like that act of selecting, I think it's 30 days, I think it's 14 days, was where we were getting so much of the engagement from this feature.
00:24:39 Speaker_00
There's so much human psychology that you all learn through all these experiments of just like how to motivate people, what motivates, what demotivates. It's like you guys should write a book on human psychology and motivation.
00:24:51 Speaker_01
I feel very much like a amateur armchair psychologist with everything that, at least as far as people who want to learn languages on their phone go. Right, right. I really understand those folks.
00:25:03 Speaker_00
So one, one kind of theme I'm hearing here so far is you guys are basically just like mining for gold, just like looking for like a vein in some mine. And once you hit it, you're like, Oh, this worked. You just go.
00:25:17 Speaker_00
crazy on just testing all kinds of things to see how far you can take that one little thread.
00:25:22 Speaker_01
Yeah. And I think I shared how this idea came partially from a modernization win that they had.
00:25:31 Speaker_01
I think there's a lot of... Because Duolingo runs, I don't know what percentile we're in, but it's got to be a very high percentile of per capita experiments run for a company based on company size. We're just constantly learning so many things.
00:25:47 Speaker_01
And there's a really great cross-sharing of Modernization say, hey, this thing worked for us. Is that something you can use? So I'd say it's rare that we go into something where it's just like, well, let's just, let's just try something.
00:25:56 Speaker_01
Typically we have some insight because of all of these experiments that we've run that. Hey, if we do this, I don't know it's one here or it's worked here. It's driven this user engagement.
00:26:07 Speaker_01
If we, you know, sort of massage that and make it, you know, try it in this, um, you know, scenario or like a different screen, you know, we, we come in at least with a strong hypothesis of this will work.
00:26:17 Speaker_01
A lot of times we do look and we're like, you know, a lot of the apps that we look at actually are games themselves.
00:26:22 Speaker_01
So it's like, all right, you're playing Royal Match or, you know, there's a new Pokemon trading card game that I'm spending way too much time on.
00:26:29 Speaker_01
You know, you look at these games and see what they're doing, and it's really good fodder for what we can do. But a lot of times you're at least going with a strong hypothesis based on what you've seen work elsewhere.
00:26:39 Speaker_00
You got it. So just to double down on that point, it's not like just random experiments. It's here's a hypothesis. We're fairly confident or has a chance to be true. Let's try it. It's not just let's just try everything.
00:26:49 Speaker_01
And I think that like how strongly you feel about the hypothesis directly ties to how hard that experiment is. Like with copy, for instance, we we've actually set up really good infrastructure for copy testing.
00:27:00 Speaker_01
And I'm of the opinion that companies should run as long as you have the user base to do it, like copy test constantly, like the amount of copy test that we've had that is just like
00:27:09 Speaker_01
that have won and you, I don't know, you just try things and you figure out what wins is definitely Legion. And have shipped like massive wins from little copy changes. Is there an example of that or just like the impact?
00:27:20 Speaker_01
Going back to that goal screen, we used to say continue, our like standard CTAs continue. And we changed that to commit to my goal. And it was like a massive win. Even just, you know, and again, it was like, okay, user's tapping on that.
00:27:35 Speaker_01
What are we asking them to do? Commit to the goal. What is that gonna lead them to do? Commit, not, you know, churn. Just that little copy change, that one time right there led to huge wins. And copy changes are so cheap.
00:27:48 Speaker_01
Like, it's just, you translate a, you know, for us, you know, we have a lot of, you know, users all over the world and a lot of UI languages, so, but like, just come up with a bunch of ideas, translate some strings. This is one where,
00:27:59 Speaker_01
know, the feedback that you'll typically get from Luis. So all of our changes at Duolingo go through product review that, you know, are reviewed by Luis. So Luis reviews every single change that, you know, we propose or every experiment that we run.
00:28:11 Speaker_01
Typically with Copy, he's just like, I don't know, test it. Like, there's nothing better than to be told by Luis, I don't think this is going to win, but Sure, if you want to. And a lot of times, to his credit, he's right.
00:28:23 Speaker_01
And a lot of times, our intuition was right, but it's just so cheap to do it. I think when the lift is smaller, it's great to have a hypothesis, but you don't need to beat it up too much.
00:28:36 Speaker_00
So on this thread, I didn't realize Luis reviews everything you're planning to change.
00:28:39 Speaker_00
And this may be the answer to the question I was going to ask, which is one of the criticisms of running a product and company this way of just experimenting constantly with all these micro improvements and changes is it can lead to something like a monstrosity of a product and experience that isn't
00:28:54 Speaker_00
consistent and cohesive and just like, that often happens. Is the solution to that having the founder basically review all the changes? Is there anything else y'all do to avoid it moving into becoming FarmVille or whatever? It's a good example.
00:29:06 Speaker_01
Yeah, our product review structure where we've got our head of product design, one of the lead, you know, the product management leaders, and then Luis in PR.
00:29:16 Speaker_01
And because they see everything, you know, and they have a really high product bar, and so that helps I think over time, we just, as PMs, have to look at, okay, where is our feature headed?
00:29:29 Speaker_01
And so we do this with the Streak, at least on a quarterly basis, to look at, okay, well, what have we learned? How has our Streak developed? And how do we imagine this going in the future?
00:29:40 Speaker_01
I don't think... It's easy to do and end in some sort of awful local maxima if you're not constantly looking at your roadmap and thinking, you don't have a clear strategy.
00:29:51 Speaker_01
For me, it's like, if you have a clear strategy for where your feature is going, hopefully all of those A-B tests are not just done to get some cheap gain. They're done with a long-term goal in mind.
00:30:03 Speaker_01
I do think though, and we do this every now and then with the streak, eventually you just hit a local maxima and you say, Like you talked about launching neutral experiments. This is a great example where it's like, all right, cool.
00:30:12 Speaker_01
Now we need to throw a bunch of this stuff out based on all the learnings. Can we reset this real estate?
00:30:17 Speaker_01
Can we reset this UI, reset this feature in such a way that is just as good as what we have now, but is way more plain or simple that we can, again, start to layer on it.
00:30:27 Speaker_01
Those are really hard experiments to get win on, you know, to win, obviously, because they are so optimized, but they're really important to do. Otherwise, yeah, you just end up with a kitchen sink of a feature.
00:30:38 Speaker_00
One other tidbit I just want to mention, an advantage you all have that other companies don't have is people want to learn.
00:30:49 Speaker_00
They want to learn the language, and so getting pushed to come back to an app for something that they want to do is not an advantage a lot of products have.
00:30:58 Speaker_00
So anything you want to add there of just like, this is why Duolingo might be a little different from what you're working on.
00:31:02 Speaker_01
I mean, that that is definitely a benefit. If I had an app that this is this is actually why I think a lot of mobile games do streaks differently, because.
00:31:12 Speaker_01
you know, to say that you played a mobile game, and as somebody who plays a lot of mobile games, you know, that you've done it for 3,000 days in a row, like, I don't know, maybe that hits a little bit different than you've learned Spanish for 3,000 days in a row.
00:31:25 Speaker_01
I think the comparison set is much larger, though, than a lot of companies give them credit for, you know, themselves credit for, and I think that there's, I mean, there's a lot of ways that companies think about their, there are very few companies, I imagine, out there in the world saying, oh, we don't do some degree of good for our users,
00:31:40 Speaker_01
Even if it's a game, it's like, I don't know, you're giving somebody a moment away from the craziness of their lives.
00:31:47 Speaker_01
And so I do think, though, that it's contingent on companies who are going to figure out if a streak works for them to figure out how can you frame the streak in such a way that a user does feel good about it.
00:31:57 Speaker_01
And it's easier for a Duolingo, but I think there's creative ways to phrase this for users.
00:32:03 Speaker_01
The other thing, maybe just on that, that I'll say is the Streak works really well for Duolingo because with language learning, it's really hard to see day-to-day progress and becoming more fluent. And fluency is not even the right word.
00:32:18 Speaker_01
It's like becoming better at Spanish or whatever. It is a years long process for someone to get better at a language. I mean, Duolingo makes it easier, but you still got to put in thousands of hours if you're going to reach C1 or C2 fluency.
00:32:31 Speaker_01
And that is really hard to track on a day to day. And so the streak works really well for us because we might not be able to tell you, hey, you now know 0.01% more Spanish, but we can show you, hey, you've gotten your streak a little bit higher.
00:32:48 Speaker_01
And so I think this works particularly well when you're an app that is doing something that's gonna be sensed or felt over a longer term to help contextualize that progress in a way that makes more sense, or at least feels more tangible to a user.
00:33:02 Speaker_00
Great, that was a great context. Empowering to a lot of companies that aren't necessarily doing language learning. Okay, so it took us on a long tangent away from lessons and experiments you ran along the journey of iterating on the streak.
00:33:14 Speaker_00
So a few things you've shared so far is just like it started being like started with this XP idea and then went to like a lesson. Then you iterated on ways to make it simpler, maybe harder.
00:33:24 Speaker_00
You added streak goals where you commit to like, I'm going to hit a certain goal of streak. What else? What else have you found that has worked, didn't work, lessons learned?
00:33:32 Speaker_01
Again, sticking with this one to seven day streak, the idea of a streak, particularly probably to this audience, is obvious. Like, oh, it's a streak. It just counts how many days.
00:33:45 Speaker_01
We've realized over time that a lot of users do not understand how a streak works.
00:33:50 Speaker_01
And it can be as big as you know as small as well i don't understand how street freezes work or i don't like my mom the other day i was talking about she's like oh my i like didn't use duolingo and i come back my street still there's like there's like certain elements of the feature that you know we can do better explaining but even like.
00:34:08 Speaker_01
What a streak is it's it's tracking how many days that i've used the app yet the more that we can make the feature easily comprehensible to users.
00:34:18 Speaker_01
The more retentive it is in it in and we've run a number of experiments to this you know a simple you talked about you asked about early easy copy changes that we made. This is my first when i experiment when when i joined doing go was.
00:34:31 Speaker_01
When you start a streak we use we have little copy at the bottom of the screen that just I don't know it's kind of like flavor copy we use it to celebrate you or give you context and I ran some tests that just like tried to in 8 words explain what a streak was that was that.
00:34:47 Speaker_01
And it was a massive win because it really required like it really dumbed down here is exactly how the streak worked. And it really helped users just understand what they needed to do.
00:34:59 Speaker_01
And I think this is something that's like you kind of constantly got to remind yourself, particularly if you work in tech and you're building cool tech features, but your user base is not.
00:35:08 Speaker_01
a bunch of tech workers to think about, all right, who is my audience? And for us, it's not just tech workers. People in America, people all over the world of all ages, of all cultures.
00:35:19 Speaker_01
And so making sure that your feature is even something as simple as the streak is understandable is critical.
00:35:27 Speaker_00
What was the actual copy do you remember it was it's still I think it's still on the app like start a day to extend your streak but miss a day and it resets something like that makes sense to me very clear and yeah words I love that OK and then when you say massive win by the way just to give people a reference point what do you think what does that look like what is a massive win in this skill.
00:35:50 Speaker_01
Well, and it's funny. I mean, this was four years ago, but I think it was like in the order of magnitude of over 10,000 DAUs for us. And actually maybe just a small bit of context. So Duolingo really cares about the metric current user retention rate.
00:36:04 Speaker_01
And actually our first ever Duolingo post with Lenny was the newsletter that our former head of product wrote, Jorge.
00:36:11 Speaker_00
still the single most popular newsletter post of all time in my newsletter across 300.
00:36:15 Speaker_01
I would highly recommend that if you are interested in this, give it a read. You know, to summarize, basically what we found is that if you want to drive DAUs and Duolingo cares, you know, our growth north star is DAUs.
00:36:27 Speaker_01
The metric that is most effective, you know, where a percentage change in that metric is most effective at driving DAUs is current user retention rate. And this is just users who are not new or resurrected, getting them to come back tomorrow.
00:36:41 Speaker_01
And so most of the work that our teams do, our retention-based teams do, is focused on Kerr. And so the retention team that I lead focused on Kerr. It just so happens that Streak is the best feature at driving Kerr.
00:36:53 Speaker_01
And so this experiment was the biggest Kerr win that we had had That was like a top three Kerwin, anyway, for us. Just this little copy, right? And that's why I say, like, just copy a thousand different ways.
00:37:07 Speaker_01
Sometimes it's not the big, beautiful feature, you know, that's going to drive the huge gains. Sometimes it's just something simple as a few words. I love this.
00:37:17 Speaker_00
OK, so when you said 10,000 DAUs, I think that references you guys measure incremental impact and absolute numbers of new daily users you're going to drive with. Attributed to that experiment.
00:37:28 Speaker_01
Yeah, and we do both. I mean, we'll also look at like a lot of times I'll look at, you know, for retention day one versus day 7 versus day 14. A lot of what I'm looking for is for us to have a better day 14.
00:37:41 Speaker_01
Excuse me, better day 14 impact than better day day one impact, because it means that users are retaining better overtime. This is particularly for users that would see a feature multiple times. I just like absolute DAU numbers because.
00:37:54 Speaker_01
As long as you're controlling for different biases, like a recency bias or a novelty bias, it's a really easy way to just have an absolute comparison.
00:38:04 Speaker_01
You start to look at percentage changes and then it's influenced by who you're treating, how many users saw the experiment, but at least an absolute number is easier in my mind to start comparing.
00:38:16 Speaker_01
Again, there are pitfalls with it, but we find that that's a pretty useful way.
00:38:20 Speaker_00
That lesson comes up a lot on this podcast than that approach to experiment. So yeah, you're in good company. Quick tangent. If there's not an answer to this, no problem.
00:38:31 Speaker_00
Coming back to the idea of just like experimenting like crazy and not creating a product that nobody wants to use anymore long term.
00:38:38 Speaker_00
Is there an example of an experiment that was positive that you all decided, no, we don't actually think this is what we want in the product that ended up not shipping?
00:38:47 Speaker_01
retention doesn't only work on the streak, although you would think most of our work is the streak. We've touched a lot of different surfaces over the years, and there was one experiment that we launched that we talked about XP earlier.
00:38:58 Speaker_01
In the lesson, the only UI elements are a progress bar at the top, and then how many hearts you have, right? So we keep it really simple, and this very much speaks to the design philosophy of Duolingo, which is simpler UI is better.
00:39:12 Speaker_01
And we decided, hey, let's add XP in there. And so let's show your XP ticking up as you're going through a lesson. That's going to make the user feel good. It's going to show you what you've earned. You're going to be less likely to quit.
00:39:24 Speaker_01
All of these good reasons to do it. And then you finish the lesson, and then we'll show you've collected all this XP. And it won. I mean, the hypothesis was a good one.
00:39:33 Speaker_01
But we realized and I remember having this conversation with Luis is like, cool, this is our most important screen in the app. It is our lesson is where users learn. And the focus here is on learning.
00:39:43 Speaker_01
And now you've added this other thing up there to distract, you know, that could be distracting for users.
00:39:50 Speaker_01
And I think the question, you know, we talked about roadmaps and strategy here, the question that he had for me, and I didn't have a good answer for at the time was like, so what else are you going to do with us? Like, what's your iteration ideas?
00:40:00 Speaker_01
Where's this going to go? Is this going to make the lesson experience more gamified? And what we realized is that honestly, it was kind of just an easy engagement win idea, but we had touched our most sacred space in the app to do that.
00:40:16 Speaker_01
And so that was a case where it's like, yeah, it was a nice win, but we'd added that UI element and, you know, at least at the time, it was less clear what we would do with it. And we realized that like long-term, it was just going to get in the way.
00:40:30 Speaker_01
And we'd rather, for simplicity's sake, pause that, you know, shut it down and keep the lesson, you know, to be the, a little bit of a learning sanctuary it was.
00:40:40 Speaker_01
Now it's funny, nowadays, I think we actually have enough XP-based mechanics and fun things that we can do that like, I think actually a lot of our,
00:40:48 Speaker_01
you know, the beliefs about the in-lesson experience have changed, so something like that could work, but at the time, you know, didn't feel good to keep that around.
00:40:56 Speaker_00
That was an awesome example.
00:40:58 Speaker_00
Hopefully we have time to talk about how the team operates, where my mind goes is like, oh, but you have all these PMs and teams that want to show impact and the performance reviews and all that stuff, and you're shipping, you're not shipping something.
00:41:08 Speaker_00
They're like, oh, look, we did a win. So I want to chat about that later, but let's keep going on things you've learned and things that didn't work along the journey.
00:41:15 Speaker_01
The other thing that I'll call out with the streak, you know, it's like we have the image of the streak is this flame, right? And you know, we have the streak flame and it's very much core to our iconography.
00:41:28 Speaker_01
It's important to acknowledge that that's a metaphor for a retention mechanic, like the idea of keeping a flame lit.
00:41:34 Speaker_01
And again, if you've, I think we've established the flame as, you know, for a lot of users as sort of their understanding of a streak, which is great, but there's a lot of people
00:41:43 Speaker_01
you know, in different cultural context and different, you know, uh, stages of life where the idea of like keeping a flame lit to show your commitment to something makes less sense.
00:41:53 Speaker_01
Um, we did some UXR in India many years ago, and this was something that just like did not resonate at all, um, there, which was, which was a really interesting learning.
00:42:00 Speaker_01
And that's something it's like, again, depending upon what your user base is, like the more global UXR you can do to understand how users are actually understanding and experience your feature, the better, because you just, again, encounter insights like this.
00:42:12 Speaker_01
And so. Even our screen design, we used to have a flame. It was mostly this flame that would light up every day. But again, it was like an indication of a metaphor for a mechanic.
00:42:22 Speaker_01
And when we redesigned it, we did this, Kurt, one of our animators, did this awesome odometer animation where it's like your number would tick every day.
00:42:31 Speaker_01
It kind of looked good, but from a product perspective, what was cool is we actually focused the design on the screen. to show your number going up. And then it would say, you know, like seven day streak, eight day streak.
00:42:45 Speaker_01
And I think that as you're thinking about designing around a streak, don't get too bought up, you know, or caught up into like, what is this, you know, like the beautiful story that you're trying to tell at the expense of it being a really comprehensible feature.
00:42:58 Speaker_01
And so as you're thinking about product design, making that product design a clear distillation of this is what we're actually tracking, you know, form should follow function here.
00:43:08 Speaker_01
You know was a was it was a learning for us and you'll see that now in a lot of places where we're showing streak where we're really leading with the number, not necessarily the flame.
00:43:17 Speaker_00
That's all. That's a theme that I'm hearing again and again is clarity. don't obsess with making it too thought like clever and don't ever think it just like clarity has a big impact.
00:43:29 Speaker_01
Clarity also doesn't have to come at the expense of delight. And this is something where, you know, you hit a milestone and duo gets, you know, it becomes, we call him a Phoenix duo and he becomes awesome and, you know, lights on fire.
00:43:41 Speaker_01
And I think there are things that you can do to still make the experience really exciting and delightful and celebratory and, and I would, I would not lose that, but just don't do it at the expense.
00:43:52 Speaker_01
And I think it's also about figuring out for what, you know, you can get away with doing more of this for users who are deeper into their streak experience than users who are starting where it's like your goal for the one day streak user is just to make sure, do they understand how this feature works?
00:44:06 Speaker_01
I mean, even something again, just like another random experiment at the bottom of the streak screen, we have a calendar. And over the years, it just looks more and more calendar like.
00:44:16 Speaker_01
And that is simply because we find that the more we make it look like a calendar, days on top, you know, little circles, the check, like the more we make it look like calendar, the more that people realize, Hey, this is a day, a daily mechanic.
00:44:27 Speaker_01
And so think about the screen holistically, but every single thing that you're doing on the screen, how can you use it to communicate? What is the point of this feature? How does it work?
00:44:37 Speaker_00
This episode is brought to you by Coda. I use Coda every day to coordinate my podcasting and newsletter workflows.
00:44:44 Speaker_00
From collecting questions for guests, to storing all my research, to managing my newsletter content calendar, Coda is my go-to app and has been for years. Coda combines the best of documents, spreadsheets, and apps to help you get more done.
00:44:58 Speaker_00
And Coda can help your team to stay aligned and ship faster by managing your planning cycle in just one location, set and measure OKRs with full visibility across teams and stakeholders, map dependencies, create progress visualizations, and identify risk areas.
00:45:12 Speaker_00
You can also access hundreds of pressure-tested templates for everything from roadmap strategy to final decision-making frameworks. See for yourself why companies like DoorDash, Figma, and Qualtrics run on Coda.
00:45:25 Speaker_00
Take advantage of this special, limited-time offer just for startups. Head over to coda.io slash lenny and sign up to get 6 free months of the team plan. That's coda.io slash lenny to sign up and get 6 months of the team plan. Coda.io slash lenny.
00:45:43 Speaker_00
I imagine one of the biggest wins was just giving people flexibility along the journey, like streak freezes and all these things. Is that a big vein of opportunity discovery?
00:45:52 Speaker_01
It is. Actually, I'm going to show you one of the most thoughtful gifts that anybody has ever given to me. This is our Duolingo Serenity or Streak Serenity prayer, my co-lead, Antonia. It's like knitted, right? For me, it's amazing. And so it says,
00:46:07 Speaker_01
Luis, grant me the serenity to accept the flexibility I need, the courage to reach perfection when I can, and the wisdom to celebrate regardless. That actually is kind of our strategy with the streak. I love the show and tell, by the way.
00:46:20 Speaker_01
That was great. Yeah. Well, I guess for podcast listeners, we'll have to get an image somewhere. This idea of flexibility versus perfection and then regardless celebration is core to how we think about the streak.
00:46:34 Speaker_01
Because I think for the streak for us, it's very much a bend, not break. If you're going to miss a day, I'd rather you come back having missed that day to an intact streak. But if you don't have to miss a day, I'd much rather you don't.
00:46:49 Speaker_01
I'd much rather you come back and use the app every day. That thing on flexibility, though, that's almost certainly been the biggest, from a mechanic perspective, the biggest DAU driver.
00:46:59 Speaker_01
One of the earliest experiments we ran was going from, you used to only be able to have one streak freeze, and then we let you have either two or three. So we tested two different arms. It was, again, another huge DAU win. This actually is funny.
00:47:13 Speaker_01
It was something that, and this is, again, a callback to that growth model post from Jorge. It actually was really bad for Kerr. because we were basically saying, hey, you can take a day off, you know, and that's okay.
00:47:25 Speaker_01
But it was really good for, this is going to be like alphabet soup, IWAR, an active weekly active user retention return rate. So basically users who had taken a day off, we were getting them to come back more at higher rates.
00:47:38 Speaker_01
And so it made up for our losses incur, but effectively what this meant is that, you know, why two streak freezes work better than one was
00:47:46 Speaker_01
I don't know sometimes people just need a little more flexibility than one day but one of the really interesting inside of this experiment was that three street freezes.
00:47:54 Speaker_01
Was actually no better than two street freezes and there were two competing things here i think this is important if you're gonna build a street you know to figure out what your flexibility mechanic is.
00:48:03 Speaker_01
we are getting more users to return after longer times away to an intact streak. But if you start taking three days off from any habit, it's just gonna be less likely that you return even four days later.
00:48:15 Speaker_01
And so we had these competing things where more users might be returning to a streak, but a lot of users were also just not coming back. We were training them to take more time off. So that flexibility, what's the right amount of flexibility?
00:48:29 Speaker_01
Again, this is another area we've run. hundreds of experiments on what is the right amount of flexibility. And we are constantly surprised here. I still don't have the answer for at every point in your streak journey, how much flexibility you need.
00:48:45 Speaker_01
One thing that I can say with certainty though, is give more flexibility when a user is starting their streak.
00:48:51 Speaker_01
Again, one of our biggest streak freeze experiment wins, I feel like I'm constantly saying this, one of our biggest wins, but they all were really, really big.
00:48:59 Speaker_01
One of our biggest streak freeze wins was when you start a new streak we give you two streak freezes again it's so funny to think back it's like how are we not doing this to begin with but at the time.
00:49:09 Speaker_01
The streak freezes kind of an overly gamified mechanic you had to buy them with gems that's our in-app currency you know cuz we wanted the whole idea of.
00:49:18 Speaker_01
you know, this to feel like you like it was really, you know, something you earned that, you know, there was a little bit of pain to getting that streak freeze.
00:49:25 Speaker_01
And so we tested though, what if we just give users when they start off their streak to streak, streak freezes, and holy smokes to that one. And sort of obvious now, you know, in retrospect, but
00:49:36 Speaker_01
If you have a one or a two or a three-day streak, it's really easy just to let it die and restart. Again, you need to get to seven days, what we've seen in the data for it to really lock in.
00:49:46 Speaker_01
And so giving users more flexibility so it's harder to lose their streak initially. And then conversely, and this is what we keep learning, eventually once people get on long streaks, you kind of don't want to give them as much flexibility because
00:50:01 Speaker_01
There's a lot of times where yeah, users don't, and I'm like this, I've got like a 400 day streak. Note that that is a lot less time than I've been at Duolingo. I have lost and restarted streaks a lot of times in my time at Duolingo.
00:50:14 Speaker_01
But you start getting a lot of streaks and you really care about this feature. You really care about your streak and most people,
00:50:21 Speaker_01
as long as you're not like backpacking through, I don't know, you know, the back country of Utah, you know, you'll, you'll be in a place where you can get service.
00:50:30 Speaker_01
And so figuring out where's the, you know, who is the user that actually could use Duolingo and not conditioning them to start taking days off that they didn't otherwise need to do is is important to figure out sort of where that line is for, for your future.
00:50:46 Speaker_00
This is fascinating. You can also buy a streak, right? Like with money, is that that's a feature, right?
00:50:51 Speaker_01
Yeah you can but it's funny this is this is also something that we buy it yeah you can buy a free sorry not a street yes you can you can buy a street freeze and the way it works is you can buy jams and then you can use those jams to to buy a street and this is something we wrestle with all we're actively working on experiment right now that's having a small hit a revenue but it's a really nice one for attention and.
00:51:13 Speaker_01
I think it's actually worth thinking about from day one as you're building a streak. Do you see this more as a monetization feature? And do you see this more as a retention feature? What's the role of monetization in this? What's the role of retention?
00:51:23 Speaker_01
I think for us, it started out much more organically. And so we have a lot of like monetization hooks that again, as the retention PM, I would love to get rid of. But again, it's sort of part of how the streak works right now.
00:51:33 Speaker_01
And so we always have this tension of hey, if we start to make it harder to buy streak freezes, then fewer people buy them, you know, buy gems to buy them. And so there's this more convoluted, you know, series of impacts that happen.
00:51:48 Speaker_00
Yeah, I love that. I love that people wanting to buy streak freezes like the ultimate sign of how much streaks matter.
00:51:56 Speaker_01
Yeah, it's streak freezes. The other big one that we've recently demonetized or introduced a free option for is getting back a lost streak. So we used to lose the streak. We had a feature back in the day called streak repair.
00:52:13 Speaker_01
We'd give you your streak back. You had to pay gems. But what we found out worked way better was a feature called earn back.
00:52:19 Speaker_01
And this is basically where you would have to do a certain number of lessons as long as you came back with a window soon after losing your streak. Do a few lessons and we just give you your streak back. And it was such a retention winner.
00:52:32 Speaker_01
And again, what we thought about was it feels like you've earned it so much more when you've done... You deserve to have your streak back. We haven't cheapened the streak.
00:52:42 Speaker_01
Uh, because you've done something and in this sense, this idea of cheapening the streak is something like from a philosophical philosophy of the streak from a philosophical level. We wrestle with all the time of. Cool.
00:52:55 Speaker_01
We're giving out more streak freezes. At what point do we cross the line and users start to realize their streak means nothing.
00:53:01 Speaker_01
Um, now everything that we've seen users are totally cool with using streak freezes and still thinking about the streak is this meaningful thing, but.
00:53:08 Speaker_01
My co-lead, Antonia, who made that awesome cross-stitch for me, she is sort of the keeper for us of the sanctity of the streak.
00:53:15 Speaker_01
And a lot of times as we, and I think this is really important to have as you're thinking about building your streak, you can almost always get engagement wins up to a certain point by just cheapening the streak, making it easier to extend, letting users have more flexibility, but you kind of got to hold the line at some point.
00:53:30 Speaker_01
It's not clear where that line is. And once you, like, you talk about one-way doors or two-way doors, There's a point where you go too far and it's a one-way door.
00:53:39 Speaker_01
And all of a sudden, those 9 million users on one-year streaks don't care about their streak anymore. And that is, I don't know, again, retention PM perspective, that'd be an extinction level event for us.
00:53:49 Speaker_01
I don't want all of these users to stop caring about their streak.
00:53:53 Speaker_01
And so to have somebody who is invested in the sanctity of the streak, and for us, it's Antonia and Luis, he's very good about this, is really important just so you make sure you don't go too far.
00:54:06 Speaker_00
That's an awesome insight. So to protect and push notifications, I think, are another example. This in general for companies, like how much is too much? Because everyone's just like, let's just send another push. It's fine. Just one more.
00:54:17 Speaker_00
And so your solution to that is a person is like the keeper and the gate, almost the gatekeeper, plus the founder of how far is too far.
00:54:25 Speaker_01
Yeah, it's good if you can have that. I think push notifications are also easier because, you know, there's a lot of things you can do around, all right, we'll set a budget cap for how many notifications we'll send.
00:54:35 Speaker_01
You know, you can... It's like a creative policy. Yeah, policies. But I think with a lot of things, it's hard, at least for the streak, it's harder to create policies for in the same way. A lot of it has to be done based on feel.
00:54:46 Speaker_01
And so you kind of just got to use your best judgment at times.
00:54:49 Speaker_00
Sweet. Okay. Any other, maybe like one or two more lessons from this journey of what Streaks has become today?
00:54:55 Speaker_01
I mean, I can, you mentioned notifications. I've mentioned this a few times. One, it's funny. You, you, you tend to think of like exactly as you say, you can just always send another notification. It's going to be some win.
00:55:04 Speaker_01
And at some point, you know, it'll, it'll be a bad experience, but it's tough to see that. There's actually a notification that we, so we send two notifications related to your core streak each day. The first is a practice reminder.
00:55:15 Speaker_01
We send it, and this is actually an interesting insight, 23 and a half hours after you practice the day before. Whoa.
00:55:21 Speaker_00
That is a lot.
00:55:22 Speaker_01
23 and a half, okay. So basically, if you practice at noon today, we'll send it to you at 11.30 a.m.
00:55:29 Speaker_00
tomorrow. And we have done- And it's, because it's like, assuming they are free in that time the day before, maybe they'll be free the same time.
00:55:35 Speaker_01
And we actually moved, we used to let users set this practice reminder time. And our thinking was, cool, you're going to say 7 p.m. That's when I really want to extend my streak each day. And then you know what? I said, this is somebody with two kids.
00:55:49 Speaker_01
Life gets in the way. Life always gets in the way. And when you think you're going to practice will change, your life will change, whatever. And what we realized is the best indicator of when you should practice was when did you practice the day before.
00:56:01 Speaker_01
We could almost certainly get more detailed. We have tried a bunch of ways to have much more complex logic, and what always wins is 23 and a half hours. That's so interesting.
00:56:08 Speaker_00
Revealed behavior versus... Yeah, exactly.
00:56:11 Speaker_01
So again, we send this practice reminder 23 and a half hours later. The other thing that we'll do, though, is we'll send what we call a streak saver.
00:56:17 Speaker_01
And this is at 10 o'clock at night, if you have not extended your streak, we'll send you a message saying, Hey, you know, like it's it's your last chance. This is this is it.
00:56:28 Speaker_01
If you don't extend your streak and you would think that like, that's kind of spammy. That's that's kind of annoying to get a notification from an app at 10pm.
00:56:37 Speaker_01
But what we found is because people care about their streak, their streak is this good thing that they attach positive emotions that they don't really want to lose.
00:56:45 Speaker_01
That notification reminding them, hey, come back and like people see this, uh, by and large as a positive notification and not a negative notification. Obviously it serves our purpose as well of getting users to come back and not lose their streaks.
00:56:57 Speaker_01
But again, I think if you can think about your notification strategy related to what is the feature that it's tied to, how do users perceive that you can.
00:57:04 Speaker_01
almost certainly not get away with more, but you could be thoughtful about notification load and when to send notifications.
00:57:12 Speaker_01
And again, for us, this late night message that's also highly impactful, super good, is actually something that could be perceived as spammy, but a lot of our users really do. Somebody who it's often late at night and I work here and I'm like,
00:57:29 Speaker_01
Forgot to do my thing about Duolingo all day. You know, here it is 1115 and I saw it down at that message is really powerful.
00:57:36 Speaker_00
Yeah, it has saved me many times. I totally know that message and I love that it's like a late night message from an app. very rarely do you actually are happy about that.
00:57:48 Speaker_00
And I love that this actually is a good example where like, it's really, it's really funny.
00:57:52 Speaker_01
All of the stories that you hear about people extending their streak, you know, if you look around a Duolingo party, uh, you know, where it's like 10, 1130, 1145, all the Duolingo employees that are like doing their lesson at the last minute, uh, you always see these like pictures of people in the club doing, or like at a concert doing Duolingo.
00:58:08 Speaker_01
And yeah, cause it's like, Otherwise, you're gonna use a streak freezer.
00:58:12 Speaker_00
God forbid you will lose your streak. That's so funny. Okay, anything else? And if there's more than one more, definitely share. But any other really interesting lessons or wrong turns or insights?
00:58:22 Speaker_01
you know, I talked about streak freezes, and we've done a lot with streak freezes, but I think if you're gonna make flexibility a thing, it's probably also useful thinking about how do you celebrate perfection.
00:58:31 Speaker_01
And so we have a feature that we have, it is the simplest thing in the world, it's called Perfect Streak.
00:58:37 Speaker_01
And it's just, if you don't use a streak freeze for a few days, we make your streak look gold, and we make your little progress bar on the calendar just look a little bit nicer.
00:58:46 Speaker_01
There's no reward for doing it, you don't get anything other than this nice little indication. And it is awesome. It is. It is. It is. It is a simple feature. It is ultra not complex.
00:58:57 Speaker_01
And it is really powerful, not only for getting users to, you know, as a bit of a reward to be, hey, you know, get to seven days, you know, without using a streak freeze and, you know, your streak becomes perfect.
00:59:10 Speaker_01
But it's also a really nice indication of users who aren't using streak freezes. Here is the thing that if you, you know, if you don't use a streak freeze, which again, kinderly, I would love for you never to use a streak freeze.
00:59:21 Speaker_01
If you don't use a streak freeze, your streak will stay, you know, perfect. It's funny, we actually just, you know, we're constantly
00:59:28 Speaker_01
you know, responding to bug reports about the streak, you know, it is, I swear to God, we we have that we have the best infrastructure on this feature, because it is so important. We had an employee who lost her a four-month perfect streak.
00:59:43 Speaker_01
It was a big deal for her because she did her lesson as she was crossing international dateline. There was a bunch of stuff going on. It was just weird in her back end. People start to care about perfection as much as they do their streak.
00:59:57 Speaker_01
For that person, it was a big deal when they lost their perfect streak. This is just an example of if you're gonna go after flexibility, which is good, finding a way to pull users back into perfection is a really important counterweight to have.
01:00:13 Speaker_00
What I'm imagining is you guys need like a Amazon style chatbot that just gives you the streak back. It's just like, okay, here you go.
01:00:20 Speaker_01
We have very much, so we have, if people lose their streak, you know, there's ways to get in contact with us, but we've actually thought about that where it's like, okay, we should just like build a self-service feature.
01:00:30 Speaker_01
And if you're, if we think that your excuse is good enough, whatever, we'll just give it back to you. Because again, it's for us, I'd much rather you be on, you know, be on a streak than, you know, have lost it, particularly if it was, you know.
01:00:42 Speaker_00
But it also can't feel that easy. I love this. I also love this point about just the power of the animation and user experience having impact. That's really interesting.
01:00:52 Speaker_00
Is that something you find often just like make like celebrating and making it feel really amazing without like copy or, you know, or like any feature is just like, holy, you're awesome.
01:01:02 Speaker_01
This is another thing where it's like when users care about the feature using not only uh animation haptics sound effects using and it's funny we don't have sound effects on the street this is probably something we'll look at in the not too distant future but like haptics are like something we have done a lot of testing like the phone vibrating yeah exactly your phone you know like there being a really cool
01:01:23 Speaker_01
Haptic pattern as you extend your streak um all of the stuff wins and it's cool because i think it wins.
01:01:29 Speaker_01
There's a few reasons why does it just makes you feel good right you know you get some cool moment your streak and and and we celebrate you we celebrate you in this visual way your phone's buzzer just like feels awesome the other thing it does is it like.
01:01:41 Speaker_01
Causes you to pause on that screen and I think there's this desire as you think through a lot of, you know, as PM's think through like, oh, how can I get users through this funnel as painlessly as possible?
01:01:50 Speaker_01
I mean, I talked about good friction earlier. There's a lot of times where I don't. I want you to stop. I want you to stop and like land on this screen. You gotta be careful not to do this for too many screens, right?
01:01:59 Speaker_01
But like for the one big ones, sometimes I just want you to pause there and enjoy the moment. Because if I can get you to enjoy the moment more, you're going to care more about your streak and you're going to be coming back tomorrow and so.
01:02:09 Speaker_01
animations that are cool and that cause you to like really soak it in haptics. You know that feel good. All of that comes together to make you really focus on that moment that all of that just gets users more connected to their streak.
01:02:22 Speaker_01
So animation in the right times works well and it's something we've had when. Quite a lot who designs the haptic stuff. Is there like a haptic designer?
01:02:33 Speaker_01
For the longest time, it would be a product designer, or it initially started as like the engineer would be like, all right, you know, would cobble together haptics based on what they felt good.
01:02:43 Speaker_01
Then it became a product design role where they would kind of use their best judgment.
01:02:48 Speaker_01
We actually just recently acquired an animation studio, Hobbs in Detroit, and now they are the sort of keepers of, you know, they do a lot of motion design work, haptics, very close to that. And so they do a lot of that.
01:03:01 Speaker_01
I do remember trying to hire for a while a haptics, like contractor, like haptics design.
01:03:07 Speaker_01
And it was the saddest hiring I've ever done because it was just like, I don't know, like it was such a specific, like, I just went through a lot of people who, uh,
01:03:17 Speaker_01
You know, it's just a really tricky space of like kind of sound effects, kind of motion design, sort of technical. Yeah.
01:03:23 Speaker_00
Um, so it's not such a unique role, very unique, specific skillset, right? And there's very few apps that really need this, this deeply. So you're almost creating this person. Yeah. That is fascinating.
01:03:35 Speaker_00
I don't want, I'm going to, that's like a whole podcast on its own. By the way, I was going to say, as you're talking about this, I love that it's a win to celebrate people that don't lose their streak.
01:03:43 Speaker_00
Like you introduce this way to make it flexible and that's a big win. And then you could go the opposite direction of if you don't use this feature, you also feel even better.
01:03:51 Speaker_01
Well, and it's funny, you talked about the danger of feature bloat. We sort of talked about the danger of feature bloat. This is actually something I'm constantly thinking about with this. We have the streak, but then we also have the perfect streak.
01:04:02 Speaker_01
We count how many weeks you've had a perfect streak. All of a sudden, we have two streak numbers that are kind of competing with each other. It's funny, we actually don't introduce the concept of a perfect streak until after you've hit seven days.
01:04:15 Speaker_01
And some of this is just because the cognitive load of additional streak features, a lot of our like cooler streak features, you got to get on a long enough streak.
01:04:24 Speaker_01
And not to say we haven't tested it because we have, because we've tested everything, introducing these features earlier. But what we've found is that pretty universally they lose when we
01:04:34 Speaker_01
introduce too many things, too many concepts to users too early in the experience. It's just hard for them to manage. Okay, sweet.
01:04:41 Speaker_00
I know that we can go down this track for hours and hours. There's endless learnings about all the things you all have done along the journey. I'm going to shift to talking about how your team operates.
01:04:52 Speaker_00
So there's a lot of threads you touch on of just how a team can do this so well, ship 600 experiments, as you said, continue to find opportunity.
01:05:03 Speaker_00
What are some maybe lessons or advice you'd have for folks that are like, oh, wow, I want to work more like this from your team's experience, how to use your team to operate that folks can learn from?
01:05:13 Speaker_01
Yeah, maybe just a little bit of context. Duolingo cares a lot about metric. Most of our teams are metric-based teams.
01:05:19 Speaker_01
We do the most work with Streak, but the metric, what we really care about at the end of the day is Kerr and DAUs because we see that DAUs hit Kerr.
01:05:30 Speaker_01
When you can be really laser focused on, my goal each quarter is to make this metric go up, I think it's much easier to make sure that you're working on the highest ROI thing.
01:05:41 Speaker_01
I think when you think more about like, oh, I want to make this feature better, I think it's easier to get lost in what better means and how you think about better. And so I do think that having a really strong degree of focus as a team on
01:05:55 Speaker_01
you know, what is the metric that I'm caring about and how is that directing my efforts is versus feature oriented.
01:06:00 Speaker_00
So basically your teams are structured around a metric slash a goal slash outcome versus we own this feature.
01:06:06 Speaker_01
So we need retention owns streak, I guess, but that's only because we've seen streak drive curve better than any feature. But
01:06:17 Speaker_01
We are not i mean we have this you know ip hook with our street freeze purchases there are other teams that work on your that can have worked on the street because.
01:06:28 Speaker_01
It's not ours to say no no we do all the iterations here we just know that it drives our metric better in the same way that like leaderboards. We have a team that focuses on how much time you spend.
01:06:38 Speaker_01
We want users to spend more time on Duolingo so they're learning more. Leaderboards is the best vector for doing it.
01:06:42 Speaker_01
So that team does a lot of leaderboards work, but every now and then I have an idea that I think will be highly retentive, and I will go in and I will pitch to them, and then we'll do some change to the leaderboard to make it more retentive.
01:06:53 Speaker_01
But I do think having that clear metric of we're trying to drive Kerr, not we're trying to just make this feature better, helps at least.
01:07:01 Speaker_01
Make sure that you know give the team clear marching orders and that focus i think is really good for prioritizing backlog.
01:07:07 Speaker_00
This is a really important point this is the same way airbnb worked when i worked there for a long time is it's here's a goal that we want your team to be responsible for. You can work on any product you need to hit this goal.
01:07:20 Speaker_00
as you said, often, like various products are most connected to what you're doing.
01:07:24 Speaker_00
But what you're describing is like, even though it seems kind of like, I imagine you own it from a bug perspective, and you kind of like are the shepherd of this part of the feature, because it hits your goal, helps your girl most, but any other team can come in and be like, hey, Jackson, we need to work on some streak stuff to help with learning.
01:07:40 Speaker_00
You're like, go for it. Does that just a tangent there? Do they work really closely with your team if they want to do some work in the code? How does that work logistically?
01:07:49 Speaker_01
Yeah. Again, this is where I say there's soft ownership.
01:07:52 Speaker_01
We're not against teams doing things to the streak, but if we're going to do something, given we probably have multiple quarters worth of a roadmap around the streak, I say probably we do, multiple quarters of roadmap for what we can do to the streak,
01:08:05 Speaker_01
if other teams want to come and mess with it, okay, we got to just figure out how is that going to work with, you know, what our plans for the streak were.
01:08:13 Speaker_01
How do we make sure a lot of times when teams are coming in thinking, hey, let's do this to the streak, they're like in context that we might have.
01:08:18 Speaker_01
And so there's as much of like a much simpler version of what we're doing now, a bit of a knowledge sharing of saying, all right, well, this is what we think about the streak. This is what we've seen work, hasn't worked.
01:08:25 Speaker_01
How does that influence some of the hypothesis that you have? And so I think getting that really, making sure the juice is worth the squeeze. Good old fashioned product management work right there.
01:08:35 Speaker_00
Uh, cool. What else is interesting about how y'all operate and how y'all work to achieve this sort of success?
01:08:41 Speaker_01
Again, my, my team lead runs is Antonio runs like the most effect, you know, really process. If you're going to run this mini experience, you have to be really process oriented and really thoughtful about which experiments am I going to run?
01:08:53 Speaker_01
When, how is that going to set up the next one? We Uh, you know, uh, there's, there's heavy Jira automation.
01:08:59 Speaker_01
I think sometimes the Atlassian suite makes my eyes bleed, but like, there's a lot of times where that degree of process helps the team unblock engineers and make them move really fast.
01:09:09 Speaker_01
And so making sure that you have really good process around how are you going to run so many experiments? It's, you know, it's, it's, it's worth investing in.
01:09:17 Speaker_00
Can you follow that thread actually, just what do you, when you say that, what does that look like? What are some elements of that process to make this work efficiently?
01:09:25 Speaker_01
all the way down to really detailed roadmaps around, all right, we're running this experiment is based on the results of this experiment or might hook into an element of this feature.
01:09:36 Speaker_01
How do we make sure that we're lining up implementation on this so that as soon as this thing runs and we're ready to go? we can start rolling out the next one. You know, I hate features just sitting around and us not again, continuing that thread.
01:09:48 Speaker_01
So it's not just thinking about what's our engineering bandwidth, but also what's the design bandwidth to make sure that we have the next iteration of this feature ready to go. You know, we're planning
01:09:58 Speaker_01
months out as we think about these feature iterations, even small ones, feature iterations. Because again, when you lose cycles, not pushing on a feature, it's just sort of lost opportunity.
01:10:11 Speaker_01
And so everything from being thoughtful about engineering roadmaps, to design roadmaps, to product roadmaps, all of that needs to come together in a system.
01:10:19 Speaker_00
So essentially mapping dependencies across function. And you're saying in JIRA, you can do this. You can do a lot of it in Jira.
01:10:27 Speaker_01
There is a non-zero amount of Google Docs that we have that sometimes does things a little bit, I don't know, sometimes it just looks a little bit nicer, it's a little bit more flexible. But Jira is our, it is where the mother load of process is.
01:10:42 Speaker_01
Great, okay, what else? Like another thing that I'll just say is like, we really resist the urge to do the big V1.
01:10:50 Speaker_01
And I think this is, you know, I shared the street goal example where a lot of times when we're exploring something, we will say, okay, well, like, that's cool.
01:11:01 Speaker_01
How do we strip away a bunch of stuff and figure out what our core hypothesis is, and then just ship that thing first as a V1?
01:11:08 Speaker_01
Cause it's it's easy and I found this time and time again it's easy to add things to features that make them win like I work in retention engagement long enough I can like add I know enough like things to pull and bells to add and whistles and you know to make something when but.
01:11:24 Speaker_01
you know, there's a lot of times where it's like, cool, they don't want because all the whistles you added, not because of what your core hypothesis was.
01:11:30 Speaker_01
And a lot of times, if you can just really simplify what the feature is, it's also much easier to ship, it's easier to design, you're not designing for a whole system, you're designing for, you know, something, something much simpler.
01:11:41 Speaker_01
And so getting everybody to think that way, ends up allow, you know, allows us to end up shipping faster, shipping simpler,
01:11:49 Speaker_01
you know, designing faster, getting faster approval, getting insight, and then doing like what I talked about with street goal, being able to run iteration after iteration after iteration, add these things iteratively.
01:12:00 Speaker_01
And then, you know, you not only by doing this, are you able to move faster, but you get confidence at each step of the way that, hey, my series of hypotheses, you know, is actually born out or if it's not cool, then we're going to drop that part of the feature and then just ship what actually matters.
01:12:18 Speaker_00
If I can try to summarize kind of the broad lesson so far that I'm hearing, and maybe you would have shared this, but I'm just thinking like if I were to try to design a company to operate the way you all operate, you essentially map all the levers that drive the business.
01:12:34 Speaker_00
So you have kind of this mapping of all the metrics that drive up to growth and daily active users. CURR ended up being the biggest specific metric to move to drive growth long term.
01:12:49 Speaker_00
So there's like imagining a tree of all the opportunities you could work on, you found like this is what is most connected to our growth long term. You basically just start mining.
01:12:58 Speaker_00
I don't know if mining is the right metaphor, but just like looking for things that move that specific metric, you just like, look and poke and explore.
01:13:06 Speaker_00
And then once you find one, you just go real deep on trying a lot of different you come up with a hypothesis and a strategy of here's how we think we can do this, and how we can move this and then you just try a bunch of stuff.
01:13:17 Speaker_00
There's also this element of the rest of development quote, there's always money in the banana stand comes to mind, where it's just like, keep working on see, there's more, there's going to be more opportunity.
01:13:29 Speaker_01
When I joined Duolingo, the PM that I took over for Anton, who used to lead the retention team, I remember saying, Dude, the streak, like, it just counts up. You guys have been testing on it for years. How much more work can we do on the streak?
01:13:44 Speaker_01
And he was like, Jackson, you child. He didn't say exactly this, but this is how I felt it. Like, Jackson, you child. We're not even 30% of the way optimized. And four years later.
01:13:54 Speaker_01
I say that with such conviction, we've made a ton of strides, but we are still so far away. Every quarter where we ship a ton of wins and improvements to the streak, it just continues to prove to me that there is so much more to be done.
01:14:12 Speaker_01
I think your framing of it is, and I would say there's a lot of thought that goes into, again, I talk about the strength of the hypothesis that you have to have as you start to build out larger future strategy.
01:14:23 Speaker_01
I do think it's really important to not just do a bunch of random stuff, but do it with intent, with a goal in mind. Otherwise, you do end up in these local maximas.
01:14:36 Speaker_01
Yeah, I mean, there's still a bunch of stuff that we haven't tried that, you know, I think we have high confidence in, you know, working out. And so we'll, we'll keep doing that.
01:14:44 Speaker_00
Are there any other, say, lasting lessons from this journey that if someone were to try to operate this way, build streaks into their product, anything you'd recommend?
01:14:55 Speaker_01
Yeah, I really do think it starts with, you know, streaks are an engagement hack. You know, you can make your app more retentive. I'm almost positive almost every app out there can make it more retentive. It is loss aversion.
01:15:09 Speaker_01
That is, you know, again, armchair psychologist Jackson, like, it's just a human thing that works on humans. But if your app is not
01:15:21 Speaker_01
something that users want to use every day or whatever, you know, cadence you want your app to be, uh, you know, to work on, it's going to be, you're only going to get so much from that streak.
01:15:30 Speaker_01
And honestly, it's probably going to distract you from what really should matter, which is making your app something that people want to use every day.
01:15:35 Speaker_01
And so if you start focusing on the streak, but you haven't made that a enjoyable experiment or experience, you're just going to waste a lot of time, honestly.
01:15:43 Speaker_01
And so I think making sure that you have your core loop of your app, figured out that, that it is giving value to users is something that they want to come back to every day. That really sets the stage for something to layer a streak on.
01:15:55 Speaker_01
So resist the temptation if you haven't, you know, if you don't think you've reached that point to go too hard down the, you know, the path of streak.
01:16:03 Speaker_00
That's a really good point. Just like a streak is not going to solve your problems if people don't actually care about the core value you're providing.
01:16:09 Speaker_01
No, and honestly, it'll probably cause more problems if what you end up focusing on is how do I make the street highly engaging, but your app isn't.
01:16:16 Speaker_01
I mean, you're just, you're wasting time that could otherwise be, you know, better spent on solving more critical problems. So that's one learning.
01:16:26 Speaker_01
You know, the other thing that I'll say is, and we met with one of our board members, Bing, Bing Gordon, a few weeks or a few months ago, rather. And he had this comment where he was just like,
01:16:38 Speaker_01
The reason why users care about your street so much is because you care about your street much you being doing go like the reason why users care about our street so much is doing go cares about the street so much like what you mean was like well.
01:16:49 Speaker_01
After every session you see a big street screen and it's animated cooler than almost any other screen in the app and sometimes you see some other screens and there's all these other feet like you don't let a user forget it you talk about them and messages and.
01:17:01 Speaker_01
And so I think it's worth thinking about, look, if you're going to build a streak, and then you're going to condit off into the corner of your app, you know, where users aren't going to see it, like, they're probably not going to care about it as much.
01:17:12 Speaker_01
And which might be fine, because there might be other lovers that you think are more important to pull on. But there's a reason why, you know, we focus as much on the street, you know, as we do.
01:17:22 Speaker_01
And that's because we want it to be top of mind for for users. And it's not by accident that users start to care about.
01:17:28 Speaker_01
And so I think just as you're thinking about building the streak, making sure that you're giving it the visibility it deserves, if you want it to have the kind of impact that Duolingo has, it's sort of an important hierarchy principle to think about as you design things.
01:17:42 Speaker_00
That's such a good point. Like you look for cues to the app of tell me what I should pay attention to, what's important. If you're just like fire explosions, you've hit a streak. Oh, maybe I should pay attention to this feature.
01:17:53 Speaker_00
And then the push notifications obviously encourage you there too. Anything else along those lines?
01:17:58 Speaker_01
Maybe final thing is, look, we ran so many tests on our Duolingo streak to figure out what worked. I mean, we have a philosophy at Duolingo of test it first. We are a lot of times willing to test things.
01:18:12 Speaker_01
I really think that if you're going to try to introduce a streak or you want to improve on the streak you have, don't get too caught up in
01:18:21 Speaker_01
The philosophy of everything i mean make sure your hypotheses feel like they're good but my recommendation is just try things and this is again you said earlier it's like this is much human psychology is anything and.
01:18:34 Speaker_01
As soon as that becomes the case like you can just got understand what users respond to and the easiest way to do that is to stop spending time you know batting around ideas in a conference room and just. try some stuff.
01:18:46 Speaker_01
Um, so huge recommendation to like, if you're going to invest in a street, try and figure out what works through testing with users, um, rather than trying to get it perfect on the first try.
01:18:57 Speaker_00
Say someone's listening and they're like, should we do street? Cause it's worth doing. What's your take on just like the chances that a Streak feature would be helpful to another consumer?
01:19:06 Speaker_01
I am well known for saying in the company that I think every Streak, every team, every app could benefit from a Streak. Now, how you implement it is very different. And I think you got to like, what is your user's use case?
01:19:19 Speaker_01
Like if they're going to come use, I don't know, tax software,
01:19:24 Speaker_01
Okay, you know, now that I say this, tax software would be a hard one, but maybe it's all about, you know, you need users to come back every day during the tax season, or how many times, I don't know, you know, now that I say this out loud.
01:19:34 Speaker_01
Times you upload your pay forms. Yeah, that is a hard use case. But the vast majority of companies, I think, have a good idea of like, all right, here is my ideal use case. I want users to come here,
01:19:48 Speaker_01
Three times a month, that would be ideal, or four times a month. You can build a streak to work. I mean, Peloton has weekly streaks because the idea of doing a Peloton workout every single day was hard for this user during COVID.
01:20:02 Speaker_01
It was just like every now and then you get on the Peloton, that was great. But the idea of a weekly streak was something that I could keep up. And so I think figure out what your usage pattern is as a user and then build your streak around it.
01:20:15 Speaker_01
You know, as long as you're not, you know, like a really, again, the tax example was probably a good counterfactual, but as long as you have some degree of frequency in your use, I think almost anything can have a streak.
01:20:29 Speaker_00
So Duolingo, it's again, a $14 billion company, this feature. possibly the most contributing factor other than the core product to that level of success and market cap.
01:20:42 Speaker_00
And it's hard to imagine another just feature of a product that has had this much impact on growth and revenue and building this sort of business. So I love that we spent this much time on it, the mother load, the mother load of advice and insights.
01:20:57 Speaker_00
So thank you again for putting it. Very fun. With that, we've reached our very exciting lightning round. Are you ready? I'm ready. First question, what are two or three books you've recommended most to other people?
01:21:10 Speaker_01
All right, I'll start with a guide to Midwest conversation. So I'm based in Kansas City. I'm a proud Midwesterner. And us Midwesterners talk in a certain way. I think you hear about Minnesota nice. But we tend not to say what we mean.
01:21:25 Speaker_01
And it is a very funny primer into what Midwesterners actually mean when they say what they say.
01:21:31 Speaker_00
So I highly recommend reading that. I like that you give that to people. Just like, here's what I might be telling you, which you may not read.
01:21:38 Speaker_01
My wife is German and I gave it to her so that she could better understand. Another book, this is a good one, Fate is the Hunters. This is a really cool book. It's a memoir of one of the early commercial airline pilots.
01:21:55 Speaker_01
And it is wild to hear the stories about what flying was back in the day. I'm a former management consultant. I flew every week for almost six years, and I never once had to worry about, am I going to make it to the other end of this flight alive?
01:22:10 Speaker_01
that was not the case back then. And so some of the stories about what it used to be like to be a pilot on some of these planes before modern aviation technology is fascinating. It makes you really appreciate what we have.
01:22:22 Speaker_00
It feels good to read a book like that, being a software PM or engineer or whatever, like how different that life is. Hardware is hard. Oh man, it's not haptic design. Okay, next, unless there's any other books you're going to share now. Okay, great.
01:22:36 Speaker_00
What's a favorite recent movie or TV show you've really enjoyed?
01:22:40 Speaker_01
Uh, so I have two kids. I watch a lot of bluey. Um, it's really good. I swear it's it brings me no shortage of joy. Um, but adult show that I heard or show not meant for four year olds that I have watched.
01:22:53 Speaker_01
I just finished the latest season of Emily in Paris. Man, wonderful. Uh, I realize it's not the highest brow of television, but just like beautiful people and beautiful cities solving problems that are not, you know, earth shattering.
01:23:05 Speaker_01
Sometimes it is nice to just Tune out. Also, I'm learning French on Duolingo. Slight plug for the app.
01:23:12 Speaker_01
I can understand a lot of the French that is being spoken and there is no better joy than having invested as much time as I have in French and actually being able to use it.
01:23:21 Speaker_00
So, huge fan of Emily. That is so funny. What a fun Venn diagram of interests. My mother-in-law loves Emily in Paris. I saw someone tweeting about like, what visa is she on? How is she still in Paris? Yeah, you best just not ask questions.
01:23:35 Speaker_01
There's a lot of questions to this show that are unasked.
01:23:38 Speaker_00
Do you have a favorite product you've recently discovered that you really like? Other than Duolingo.
01:23:44 Speaker_01
Last week I went to Home Depot and I bought a new ladder and ladder innovations you don't think, you know, of often, but you can make one of the legs go a little bit further than the other leg.
01:23:55 Speaker_01
And as somebody like myself who has a house that is built on a slight slope, Every time I go up on my ladder, I take my life in my hands. But with this ladder, I'm always even.
01:24:04 Speaker_01
I cleaned my gutters twice last week, just because of how awesome this ladder has, how much this ladder has changed my life. So ladder innovation, I don't think it gets talked enough about, and so I'm happy to give it the spotlight it deserves.
01:24:18 Speaker_00
I appreciate you doing that. That's the first ladder recommendation we've had on the podcast. Two more questions. Do you have a favorite life motto that you really find useful in work or in life that you share with folks?
01:24:26 Speaker_01
This probably will not be much of a surprise based on how I've talked about our willingness to test things, but you miss 100% of the shots that you don't take.
01:24:34 Speaker_01
I'm a big fan of just trying things, even if your possibility of success is not 100, because you learn a lot along the way, so.
01:24:44 Speaker_00
Final question. Do you have any fun traditions at Duolingo amongst either the PM team or the company in general that might be delightful to share?
01:24:53 Speaker_01
We have way too many traditions to count. I will share the weird tradition that we do at every retention stand-up. And this started during the pandemic. We obviously used to stand up in person.
01:25:06 Speaker_01
And then when we went remote, we did this thing where whoever's the last person to go would count down three, two, one, and then we'd all try to clap at the same time, which was kind of fun and dorky, but we fell in love with it.
01:25:19 Speaker_01
And four years later, we're still doing it. Recently, we've added, we all say yee-haw in unison afterwards. I can't tell you why, but trying to synchronize a clap via Zoom and then all shouting, yeehaw.
01:25:35 Speaker_01
I did this in a phone booth the other day, and after I came out, someone told me, you know that those aren't as soundproof as you think, but you know, when you get a good opportunity to give a yeehaw, you kind of can't pass up on it, so.
01:25:47 Speaker_00
I love these little things. Like they're so, you know, they sound so minor, but they're such important elements of team culture and tradition and so important for PMs to find ways to just have fun and do something ridiculous.
01:25:58 Speaker_01
I will say it took a while to get people behind shouting yeehaw. But now that we have people doing it, you can't, you can't take it away from them. We all love it.
01:26:09 Speaker_00
Oh man. I called our all hands for a while, y'all hands. Feel free to steal that. You get it, Lenny, yeah. I get it. Jackson, this is incredible.
01:26:18 Speaker_00
I feel like people are gonna listen to this with notebooks and just like, okay, here's a bunch of ideas we should try with whatever we're working on, whether it's Streaks or not. Thank you so much for being here. Two final questions.
01:26:28 Speaker_00
Where can folks find you online if they wanna reach out, learn more, learn more about Duolingo? I know you're hiring product managers, so share more there. And finally, how can listeners be useful to you?
01:26:37 Speaker_01
Yes, you can find me on LinkedIn. That is where most of my online social media is. So Jackson Shuttleworth. And then how people can be useful to me. Yes, as you said, we are hiring. We're actually hiring for my team.
01:26:50 Speaker_01
Are you interested in thinking about Streaks as much as we do? We might be the right home for you. So please, you can apply on our website. We're also hiring for a number of other product management roles and they're all as thrilling as this work is.
01:27:05 Speaker_01
And then you know i'm always interested about how other companies have implemented streaks and what they've learned and so what i'd say is if you're a company who's implemented a streak maybe in a different way than duolingo has or you know you found a whole ton of success and you know another vector another element of the feature that we didn't talk about today i would love to know more i used to catalog basically every streak i found out there.
01:27:30 Speaker_01
And as it's become more of a popular feature, it's just been hard to keep up on so. You know, if you have interesting Streak insights to send my way, I would love to hear them.
01:27:41 Speaker_00
I love that. A collection of all the best ways of doing Streaks. Jackson, I just want to say congrats to your team and you for having so much impact. This is like the dream of a lot of PMs and teams is to see this much impact and continue to ship wins.
01:27:55 Speaker_00
And so congrats. Nice work. Thank you very much. And with that, Jackson, thanks so much for being here.
01:28:01 Speaker_01
Yeah. Thank you, Lenny. This was a lot of fun. Same. Bye, everyone.
01:28:07 Speaker_00
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
01:28:15 Speaker_00
Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at LenniesPodcast.com. See you in the next episode.