Today, we're talking about why your data might be leading you astray, Simpson's paradox, which you may have never heard of before, but it's very interesting, and the danger of misleading results.
Here's a story that should be a wake-up call for any B2B SaaS CEO.Even when you think you're doing everything right, your data might be leading you towards decisions that could undermine your growth.So we'll get into that.
All right, imagine you're the CEO of a B2B SaaS company focused on driving growth.Your CMO proposes shifting from your traditional inbound sales motion to a product-led growth strategy.
Being cautious, you decide to run a test rather than jump straight in.So you want to run a test in Q1 and see what the results look like.Here's what happens.Q1 results.Inbound, 582 opportunities, 183 deals, which is a conversion rate of 31.44%.
The product-led growth strategy, 140 opportunities, 45 deals, a conversion rate of 32.14%.So an improvement.The PLG actually had a higher conversion rate.You're encouraged by the results.So you expand the PLG test in Q2.
So what do the Q2 results show?Inbound, 48 opportunities, 12 deals, a conversion rate of 25%.PLG, 411 opportunities, 104 deals, a conversion rate of 25.3%.So again, the PLG performed better than the traditional inbound motion.
After two quarters, it seems like a no-brainer.PLG is outperforming inbound.The higher conversion rate suggests PLG is the future, right?Not so fast.When you combine the results from both quarters, the data tells a different story.Combined results.
Inbound over Q1 and Q2 combined.630 opportunities, 195 deals for a conversion rate of 31%. The product-led growth motion, 551 opportunities for 149 deals, a conversion rate of 27%.
So when you combine the two together, inbound actually had a conversion rate of 31%, PLG had a conversion rate of 27%.Suddenly your traditional inbound motion is performing 15% better than PLG.How can that be?
This is Simpson's Paradox, a statistical phenomenon where trends that appear in separate data sets reverse when you combine them.I've been hearing a lot of talk lately about how easy it is to use data to guide marketing decisions.
But the truth is, even simple tests can lead you astray if you're not careful.This example might seem straightforward, but that's the point.Even the simplest decisions can be wrong for reasons most people would overlook, like Simpson's Paradox.
Though I use PLG to highlight the challenge, Simpson's Paradox can occur in any area.College admissions, medical treatments, income distribution, sports, A-B testing, and many more areas.
In fact, in this example, the data comes from a real life instance in baseball.And both in 1995 and 1996, David Justice had a higher batting average than Derek Jeter.But when you combine the two years, Derek Jeter comes out on top.
The data flips when it looked at holistically.So what's the lesson in all this?Well, it looks like straightforward data could be deceptively misleading.Before making any decisions, ask yourself, are we analyzing the data in the right way?
Are we weighing results properly across different cohorts and time periods?Are we sure the short-term trends won't reverse when looked at over time? Making the wrong call based on a faulty interpretation of the data can be costly.Always dig deeper.
Your growth depends on it.
Sign in
