4 April 2012

# Go Figure: Silly Stats in the Social Media Fantasy League

by Freddy Tran Nager, Founder of Atomic Tango + Guy Who Questions The Numbers; photo via Wikimedia Commons

Imagine that you own a basketball team, and you send your talent scout out to find a new player. He comes back and presents two options:

• Player A: averages 9 points a game, including 1 three-point basket per game
• Player B: averages 24 points a game, including an average of 2 three-point baskets per game

Both play the same number of minutes per game, and both want the same amount of money. To your surprise, your scout recommends hiring Player A. “Why?!” you ask incredulously. “Because,” the scout replies, “Player A gets 33% of his points from three-point shots, and Player B gets only 25% of his points from three-point shots. That means Player A gets 32% more points from three-pointers than Player B.”

As you stare in stunned disbelief, the scout takes it as a signal to continue…

“Now you know that three-point shots are worth 50% more than two-point shots, so if we multiply the 50% by 32%, that makes Player A 1600% more valuable than Player B.”

Of course, that makes your choice easy: you fire the scout.

Sounds silly, doesn’t it? But that’s how some people are measuring social media “effectiveness.”

I came across this recently in an article that claimed, “Top B2B Firms Gaining 230% More Leads via Social Media Than Peers.”

“Best-in-Class B2B companies generate on average 17% of their leads from social media channels, roughly 230% more marketing-generated leads than other companies (5%).”

That’s some mad mathematical mayhem there – what financial-analyst Ron Shevlin would call “quantipulation.” Where did that figure of 230% come from? I tried to do the math: 17 is 240% more than 5, so somewhere along the way, we lost 10% and…

Hold on there: this whole case smells like something beached itself and died. I find myself suddenly remembering Darrell Huff‘s classic book, How to Lie with Statistics, which demonstrated how you can say anything you want with numbers — particularly if you use those ever deceptive percentages.

So I contact Shevlin, who’s a professional marketing analyst in the financial services industry. He cuts to the chase:

“NEVER EVER calculate the percentage difference between two percentages.”

Thank you, Sensei, for dispensing with that nonsense.

The “So What” Factor

As I tell my students, the most important question to ask when analyzing social media in business is, “So what?” It’s the perfect response to any of these statements:

• “My video went viral!”
• “Our free burrito offer got a bunch of likes!”
• “According to Klout, I’m the world’s most influential person in three categories!”

Ask “So what?” — and the answer better have a \$ sign attached.

So let’s look at our 230% overachievers, shall we? And let’s be good sports and grant them that number. So here it goes: so what if you got 230% more leads?

• What are the absolute numbers? 17% sounds like more than 5%, but you don’t need to be a financial analyst to understand that 17% of 100 is a lot less impressive than 5% of 1,000,000.
• What was the ultimate value of those leads? Did they all convert into sales, or were they junk? And if they did convert, at what dollar amount?
• What was each of those firms selling? Some products and services are a lot easier to sell through social media than others. (Say, an MP3 download vs. nuclear accident clean-ups.)
• Are the top firms getting more leads because they’re good at social media, or because they’re top firms?
• If a higher percentage of a company’s leads comes from social media, couldn’t that mean that they suck at traditional media? Or, conversely, perhaps the 5-percenters are so good at traditional media, that they don’t need to bother with social. (There’s a company like that you may have heard of: Apple.)
• How much did the 17-percenters spend in time, money, and opportunity costs to land those leads? What did the 5-percenters spend? Of course, EVERYONE knows by now that social media isn’t free and can be even more expensive than traditional media, right?
• If the firms had invested all that time and money into traditional media – from which 83-95% all leads in this case come from – what would the results have been?
• And, wait, if non-social media is generating 83-95% of all leads for these firms, why is this even a story? I know where I’d invest my time and money, don’t you?

Now don’t get me wrong: social media has actually generated leads for my own B2B agency, almost entirely through LinkedIn. So yes, it can be productive and lucrative. For some… at some times… under some situations… for some prospective clients.

The key is discerning what that “some” is, and we can’t discern squat from distorted stats. They’re no way to pick a basketball player, and certainly no way to run a business.