Making "Bad" 6% Less So

Not to seem ungrateful that we're now just a few more months away from seeing the 6% increase in PPM samples which Nielsen promised for 2014 last November, but - as my post "The Canary Has A Name" suggests - that won't come even close to realizing the dream of true PPM-based cross media measurement with reliable estimates a reality.

As Paragon Research's Larry Johnson recalls "when Nielsen and Arbitron were first collaborating in the early 1990s about rolling out radio PPM jointly, the sample size was to be roughly three times what Arbitron has been offering.  The sample size becomes even more crucial if Nielsen is going to measure multiple platforms with a PPM device.  If you’re going to accurately measure outlets with small, niche audiences, the sample size must increase dramatically.  For example, cable television gets cut up into very small pieces given the huge number of cable channels available.  It’s interesting that Nielsen uses a mixed methodology utilizing both diaries and meters in many of the television markets they rate."

Researcher Mark Ramsey offers a brief statistics course in a two year old article to underscore why it's important that Nielsen make the most of their abilities now that that they own PPM: 

Small samples yield extreme results more than large samples do. Every broadcaster knows this, of course, but the impact is far worse than you think.

Daniel Kahneman discusses this issue in some depth in his book Thinking, Fast and Slow:

Imagine a large urn filled with marbles. Half the marbles are red, half are white. Jack draws 4 marbles on each trial, Jill draws 7. They both record each time they observe a homogeneous sample—all white or all red. If they go on long enough, Jack will observe such extreme outcomes more often than Jill—by a factor of 8.

By a factor of 8!  Extreme (and erroneous) results are 8 times more likely with 4 rather than 7. And how many meters are tuned to your station right now?

Ramsey asks:   "What does it say about the veracity of our message and our messengers when we’re using numbers we know to be flat-out false?"

Another researcher, Kurt Hanson recalls his attempts several decades ago to sell radio on a better sample size in a service called Accu-Ratings, and puts it this way: "introduction of Portable People Meters (PPMs) solved the wrong problem:  Diaries weren’t being filled out precisely to the minute (in fact, lots of respondents filled out their diaries in one sitting at the end of the week), and PPMs “fixed” this problem, but because meters were so much more expensive than paper diaries (hundreds of dollars vs. maybe a dime), Arbitron used far fewer of them per market.  So the sample size issue got *worse,* not better!"

If today's multi-media ratings firms are serious about credibly measuring the increasingly-fragmented pie, they have to see the potential of signing many, many new clients which hopefully will rapidly grow their revenues, while keeping radio and television's rates stable at current levels.

I hope they also understand that to achieve that lofty goal sample sizes must be at least tripled or quadrupled, not merely increased in small increments.