How Big Do Samples Need To Be?

The best way to be sure that a survey sample is reliable:  do repeated measurements consistently produce the same results?

For example, in Arbitron market #67 Fresno - where thankfully both owners still subscribe to ARB so we can at least get some idea from the real world - the top country station (Peak's KSKS) has gone from #3 in 12+ monthly trend cume rank in May 2011 to #5 to #2 to #3 to #3 to #8 to #7 to #4 to #3 to #5 to #6 to #8 to #3 in May 2012.

The second tier country station (Clear Channel's KHGE) wobbled even more over the same period, from #21 to #11 to #7, then #16, #14, #21, #16, #10, #15, #18, #12, #17, #9.

I chose 12+ cume rankers to trend, since those should be the largest, most-stable numbers to track, less susceptible to weighting-driven TSL/AQH wobbles and yet it's still quite obvious that something is unstable about even those numbers.

•  Which are the outliers in the stats and which are reality? 
•  Is a year enough time to allow to see?  
•  How many years' averages will it take to know for certain?

Pity the poor media buyer and radio seller trying to make sense of them.

My conclusion, though I am not sure Arbitron (or BBM) would agree:  the Fresno sample isn't large enough, given the complexities of ethnic/non-ethnic sample balance, let alone Spanish only/bilingual/English only language and cell phone/land line home phone preference trends.

Nielsen just announced that it plans to double its TV diary and quadruple it's metered panel samples.

You'd think that would make a huge difference, and yet it may not since the TV ratings company in the USA did the same thing only five years ago with the goal of tripling it by 2011 and yet they seem to "need" to do it again now and state that they plan to move as fast to do so as their clients want.

Nielsen has competition which is clearly driving the changes; Arbitron does not. 

Same in Canada, where BBM is the only option for broadcasters, since they "own" it.

If North American radio wants to know how big samples must be to be consistent and credible in our markets + how fast our ratings supplier can get there, we're going to have to keep the pressure on ARB and BBM, with high hopes they care as much as their clients do, to make changes in spite that.

Just Because You Can Find A Willing Lawyer Doesn't Mean You Have A Case

No one is more upset about Arbitron's sample issues than I am. The only stat growing faster than their profit margins is the number of places with totally unacceptable proportionality problems and missed sample targets in more and more cells and I wish they were doing more to address this in the hundreds of diary markets instead of bragging about improving proportionality in the handful of PPM cities coming online.

But, what's the point of harassing ARB with complaints to the FCC, the AG, the courts? That's like suing Wendy's because The Baconater has more fat than you'd like to have in your lunch. Have a salad, or go somewhere else.

Cumulus is not renewing their ARB contracts which expire at the end of the year and it sounds like at least Nielsen and Eastlan, if not others, have been vying for that business.

That's a good thing. Healthy competition and smarter, more state-of-the-art approaches are needed. More savvy researchers will solve these problems, not more attorneys.