How Big Do Samples Need To Be?

The best way to be sure that a survey sample is reliable:  do repeated measurements consistently produce the same results?

For example, in Arbitron market #67 Fresno - where thankfully both owners still subscribe to ARB so we can at least get some idea from the real world - the top country station (Peak's KSKS) has gone from #3 in 12+ monthly trend cume rank in May 2011 to #5 to #2 to #3 to #3 to #8 to #7 to #4 to #3 to #5 to #6 to #8 to #3 in May 2012.

The second tier country station (Clear Channel's KHGE) wobbled even more over the same period, from #21 to #11 to #7, then #16, #14, #21, #16, #10, #15, #18, #12, #17, #9.

I chose 12+ cume rankers to trend, since those should be the largest, most-stable numbers to track, less susceptible to weighting-driven TSL/AQH wobbles and yet it's still quite obvious that something is unstable about even those numbers.

•  Which are the outliers in the stats and which are reality? 
•  Is a year enough time to allow to see?  
•  How many years' averages will it take to know for certain?

Pity the poor media buyer and radio seller trying to make sense of them.

My conclusion, though I am not sure Arbitron (or BBM) would agree:  the Fresno sample isn't large enough, given the complexities of ethnic/non-ethnic sample balance, let alone Spanish only/bilingual/English only language and cell phone/land line home phone preference trends.

Nielsen just announced that it plans to double its TV diary and quadruple it's metered panel samples.

You'd think that would make a huge difference, and yet it may not since the TV ratings company in the USA did the same thing only five years ago with the goal of tripling it by 2011 and yet they seem to "need" to do it again now and state that they plan to move as fast to do so as their clients want.

Nielsen has competition which is clearly driving the changes; Arbitron does not. 

Same in Canada, where BBM is the only option for broadcasters, since they "own" it.

If North American radio wants to know how big samples must be to be consistent and credible in our markets + how fast our ratings supplier can get there, we're going to have to keep the pressure on ARB and BBM, with high hopes they care as much as their clients do, to make changes in spite that.