What Every Music Director Knows That Arbitron Seemingly Doesn't

Big kudos to Lincoln Financial Media's Don Benson for his service in the past year as chair of the Arbitron Affiliates Advisory Council.  No one could have done more to call attention to the constant elephant in the room in radio's relationship with ARB, sample sizes.

The challenge appears to be that ARB would like the AAAC to advocate to their radio brethren that the only way to get sample sizes up is to increase the rates they pay and of course meanwhile radio tells their reps on the council that given that the ratings giant's profits have been growing faster than the economy even during these recessionary times and are locked in at the same levels for the next few years, so they see ARB as a parasite that is slowly killing its host.

Once upon a time, when radio research companies were these distant experts who also charged humongous fees for perceptual studies and music testing because they were the masters of techniques that were intellectually above the heads of mere mortals.

Today, that's no longer true, as online music testing and perceptual studies are routinely done in house using email databases and anyone who sees the results week after week can quickly observe the difference a large, representative sample creates in the results versus a too-small one.


This ranker, shared with permission, from a recent test tells me that a total sample of 126 is a pretty reliable indication of how the average listener to this station feels about these songs, especially so if the station looks at a weekly trend report from different people of about the same size sample over multiple weeks and the stats remain fairly stable.

Another station whose data I watch each week just had less than half the sample size of the first one.  Hopefully, the database manager/music director of this station knows that those columns with so many 100%'s - which look a lot like PPM rankers in some narrow cells - are not worth making decisions on.

I'd prefer to see a sample of 250 or more, so all of the narrow demo tabs are also well-balanced and at least the minimum of 30 persons that ARB's PD Advantage and Maximiser will allow you to run a report on. 

49 people with just five or six in the narrow cells? = garbage data. You'd be better of with no research than acting on this particular music test.

Yet, media buyers increasingly are looking at very granular cume and TSE numbers for individual radio stations based on samples that small and expecting them to be consistent and stable.

Hopefully, 2013 council Chair Craig Jacobus and Vice Chair Joel Oxley will be just as vociferous as Benson and his committee has been in 2012 and - for the good of all of us - finally move Arbitron toward not just listening to radio's long-stated concerns, but ACTION on out-dated paper-based diary methodology and sample size increases in all size markets.

See:  How Do You Keep Online Music Testing Sample Sizes Robust? (pdf) A&O music specialist Mark Patric shares his approach.  What's yours?