NHL TV Ratings, Proceed with Caution

The Sports Business Journal has an article up on NHL attendance and TV Ratings. The information is both interesting and also potentially very flawed. I will discuss both angles. First let's look at the reported data for the 24 US based franchises. (Teams in bold have won the Stanley Cup post-lockout.)

Comments: Generally speaking there is a big split between cold weather and warmer weather markets in the TV ratings. But there are some big differences within New York between the Rangers and the Devils and Islanders. Winning also helps you generate better TV ratings.

Things That Make You Go Hmmmm

Now why should these numbers be treated with great caution? Sample size my friends. The local market Nielson numbers are generated from some rather small samples of households. For example the large Washington D.C. market has a sample of only about 600 households. Contrast that with a standard major poll taken during a US presidential election in which between 800-1,200 voters are consulted.

In a typical Presidential Election poll a big sample size of 1,000 results in a margin of error of +/-3 which means the point estimate could be off by 3 points in either direction. Now if you do an election poll with ONLY 500 respondents your point estimates can be off in either direction by 5 or 6 points. That's a pretty sobering thought when you realize that only the Buffalo or Pittsburgh markets produce a rating greater than 6% of households on a regular basis. For most NHL teams their entire rating number is within the margin of error for the sample of their metro area.

The other reason to use caution is what we call selection bias. The science of sampling is built on probability theory and randomness. But even a very diligent polling firm will get a "bad sample" every so often. In political poling a "bad sample" is one where you dial phone numbers randomly but you have bad luck and end up with an abnormally large number of Democrats or atypical number of Republicans. In hockey terms a bad sample might involved a metro area where Nielson ends up with either too many or too few hockey fans in their pool of households.

The problem with "selection bias" is that it is hard to know when it is present in the Nielson ratings. In political polling you can see when you have a way more Democrats or Republicans than other random polls. The other major difference is that most political polls draw a totally new sample every time--whereas the Nielson ratings use the same households for an extended period--if they draw a bad sample that bad sample isn't replaced for months.

Now in the political polling business roughly speaking about 1 in 20 samples ends up being a bit unrepresentative. If the same "selection bias" error rate applies to Nielson ratings that means that one of those 24 NHL market ratings is pretty much garbage--the problem is that we don't know which one.

Finally, another big problem for sports teams and NHL clubs in particular is that the rating are extremely vulnerable to the behavior of just ONE OR TWO PEOPLE in the sample. Consider for example the Atlanta market where the sample size is about 500 households. The current rating for Atlanta is .23% which means that on a typical night just 1.15 out of the 500 Nielson housholds are watching the Thrashers. Now if one or two additional households suddenly became fans of the Thrashers, the club's Nielson ratings would "explode" and show 100-200% growth. The Nielson numbers are very sensitive to tiny shifts in the viewing habits becaue the samples for any one metro (outside of NYC, CHI and LA) are rather small. So exercise some caution when reading the chart showing the % increase or decrease. The viewing habits of just one or two people might causing that shift.

Conclusion:

The Nielson numbers are interesting, but should be consumed with a healthy awareness for the potential for errors. To be perfectly blunt the Nielson rating are on shaky ground statistically when it comes to estimating NHL TV audiences--but they are pretty much the only data out there--so people keep using them.