Challenge those numbers

I've been remiss in reading the Numbers Guy column at the Wall Street Journal. (Actually, I've been remiss in reading almost everything I normally try to catch.) So it is with my head hanging low that I bring you news of the April 22 column this late.

But you gotta read it.

Carl Bialik looks at how aging statistics end up being recycled in media reports for years -- with no one looking at where the numbers came from or whether they still hold true.

He uses three great examples. The first is from an Associated Press article that said, "Canadian police estimate that more than 100,000 Web sites contain images of child sexual abuse."

Bialik did a little digging:

It turns out the Canadian police were citing the statistic from a three-year-old article in a magazine that stopped publishing in 2003. The magazine's source is a U.S. agency that no longer exists. And the agency that has replaced it can't track down the original source of the stat. The lesson: An old stat can get new life when "experts" repeat it, especially when there is no conflicting version of the number.

But even the source doesn't matter that much.

Even if the number wasn't so old, there are still plenty of reasons to question it. Counting Web sites is not an exact science (Google provides only estimates of hits for searches). Also, it's unclear how active the sites were. If just one site carried child porn, but it had the traffic of, say, Yahoo, that would be far more troubling than 100,000 unvisited sites.

His most recent column deals with another dubious stat, and this story is a good read, too. It starts with the Texas House trying to pass a law restricting gay and lesbian couples from serving as foster parents.

Of course, the topic was covered endlessly by quote machines on CNN, and one of them served up this: "Children in foster homes with same-sex parents are 11 times as likely to be sexually abused as those with heterosexual parents."

Bialik traces the statistic's path:

To get on CNN, that number snaked through a twisting path, from a little-noticed Illinois study published by an antigay scientist/activist in a psychological journal, to several conservative Web sites, to, finally, the attention of a Texas activist who presented her misinterpretation of the study on national television, essentially unchallenged.

My main point (besides "Read the Numbers Guy!") is that we have to get in the habit of looking at numbers -- not just at checking that they compute, though that's very important. But we have to make sure these stats make sense, prove what they say the do, and have some basis in reality.

To let them go by because we're scared of numbers is even more dangerous once we see that publishing them once lets others recycle them for years.