Margin of error

From an article in the NYT about the Pentagon report on repealing Don’t Ask Don’t Tell:

About 115,000 of the 400,000 active duty and reserve troops who received copies of the survey responded to it, according to the report. The survey had a margin of error of plus or minus 1 percentage point.

Ah, fun with statistics. Something doesn’t add up here. 115K of 400K might seem like a great sample size, about plus or minus .25, actually… but that margin of error would only be good if the survey had only been sent to those 115K of the 400K and not the other 275K. As is, the response rate is only 28%, which sounds typical of a mail survey with little or no followup.

In other words, this survey could have a half-ton of sample bias, particularly non-response bias. I wonder if they called randomly into the pool of nonrespondents to check for this. The report is not public (yet) – anyone seen more data on this?

Leave a Reply