Who Needs Polling?

Discusses how public opinion polling is being undermined by unsound methodology and an uncritical media.

In a recent set of columns, political commentator Arianna Huffington advised Americans to hang up the phone when polling organizations call. She said that polls are unreliable because of low response rates, leading questions and dubious sampling methods. Even though there are many high-quality polls being conducted, there are also enough questionable surveys to warrant consideration of the criticism. In fact, growing numbers of pundits are expressing doubt about the credibility of what the media passes off as reliable research today.

From “American Idol’s” self-selecting phone polls to Internet forums to overnight spot polls, people have many new ways to voice their views and the results are often put forth as legitimate reflections of American public opinion. Perhaps that’s fine when the topic is who is to be the next pop sensation. Unfortunately, such scientifically unreliable methods are also used to solicit and represent Americans’ feelings on serious matters that affect public policy. And they are conveyed through the media without any indication of their quality and reliability.

Without sound methodology – in particular, a careful effort to obtain a random, representative sample – there can be no confidence that a poll actually reflects the reality of public opinion. With online polls, call-in surveys and the like, the people who decide to participate are often those who have a particular point of view they want to advance, and therefore in the aggregate not representative of the population as a whole. Even adhering to sound methodology can’t produce 100 percent certainty. But findings based on anything less are hardly more than organized speculation.

Overnight polls – where the pollsters call phone numbers until they reach a set number of households rather than calling back the original households in the random sample until the desired number is reached – also seriously undermine accuracy and representativeness.

But methodological shortcomings are not the only problem. Many polls are conducted because a sponsor has an idea that they believe can be substantiated and advanced through the process. This can affect how questions are asked and which questions are not asked at all. It also affects what information is released and how it is packaged.
And so for all of these reasons, those who slam polls have a valid point, albeit one that throws the baby out with the bathwater.

The core of the problem is really this: the media often do not distinguish between public opinion research founded on sound methodology and findings based on dubious or not much methodology at all. Describing a poll’s methodology in full may not be feasible, so the reporter should make a judgment and say whether or not the poll has serious merit and is conducted by a reputable organization or is simply an unscientific survey primarily done for entertainment purposes.

Ironically, it is during an era of unprecedented technological advances, which provide all kinds of new ways for individuals and organizations to advance their opinions, that reflections of public opinion are often distorted. That is why organizations engaging in deep, probing and unbiased public opinion research are needed more than ever.
Public Agenda never holds back findings that don’t fit a certain point of view. Public Agenda’s only commitment is to unbiased, thoughtful exploration of issues, not the advancement of an ideology. Our commitment is to the public itself, not to
a particular point of view. Thorough, sound research is not easy.

It takes time and resources, and it requires sticking with issues over the long haul. But without this investment, all one has is anecdote and intuition. Sound public opinion research has become one of the most important ways the American people are able to contribute to public debate – and the citizens’ voice has never been more needed than it is today.