Simple Poll Truths
What are the simple truths for polls?
1. The margin of sampling error is only one of an infinite number of errors pollsters can introduce in their polls.
2. The margin of sampling error can be reduced to a very small interval by just increasing the sample size of the poll. Most pollsters choose large intervals that make it statistically impossible for the polls to draw conclusions with any confidence in close races.
3. Polls should use likely voter models to predict results on election day. Not only do many choose less accurate adult and registered voter models, they change the models from one time point to another during the election cycle.
4. High quality individual polls are better than the error prone mashing of polls in aggregate averages for predicting results on election day.
Imagine if runners' results for the 100 meter dash were measured over a year and used to predict the results for their performance in the olympics. The times were only given in a plus or minus 2 seconds range (when a fraction of a second can determine the winner). The runners were measured for performance in the 100 meter hurdles in the first half of the year (because it made certain atheletes news organizations wanted to write stories about competitive), only measuring their performance in the 100 meter dash right before the olympics when results needed to be more accurate. And all the results were then mashed together in one 100 meter aggregate average that then needed to be statistically reweighted and adjusted to try and remove the mash up error.
We would not as sports fans put up with such an inaccurate, error prone system for predicting olympic results. However, this is the current state of political polls in the United States.
Until the public and news organizations that use political polls demand more, I fear that not much will change in the political polling landscape. It is often said that one can use statistics to support any result they want. I think it is more accurate to say that pollsters and news organizations can choose to introduce whatever errors in their polls they want to support their own narratives and political biases.
Consistency in polling is key to improving political polling moving forward. At the very least, polls themselves should not be changing their methodology and samples over the course of an election cycle. Then at least methodologically identical polls could be compared in time to see trends in the electorate after important events. Consider the following Morning Consult series of polls from the realclearpolitics site I captured on July 24th, 2024:
If I plot the results of these polls with similar methodology and include major election cycle developments, I produce the following graph.
.gif)
This period of time was marked by a poor Biden debate performance (Biden's cognitive decline on full display), an assasination attempt at a Trump rally, and a Republican National Convention with generally positive coverage in the news media. The press was almost unanimous in its reporting of Democrat panic following the debate, with increasing calls for the candidate to voluntarily drop out of the race by the press and politicians. Biden, however, was steadfast in his defiance of the calls and continued to insist he was staying in the race until his announcement on July 21st that he would withdraw.
Now I don't pretend to know the internal polling that Biden was looking at, but from the Morning Consult results, it is clear that in the first three polls following the debate, Biden's support did not drop outside the margin of sampling error. The race was statistically too close to call and the resistance of Biden and his allies to drop out was understandable. It appears that Trump's response to the assassination attempt and the generally positive coverage of the Republican National Convention was enough to finally separate the two candidates statistically in the last poll. There were probably many other factors contributing to Biden's decision to drop out of the presidential race. But Trump taking a statistical lead in the polls had to be an important factor.
And the advantage of the large sample in reducing the margin of sampling error to +/- 1 is clear as the two candidates at that point were only seprated by 4 points. Well within the margin of sampling error of many other polls.
Of course it would have been even better if Morning Consult used a high quality likely voter model. And it would be even better if there were a group of say the closest five polls in predicting past presidential elections all displayed on the same page for comparison.
But I can already hear the Nate Silvers of the world saying those five polls have to be averaged together...
Hope you enjoyed this article on the simple truths of political polls.
The Doctor of Truth