Aggregate Averages
Are poll averages legitimate ways to measure poll results?
I screen captured the Virginia General Election poll table above from the Realclearpolitics site on July 20th, 2024. It shows a change from the table from the previous section I captured on July 17th. At the top of the table outlined in green, the RCP average is now +0.4 for Trump in the spread section. The July 17th table was a tie. The only change was to include the new Emerson poll that averages in the Emerson +2 spread for Trump. Can we say that Trump is now ahead in Virginia because of the RCP average?
From a previous section, we can see that the new Emerson poll should be considered too inaccurate to make any statistical claims based on the margin of sampling error. Trump's actual support will fall between a 42%-48% confidence interval. And Biden's will fall between a 40%-46% confidence interval. The confidence interval is just too large to say much of anything about who is actually ahead or if it's too close to call.
But what about that +0.4 spread of the RCP Average? It doesn't have a sample or a margin of sampling error because it is an average of the polls below. These polls have different types of samples, different sizes of samples, different sampling errors, and were taken months apart from each other. The methodology of each poll is different in various ways from the variation in questions asked to what lengths each poll takes to ensure they are accurately modeling their chosen samples (adults, registered voters, or likely voters). This average introduces large additional errors in predicting election day results. Let's look at what an average that attempts to quantify those errors says about poll aggregates.
ABC News has averages under their 538 brand. They discribe their presidential polling averages here in an article dated April 25th, 2024. They state that when considering their averages:
"there's considerable uncertainty in those numbers. To communicate this, we are also publishing uncertainty intervals for our horse-race averages for the first time. These intervals — represented by the red and blue shaded areas around each line — are kind of like the range of possible precipitation reported in a weather forecast, showing you could get anywhere from, say, 1 to 3 inches of rain in an upcoming storm."
So their carefully calculated averages are no better than a weather forcast for the amount of rain? 538's description continues:
"Our uncertainty intervals take into account the variability of the polling data and the uncertainty we have about the various adjustments we are making, which are detailed later in this article. Right now, that interval shows that Biden's support could be anywhere between 39.4 and 42.2 percent, while Trump’s range is from 40.3 to 42.8 percent. Read on for more information about all the sources of uncertainty we are (and aren't) taking into account for these averages." Or put another way, 538 admits that they are introducing all sorts of errors by trying to come up with a poll average. There is no such thing as a margin of sampling error associated with this made up statistic, so they will try and make up a confidence interval statistic by adjusting and weighting all sorts of data to make their error prone statistics more legitimate. Worse yet, they state that they are averaging polls together of varying quality. And they have to rate the quality of each poll and reweight the average to make up for all the error introduced by including the low quality polls. "These ratings distill the empirical record and methodological transparency of each pollster into a single rating (from 0.5 to 3.0 stars) that tells our model how seriously to take its polls." Here is a screenshot of their ratings page I captured on July 24th, 2024 and last updated February 22, 2024.
"Our uncertainty intervals take into account the variability of the polling data and the uncertainty we have about the various adjustments we are making, which are detailed later in this article. Right now, that interval shows that Biden's support could be anywhere between 39.4 and 42.2 percent, while Trump’s range is from 40.3 to 42.8 percent. Read on for more information about all the sources of uncertainty we are (and aren't) taking into account for these averages." Or put another way, 538 admits that they are introducing all sorts of errors by trying to come up with a poll average. There is no such thing as a margin of sampling error associated with this made up statistic, so they will try and make up a confidence interval statistic by adjusting and weighting all sorts of data to make their error prone statistics more legitimate. Worse yet, they state that they are averaging polls together of varying quality. And they have to rate the quality of each poll and reweight the average to make up for all the error introduced by including the low quality polls. "These ratings distill the empirical record and methodological transparency of each pollster into a single rating (from 0.5 to 3.0 stars) that tells our model how seriously to take its polls." Here is a screenshot of their ratings page I captured on July 24th, 2024 and last updated February 22, 2024.
It looks very scientific and legitimate. Right? With such an accurate statistical analysis of the quality of polls, they would never just throw out their model and completely change their rating system, would they?
As one should be aware by now from previous discussions of polling firms, consistency and accuracy is not very important to many of them. So of course they would change their polling system if it better supported the goals and narratives of their clients.
Below is a screen capture from an archive site. Last updated March 13, 2023. It represents how 538 used to present their poll firm ratings before Nick Silver, the founder of 538, left Disney/ABC. Instead of a 3 point rating system, poll firms received a letter grade. And instead of a transparency score, firms were rated on how much political bias they had in their poll results, Democrat or Republican.
Letter grades and bias scores are much easier for the general public to understand than 3 point scales and transparency scores. But notice how with ABC News' new rating scale (above), ABC News/The Washington Post has moved up to a 2nd place ranking. And their Democrat party bias is no longer displayed. A very convenient outcome for the parent company of 538. And all the top three rated (perfect 3.0) polling groups in the rankings above have Democrat leaning biases outlined in green below.
Nate Silver wrote an article, a portion of which is quoted below. It is dated July 1, 2023 and titled, "Polling averages shouldn't be political litmus tests"
He notes that:
"This past week, the new Editorial Director of Data Analytics at ABC News, G. Elliott Morris, who was brought in to work with the remaining FiveThirtyEight team, sent a letter to the polling firm Rasmussen Reports demanding that they answer a series of questions about their political views and polling methodology or be banned from FiveThirtyEight’s polling averages, election forecasts and news coverage."
It appears to be his conclusion that ABC News' focus on trying to get Rasmussen to be more transparent about their associations and methods is actually a way to exclude them from the rankings by making demands on them that they will not be able to meet. They are attempting to drop a Republican biased polling firm without holding Democrat biased polling firms similarly accountable.
He writes, "...this looks like a fishing expedition, with Morris hoping to catch Rasmussen in some sort of venial methodological sin that is probably fairly common within the industry. Or, because the questions are onerous, the tone of his email is hostile, and Carroll was only given a day-and-a-half to respond just before a four-day summer weekend, he was hoping that thy [sic] wouldn’t be answered at all — so he could say 'See! They refused to answer my questions!'. Either way, this is the letter you get only once someone has already made their mind up."
And in fact, Rasmussen was dropped from 538's rankings shortly after Nate Silver warned that this was ABC's intention.
What does this mean for making up an aggregation statistic averaging polling results?
Trying to aggregate polling data introduces large amounts of error into the usefulness of the poll average statistic. The polls vary on an almost countless array of characteristics from populations modeled, samples, methodology, dates completed, and quality. Just averaging all these polls together means there is very little predictive value for election day results compared to individual high quality polls which are careful in how they choose their populations, samples, and methodology.
538 attempts to mitigate this inaccuracy by using statistical methods to move the needle back in the perceived direction of election day results. However, each of these statistical methods introduces more error that must be mitigated. The example provided here is that averaging in low quality polls requires that the low quality polls are statistically removed from the average by creating poll quality rankings. However, even these ratings systems appear to be arbitrary and highly susceptible to introducing political bias in the results. Which brings up the question, why attempt to smash all these polls, regardless of quality, into one average in the first place? I believe that it is not worth averaging the polls if the average creator must statistically remove the many resulting errors (introducing even more errors at each step).
It is far better to use high quality individual polls which can choose their populations, samples, and methods carefully. And it should be fairly straightforward to come up with a ranking because there is an election day result one can eventually compare the polls with. Rankings should be based on how close the polls are to actual election day results. What other parameter could be more important?
The simple truth is that aggregate poll averages are not as accurate as a well modeled and executed individual poll. Averages should not be used to complete the sentence, "If the election were held today..."