Choice of Sample
Another potential Sampling Error
Above is a screen capture from Realclearpolitics from their local Virginia election poll page. I took the capture July 17th, 2024.
From the earlier discussion on Margin of Sampling Error, readers should be able to note that the MOE's given have very large intervals. And although the VCU and NY Times/Siena spreads indicate a +3 for Trump or Biden, because of the small samples, all of the polls should really be read as, "because of our large confidence intervals, we have no idea who is ahead".
I have outlined the Sample section in green. Each poll has either an A, an LV, or an RV after the sample size. What do these acronyms stand for?
Clicking on the VCU link, we get the following methodology paragraph:
"The VCU-Wilder poll consisted of telephone interviews with 809 Virginia adults, age 18 or older, with about a third conducted on landlines and the rest on cellphones. VCU weights the results based on Census Bureau data so the raw numbers of the poll do not reflect the weighted percentages."
So A stands for adults as noted in red. If you look at the methodology sections of the other polls, one finds that LV stands for Likely Voters and RV stands for Registered Voters. These are very different samples, although they are used in the RCP average and discussed in their polling results as if you can easily compare them.
The VCU methodology blurb says that VCU weights results based on Census Bureau data. Let's look at census population numbers from the last presidential election held in 2020. The total adult population of Virginia was around 6,740,000. However, not all adults are citizens. And not all adults take the time and effort to register to vote. So the Virginia Department of Elections lists only 5,975,696 people as Registered Voters.
And, not all registered voters take the time to go to the polls or fill out an absentee ballot. So the same Department lists only 4,486,821 or 75% of the registered voters as actually casting a vote in the November 2020 Elections (a high percentage compared to off year, non-presidential elections). A poll using a Likely Voter model attempts to provide results for this last subset of registered voters who actually cast a vote.
VCU models their sample to provide results for adults. So VCU's sample is for a much larger population than those adults who actually vote (by more than 2 million people). This is not really an election poll. It is an opinion poll. As it includes close to 1 million people who can't vote because they aren't registered. And even those registered to vote only cast a vote at a 75% rate in the last presidential election.
Choosing the wrong sample to model your poll after is another type of sampling error. Choosing a sample of all adults or even registered voters is a choice that polling firms make that introduces known errors in their sample compared to likely voter polls. Only likely voter polls attempt to adjust sampling models to more accurately reflect expected voter turnout on election day.
Yet here is how the news organization in the VCU link reports the results of the poll:
"Former president Donald Trump leads President Biden in Virginia by three percentage points in a new Virginia Commonwealth University-Wilder School poll...The results, which are within the poll’s 4.8-point margin of error, show Biden dropping from the 42 percent support he enjoyed in the last VCU-Wilder School Commonwealth Poll, released in January, while Trump was unchanged at 39 percent...A New York Times-Siena College poll conducted in Virginia July 9-12 showed the opposite result, with Biden leading Trump by three percentage points — 48 to 45 — among likely voters, though that result is also within the poll’s margin of error."
And the big talking point The Washinton Post article mentions is:
"'Virginia is in play,' said former governor L. Douglas Wilder (D)..."
While this is an interesting narrative that the noted shift from +3 Biden to +3 Trump means Virginia can now be won by Trump, the two poll results just aren't valid statistical evidence. The large Margins of Sampling Error mean we have to treat the quoted polls as too inaccurate to measure the differences between the candidates. And even if there were a statistically valid difference, the all adult population model of the VCU poll really doesn't have much accuracy in predicting election votes. It is inaccurate by millions (in this Virginia example) in overcounting people who can't or likely won't vote in the next presidential election.
Let's see what it may mean when polls choose the wrong sample.
The Census Bureau's 2020 Presidential Election Report notes that compared to the Citizen Voting-Age Population (CVAP), the actual population who cast a vote in the election contained the following over represented groups:
Those ages 45 to 64.Those age 65 and older.The non-Hispanic White population.Those with some college or an associate degree.Those with a bachelor’s degree or higher.
So any model which attempts to poll likely voters should overweight these groups in their methodology compared to the CVAP.
Conversely, if using a citizen adult voting model, polls should be adjusted so they approriately overweight the opposite groups compared to the actual voter population. Specifically:
Those ages 18-29 (which includes college age adults).
Blacks, Asians and especially Hispanics.
Those with only a high school educaton.
Those who make a family income of less than $50,000 a year.
These groups/characteristics of a population are known as demographics. And those demographics may sound familiar as they are often associated with voting blocks with a preference for Democratic Party candidates. By overweighting these groups in an adult model vs. a likely voter model, the adult poll may skew its results with a Democratic Party preference error. I'm not saying that the adult polls are not important. But as I said earlier, this type of poll with a known bias towards these groups is an adult opinion poll. It is not an election poll capable of accurately being prefaced with, "If the vote were held today..." And adult model polls should not be included in an election section averaged in with likely voter polls.
Also, it should be noted that the CVAP removes the non-citizens from the adult population. So the CVAP is somewhere between all adults and the registered voter population. A breakdown of the report also shows similar over representation patterns when comparing registered voters to those who cast a vote on election day. So registered voter polls will show a similar bias but of a lesser magnitude than adult voter polls.
In addition, it is also possible that with a changing electorate, these bias errors could shift toward Republican Party preference error as well. For example, Trump has made large gains with some demographic segments that traditionally vote with the Democratic Party, such as Hispanics. However, both types of party bias in adult and registered voter polls would be minimized by using likely voter polls with accurate models in their methodology.
Something else to consider, once the polling firm chooses a sample, they would want to be consistent and use the same sample over time right? This would allow the polling firm to compare results in successive polls which use the exact same methodology. However look at the screen capture below from Realclearpolitics. It is an expanded list of general election Trump vs. Biden polls I captured on July 24, 2024.
I outlined three successive CBS News polls taken over a month in June and July. CBS News changed their sample from Registered Voters to Likely Voters during this time. Why would they introduce this inconsistency and error into their polling? Maybe they liked the bias of Registered Voter polls which showed the candidates close from 6/17-6/21. Maybe they want to shift to Likely Voters in July polls so they have a better chance to have results closer to actual election day results as the election day nears. However, consistency and accuracy in election cycle reporting is not the priority of CBS News when they switch out their samples like this.
The simple truth is, there is just no reason to use an adult poll or even a registered voter poll to estimate a candidate preference on election day by actual voters. If the objective is to have an election poll, not an opinion poll, then the sample chosen should be a likely voter poll.
The last bit of jargon from the realclearpolitics screenshots I'll talk about is the RCP Average.
The last bit of jargon from the realclearpolitics screenshots I'll talk about is the RCP Average.