It’s cherry-picking season already.
By Nate Silver
President Trump during a campaign rally on Saturday in Melbourne, Florida.
Here’s what we can say for sure: It’s unprecedented for a president to face so much opposition from the electorate so soon. Recent polls show that anywhere between 43 and 56 percent of Americans disapprove of President Trump’s job performance. Even if you take the low end of that range, Trump’s numbers are much worse than any past president a month into his term.1
But beyond that, there’s a lot of seeming disagreement in the polls about exactly how unpopular Trump is — and even whether his disapproval rating exceeds his approval rating at all. Moreover, the differences between Trump’s best surveys and his worst ones span a critical range. Take one group of polls, and the country looks about evenly divided — a lot like it did during the 2016 election, when Trump narrowly lost the popular vote but nonetheless won the Electoral College. Take another group, and his electoral fortunes look much bleaker, with Trump already unpopular enough that the House of Representatives could be in play despite Republicans’ advantages from gerrymandering and the geographic distribution of their voters.
What’s the real story? The differences between the polls aren’t random, or at least they don’t appear to be based on the relatively limited amount of data we have so far. Instead, Trump’s approval ratings are systematically higher in polls of voters — either registered voters or likely voters — than they are in polls of all adults. And they’re systematically higher in polls conducted online or by automated script than they are in polls conducted by live-telephone interviewers. Here’s every approval rating poll that we can find for Trump so far this month:2
POLLSTER POPULATION* METHOD APPROVE DISAPPROVE
Rasmussen Reports LV Automated + online 55% 45%
Morning Consult RV Online 49 45
Zogby LV Online 48 43
Fox News RV Live caller 48 47
Emerson College RV Automated 48 47
Harvard-Harris RV Online 48 52
YouGov RV Online 46 49
Ipsos Adults Online 46 50
SurveyMonkey Adults Online 46 53
CNN / ORC Adults Live caller 44 53
YouGov Adults Online 43 47
Public Policy Polling Voters Automated + online 43 53
IBD / TIPP Adults Live caller 42 48
Quinnipiac RV Live caller 42 51
Pew Research RV Live caller 42 54
Gallup Adults Live caller 41 55
CBS News Adults Live caller 40 48
Pew Research Adults Live caller 39 56
45 50 A big spread in Trump’s approval ratings
*LV = likely voters; RV = registered voters.
Based on polls conducted between Feb. 1 and Feb. 19 2017.
Sources: RealClearPolitics, Huffington Post Pollster, @Politics_Polls
Trump’s overall average is 45 percent approving and 50 percent disapproving. You could come up with fancier ways of calculating the average, such as by weighting the higher-quality polls more heavily. Indeed, we’ll be releasing our own FiveThirtyEight-style approval ratings averages soon. But the more important difference, as I mentioned, is that the average varies based on the sample population and the type of survey:
SURVEY POPULATION APPROVE DISAPPROVE
All adults 43% 51%
Voters (registered or likely) 47 49
Live caller 42% 52%
Online or automated 47 48 Trump’s average approval ratings by type of poll
Based on polls conducted between Feb. 1 and Feb. 19, 2017.
Sources: RealClearPolitics, Huffington Post Pollster, @Politics_Polls
All adults vs. registered voters vs. likely voters
Trump has a fairly poor 43 percent approval rating — and a 51 percent disapproval rating — among polls of all American adults, but he improves to a 47 percent approval rating and a 49 percent disapproval rating among polls that survey registered voters or the narrower group of likely voters. That’s a reasonably big difference. So which polls should you use?
Traditionally, approval rating polls are conducted among all adults, so those are probably better for making historical comparisons. And there’s something to be said for inclusivity if your goal is to assess the extent to which Trump has a mandate with the public. He is, after all, the president of all Americans and not just those who are registered to vote or who do so regularly.
But for forward-looking, predictive purposes — to assess the effect that Trump will have on the midterms, for instance — the voter-based polls are probably more useful.
While there can be good reasons for using polls of voters as opposed to those of all adults, however, I’d be wary of making too much about the difference between registered-voter and likely-voter polls. At this early stage, it’s hard to predict what the likely voter electorate will look like in 2018. Midterm voters are typically older and whiter than registered voters overall, which should help Republicans. But they’re also better-educated, which should help Democrats. Furthermore, the “enthusiasm gap” can vary quite a bit from election to election, although it usually favors the opposition party in the midterms (i.e., Democrats in 2018).
The differences between these various types of polls may also narrow as we collect more data. So far, the only pollsters surveying likely voters are Rasmussen Reports and Zogby, and they aren’t very good pollsters. And the only pollsters we could find releasing numbers among both all adults and registered voters — which provides for the most direct comparison between those groups — are YouGov and Pew Research. YouGov’s poll showed Trump’s approval rating lower among all adults than among registered voters, but his disapproval rating was lower also. That’s a fairly typical pattern: Adults who aren’t registered to vote are often politically disengaged and may have indifferent views toward Trump.
Live-caller vs. automated script vs. online polls
Polls conducted with live-telephone interviewers, such as Gallup’s, show worse numbers for Trump — an average of 42 percent approving and 52 percent disapproving — compared to polls conducted online or by automated telephone calls, which have his numbers almost even at 47 percent approving and 48 percent disapproving.
One theory about this is that the online and automated polls reveal “shy” or “hidden” Trump support, with people more willing to reveal their true feelings about the “politically incorrect” Trump in online or automated polls where they have greater anonymity. It’s a plausible theory, but I’m not sure it’s really supported by the evidence. Trump didn’t overperform his polls overall during the Republican primaries, and while he did so in the general election, the overperformance was concentrated among white voters without college degrees, not the group you’d expect if the “shy Trump” theory is right.3
There are also arguments against using online and, especially, automated polls. They don’t have as long a track record of success, even if some of them fared reasonably well in the 2016 general election. Moreover, they may have trouble reaching a representative sample of the population. Automated polls only call landlines, which means they miss the roughly half (!!) of the American population that uses mobile phones only.4 And most online polls don’t use probability sampling, the traditional means of taking a random sample of the population.
Without getting too far into the debate, we’d probably recommend using a mix of live-caller and the higher-quality online surveys, while being suspicious of automated polls.
A further complication is that there’s a relationship between how a poll is conducted and who it’s surveying. For whatever reason, most live-caller approval ratings polls survey all adults, while most automated or online polls survey registered or likely voters. Thus, there’s some cross-contamination between the problems I’ve mentioned here, and it will take more data to sort it out completely.