The Government Report That Made Me Stop Trusting Our Statistical Agencies

I’ve waved people off this degree of mistrust. I no longer can do so.

I’m often asked whether, in the age of Trump, we can trust government data. My answer has always been that we can because our statistical agencies have layers of insulation from such interference. To be clear, that doesn’t mean the system is free from any interference. The Trumpies’ hostilities to facts and their maintenance of their alt reality requires them to harass, fire, and defund those wihin in the gov’t statistical infrastructure. But I was skeptical that they could cook the books.

Until now.

This is the new Census publication that flipped me:

The analysis, which you’ll note is unsigned—Census reports have authors, so that’s the first clue that something has gone very awry—purports to find that 24,000 non-citizens voted in the 2020 election. But the work is fraught with serious problems and is wholly unreliable.

Most importantly, it is highly inaccurate. The data used to identify nonvoters is inadequate to that task, as I’ll explain. The fact that there are no confidence intervals around the estimates—e.g., the margins-of-error you get in polls—is another clue that nefarious forces are at play. This is highly unusual in Census analyses.

These concerns are strongly amplified by NPR reporting that uncovered the following:

Work on the analysis was not conducted by career civil servants at the Census Bureau, and the team behind it included individuals affiliated with the America First Policy Institute, a think tank started by officials from Trump's first administration, according to a Census Bureau employee who NPR has agreed not to name because they fear retaliation at work.

I’ve never heard of such a thing and it is the clearest sign yet of Trumpian infiltration of the US statistical infrastructure.

Why do I think the analysis is inaccurate (beyond the obvious signs denoted so far)? Because I, and more importantly, statistical matching experts, do not believe that the process employed by whomever wrote this report could reliably match voter records with Census records of noncitizens.

That is, the “analysts” did not have a single database that recorded both noncitizen status and whether you voted. It had separate lists that it merged, a notoriously challenging process that depends on reliable linkage information that allows analysts to be certain they’re linking the records of the same person.

For the details on the challenges of this process, we’re fortunate that Amy O’Hara, who used to oversee such processes at the Census, has scrutinized the new report, and finds that it “lacks methodological rigor, transparency, and accountability, and the results should be discarded. Moreover, the Census Bureau should answer for the process concerns raised by the publication of this unsigned and shoddy report.”

She writes:

…Matching methods are most effective when data have accurate and complete Social Security Numbers (SSNs). Although documentation is lacking in [the Census report], it is my understanding that commercial vendor voter data do not have SSNs, which substantially weakens the accuracy of linkages to other administrative data.

Without SSNs, matches must rely on combinations of other identifiers such as name, date of birth, and address. For a variety of reasons—spelling variations, typos, missing data, relatives with similar names, common first name / surname combinations, and more—these characteristics produce much weaker match certainty. Here, illustrative examples are helpful.

—With an SSN we could be fairly certain that a record for Amy O’Hara in one database is the same Amy OHara (no apostrophe) in another. Without an SSN, we need to rely more heavily on other clues that offer less clarity.

--Is this John Smith on Main Street the same as that John Smith on Main Street? What if one John Smith’s record shows a birthdate in 1977 and the other in 1997? Is one of those a typo or are they two different people? Perhaps the second John is a son of the first?

--What if there are two unrelated people named Maria Jones living in the same county? What if one is a citizen and one is a visa holder? How can we be sure that the linkage process matches records to the right Maria?

Another measurement expert and former Census analyst, John Abowd, concurs that the matching process was highly inaccurate (my bold below).

False positives are a statistical phenomenon that is inherent to record linkage. No system can avoid them…For commercial data like these authors used, the false match rate even in the first four passes is 0.146%, which is more than seven times greater than the reported 0.019% claimed noncitizen voters. On subsequent passes, this false match rate rises as high as 13.1%, and overall, it is estimated at 0.185%. The alleged “noncitizen voters” are therefore much more likely to be data errors than noncitizen voters.

You’ll note in the Abowd quote that the Census reports just under 0.02% noncitizen voters (of which I’m confident that any confidence interval would find to be statistically indistinguishable from zero). Much reporting on this dastardly—and truly depressing—statistical abuse has focused on this small number—the 24,000 reported in the document, arguing that, even were the analysis to be correct, that number is too small to change the election outcome. I saw one tweet to the tune of, “Congratulations, Trump. You proved you lost the election.”

I take little solace in that critique. First, note that the Census ominously subtitled this “A Beginning Analysis,” meaning that they’re going back to this well. Second, Trump and Lutnick (Commerce Sec’y; the Census Bureau is within the Commerce Dept.), both of whom posess quantitative skills that are, shall we say, limited, have been touting this slop as official gov’t evidence that Trump won in 2020.

For better or worse, I am congenitally non-alarmist, but I fear they could use this report to militarize the midterm voting.

I was alerted to this report by the vigilant and careful reporting of Catherine Rampell, who flipped the same way I did, writing, “No, you should not default to trusting the government data. There’s clear evidence the Trump administration is cooking at least some of the books.”

“At least some” is an important caveat. I don’t take this debacle to mean that the admin has figured out how to manipulate, GDP, inflation, and unemployment reports. But, like Ms. Rampell, I’ve lost my confidence in that assertion.