Impartiality in Politics: An Analysis of iSideWith.com’s Popular Quiz
The popular political quiz website iSideWith.com, created by Taylor Peck and Nick Boutelier, boasts that more than 53 million voters have used their tool to find a candidate or party affiliation. Visitors can fill out over a hundred yes-no questions on their political stances that will then match them with candidate and party positions.
It’s a clever way to encourage clinical politics — divorcing emotions and identifying with candidates purely on principles and policy. Having to gather my thoughts to fill out the questions was rewarding in its own right — and I encourage those who might be interested to try it (I see it as an especially helpful tool for lower-level elections). What fascinates me about the site, however, is not so much its conceit, but rather its claim to be “nonpartisan” and impartial.
The ideal of impartiality is evasive when it comes to politics. One of my non-profit’s first projects, a video game about misinformation we did in partnership with the Pulitzer Prize winning organization PolitiFact, aspires to this ideal. Described by some as “Tinder for fake news”, you’re presented with a statement made by a politician and asked to swipe left if you believe it is factually inaccurate or right if you believe the claim has verity. Like iSideWith.com, the questions are dichotomous, binary, true or false, yes or no — and while that may sound simple enough, I can attest that more time was spent building bots to play my game randomly or semi-randomly to capture invisible and subconscious bias than was spent programming the game itself.
To this end, I thought it would be interesting to subject iSideWith.com’s popular quiz to one of the most basic of these tests, not because I believe they are acting in bad faith or pursuing a partisan agenda, but rather because having worked on something vaguely similar, I am floored by the logistical magnitude of how one would design such a quiz around minimizing intractable bias.
So first let us run a simple experiment. If you recall from above, results in the quiz return a percent match with each candidate, 0–100%. I submit that if the quiz were truly nonpartisan and impartial in the ideal sense, the following statement would be true:
If all of the yes-no questions are filled out randomly, no one candidate would have a statistically significant advantage over another.
This seems reasonable, however, we will need to repeat the test many thousands of times to ensure any differences are not a result of chance and chance alone. For this, I created a short bookmarklet, a way to quickly inject javascript into a page client-side. This code is below:
It simply exposes all of the questions in the quiz and fills them out randomly. Out of respect for iSideWith.com I will not document how I used this in tandem with a web scraper to repeat many thousands of times, as I don’t wish for their servers to be overburdened, but if you’re curious about the process for another project you have in mind, feel free to DM me. I will also mention I was careful to only use one ID the whole time so that their metrics were unaffected (equivalent to re-taking the quiz and and changing the answers). Here is the raw data in excel with almost 3000 results, and embedded below, the averages and standard deviations for each candidate and party affiliation:
The first thing I noticed was that all candidates (except for Marianne Williamson, more on her later) are over 50% and very significantly so,- given the nature of the quiz I expected all the candidates to converge toward 50%, but this overage may actually be an indication of iSideWith attempting to correct biases. In my own little thought experiment of how I might approach building the mechanics of the quiz, a consideration I had was to just repeat our random test many hundreds of thousands of times, and then artificially compensate a candidate’s match percentage globally for whatever the differences were. You wouldn’t necessarily want the added complication of lowering other candidate’s scores around each adjustment, so this methodology could explain why the averages are so much higher than 50%.
First thing first, we don’t need a t-test to see that the differences between democrats and republicans aren’t significant. This is truly impressive given the fact that with a republican incumbent, you have many more, and much more diverse, democratic candidates on the ballot. Next, we’ll do a simple ANOVA:
While there is some variation, only two results stand out to me as being significant. T-tests between each candidate’s average and the global average reveal the following disadvantages are statistically remarkable: Andrew Yang whose two-tailed P-value is about 0.0111, and Marianne Williamson whose two-tailed P-value is a exceptional 0.0001. To say this another way, the chance that Williamson’s “disadvantage” here is fair and only a result of chance is less than .01%. Still, take this with a grain of salt:
All in all, these discrepancies are relatively nit-picky and the consistency of the results both surprised and heartened me. I also imagine that the resulting percentage we get is rounded (and understandably so), adding to overall error. I repeated the test (with a smaller sample size of around 1,000) for the 2016 election and found no significant differences between candidates (other than a slight advantage to Jill Stein that was barely statistically relevant). I should also mention I noticed that both Donald Trump and Gary Johnson’s scores (candidates who are involved in both the 2016 and 2020 elections) were slightly different between the 2016 and 2020 quizzes with the same questions filled out, which may further support the possibility that there is a behind-the-scenes biasing mechanism in place to balance all the candidates.
As sensible as this seems, one must also ask the more fundamental question of whether this balancing act is, in fact, an act of partiality in itself. Perhaps the initial hypothesis put forth, that purely random answers should generate perfectly representative results across the board, is a flawed foundation since it presupposes all questions should equally represent all candidate’s positions.
I know what you’re thinking — wait, isn’t that the whole point? Isn’t that the very definition of being impartial and why you ran this little experiment? Well, if you’ll permit me a gratuitous example, suppose someone were to run for president with objectively revolting policies — eating puppies, let’s say. Is it then incumbent on iSideWith, and us as a nation in general, to tip the scales in the name of nonpartisanship to grant this candidate a platform mathematically equivalent to that of the rest of the field? If Marianne Williamson is, in fact, objectively disadvantaged as seen in our little experiment, couldn’t that result, in theory, be entirely reasonable and not evidence of bias in the least?
I would argue that instead of expecting questions to represent all candidates positions fairly, equally representing topics that Americans find relevant according to recent polls, or even relativizing the representation of a candidate’s position by their current polling numbers would be effective alternatives, albeit sadly impractical in today’s political climate. Imagine the scandal, if, say, the results above showed a significant p-value that Trump was disadvantaged. I’m just as guilty — I was myself salivating to find some discordance in the results of this modest little experiment until I found the results almost suspiciously equitable.
The balancing and counter-biasing mechanisms I used to adjust my PolitiFact Game and the similar methods I suspect (again, I do not know, but the results of a sophisticated relational database are rarely that neat in my experience) were in some way utilized in iSideWith’s true or false quiz are important and useful, but I worry we may be letting our fear of being perceived as bias compromise some degree of journalistic and scientific integrity. I know for myself, I consciously implemented these measures partially to better defend against inevitable accusations of bias and partisanship. Despite the (undeserved) accusations of its left-leaning bias, I think an argument can be made that PolitiFact possess, if anything, a conservative bias simply because they want so badly to defend themselves against attacks on their legitimacy. It’s natural to imagine how this fear could weigh heavily and influence how they curate content.
There are those who say that bias and subjectivity, especially of the political variety, is unavoidable — that we are compelled by our nature towards partiality. While this may be true in some abstract sense, I’ve witnessed too many lazily embrace this notion without any careful consideration for what should constitute journalistic and scientific bias, even in something as simple as a binary quiz. And whether you agree, I hope you’ll at the very least acknowledge that it’s never quite as simple as true or false.
I am in no way affiliated with iSideWith.com. I hope they’re not upset with me for scraping some of their data.
