Here is a list of caveats to the measure of factuality from fact checker ratings. More caveats will be added when they come to light, and when Brash comes around to writing about them. Also stay tuned for discussion of caveats in Malark-O-blog. Thanks to friends, family, colleagues, and assorted redditors who raised these issues.
Factuality, truthfulness, or both?
It's easy to assume that a politician's grounding in facts bears on how truthful that politician is. But factual accuracy is better understood as a function of both truthfulness and erudition. Although a lot of deception goes on in politics, sometimes a politician is simply misinformed. In any case, we should prefer politicians who are both better informed and more truthful.
What is bias?
When the desire for fair and balanced news took hold of some journalists in the late 20th and early 21st centuries, it paved the way for a lot of misunderstanding about the word bias. When Malark-O-Meter talks about "bias", it's bias in the statistical sense. Thinking of throwing darts at a dart board, trying to hit the bullseye. Accuracy refers to how well your hits are centered on the bullseye. Precision refers to how well you group your shots. It is possible to be accurate without being precise, and it is possible to be precise without being accurate. Bias is the distance between the center mass of your grouping and the bullseye. In this sense, bias isn't morally wrong necessarily.
Are fact checkers biased?
Many believe that some fact checkers are politically biased. Political bias influences the statements that fact checkers rate, whose statements they rate, and how they rate them. Others recognize that fact checkers typically rate contentious statements, which may lead factuality scores to underestimate the true factuality of politicians. Often, Malark-O-Meter will rate and compare individuals' factuality under the assumption that fact checker data is not so biased that it is useless. In the long term, however, Malark-O-Meter will execute and report on research bearing on the existence and strength of these biases and, if necessary, develop methods to overcome them.