Measuring uncertainty
Why measure uncertainty?
Rather than answer questions like, "Is Barack Obama more or less truthful than Mitt Romney?", Malark-O-Meter prefers to answer questions like, "Given the amount and type of data we have, how certain can we be that Barack Obama is more or less truthful than Mitt Romney given a set of assumptions we have made about how to measure factuality?"
Rather than ask, "Was Barack Obama less factual during his first presidential debate than he usually is?", Malark-O-Meter would ask, "What's the likelihood that Obama was less factual during the debate compared to the likelihood that he was more factual? And can we say with any confidence that his factuality was any different than it usually is?" These are good questions because they recognize that, even if the assumptions behind our measures of factuality hold true, the measures themselves are still imperfect.
Rather than ask, "Was Barack Obama less factual during his first presidential debate than he usually is?", Malark-O-Meter would ask, "What's the likelihood that Obama was less factual during the debate compared to the likelihood that he was more factual? And can we say with any confidence that his factuality was any different than it usually is?" These are good questions because they recognize that, even if the assumptions behind our measures of factuality hold true, the measures themselves are still imperfect.
Where does uncertainty come from?
A lot of things produce uncertainty. Malark-O-Meter tends to focus on one: sampling error. Sampling error arises when we try to measure some characteristic of a population, but the costs of doing research force us to measure only a sample from that population. The characteristic could be average height in the United States in 2012, making the population everyone who lived in America during that year. For Malark-O-Meter, the characteristic might be the average factuality of the statements that someone makes, the population being all of the statements that person made in a given period of time.
How do we measure uncertainty
Given some assumptions about what the population measure might be, statisticians can calculate the probability that a sample of a given size will produce a given estimate of the population measure. From the distribution of these probabilities, we can calculate confidence intervals. Confidence intervals are the range of estimates surrounding an observed measure that we can be X% confident we would obtain if we were to collect many many samples of a given size. We can also calculate the probability that the estimate will be greater or less than some number. The awesome thing is that we can calculate sampling distributions and confidence intervals for almost any measure (or parameter) that interests us, including but by no means limited to a measure comparing two or more individuals, or comparing an individual to himself in different contests, or one that measures the association between an individual's age and his or her factuality.