Uncertainty in individual factuality scores
These methods apply to analyses of any one of our malarkey scores.
Estimating the probability distribution
Step 1: Sample the percentages of statements in each category of a each available report card from a Dirichlet distribution with parameters equal to the total number of events each category plus one.
Step 2: Caclulate malarkey from the sample report card percentages.
Step 3: Repeat steps 1 and 2 many times. The more the better. When Brash started doing this, he did it 10,000 times. Now he does it at least 100,000 times.
Step 4: The set of proportions of factuality scores at a each given value estimates the probability distribution of the factuality score in question.
Step 2: Caclulate malarkey from the sample report card percentages.
Step 3: Repeat steps 1 and 2 many times. The more the better. When Brash started doing this, he did it 10,000 times. Now he does it at least 100,000 times.
Step 4: The set of proportions of factuality scores at a each given value estimates the probability distribution of the factuality score in question.
Estimating the confidence intervals
From the estimated probability distribution, calculate values at the 2.5th and 97.5th percentile.
Voila. 95% confidence interval.
Voila. 95% confidence interval.
Extensions to aggregate malarkey and comparisons
The method is essentially the same when estimating the uncertainty in aggregate malarkey, and in comparisons between two individuals or groups.