How can we measure the accuracy of judgments and determine which ones to trust?

(Forthcoming in Psychology Today)
​

​THE TL;DR KEY POINTS

• I recently published an argument for calibrationism, the idea that judgments about the world are trustworthy only if—among other things—there’s evidence that they are produced in ways that are “well calibrated”
• A set of judgments are well calibrated just in case they assign, say, 80% probabilities to things which are true 80% of the time, 90% probabilities to things which are true 90% of the time and so on
• In this post, I provide a tool for measuring calibration, and I share some ideas about what to do and what not to do when measuring calibration, using my own experience as an example 

---

PRELIMINARY CLARIFICATION: WHAT IS CALIBRATION AND CALIBRATIONISM?

​
​As I mentioned elsewhere, I recently published a paper arguing for calibrationism, the idea that judgments of probability are trustworthy only if there’s evidence they are produced in ways that are calibrated—that is, only if there is evidence that the things one assigns probabilities of, say, 90% to happen approximately 90% of the time.

Below is an example of such evidence; it is a graph which depicts the calibration of a forecaster from the Good Judgment project—user 3559:

​The graph shows how often the things they assign probabilities to turn out to be true. For example, the top right dot represents all the unique events which they assigned a probability of around 97.5% to before they did or didn’t occur: that Mozambique would experience an onset of insurgency between October 2013 and March 2014, that France would deliver a Mistral-class ship to a particular country before January 1st, 2015 and so on for 17 other events. Now, out of all of these 19 events which they assigned a probability of about 97%, it turns out that about 95% of those events occurred. Likewise, if you look at all the events this person assigned a probability of approximately 0%, it turns out that about 0% of those events occurred. 

However, not all people are like this, below is a particular individual, user 4566, who assigned probabilities of around 97% to things which were true merely 21% of time, such as Chad experiencing insurgency by March 2014 and so on.

Read More