TL;DR KEY POINTS
THE BACKGROUND My life is great in a lot of ways: I’m lucky to have a wonderful family and many projects and activities which I enjoy, for example. Thankfully, then, I'm very happy with my life overall. But despite that, one thing about my life can at times be very difficult and frustrating: being a judgment and decision making scholar. A judgment and decision making scholar—or a “JDM scholar” for short—is someone who professionally studies judgment and decision making: that is, someone who studies how we do, or how we should, make judgments and decisions. I’m a JDM scholar because I want to improve judgments and decisions, both the ones from myself and those from others in their domains. And there are many domains where judgment and decision making could be improved: a JDM scholar could reduce false death sentence convictions in law, fatal misdiagnoses in medicine or disastrous policies in politics, to take a few of countless examples. But despite both the possibility and promise of improving judgment and decision making in these domains, there are many reasons why this can be difficult or impossible for JDM scholars. I will explore some of them here, as well as why I think they sometimes stem from assumptions that are specious—that is, superficially plausible but actually wrong. (A caveat, though: while this blogpost talks of "people", it is not written with any specific "people" in mind--unless otherwise stated.) REASON #1: PEOPLE MAY THINK YOU ARE USELESS JDM scholars are often generalists: they study how to improve judgment and decision making in ways that are "domain general"—that is, applicable across many domains rather than just one. However, judgment and decision making, as it happens in practice, always happens within a specific domain, whether it is medicine, law, politics or something else. There is simply no such thing as “applied judgment and decision making” without it “applying” to "some" specific domain. But because of this, any time a JDM scholar aims to improve JDM in a given domain, those within the domain could criticize the scholar as lacking any specialization in that domain. They might say, “She’s not a doctor”, or “a lawyer”, or “a political analyst”, for example. Concomitant with this is often the claim that the JDM scholar lacks “experience” in the relevant domain, where experience is typically operationalized in terms of the number of years working in the given domain. This, then, is one reason why being a JDM scholar is difficult. Sometimes this reason is right; it's known meteorologists are often accurate about their domain, and it's not obvious to me that JDM scholars generally have much to contribute there. But sometimes this reason is specious. First, the available evidence suggests years of experience is a weak predictor of judgment accuracy in some domains; for example, Philip Tetlock found that years of experience in geopolitical forecasting didn’t predict accuracy at all, while another study found it had only a mild influence. The problem with some (but not all) domains is that—as I argue elsewhere--they do not rigorously track judgment or decision quality, nor what correlates with it. It is analogous to how pre-modern physicians practiced bloodletting for millennia without rigorously studying whether it worked. Only with the development of modern medical science did we realize what truly worked and what didn’t. The same point holds with judgment and decision making, and that brings me to a second point: the same literature shows that other things aside from domain experience or specialization matter more. For example, the literature suggests far stronger predictors of accuracy in domains like geopolitics are past records of accuracy, use of reference classes and other domain general features that are studied by JDM scholars. So although the JDM scholar often lacks the specialization or personal experience for a specific domain, what they do have is insight into the decades of rigorous scientific research about how to best make judgments and decisions, and sometimes in the specific domains too. Consequently, while it is sometimes correct to claim that a JDM scholar is useless simply because they are unspecialized or inexperienced, as with meteorology, there are other cases where this claim is like a medieval bloodletting doctor claiming that a modern medical student is useless because the doctor has years of experience and specialization in practices whose efficacy has not been rigorously studied. Unfortunately, however, it gets worse than that for another reason… REASON #2: PEOPLE MAY THINK YOU ARE WRONG Not only will people think you are useless, but people will think you are wrong. Of course, sometimes the reason for this is that JDM scholars are wrong, and as JDM scholars, we have to be genuinely open to letting others correct our views through open-minded debate. However, a lot of the time people think you are wrong for another reason: namely, that humans are programmed to think in ways which are intuitively compelling but demonstrably unreliable. This comes up in countless ways—in availability biases, in representativeness biases and in many others. In my own work, it comes up especially strongly for the so-called Monty Hall problem—a brain-teaser where my experiments found that every untrained participant got the wrong answer. And when I taught logic at Stanford, I clearly remember one of my own students insisting--so strongly—on why I was wrong about the correct answer to the very problem which I specialized in! She, however, was one of my better critics: although she thought I was wrong, she was at least willing to engage and argue with me about why that is so. Much of the time, people simply assume you’re wrong from a distance, foreclose any possibility of questioning or challenging that assumption, and then they move on. This is problematic, since sometimes seeing the correct answer or reasoning in a given context requires the time, patience and open-mindedness to debate or undertake some training—and even then it might not work. For example, only a third of the trained participants in my experiments got the right answers to the Monty Hall problem (and some of them probably cheated even then!). Note that this differs from other specializations. For cryptologists or engineers, for instance, it’s obvious to the non-specialist that they do not know what the right answer is; the average person would have no idea how to decode a given string of symbols or how to engineer a rocket ship. It’s different in JDM research because the non-specialist is psychologically programmed to think in ways where they believe they already know what the right judgment or decision is; as David Mandel puts it, they already have an intuitive theory of how to make judgments and decisions. This is not to say those theories are never right; sometimes they are and can contribute beneficial insights, but there are also many ways in which they can potentially be wrong as well. Unfortunately, though, all the ways in which these theories could be wrong are not obvious: if they were, then JDM scholars wouldn’t need to study them for years or decades at a time. But unfortunately, it gets even worse than that for another reason… REASON #3: PEOPLE HATE MAY HATE BECAUSE YOU THINK THEY'RE WRONG Not only will some people think you are useless and wrong, but they will sometimes hate you because you think they are wrong. For example, if you are a good JDM scholar, then you have to think uninitiated people are wrong when they first encounter the Monty Hall problem, no matter how confident, competent or qualified they themselves think they are. And of course, many of them would dislike you for thinking they are wrong—both because it insults them and because you appear arrogant. They might also be convinced your qualifications contribute only arrogance to your outlook instead of merit to your views. And unfortunately, this same phenomenon can come up in many other domains, because the patterns of thinking that lead to incorrect or correct reasoning in the Monty Hall and other problems are also potentially present in countless other domains, including medicine, law and many more (as I discuss here). Yet not only would they hate you for thinking they are wrong, but they might feel that if you are right about their wrongness, then their job security, self-perceived performance or other things might be at risk—in which case they could hate you even more. What’s more, we could see how the hate would be only worse if you think there may be horrible consequences because of their wrongness—like how innocent people die because of false diagnoses or false death sentence convictions. SUMMARY AND SOLUTIONS? To summarize, then, being a JDM scholar is hard for various reasons: people may think you’re useless because you’re “unspecialized” or “inexperienced” in their domain, because they are psychologically programmed to think you are wrong and because you think they are wrong in ways that can make them hate you.
Consequently, being a JDM scholar might entail that if you try to help others by explaining how their judgments and decisions could be improved in their domain, then they might just think you’re a "useless, inexperienced, incorrect and arrogant [insert your preferred personal swearword here]"—far from an outcome that benefits anyone. That, then, brings me to the final part of this blog: what are the solutions? So far, my answer is mainly, “Yeah, I don’t know”. That said, I surmise two things may help when trying to improve judgment and decision making in a given domain. The first is finding people in that domain with three things: 1) the time, 2) the patience and 3) the open-minded humility to explore how JDM science can benefit them. We might call such people the “champions” of those domains, like how Brian Nosek uses the term to refer to the advocates of “open science” in specific disciplines. (In fact, that more I think about it, the more I realize all my JDM complaints might apply to metascience scholars too.) Unfortunately, however, many people lack at least one of the qualities to be champions of JDM science in their domain, and sometimes through no fault of their own—often we are too busy to take on extra work, for example. Whether there are JDM champions in a given domain is not something JDM scholars can easily control, however. The second thing that JDM scholars can do--and something that they can control--is how to make their communications as tactful and accessible as possible. Typically such tact may often require passing two tests. The first is the "no criticism" test: for example, no one should read your work and feel criticized by it. Sometimes such criticism is appropriate—like in a typical philosophy department—but sometimes it only counter-productively alienates people. The second test is the "positivity" test: people should read your work and feel positive afterwards. And additionally, JDM work should often pass an "accessibility" test: a non-specialist should be able to read your work and understand both what you are saying and why it might add value to their domain. However, these things are not easy, and I myself am guilty of violating both of these requirements at times. But as JDM scholars, perhaps aspiring to these standards might make life easier for us, as well as improving our likelihood of benefitting those we aspire to help.
0 Comments
Leave a Reply. |
AuthorJohn Wilcox Archives
December 2024
Categories
All
|