In 2007, Eliezer Yudkowsky wrote an interesting article advocating for what he called “defying the data.” The idea was fairly simple – say you have some theory explaining how the world works. A new study is published with data that can’t be accounted for with your theoretical framework. How should you respond?
One response is to abandon your theory in favor of the new data. Another response is to keep your theory intact and, as Yudkowsky says, “attack the experiment – accuse the researchers of dishonesty, or flawed design, or conflict of interest.” But there is a third possibility – that of simply defying the data. As Yudkowsky put it,
Nonetheless I said it outright, without apology, and with deliberate insolence. I am keeping my theory; your experiment is wrong.
If an experimental result contradicts the Standard Model, this is an important fact. It needs to be openly acknowledged. An experiment that makes traditionalists want to discard the data – or even an experiment that makes traditionalists very skeptical of the data – should be a high priority for replication. An experiment worth defying should command attention!
But it is not socially acceptable to say, “The hell with your experimental falsification, I’m keeping my theory.” So the data has to be defied covertly – by character assassination of the researchers, by sly innuendos, by dire hints of controversy. The data has to be dismissed, excused away, swept under a rug, silently into the dark, because you can’t admit you’re defying the data. This is not a good way of focusing attention on an anomalous result. This is not a good way to ensure funding for replication attempts.
This makes a great deal of sense. If a theory has been well-established and upheld by multiple studies and experiments, then one really striking appearance of contrary data shouldn’t amount to much. Experiments and studies, even meticulously crafted ones, can go wrong for all manner of reasons. In 2011, some scientists seemingly found data showing neutrinos moving faster than light – a finding that contradicts well established physical theories. In this case, the perfectly sensible response is to defy the data. After all, what’s more likely? That all of established physics has been completely upended – or that the researchers at CERN had a slight measurement error? (Spoiler alert – it turned out to be the latter.)
Now admittedly, my own defiance of data in this blog post will not have anything like the precision that can be found in physics experiments, either in the measures used or even the definition of the underlying phenomenon. Still, there was a specific moment years ago that instantly provoked in me a response of, “I defy the data” – and it relates to what standard of living data is purported to show.
At the time, I was in college, and very much in a financially strapped period of my life. I split a small apartment with two other people, working on my economics degree while also working full time hours at Barnes and Noble to try to limp by financially. Around that time, I had heard a claim, often repeated in various forms, along the lines of, “young people in this generation actually have a lower standard of living than their parents did at the same age!” And I immediately, without hesitation, decided to defy that data.
By a fine coincidence, only a short time earlier, my parents had mailed me a variety of keepsakes from my youth. One of them was a DVD onto which they had transferred the contents of a VHS tape made years earlier, when I was three and a half years old. This would have placed the footage at around 1987. My dad had recorded this video to send to his own mother, so she could see family movies of how we were getting on. One of the things that struck me most watching that video when it was sent to me was how I was now almost the same age as my mother was when it was recorded. This video gave me a glimpse into what my parent’s life looked like when they were the same age I was at that time.
How did our lives compare? Was I experiencing a lower standard of living compared to them, at that time? Absolutely not! Now, I’m sure if you did a comparison of our income brackets I’d have looked like I was far down the ladder compared to them. Even though that video was recorded around 1987 and I was watching it in 2011, my nominal income was lower than theirs – and when you adjust for inflation, it would have been no contest. But not only was my standard of living not lower than theirs, you’d have had to pay me a considerable amount of money to even be willing to consider living at the standard of living they had, with the goods and services available to them in rural Oregon in 1987! (Actually, I’m not sure there is any amount of money you could pay me to make me willing to do so – because so much of what I enjoy about my living standards both in 2011 and now couldn’t have been made available to me for any amount of money in 1987.)
I could spend all day and night listing all the things available to me in 2011 that my parents in 1987 couldn’t have had in their wildest dreams, but I’ll just use one comparison for now. My dad was behind the camera for almost the entire video, but occasionally you could see him reflected in a mirror. That home movie was recorded using a heavy and cumbersome camera that had to be mounted over his shoulder and produced low resolution, low quality output. They didn’t even own the camera – they just rented it for a few days to make this video, because a camera like that was far too expensive for them to actually own outright. And as I watched that video, I had a smartphone in my pocket that weighed only a few ounces and could record vastly higher quality video that could be instantly transmitted anywhere in the world. Though that phone would be considered obsolete today, just the sight of it in 1987 would have had people debating on the need to reintroduce witch burning laws!
People can debate with each other until they’re blue in the face about how to best measure living standards. I don’t have a brilliant new measure to offer here. But I will say if your measure of living standards comes to the conclusion that I was at a lower standard of living in 2011 than my parents were in 1987, your measure is terrible and should be discarded. If that’s what your data say, then I defy the data.
