By Jim Kobylecky
Evidence is hard. But repeatable evidence is even harder— to obtain at least. Every engineer (we’ll call her Ms. Murphy) is familiar, in fact if not in name, to the idea that you can’t step into the same river twice—that “same” river has already flowed past and you’re dealing with a new batch of river every time you try. Every manager is familiar to the phenomenon that he or she has to pin Ms. Murphy to a fact-based river forecast anyway—too much money is at stake to be cute about it. The investors won’t respond well to a genial shrug at the next board meeting.
Basing billion dollar decisions on the evidence of social science is even harder, more necessary, and more misleading. At a recent Linus Pauling Memorial Lecture (Mentor Graphics being a major sponsor and inviting me along), Dr. Nancy Cartwright of the London School of Economics and UC San Diego applauded the movement to establish evidence-based public policy, but exposed where blind faith that its tools, such as Randomized Controlled Trials (RCTs), provide necessary and sufficient proof (guaranteeing results) can lead us into expensive, demoralizing failures.
Think about Rube Goldberg devices (if you’re too young, then use the web already and check them out). Basically a definable action leads to a desirable result. But if you walk away and try the same definable action somewhere else and don’t carry the exact same “in-between” with you, results will more than vary.
In between the cause and effect of a Rube Goldberg device is an incredible, and funny, chain of bizarre yet plausible principles and events. If you simply cover up that messy part, you can think you have found Nirvana—as long as you never try to repeat it.
But whether it’s health care, class size, educational reforms, or helmet laws, that’s what we keep trying. We have to do something, and it’s too expensive to try the Wrong Thing.
(I can’t do Dr. Cartwright’s talk justice in this brief look. It was very challenging and occasionally had me whimpering under my seat. Terry’s Takes, , is an alternative explanation video blog by the director of the Institute for Science, Engineering and Public Policy, which organizes these lectures. Dr. Cartwright has written books and many articles. I’m told her first book is the most accessible.)
But back to Health Care, or Helmet Laws, or Educational Initiatives. We have to do something! Every other talk show host starts with “It’s really simple, people.” But Dr. Cartwright stressed doing our homework, and lots of it. RCT’s are a laudable attempt to provide “what’s” without resorting to “how’s” and “why’s.” But if we don’t examine how and why that particular Rube Goldberg action produced that result IN THAT CASE, we are still rushing blindly when we apply that action again.
It’s almost like we’re back to doing a “gut” feeling approach. Because we are. We always are. So own up to it, and integrate the evidence with the brain’s ability to make connections we aren’t terribly aware of. Go to the original site. Go to the target site. What’s different? What tastes the same? Get on the ground. Experience the environment. Take a bunch of people, poll for impressions and cautions.
Then go small. Bet on pilot studies. Expand the evidence base. See if there isn’t a huge chaos between the two cliffs just waiting for your blind leap. Build the bridge. Play the odds. Bet smart. Keep your mistakes manageable and learn from them.
Easy for me to say. It’s just up to you and your two buddies, Rube and Ms. Murphy to make happen. Remember, I’ll be right there behind you, second-guessing you every step of the way (grin, I hope).
Originally published on Chip Design’s Koby’s Kaos blog.