The problem with questioning the morals of RCTs in a blanket way is the assumption that the thing being questioned is really great and life-saving to begin with. Thanks for using blood-letting as an example. In the control group for that experiment, are the people not getting their blood drained really having medical treatment withheld? Well, I guess they were for the standards of the time, but if it weren't for testing these things, we wouldn't find out that they don't work to begin with.
We tend to think that we have everything figured out, but within recent memory people / doctors were 100% certain that stomach ulcers were caused by stress and smoking. Boy were they surprised to find out that they were caused by bacteria or too much Advil - oops. And that's from the 1990s.
Great post – very thorough and well-researched. Thank you for writing this.
One thing I don't really agree with, though, is your conclusion that "why projects work is, fundamentally, a better question than whether they work."
Once we understand that something does work, sure, let's figure out why. However, many things—most things, in fact—actually don't work, including initiatives we may expect to work. You mention the example of microcredit: Isn't it very useful to know that in fact the impact of microcredit is very limited? Without RCTs, we'd probably have never known this.
I like your conclusions here and I feel better informed about the works of Duflo et al. which I read a while back. I always wish your conclusions were laced with a few more recommendations (possibly for further study?).
The problem with questioning the morals of RCTs in a blanket way is the assumption that the thing being questioned is really great and life-saving to begin with. Thanks for using blood-letting as an example. In the control group for that experiment, are the people not getting their blood drained really having medical treatment withheld? Well, I guess they were for the standards of the time, but if it weren't for testing these things, we wouldn't find out that they don't work to begin with.
We tend to think that we have everything figured out, but within recent memory people / doctors were 100% certain that stomach ulcers were caused by stress and smoking. Boy were they surprised to find out that they were caused by bacteria or too much Advil - oops. And that's from the 1990s.
Great post – very thorough and well-researched. Thank you for writing this.
One thing I don't really agree with, though, is your conclusion that "why projects work is, fundamentally, a better question than whether they work."
Once we understand that something does work, sure, let's figure out why. However, many things—most things, in fact—actually don't work, including initiatives we may expect to work. You mention the example of microcredit: Isn't it very useful to know that in fact the impact of microcredit is very limited? Without RCTs, we'd probably have never known this.
I discuss this in some more depth in a recent post of mine: https://inexactscience.substack.com/p/most-ideas-fail-and-thats-fine
I like your conclusions here and I feel better informed about the works of Duflo et al. which I read a while back. I always wish your conclusions were laced with a few more recommendations (possibly for further study?).