How would you want to evaluate the risks and effectiveness of a new drug that was developed by a pharmaceutical company at the cost of millions of dollars? I’d be suspicious of an observational study: even if conducted by professionals, there just seem to be too many ways for things to be biased.
Right. And it’s not just observational studies. The data from any big study can be analyzed many ways. The more at stake, the greater the chance of what Andrew calls bias and I call making choices that favor the result you prefer. Independently of Andrew, Hal Pashler and I came up with what I call the Pashler-Roberts Law: The more expensive the research, the less likely the researchers will be honest about it.
As Andrew says, this is a big virtue of self-experimentation. Because it’s free, it’s easy to be honest, especially about failure. The cheaper the better is a broad truth about science that’s hard to learn from books or classes or even talking to scientists.