The term "paradigm shift" is bandied about far too often in books and articles about healthcare. Thomas Kuhn in his classic book "The Structure of Scientific Revolutions" coined the term to describe a change in the basic assumptions within the rules of science.
Kuhn and his paradigm shift came to mind recently after I finished two books that truly stunned me: "Automate This: How Algorithms Came to Rule Our World" by Christopher Steiner (2012) and "Big Data: A Revolution That Will Transform How We Live, Work, and Think" by Viktor Mayer-Schonberger and Kenneth Cukier (2013).
As a practicing pathologist and medical school professor, the paradigm I followed for years was to generate hypotheses that would affect treatment, prognosis and classification of diseases.
The gold standard for testing such hypotheses was the double blind clinical trial that used randomization to insure the only difference between the control group and the treated group was the therapy under investigation. One did not accept a treatment as truly evidence-based until it had survived this rigorous and expensive experiment.
These two books convincingly make the claim that big data and algorithms turn my beloved paradigm on its head. Rejecting the biomedical model that concentrates on finding the cause of disease and assuming the disease world can be truly understood by scientific experiments, this new approach goes after correlations, not causality.
Big data and algorithms believe diseases may never be truly understood, but such correlations can still provide us with actionable strategies to deal with complex diseases that often exhibit emergence.
As Mayer-Schonberger and Cukier put it, "Big data is about what, not why." Big data refers to studies one can do at a large scale that cannot be done at a smaller one, to extract new insights or create new forms of value, in ways that change the study of diseases and everything else.
Meanwhile, Steiner defines algorithms as a list of instructions that leads its user to a particular answer or output based on the information at hand, and bots as multiple linked algorithms all aimed at performing one task that can roam the Internet in search of data.
Both books contain numerous examples of how this paradigm shift will revolutionize medicine. For example, Google published an article in Nature comparing the 50 million most common search terms in America with CDC data on the spread of influenza between 2003 and 2008.
By examining 450 million different mathematical models, they compared their model of 45 search terms against actual CDC documented flu cases between 2007 and 2008. The Google model successfully predicted the spread of flu a full week earlier than the traditional CDC approach.
Steiner describes the University of California, San Francisco Medical Center's experience with a $15 million pharmacy robot that has "filled two million prescriptions without making a single mistake. And there's no human contact between the pills and their packaging, eliminating the chance of contamination."
When Steiner looked at the literature documenting human error in filling prescriptions, he found estimates of 37 million to 51.5 million errors annually.
I am convinced big data and algorithms will disrupt healthcare in ways that are only now becoming appreciated. There also are definite risks and unintended consequences associated with this new powerful approach. Our current methods of trying to ensure privacy will need to be revised, and big data has limitations.
David Brooks, in The New York Times, identified some of the things big data can't do: take into account people's irrational behavior, understand the context of the problem, and realize that more data sometimes means more errors and more spurious correlations.
A Frost and Sullivan white paper recently estimated that in 2011 only 10 percent of American hospitals used health data analytic tools. I predict the use of this powerful new approach will create winners and losers among hospitals. Anyone responsible for the viability of a hospital system would be wise to read these two books and develop a data analytics program as soon as possible.
Kent Bottles, M.D, is a Senior Fellow at the Thomas Jefferson University School of Population Health.