Tips on How To Read a Study – and Why You Should Care
Posted Aug 17 2009 11:17pm
Every day you read about or hear about a new study that could dramatically change the way we treat a disease, or live our lives. But much of the time this week’s studies seem to contradict last week’s studies. One day coffee is good for you, the next it is bad. So how do you make sense of all this?
Well, first of all don’t worry, a lot of medical experts are trying to find the answer to that question too. In 2004 Dr. John Ioannidis published a study in the Journal of the American Medical Association. Dr. Ioannidis looked at 45 large studies published in well known medical journals between 1990 and 2003. He found that in approximately one third of the cases, the findings of one study were later completely contradicted, or seriously called into question by later studies.
So if the researchers themselves can’t make their minds up, what are the rest of us to do.
The best place to start is by asking yourself the following questions of every piece of medical research you see.
1) Is this relevant to me? If the study is about a problem you don’t have, a disease or condition you are not being treated for then move on. There’s no point cluttering your head up with information about something you don’t have, particularly as that information might not be correct anyway.
2) Did this study look at people or poultry? If the study was done in mice, rats, fruit flies etc, and you are not also a mouse, rat or fruit fly then move on. It’s not relevant to you. In time they may do the same studies in people. But that could be years away and even then there’s no guarantee that they will find the same results. Why waste time worrying about something that doesn’t concern you when you could just as easily waste that time worrying about who to vote for on American Idol.
3) How many people are involved? When it comes to medical studies, size really does matter, and bigger is almost always better. Unless large numbers of people were involved don’t waste your time on it. Looking at small numbers really only tells you if the drug/treatment etc is safe for those people, at that time, under those conditions. Wait till they test it on thousands of individuals before you start to pay attention.
Also look at the kinds of people the research was done on. Some studies are only carried out in women, some in men. Some focus on people over 65, others on people under 45. Some studies look at particular racial or ethnic groups, such as Alzheimer’s in Japanese Americans living in Hawaii, or breast cancer in Ashkenazi Jewish women. So, a basic rule of thumb is that the less the people in the study look like you, the more reason to look at it with a questioning eye.
4) How was the study done? The gold-standard for medical research is called a double-blind, placebo controlled study. That basically means that neither the patients nor the doctors/researchers know who is getting the real treatment, and who is getting the placebo. That way neither can interfere with the results.
Other methods take a different approach, often because it’s the only practical way to get the information they are looking for. Observational studies for instance compare people who take, for example, vitamin C supplements to those who don’t and see which group gets more heart disease. These studies have to try and exclude all the other factors that could influence their findings such as lifestyle habits, diet, environment etc. In the end they may be able to suggest a link between people who take vitamin C supplements and reduced risk of heart disease, but they can’t prove it. Next year another study may come along and show just the opposite.
5) If the study was a clinical trial, what phase was it? Phase I and II clinical trials are carried out in small groups of people, to determine if the treatment is first safe, and then if there is any sign it actually works. It is only in Phase III trials that a drug is tested in large enough numbers, and compared either against another treatment or a placebo, to determine if it is truly effective.
6) Has the study been published? The best science is usually reported in respected medical journals. If you hear about a new drug or treatment through research presented at a medical meeting, don’t rush out to buy it or take it. Wait until that research has been published. That means the study has been peer-reviewed, a process that usually weeds out weak or badly done scientific work.
7) Who funded and promoted the study? Remember, health care is a multi-billion dollar industry. There is a lot at stake. A study that is favorable to a new drug or treatment can boost sales for that medication, and the stock price of the company that makes it. So check to see who is behind the study.
Remember the diet drug known as phen-fen? Many newspapers and magazines, including the New York Times, published favorable reports on the safety and effectiveness of phen-fen based on a 1999 study and accompanying editorial published in the Journal of the American College of Cardiology. It was later learned that both were written by a paid consultant for Wyeth, a company that made one half of the phen-fen combination.
How does this fit in to previous research? A single study rarely offers definitive evidence of the effectiveness of a new treatment. Look at previous research to see if these latest findings match those, or how they differ. Never change your habits based on one study.
9) What is the risk? Even if the research is good, the findings valid, you have to ask yourself what difference does it make? To do this you have to consider two kinds of risk, relative risk and absolute risk. For example, if a study says eating pork increases your risk of cancer of the eyeball by 43 percent, that sounds like a lot, but it is a relative risk. You also have to look at how many people in the study got cancer of the eyeball. If there were 200,000 people in the study and only 500 developed that disease, then the relative risk is 500 in 200,000 or 0.25 percent, roughly 1 in 400. It’s a much smaller number. However, reporters will often give the larger number because it’s more dramatic, or because they just don’t both to do the math.
10) Is the study being hailed as a medical miracle or breakthrough treatment? Be very skeptical of any treatment that promises miracles, cures or breakthroughs. Much as we would like it to be otherwise, science tends to work very slowly. Each study builds on the last one, creating a body of work that eventually, hopefully, tells us what works. Studies that claim to have all the answers, or research that claims that one drug can cure all cancers are usually as reliable as the supermarket tabloids that report on aliens and Elvis being seen in the White House.