This article is about evidence based medicine and how a worthy concept has flaws, shortcomings, and is occasionally misused. The impetus for this article came from reading how Dr. Oz has been criticized for turning his back on evidence based medicine by a group of physicians who demand that Dr. Oz be removed from faculty at Columbia University.
We wrote in this post nearly a year ago that Dr. Oz was being criticized by members of Congress for promoting weight loss supplements. Now a group of physicians is saying that Dr. Oz displays an “egregious lack of integrity” for promoting “quack treatments”. He is being called a “fake” and a “charlatan”.
Interestingly, at least in the news accounts I have read, those physicians criticizing Dr. Oz do not point to or cite any evidence to support their claims about Dr. Oz. In that sense, they are guilty of the very “crime” they are saying Dr. Oz is committing.
This article isn’t to defend or criticize Dr. Oz as I have only seen a few clips of his television show. The purpose of this article, though, is to use this news event to talk about evidence based medicine – what it is and what it is not. We encourage you to read this editorial on Evidence Based Medicine written by Dr. David Sackett and published in the British Medical Journal (BMJ). Dr. Sackett is a Canadian physician and one of pioneers of the evidence based medicine concept. His editorial is called, Evidence Based Medicine: What it is and it isn’t. Health care professional especially should read his editorial. Dr. Sackett provides a humane and thoughtful model of evidence based medicine that does not rely solely on medical studies. His model of evidence based medicine is different from the one that is unfolding before our eyes in the United States.
Let’s start this discussion with this quote from Marcia Angell, MD. Dr. Angell is former editor-in-chief of The New England Journal of Medicine and author of the book, The Truth About Drug Companies: How They Deceive Us and What to Do About It.
… Similar conflicts of interest and biases exist in virtually every field of medicine, particularly those that rely heavily on drugs or devices. It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgement of trusted physicians or authoritative guidelines. I take no pleasure in this conclusion, which I have reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.
Her words give us reason to pause when we talk about medical evidence. Here’s another quote from Dr. Angell.
… Over the past two decades the pharmaceutical industry has moved very far from its original high purpose of discovering and producing useful new drugs. Now primarily a marketing machine to sell drugs with dubious benefit, this industry uses its wealth and power to co-opt every institution that might stand in its way, including the US Congress, the FDA, academic medical centers, and the medical profession itself.
I tend to agree with Dr. Angell. Medicine has become a tainted and to some degree a corrupt profession. It is really hard to know how much of published medical studies should be believed. Mainstream medicine has a problem. It has been hijacked by businessmen and bureaucrats. But, its problems go beyond that.
Problems of Mainstream Medicine
Here’s a problem mainstream medicine currently confronts and the reason why Dr. Oz is perhaps being attacked. More than 30% of individuals now go outside mainstream medicine for at least some of their health care needs. More than THIRTY PERCENT and that number grows each year. That’s an National Institute of Health number. Individuals are walking away from mainstream medicine. Why?
I think it in part has to do with the fact that mainstream medicine largely offers two main forms of treatment that most people would prefer not to receive – surgery and medication. Would you? To some degree physicians have been conditioned to think that every sign and symptom requires a pharmaceutical solution and patients sense that. “If I go to the doctor he is going to just be me on a medication.” They typically don’t mind being on a medicine for an acute problem but many balk at the idea of being on something chronically. So patients are going elsewhere.
The pharmaceutical companies have infiltrated medical education at all levels. The industry funds medical research and education. It funds much of the FDA unknown to most. Yes, it funds the very federal agency that is suppose to regulate it. Conflict of interest? You bet. How much should you trust the FDA when it approves a drug? It’s hard to know. The industry is also one of the biggest, if not the biggest, contributor to political campaigns. Its way of protecting its growing interests.
Does Medicine Cure Anything?
How many cures do we ever really see?
More and more people are failing to buy into the hype that mainstream medicine is winning the battles against heart disease and cancer when all they see around them are a proliferation of heart and cancer hospitals. If we are winning the battle on these diseases why are we building more and more such hospitals?
Yet, it is well documented but no so well-known, that certain cultures across the globe rarely get heart disease and cancer. Why? It is nutrition related. We already know how to prevent heart disease and most cancers. But, there is no money in that. Our treatments for heart disease are not as effective as touted and chemotherapy rarely cures cancer, but both generate a lot of revenue and creates jobs. Imagine if we cured heart disease and cancer. Much of medicine no longer has a purpose. Much of the pharmaceutical industry no longer has a purpose, either.
Medicine really rarely cures anything. Perhaps it really does not want to. Perhaps it simply wants to manage disease as expensively as possible for as long as possible. Dr. Oz has brought some light to this and seems to support alternative less expensive treatments, and he is now being attacked. Can mainstream medicine allow a highly reputable physician suggest alternative therapies, after all?
Evidence Based Medicine: Sounds So Pure
Now back to evidence based medicine. Evidence based medicine is based on the premise that the evidence is reliable, valid, and trustworthy, and that the evidence is used appropriately. But, is it?
The term evidence based medicine seems so obvious, so practical, so pure, so logical, so perfect, so much so how can anyone argue against its use. But, evidence based medicine has some flaws and shortcomings. Dr. Sackett recognizes these concerns and tried to address them in his BMJ editorial. Traditionally, medical decision-making involved the consideration of medical evidence, physician experience/judgement, and patient desire – a model Dr. Sackett supports. We are now trying to gravitate to relying solely on medical evidence.
Medicine as an Art
Medicine is an art and a science. In fact, not too long ago the office building that housed physician practices in cities all across the country was frequently called the Professional Arts Building or the Medical Arts Building – not the Professional Science Building. Interesting, don’t you think?
In fact, any profession that deals with people is an art. Teaching is an art. Practicing law is an art. And, medicine is an art no matter how much of a pure science we try to make it. Take into consideration that in drug trials evaluating the use of antidepressants that over 50% of the subjects improve with a placebo. So it makes you wonder how valid it is applying the scientific method to an entity (human beings) that responds unpredictably to what science would suggest.
Therefore, is it wise to take solely a strict scientific approach to the practice of medicine? This strict adherence to scientific studies is not what evidence based medicine is about according to Dr. Sackett, but that is what it is becoming in the United States. We have adopted a perverted version of evidence based medicine (different from the one Dr. Sackett touts) that wants to rely solely on the medical evidence and weigh less physician experience and patient desire in the decision-making process.
Flaws of Evidence Based Medicine
Why is relying on medical evidence a problem you might think?
Study Bias, Statistical Manipulation, and Conflict of Interest
Number one, any study can be designed, and the results can be reported in such a way to support the researchers’ biases. How much should you trust the results of a drug study funded by a pharmaceutical company, or a study on a nutritional supplement funded by a supplement company? A never before used statistical analysis of a recent study suggested that testosterone increases the risk of heart disease even though the raw data showed that men on testosterone actually had half the rate of heart disease. This was a classic example of statistics being manipulated to make the point the researchers probably intended from the outset.
Strength of Medical Evidence
There are different types and strengths of medical evidence. Meta-analyses and double-blind placebo controlled studies provide the strongest form of evidence, while expert opinion and case studies provide the lowest strength of evidence. In between are several other layers of evidence.
But some times expert opinion and case studies represent the best evidence currently available. Not all medical problems lend themself to double-blind controlled placebo studies. For instance, it would take 30 years or more to do a really good double-blind controlled placebo study on testosterone, and would be terribly expensive. Are we going to wait that long to make decisions on testosterone therapy when we have numerous lower forms of evidence that supports its use? But, medicine in the US more or less has taken a position that we should not do anything until we have results from a double-blind placebo controlled study.
In the meantime people suffer from disease as we wait for what is felt to be “convincing and overwhelming” evidence. Dr. Sackett says physicians should use the best available evidence and ask if that evidence applies to the patient in question, if not physicians should work their way down to the next lower best available evidence, and so on.
Science Cannot Answer Everything
Science cannot answer everything. At times you have to employ common sense. Piaget said, intelligence is what we use we don’t already know what to do. I think much the same can be said about common sense. Common sense is what we use when we don’t have all the information we need. In other words, there will always be holes in our scientific knowledge and we use common sense to fill in those gaps. What else can we do? Intelligence and common sense enable us to manage uncertainty when we don’t know what to do because we don’t have all the information we want or need (findings of double-blind placebo controlled study, for instance).
Real Patients Differ From Those in Studies
Evidence based medicine naturally evolves into guidelines which are guidelines (not mandates) designed to cover the average patient. But, in reality few people represent the average patient. Many patients have something unique about their circumstances that pigeon-holing them into some category or guideline becomes challenging. In other words, many patients are outliers and the answer to their situation is not always conveniently addressed in a scientific study. It is not always possible to take a population-based study and apply the results of the study to the individual patient sitting across from the physician in the office.
If we truly, truly want to be scientific we should only apply the results of a study when all the conditions that were met in the study can also be met and applied to the individual patient. That rarely happens. Studies are performed under “ideal or sterile conditions”. Real life is not so ideal – not so perfect.
Just like much of education teaches to the mean, much of medicine treats to the mean. Practice guidelines based on medical evidence typically are designed to get patients to average risk. Some of my patient don’t want to be at average risk. They want to be at optimal risk. In other words, some practice guidelines are too soft to be effective. Walking 30 minutes most days out of the week is better than do nothing, but it’s not going to get you healthy. Sorry! But, that’s what we recommend because of “evidence”.
Average risk is fine if the disease in question is rare to begin with. But, do you really want to be at average risk for heart disease, the number one killer for the past couple decades despite all the “advancements” in its treatments? Let that sink in. Despite all the advancements in procedures, diagnotics, and drug therapy, heart disease remains the number one killer and the deaths related to it increase each year. That’s why we see a proliferation of heart institutes – because we are losing the battle.
Medical Evidence is Frequently Conflicting
Medical evidence is often conflicting. Now the main premise behind the scientific method is that it yields results that are reproducible. Well, if that is true then why do have numerous studies trying to answer the same questions on the same problems over and over again? Doing so seems to be self-admission by the medical profession that studies sometimes are lacking and do not adequately provide answers. Why is it that in even years there is a link between caffeine consumption and breast cancer? And, in odd years there is no link between the two? Why the conflict if science provides the ultimate answer? Which study should you believe?
Here are a few reasons for conflicting studies but there are more. First, conducting a study presupposes that enough knowledge is already known to properly design the study to answer a question. But, that is not always true. In other words, we are not always smart enough to know what we don’t know, or know enough to know what needs to still be answered. The questions trying to be answered in research studies are not always relevant or get to the very essence or core of a problem. Secondly, most studies look at specific demographics and the results may not apply to other demographics. Are we being scientific when we extrapolate the results of a hormone study or heart study on 60-year-old women and apply them to the 50-year-old female? Similar studies on 50-year-old women might yield different results.
Or, studies only assess a limited number or risk factors. Not all risk factors of a disease are known. There are at least 17 risk factors for heart disease, and most certainly others that remain unknown, but only a handful are ever assessed in most studies. That affects the results, though well-controlled randomized studies are supposed to inherently mitigate against those uncertain factors. But, do they always? Plus, there is misinterpretation of results, or cases where other explanations to explain the findings are not considered by the researchers.
For instance, a recent retrospective study concluded that fish oil increases the risk of prostate cancer. The data from the study were actually used from another study looking at a different problem, but since they had data on fish oil and prostate cancer it was decided to draw some conclusions on fish oil and prostate cancer. So the study was not specifically designed to draw an association between fish oil and prostate cancer.
First, fish oil consumption in the study was based on patient self-reports so the exact amount of fish oil consumed is debatable. Secondly, the source of the fish oil was unknown. Most people buy cheap fish oil which is more likely to be contaminated with mercury and other toxins. Perhaps it was one of these toxins that was responsible for an increased risk in prostate cancer, not the fish oil itself. But, the researchers never raised that possibility. But, guess what? Patients have stopped taking fish oil supplements. But, eating fish apparently is still ok.
The publish or perish mind-set of academia allows for bad research as bad research is preferable to no research. Unfortunately, bad studies do get published at which point they then become “gospel truth”. Journals are under pressure to publish studies. Medical journals do not remain viable unless they have a critical mass of studies to publish – each and every month. So bad studies do get published. Also, medical journals are rarely supported entirely by subscription fees – even peer-reviewed journals. They are partly funded by pharmaceutical companies and medical device companies through paid advertisements raising the possibility of conflicts of interest. How likely is a journal to publish a negative study on Crestor if the journal is getting ad revenue from its manufacturer?
Physician Experience and Patient Desire Still Matter
Finally, what about physician clinical judgment and patient desire? Do they not factor into the medical decision-making process any longer? Dr. Sackett says in his editorial says evidence based medicine was not intended to ignore a physician’s judgement, or a patient’s desire. Yet, we see that happening more and more in this country where a patient’s desire for a treatment or diagnostic study is becoming increasing ignored. If a patient would like to try a treatment for which there is only scant medical evidence should that treatment be denied, or at the least not even be given some consideration?
Now the issue of who should pay for such treatment and testing is a different discussion. And, I do think insurance companies (or any entity paying the bills) have the right to decide what and what they will not pay for. Which takes us to the next point. As a patient you can get nearly anything you want done in medicine – if you are willing to pay for it yourself. Remember that. We’ve created this “someone else must pay for something” mindset in this country and it is not healthy. Your health is your responsibility.
And, in the end much of this evidence based medicine concept is about controlling costs though it is cloaked as a superior form of medical decision-making. It gives insurance companies “cover” and reasons to deny tests and treatments.
In his editorial Dr. Sackett say this, “Some fear that evidence based medicine will be hijacked by purchasers and managers to cut the costs of health care. This would not only be a misuse of evidence based medicine but suggests a fundamental misunderstanding of its financial consequences.” That was written in 1996. Unfortunately, we are seeing the misuse of evidence based medicine in this regard.
Back to patient desire.
I’ll give you one example at how patient desire is being ignored. Not too long ago a 35-year-old male came to me with several classic symptoms of low testosterone without any other possible cause to explain his symptoms. He saw an endocrinologist who refused to treat him because his total testosterone was 325 with the reference range being 300 to 1,000. A testosterone level of 325 is something you expect in a 70-year-old, not in someone half that age.
In the eyes of the endocrinologist, he therefore, had a “normal testosterone” level. The endocrinologist probably felt he was practicing evidence based medicine because the lab tests were “normal”. Yet, interesting the reference range is not based on any science or evidence. So how can a physician possibly take an evidence based approach to this topic when the reference range has no science behind it?
The reference range was simply arbitrarily established using the two standard deviation statistical model method which means only 2.5% of men can ever be testosterone deficient based on lab values. That is why the reference range for testosterone and most labs are so broad. The reference ranges simply tell you where your results fall relative to others. The ranges do not, however in most cases, describe what a healthy or desirable level should be. Many doctors are not aware that most lab ranges are statistically based – not evidence based. Imagine if we only treated the 2.5% of people with the worse vision.
The patient asked the endocrinologist, “but what about my symptoms?” The endocrinologist responded, “you just have to accept that’s the way it is going to be for you and live with those symptoms.” Kind of a rotten thing to say to a patient seeking help, don’t you think? The patient desired to be treated and was willing to accept any risks related to treatment. Plus, there is a treatment available (testosterone therapy) that has been around for 70 years and for which the side effects are well-established unlike many new drugs being prescribed like candy these days including the many new diabetic medications this endocrinologist now prescribes. What harm would have there been if the endocrinologist decided to treat him for low testosterone and monitored him appropriately? Really none.
These are some of the potential flaws and shortcomings of evidence based medicine.
Physicians Versus Alternative Care Providers
Years ago a colleague of mine and a leading sports medicine and knee specialist was asked by an athletic trainer for his opinion on chiropractors who are frequently criticized by mainstream physicians and even called “quacks” by some.
He said this, “Well, it (chiropractic) must work well enough in enough patients that patients keep going back to them. It is foolish for those of us who are physicians to think we have all the answers to medical problems. We don’t. Some patients I operate on do not get better. In fact, I make some of them worse.”
There are not too many physicians with that point of view and who are willing to be so honest. In the end all that really matters is do patients get better or not regardless of how one skins the cat?
We should all take a look at the history of the healing arts. Allopathic physicians (MDs) were among the last to jump on the scene. Long before there were MDs there were acupuncturists and naturopaths and herbalists that rely less on pure science and instead honed their skills and adapted their practices through a process called trial and error. Is trial and error any more or less a valid approach to healing and finding answers than the so-called scientific approach?
Perhaps when we look at the healing arts in historical terms we find that MDs really represent “alternative medicine” and everyone before them represent “traditional medicine”. Maybe we have it backwards on the topic of “alternative” and “traditional” when it comes to medicine. Just more food for thought.
One last quote from Dr. Angell relating to conventional and alternative medicine.
It is time for the scientific community to stop giving alternative medicine a free ride. There cannot be two kinds of medicine – conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work. Once a treatment has been tested rigorously, it no longer matters whether it was considered alternative at the outset. If it is found to be reasonably safe and effective, it will be accepted.
That statement makes sense, though I do not think she is aware that some alternative therapies have already been tested. The problem is those studies are not typically published in journals that physicians read. So they go unnoticed by the “medical community”. Such studies are published in the chiropractic, acupuncture, nutritional journals and so on, and are no more biased than those published in medical journals.
There’s plenty of research on nutritional supplements including herbs. In fact,the same publisher of the PDR (Physicians’ Desk Reference), which is the “bible” physicians use to look up drug information, also publishes the PDR for Nutritional Supplements and the PDR for Herbal Medicines. Both books contain lists of references and grade the quality of evidence for each supplement and herb discussed. Many physicians do not know that these two sources exist.
Certainly, medical evidence should be considered in the decision-making process, but it is only one tool to be utilized in that process. And, certainly alternative therapies need to be tested.
Dr. Sackett discusses the value of evidence based medicine, but the model only works if the evidence can be trusted, and if the evidence is used appropriately. Dr. Angell indicates that the evidence cannot be trusted, and we now demostrate why it may not be trustworthy.
Evidence Based Medicine: is it Cookbook Medicine?
From where I sit the evidence based medicine that is being implemented is different from the one Dr. Sackett describes and it appears to be used in attempt to make the practice of medicine rather cookbook – something Dr. Sackett suggests against – a cookbook formula that doesn’t require physicians to think, but rather simply follow orders.
On the surface evidence based medicine seems like the perfect solution to help guide the management of patients. But, it’s clinical usefulness is only as good as the veracity of the evidence. Plus, it risks taking the decision-making process out of the hands of patients and their physicians and puts it into the hands of bureaucrats and academicians neither of whom treat real patients, nor bear the responsibility that goes along with that privilege.