The tough criteria used by the AHRQ panel to assess diagnostic and biomarker studies ended up eliminating well over 90% of ME/CFS studies from consideration. That shocking result has lead some to suggest that the chronic fatigue syndrome field should be given something of a pass when it comes to reviews. They argue that the historical neglect of the ME/CFS field should be taken into account when review panels assess it.
Should reviewers relax their standards for chronic fatigue syndrome?
So now we’re asking you what you think. Should independent review panels relax their criteria for disorders like chronic fatigue syndrome?
Perhaps it goes deeper than that? Perhaps they shouldn’t assessing ME/CFS at all? Perhaps they’re too unschooled in the nuances of ME/CFS to do it justice? Perhaps they’re too biased?
Or are they just what the field needs? Do they provide the push it needs to get it on the right track?
You decide…(see Poll at the end of this blog)
The Independent Review Question
The upset over the idea that chronic fatigue syndrome research was going to be judged by non-experts was understandable. ME/CFS is a controversial field. There’s certainly concern that an independent review will turn back the clock and label ME/CFS a behavioral disorder. Until ME/CFS is put on a solid physiological basis that fear is probably not going to go away.
Can outside reviewers be trusted to understand ME/CFS enough?
It wasn’t surprising then, that when given a chance to undergo the kind of high level review that other disorders routinely engage in, many in the community balked. ME/CFS, they felt, could only be understood by ME/CFS experts. Letting outsiders in was like letting the fox into the henhouse.
A few, on the other hand, embraced the reviews. Dr. Ron Davis felt the chance to undergo a high level analysis was an opportunity not to be missed. The very fact that independent groups were going to be assessing ME/CFS to him meant we were getting somewhere. Dr. Bateman in what ended up being a rather lonely stance, also asserted that it was time for ME/CFS to stand on its own two feet.
Mistrust Reigns
But mistrust and fear drove many. Mistrust surely, for instance, drove the petition that requested the NIH not engage in an analysis, but simply declare that Canadian Consensus Criteria the definition for ME/CFS.
A feeling of invalidation may have played a role as well. ME/CFS experts, had after all, worked for years to put together the definition. It had garnered widespread support. Why didn’t the NIH just accept it? (Can’t we get any respect?)
The petition signers hoped that force of numbers would compel the NIH to do something it probably doesn’t do very often – accept consensus opinions in lieu of hard research. The NIH brought in a respected and presumably objective group to oversee the issue – a good idea so long as the group was trusted – but the trust was not there.
Distrust is understandable – but is it helping or hurting?
That mistrust squashed alternate interpretations. The IOM Workshop could have been viewed as a sign that the feds were finally getting serious about ME/CFS. Or as an opportunity to get the CCC declared the official definition. Or simply as an opportunity to get clear on what needed to be done to do that.
The P2P effort with no ME/CFS experts on board was even worse. Some advocates rejected the very basis of the P2P effort itself – the idea that it’s fruitful for non-experts in a field to give independent analyses of that field.
Mistrust of non-experts, in particular, seems almost endemic in our community. What do you think? Should independent reviewersbe allowed access to ME/CFS, and if they are, should ME/CFS be given a special break?
Arguments For Treating ME/CFS Differently Than Other Disorders
-
Should ME/CFS be treated differently from other disorders?
Because ME/CFS has been underfunded for years ME/CFS studies can’t be expected to meet the rigorous standards that better funded diseases do. The ME/CFS research field simply lacks the infrastructure – the money and academic environment – that fosters that. Reviewers should recognize that, give the community a break and use less rigorous standards to analyze it.
- Otherwise good research that doesn’t quite meet the standards ends up getting dismissed. This is particularly damaging with regards to the biomarker studies. The AHRQ could have identified promising areas like reduced aerobic capacity or dysfunctional natural killer cells. Then the P2P report could have recommended funding research into those areas. That’s probably not going to happen now. Somehow those areas should have been highlighted.
Arguments Against Giving ME/CFS a Break
- ME/CFS has to stand to stand on its feet and meet the most rigorous standards of medical research at some point. The sooner it can do that the better for everyone. Without efforts like these the field can be stuck in a rut for decades. In the end when studies don’t meet standards, the patients lose out the most. Even though they are painful these reviews are a good thing.
-
Or will reviews propel the field to the next level?
Outside review groups are a staple of medical analysis. Few researchers probably enjoy being put under the microscope by outsiders, but most probably acknowledge that independent analyses provide a foundation their fields can build on.
- It’s impossible to give any one disorder a break anyway. Once one disease gets a break others will demand one as well and the reason for having independent reviews will disappear. Because asking groups like the AHRQ to relax their standards is an impossible task, it shouldn’t even be attempted.
The Poll
What do you think?
This is really bad news. What they should be doing is releasing much more funding for research, along with guidelines. After a time, start enforcing the guidelines.
This is what gives the feds a bad name and leads people to confirmation biases. It is the “ivory tower” phenomenon.
My hope and my guess is that these projects were initiated to prepare the foundation for a big grant. The feds often do a background review before they do that. P2P was supposed to identify the gaps in research. Unfortunately they disallowed so many studies that it’s going to impossible to target any one area with regards to biomarkers. That’s the big loss of this study… All they can say is more funding is needed. With regards to the definitions they can do more.
Who knows who these reviewers are and what their credentials are?
Even the so-called experts haven’t really come up with anything other than “ME/CFS patients have these problems” but so far haven’t come up with a definitive biomarker or valid medication.
If most of the research isnt counted then the health system should look at that as a failure on their part due to lack of funds and put a snappy $100m into research to start the ball rolling, thats if they were serious and really wanted to understand this illness.
I agree!
Very true!
It was not until 20 minutes ago that I realized what a loss the biomarker aspect of the AHRQ draft was. P2P is supposed to identify gaps in the research – but they excluded so many studies that that’s really impossible. Instead of the AHRQ validating work on exercise studies or brain studies or autonomic nervous system dysfunction – basically telling the P2P panel that this work is good enough to follow up on – they’re not leaving any one area for the P2P panel to recommend.
A major grant on autonomic nervous system problems could have come out of this. I wonder if that’s even possible now. My guess is that it’s not. It’s understandable that the feds want an objective analysis of the field before they produce a funding package – they do that, I believe, for everybody – but the filter was so tight that almost nothing got through.
It’s a big loss.
That’s exactly it!
In an immature field which has received next to no funding, an evidence-based review on diagnosis and treatment is simply the wrong way to go, as is the P2P format.
The NIH could have done a review on the existing studies on etiology, allowing for small studies and less-than-perfect studies given the massive lack of funding for this research field. This way they could have created a good overview of the existing biomedical studies and decided on which seem to be the most promising research leads – in order to then issue a well-informed RFA.
(Or they could have just issued that RFA after the 2011 State of the Knowledge Workshop on ME/CFS which in fact did give a rather good review of the state of the research field, the gaps and the promising leads….)
To just skip the etiology phase in such an immature field and go straight to diagnosis and treatment isn’t logical or effective. How could there be evidence based diagnostic markers and treatment when researchers are still struggling (with almost no money) to determine the cause, disease mechanisms and subgroups of ME/CFS?
THIS is what the NIH should be focusing on. One has to start at the beginning. With a 20-fold increase in funding, bringing us up to the normal level for similar diseases (such as MS) and an RFA focused on disease mechanisms, subgroups and biomedical treatment, we could actually get somewhere.
Great analysis Anne! I agree with all of it..
This whole exercise is an example of circular reasoning. The primary input to high-quality research is money. This determines the number of patients in samples, the people you can hire, the tests you can perform, etc. The correlation between quality assessments and funding is very strong. If you want to support prior funding decisions, and impede change, this is the way to do so without appearing to dictate review outcome. This is a classic maneuver in organizational politics which doesn’t even leave anyone with a clear target to blame. (Who is personally responsible in this case? Anyone you can name?)
By eliminating low-quality research in a poorly-funded field you can reinforce the effect of previous funding decisions. “The science isn’t there” and it is never going to be there unless funding decisions change or a miracle happens. Prior errors in problem definition will also have a disproportionate effect on such a review. We have abundant reason for negative opinions of this process.
(If you want an historical example look at the response to F. Peyton Rous’ discovery of the first retrovirus, not even known to be a virus at the time. No prior references, no replication, strongly disputed by all contemporary medical authorities, when not ignored. Only applied to chickens, certainly not human cancers. Longevity was a major factor in Rous’ Nobel Prize.)
If science were about reproducing results that “everyone knows” (or at least “everyone who counts”) this would be science. In that case we could close a great many laboratories, and concentrate on funding committees and administrators who created the current situation.
I am not sure this captures the essence of the debate.
In other fields, typically what happens is, yes, we start with definitions made by a consensus of experts working in the field. The government is rarely involved, but even if it is, they don’t typically go grab a lot of people from other fields (usually at such a definition convention non-experts might include one or two statisticians, epidemiologists, people of that sort with a specialized kind of expertise, but often even these have experience in the field in question: there might be a logistics support person with no experience, but with interest in the field). Then we do some preliminary science. This is the step we are on in ME/CFS: preliminary science with consensus definitions (all the definitions we are using are consensus definitions, including Fukuda and Oxford).
Then preliminary science gets funded to do better studies, then biomarkers are found, then better definitions get written, then we can do full-on evidence reviews.
You don’t do evidence reviews optimized for well-designed, well-powered RCTs of treatments, on a whole heap of pilot studies exploring the pathology and a few pilot studies of treatment, with one or two poorly-designed RCTs thrown in. You don’t use parameters that look like they were taken from a CBT/GET study, and use those to assess all the kinds of studies there are.
It has nothing to do with giving ME/CFS a free pass. It has to do with doing the right kind of science at the right time.
I agree that the field wasn’t ready for this kind of analysis and that a different kind of analysis would have better.
On the other hand the AHRQ draft did point out study design problems that the field – at least some of it – can use to produce better studies in the future.
Hopefully, the report will prepare the way for a grant which will allow us to do the kinds of studies that would make into the next report of this type. Even though the timing really was off we could end up getting some funding out of it.. I don’t know they would do a study on identifying research gaps if they’re weren’t going to try to fill them.
I left my comments in the box, after the questionnaire, and cannot remember them exactly!
Hah. If the CDC had applied rigor in the first place we wouldn’t be in this mess.
Holmes criteria should have picked the neurologic signs instead of “fatigue” something that exists in virtually every illness. Then the fools kept expanding the case definition. If you read the paper setting forth the Fukuda definition, he fully expected it to be subdivided. Instead, we wound up unable to distinguish between Hepatitis C and whatever the hell the rest of us have. A lot of people with Hep C were misdiagnosed as CFS between 1988 when the Holmes Criteria were published and 1991 when the first Hep C test came out.
Please apply rigor. Give us funding and for God’s sake do what competent epidemiologists do and start finding subgroups. Oh and even though the CDC now says all boomers should be tested for Hep C for years their website said not to bother doing the test and last I heard *still* says not to work up for autoimmune disease or MS. Byron Hyde says the top missed diagnoses in CFS are Lupus and MS, and that 50% of the people who came to his practice for 1 year were misdiagnosed.
Please give us some rigor, and some damn common sense! The CDCs shameful handling of ME/CFS should go in every textbook of epidemiology as an example of how NOT to do epidemiology…
“Common Sense” — that is the key phrase. All these profoundly abnormal symptoms expressed by a multitude of people that typically demonstrate 20+ symptoms, with (most likely) an overlap of 10 symptoms with hundreds of thousands (if not millions) of other chronically-ill people.
I interpreted your mentioning of the “Holmes criteria” as “Sherlock Holmes”. Doesn’t take a genious to realize that the patients from Dr. Dan Peterson, Dr. Jose Montoya, Dr. Nancy Klimas, et al. all tend to demonstrate the same patterns of what they got, and how they got them (typically after a viral infection of Herpesviruses — as subgroups). But, more importantly following some sort of pathogenesis.
Also, all of us spend all our savings to see doctor upon doctor upon doctor in complete desperation, essentially becoming victims of the medical world that feed on the desperation from us. 20, 30, 40 doctor visits…
Chronically-ill = cash cow (tests, procedures, dr. visits, etc.). Shit! If I was a doctor or test-provider, I wouldn’t want these diseases to be figured out and cured. Heck no. I would want the $$ to role in…
I wasn’t sure if you know this from your post. The Holmes Criteria are the first, 1988 CDC case definition.
yes I do, but thanks
We have the multi-site CDC studies and the big CFI epidemiology coming up as well…I hope they can dig out some subsets…
I agree that medicine should regard this as a huge failure and that lots and lots of funding should come our way. But how can we do studies if we don’t even know if ME and CFS are the same thing?
I think the most crucial study right now is to take that Lake Tahoe original cohort and see how many have died, what they died of, what illnesses the rest have and if any have completely recovered. I knew back in 1994 when the Fukuda definition came out that we were going to have to see what the original cohort is dying from, in order to figure out the mess. That’s how Hep C was discovered after all. People were dying from liver failure who were negative for Hep A and Hep B. Back then it was called “Non-A Non-B Hepatitis”. Nice, clear endpoint. Give us some nice, hard pathology and biomarkers will become clear.
The Simmaron Research Foundation is engaged in a study that’s looking at “old-time ME/CFS patients to see whats happened with them. They will certainly have some of that original cohort in the study. ..
It is premature to be evaluating our research so strictly as we have not yet had a clear, pertinent definition–i.e. the Canadian Consensus Definition being best so far. You have to start with the best working definition and criteria, followed by research based upon it, followed by more research and then revision of the definition however the research indicates, etc. This is my best suggestion for the kind of sensible sequence needed to bring clarity to what ME/CFS is, what the biomarkers might include and what research could help establish these.
Read more: Should the Feds Give Chronic Fatigue Syndrome A Pass On Rigor? http://www.cortjohnson.org/blog/2014/11/14/feds-give-chronic-fatigue-syndrome-break/
The criteria used by the AHRQ panel is one tool in a very large tool box. The best scientists I know – and I work with many – rarely examine a problem with one tool and call it a day. Few issues are so black and white in science that a one-shot approach can capture the nuances – a truly interested analyst will recognize that human health is best understood through an approach that is both an art and a science.
Research in the AHRQ review that did not meet the criteria went unexamined and that is a shame. Much of that work had meaningful results that are still meaningful, even when you allow for some of the missing criteria. In other words – the results would have stood.
This was a lazy approach and not particularly innovative.
Two cents.
I think you’re right. I think the AHRQ has a certain way of doing things that didn’t mesh well with ME/CFS. They apply a standardized approach that probably works well for most disorders but not this one.
For the life of me I can’t figure out where ME/CFS fits in with the P2P process, or the NIH Office of Diseae Prevention, which is running this workshop in conjunction with the Office of Research on Women’s Health. Just read the strategic plans for both of these departments and you are left with WHAT? http://orwh.od.nih.gov/research/strategicplan/ORWH_StrategicPlan2020_Vol1.pdf
https://prevention.nih.gov/about/strategic-plan.
Furthermore, I have periodically followed the output from AHRQ over the last 20 years, and it’s subdivision, the US Preventive Task Force (USPTF). More often than not the results of their “evidenced based review” results in “there is not enough evidence to reach any conclusions”. Yes, sometimes they do come out with OK recommendations for future research, but it seems that this research rarely happens or never meets their standards–so 20 years later we see the same conclusions. My recommendation is shut down AHRQ and use the money to fund the research that will advance science.
As for ME/CFS, NO the evidence review by a “jury” of non experts using the AHRQ findings is very unlikely to result in any advancement in ME/CFS research. The only ones standing to benefit will be the medical and disability insurance industry.
🙂 At least we’re not alone.
They were not experts in ME/CFS – they were experts in their field – analysing studies. If that’s the goal of a project would you really want ME/CFS experts – who do not have expertise in analyzing studies – doing the analysis?
Wikipedia on AHRQ–http://en.wikipedia.org/wiki/Agency_for_Healthcare_Research_and_Quality
If anyone can point out anything good that has come out of AHRQ, other than helping to provide justification for insurance companies to deny coverage for treatment, please let me know what it is.
This may not be enough to assuage your feelings about the AHRQ but I can think of a couple
As far as I can tell nobody seems to be working on whether the disease is progressive. Well, I have a progressive form of it. Cort, do you know of any studies on progression? Byron Hyde says ME can progress, but that’s all I know.
My own illness was fluctuating for the first 12 years, but the last crash, 14 years ago, never let up and has gotten worse to the point where I am left housebound. I now have obvious damage on MRI and my left optic nerve is degenerating. So, they worked me up for MS (3rd time), and still, whatever it is is not MS. I also have a resting pulse around 120, something new over the last few years.
(BTW, something to know for others with abnormal MRIs. They should do the MRI with contrast, because MS will pick up the contrast.)
Few long term studies that I know of. There actually was one of the original cohort in Incline that found a substantial number of people recovered. That was a bit surprising. Bell did a review of adolescents that suggested that most improved. He noted that treatment didn’t matter whether they improved or not. I don’t think any one has really done a thorough job of it though. The Simmaron Research Foundation examining those long term patients should be helpful
Dr. Peterson certainly sees a lot of progression in his group – Corinne for one. Corinne is the same way – she doesn’t in any of the boxes – she has ME/CFS. So you still don’t have MS – I guess it sucks to keep falling though the cracks since it limits options.
I think the consensus is that in general most people improve a bit over time but a good portion obviously don’t. That study will be fascinating.
Jason et al have done a study on people who have been diagnosed with ME/CFS and died – deaths from heart failure occurred 30 years younger than in the general population, for deaths by cancer and suicide, the difference was ten years. His group also studied the outcomes for patients from keeping activity within perceived energy limits and found improvements over a year in activity levels. We need research into complete rest as an initial response, and to study the correlation between the conservation of exertion/attempting to increase exertion and the progression or otherwise of the illness. My own experience over nearly 30 years of ME/CFS is that I became aware, very early on, that exertion made me worse. Because of recognising this and treating the illness as a logistical problem, I have improved greatly over the years. I have also had protracted relapses, (over a year to achieve previous level of recovery) once because of taking up cycling for a few weeks, once because of looking after a rescue dog over Christmas, involving several short walks each day, and most recently rushing through the London Underground to get to the Invest in ME Conference! Protracted resting always results in improvement. But who is going to fund research into rest? That can’t be patented or charged for.
Whoever,whenever a study of great Resters occurs,I Am Ready,! This disabled body CANNOT tolerate standing aside from brief periods.And.lying down is often a must..U am so tired of being TIRED!
This is a repeat comment as my earlier comment did not appear. CFS should not be given a pass. That would be an abuse of the P2P process.
The whole P2P should be given a pass, its the wrong instrument, and methodologically flawed for a disease at the stage of research where CFS is right now. It can only be biased, especially the literature review, purely from methodological considerations. This has nothing to do with other biases. The problems we see are a result of a flawed process.
The P2P process is inappropriate for its claimed purpose. Even if we get a favourable result, which is unlikely, it will be for the wrong reasons.
The first goal should be to immediately raise ME/CFS funding to a minimum of $100 million a year and fire the people obstructing funding on the grounds that the illness is psychosomatic (this happened to Dr Lipkin when he sought funding).
The second goal should be to adopt the CCC criteria. The Fukuda criteria can continue to be used for patients that don’t meet the CCC criteria.
After a few years, larger higher quality studies will start coming out, and then we can decide how to continue and then perhaps a P2P process would make sense. Right now, it doesn’t.
Agreed!