“Because of limitations in the evidence base, we did not have high confidence in any of the findings from this review.”
Some think the AHRQ report is ‘irredeemably flawed” but it could provide a powerful tool for change for ME/CFS
The reports from ME/CFS advocates are in. The AHRQ report is “appallingly bad”, “scientifically indefensible and irresponsible”, and is “fundamentally and irredeemably flawed”. It contains “multitudes of mistakes,” has “serious and sometimes insurmountable flaws”. In short, it’s a dangerous document put together by a bunch of numbskulls – a document that every responsible ME/CFS patient should do their best to see gutted.
I see it differently. I believe it’s the single most effective tool for change that advocates have been handed in decades.
Background
“The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of systematic reviews …..provide comprehensive, science-based information on common, costly medical conditions, and new health care technologies and strategies.”
The Agency for Healthcare Research and Quality (AHRQ) were handed the task of developing an evidence-based assessment of diagnostic and treatment efficacy in ME/CFS for the P2P project. That is what they do. They’re a busy, well respected organization that is often used to do analyses of medical issues.
They’ve done hundreds of “evidence-based” reports over time. In September they published five evidence-based reports on different disorders, three technology assessments, and four other reports.
Yes, they don’t “know” ME/CFS, but they’re not experts in any of the other disorders they analyze either. In this case, it’s not needed or even desired. The AHRQ’s clients are paying for a purely objective analysis of the evidence. I assume that they know what they’re doing.
Let’s see what they reported.
First Cut
“The limitations in applicability, as well as the limitations of the evidence base, make it difficult to draw firm conclusions with implications for clinical practice.”
Let’s just cut to the chase. The AHRQ essentially reported that this field is such a mess that thirty years after this disorder burst into the scene the AHRQ group couldn’t come to any firm conclusions about such basic factors as how to define or diagnose this disorder or how to treat it.
Citing poor study quality the panel was unable to come to ANY firm conclusions regarding the diagnosis or treatment of ME/CFS
There’s a lot of upset over the many studies that didn’t make it into the analysis. That was indeed shocking and a future blog will look at those studies. The real takeaway message from this report, though, is that this field – at least with regard to diagnosis and treatment – is a mess.
“Intervention studies were scarce and most were either fair- or poor-quality and measured outcomes using heterogeneous methods making it difficult to compare results across studies.”
Is that any surprise to anyone? Do you feel well-taken care of at your local doctor’s office? Do you know of any validated treatments? Are you going from one practitioner to the next hoping someone has a clue? Of course, this field isn’t up to snuff. If it were, we of all people would know it.
But then again, how could a complex disorder like ME/CFS be in good shape on a $5 million/year budget at the NIH? How could treatment be in good shape with just one drug in thirty years making it though the FDA – only to fail? How could it not fail to meet standards of scientific rigor with a variety of definitions vying for prominence? The field is so unformed that some researchers (Wyller in Norway) feel comfortable creating their own definitions! Relative to most of the medical field, with regards to diagnosis and treatment, Chronic Fatigue Syndrome is an under-funded, under-researched mess. That’s what this report essentially says – and it’s right.
Welcome to the big leagues, ME/CFS. This is the kind of rigorous, independent review that ME/CFS has not gotten in decades, if ever. It’s an eye-opener.
On to the report:
Study Criteria
This section outlines some of the criteria they looked at. You might want to skip through it and go directly to the results. I don’t understand much of it. The point is that this was a very rigorous analysis.
The investigators – all PhDs – extracted data on study design, setting, inclusion and exclusion criteria, population characteristics (including sex, age, race, and co-morbidities), sample size, duration of follow-up, attrition, intervention characteristics, case definition used for diagnosis, duration of illness, and results.
Quality
Two investigators independently assessed the “quality” or “internal validity” of each study. If discrepancies occurred, a third independent investigator was brought in.
Study quality was assessed using a variety of measures
Two kinds of study quality criteria were used: diagnostic study quality and clinical quality.
Diagnostic Study Quality – The quality of diagnostic studies was assessed using questions adapted from the AHRQ Methods Guide for Medical Test Reviews for Chronic Fatigue Syndrome. They assessed whether a representative sample was used, whether the study used a credible reference standard, whether thresholds were pre-specified, and more.
For diagnostic accuracy studies they extracted relative measures of risk (relative risk [RR], odds ratio [OR], hazards ratio [HR]), ROC, and AUC – when available. (If you understood what these are you understand more than I do.) They wanted to quantitatively pool the results, but they couldn’t because differences in methods, case definitions, and outcomes prevented them from doing so.
Clinical Trials Quality – This was assessed based on the presence of such factors as the similarity of the groups assessed (healthy controls vs patients), reporting of dropouts, crossover, adherence, the use of intent to treat analyses, and more.
The “internal validity” and study design, patient populations, interventions, and outcomes determined whether they could do meta-analyses. They calculated “pooled relative risks” or “pooled weighted mean differences” depending on the study type and “heterogeneity of effects between studies”. They used “subgroup analysis”. They did a lot of statistical stuff, most of which is gibberish to me.
Study Quality
They defined what it meant to be a good or fair or poor quality study.
- Good-quality studies clearly describe the population, setting, interventions, and comparison groups; clearly report dropouts and have low dropout rates; are non-biased using blinding; and appropriately measure outcomes and fully report results. They are presumed to be valid.
- Fair quality studies have enough issues that it’s difficult to tell if they’re valid or not. Some are probably valid while others are not.
- Poor quality studies have a serious enough flaws that the results found are probably more a result of the flaw than anything else.
Results
“Toto – I’ve a feeling we’re not in Kansas any more.”
Less than 10 percent of the studies their broad search turned up were included in the study
They started out with a broad sweep, doing a full text review of no less than 914 articles and then excluding a stunning 90% + of them from further analysis for following reasons with number of studies excluded for each reason:
- Did not address a key question or meet inclusion criteria – 301
- Wrong population – 76
- Wrong intervention – 9
- Wrong outcomes – 84
- Wrong study design – 131
- Wrong publication type (presumably letters, opinions?) – 157
- Inadequate duration – 57
- Systematic review not meeting requirements – 27
That’s a pretty astonishing filter. Some bloggers are crying foul, but the reductions appear to be simply the result of strict inclusion factors, one of which was this:
“Articles that attempted to define an etiology on the basis of a biochemical marker or a particular physiologic test were not included in this review because the intent of these was to identify an etiology rather than understand how the specific test could distinguish patients that would respond to treatment.”
I don’t really understand what that means, but I assume that they do and that’s it’s a standard exclusionary factor in these kinds of analyses.
They whittled the number of studies in the final analysis down to 28 diagnostic and 36 treatment studies.
Diagnostic Studies
“No studies evaluated a diagnostic test for ME/CFS using an adequate size and spectrum of patients and no studies demonstrated an accurate and reliable method for identifying patients or subgroup of patients with ME/CFS.“
Not one diagnostic study met their criteria
Not one diagnostic study met their standard of quality. Most of the studies were assessed were of fair or poor quality.
They noted that including ME/CFS patients and healthy controls – as almost all ME/CFS studies do – is just the first step. Studies next need to distinguish between patients who have symptoms similar to ME/CFS but don’t actually have the disorder. You could probably count the number of studies that have done that on the fingers of one hand.
Self-Rating Scales Fail
The AHRQ judged that even the SF-36, a primary fatigue scale used in much ME/CFS research, and generally proposed as having been validated, has not been validated at all. They asserted that the effectiveness of other scales were impossible to determine because the studies they were used in were too small, too poorly designed, etc.
Diagnostic Biomarkers
If you were looking for diagnostic biomarkers – laboratory tests that define ME/CFS – four studies (just four!) made the grade for analysis. They included two cortisol tests, pro-inflammatory cytokines in response to stress, and RNase L. The AHRQ must have dismissed dozens of studies in this category.
This was the most disappointing section of the report for me and a cause for concern. In a follow up blog I’ll look at which diagnostic biomarker studies were excluded and why.
Remarkably only four diagnostic biomarker studies were included in the report
That didn’t mean these four were quality studies. All were judged too small to provide trustworthy evidence, and none included groups with “diagnostic uncertainty”. (The CDC is the only research group that I’m aware that regularly includes fatigued patients who do not meet the criteria for ME/CFS.)
With RNase L – a controversial factor if there ever was one in ME/CFS – making the cut, it’s hard to argue that bias against certain types of biomarkers was involved. [This can be said for the treatment trials as well. With a homeopathy study making the cut, it’s hard to argue that the reviewers weren’t willing to look at any kind of treatment. The knife probably cut equally – on every side.]
Studies that we would have thought at least preliminarily identified subgroups using exercise testing, cerebral blood flow, heart rate variability, acid accumulations, and, as the report noted, “many others”, simply did not include the diagnostic testing outcomes the AHRQ required (ROC/AUC, sensitivity, specificity, or concordance) and therefore were not included.
When diagnostic studies on serum parameters and cardiopulmonary functioning did meet those requirements, they failed in other ways. For one, they failed include a broad enough spectrum of patients to ensure that the test was actually diagnostic for ME/CFS.
Welcome again to the big leagues. This is not “pick on ME/CFS” time. This is what the AHRQ does, day in and day out. They’re engaged to clear out the cobwebs and bring clarity to complex issues such as how effective treatments really are. They appear to be highly trusted by the medical profession to do that.
Treatment Trials
“Most of the evidence available surrounding treatment is insufficient to draw conclusions. Because of limitations in the evidence base, we did not have high confidence in any of the findings from this review, and only had moderate confidence in the benefit of CBT (fatigue and global improvement) and GET (function and global improvement).”
The authors did not have high confidence in any of the findings in the report
In the treatment section they assessed 9 drug, 14 CBT, 7 CAM (complementary and alternative medicine), 6 exercise, and 5 trials that combined more than one treatment. Twenty of the thirty-six trials (55%) assessed were either CBT or exercise trials, but this no surprise, and surely reflects the fact that these trials tend to be larger and better funded and more of them have been done; i.e., they were more likely to make it into a report of this type.
Only seven the thirty-six studies were judged to be of “high quality”. That’s one high-quality ME/CFS treatment trial occurring about every four years!
Definition Heterogeneity Signaled Out
“Acceptance of a single case definition and development of a core outcome set would aid in better studying the interventions to allow for more meaningful guidance for clinicians, policy makers, and patients.”
One of the reasons for the AHRQ’s inability to provide firm answers was the muddying effect caused by the different case definitions used. The AHRQ panel essentially didn’t trust any of the definitions. Nor did they suggest one was better than the other. The lack of data present for the Canadian Consensus Criteria surely made it impossible for the panel to quantitatively assess how effective it is.
The evidentiary base needed to separately assess the impact of the definitions on treatment efficacy was simply not present. (The exception to that may be the use of the Oxford definition in behavioral studies.). The definition problem is a huge one and it’s one place this report may have a real impact.
The authors essentially stated that until a single definition and (validated) core outcomes are produced and used, it’s going to be very difficult for ME/CFS treatment trials to produce “meaningful guidance” for anybody – patients, clinicians, or policy makers.
Methodological Problems Exclude Drug and Alternative Treatment Trials
“Across all intervention trials, heterogeneity in the population samples (different case definitions used for inclusion), outcomes evaluated, and tools used to measure these outcomes limited the ability to synthesize data.”
All the drug and alternative medicine trials except one provided “insufficient” evidence on their effects on ME/CFS for the panel to say anything about them. Almost all were single studies, and a single study means nothing in the medical world. It must be validated.
An immense disparity existed between the attention given to behavioral compared to non-behavioral approaches in ME/CFS
Note the disparity between the number of attempts to validate treatment efficacy between behavioral and all other trials. In contrast to the 14 CBT and 6 GET trials only one other intervention (Ampligen/rintatolimod) had more than one study devoted to it.
Two trials of Ampligen (rintatolimod) and one of valganciclovir suggested improvement, but the studies were not definitive and were limited by inconsistencies in methods and findings, small numbers of participants, methodological shortcomings, and lack of long-term follow-up.
Several other trials (IVIG, Isoprinosine, galantamine, hydrocortisone) simply didn’t provide evidence for significant improvement.
Numerous other trials did not make the cut. We’ll be looking at those in an upcoming blog.
Behavioral Trials
Both CBT and GET provided “some benefit” – hardly a ringing endorsement of them – primarily to fatigue and functioning. The report did not state, as one blogger has reported, that both were “effective” treatments for ME/CFS. These studies had their own problems (multiple methods of evaluating their outcomes, mixed results on the same measure when using different tools.)
The authors had “low confidence” that CBT is able to improve functioning
In the end the AHRQ only had “moderate” confidence that CBT is able to reduce fatigue and provide “global improvement” in ME/CFS. Based on the evidence presented, they had low confidence CBT is able to improve overall functioning, enhance quality of life, increase working hours, or reduce work impairment.
The AHRQ also acknowledged the dangers associated with exercise and GET. They noted that harms were not well reported in GET studies and several factors indicated they could be significant. Those factors included the high degree of harms reported in one GET trial, high dropout rates in another, and higher withdrawal rates whenever the arms of trials included exercise.
AHRQ also reported that several studies find that exercise worsens symptoms, and that an ME Association survey reported that GET had higher rates of symptom exacerbation than other treatments. They also noted that one study indicated that ME/CFS patients who remained within their energy envelope and avoided overexertion significantly reduced their fatigue levels.
Applicability
“The main limitation of the evidence base in our review was poor study quality.”
Rigorous research studies always include a shortcomings section which spells out possible limitations to their findings and the AHRQ did as well. This part of the report indicated how applicable the AHRQ authors believe their findings are to the Chronic Fatigue Syndrome community (patients and doctors). How well, in other words, do they reflect what doctors encounter in their office?
Several factors suggested the report was accurate. The use of all case definitions allowed them to cover a broad swath of patients seeing doctors for Chronic Fatigue Syndrome. They felt that the interventions they looked at probably covered those “commonly” used in medical practices. [Their report did not reflect many of the treatments used in ME/CFS expert’s practices, primarily because many of these treatments have not been studied in ME/CFS].
Multiple Definitions Limited Applicability
“This (the Oxford definition) has the potential of inappropriately including patients that would not otherwise be diagnosed with ME/CFS and may provide misleading results.”
The report cautioned that studies using the Oxford definition may not contain ME/CFS patients.
The use of multiple definitions was a major limitation. In very clear terms, they warned that studies containing the Oxford definition, “in particular”, might not contain any ME/CFS patients. They also noted that “ME” and “ME/CFS” definitions selected out a more severely ill set of patients. (Unfortunately they did not note how many of the CBT/GET studies used the Oxford definition, nor did they break down the success rates of those studies relative to that definition).
Some other limitations also suggested the report may not apply to the entire ME/CFS community. Small study sizes, inadequate methodologies, lack of study replication, etc. rounded out some of their concerns regarding how applicable their findings are.
Expert Bias Warning – They asserted that, given the lack of a diagnostic standard, ME/CFS is at inherent risk of something called “expert bias”. “Expert bias” is apparently a known factor in disorders that do not have validated tools.
They singled out the post-exertional malaise symptom as a possible example of “expert bias”. Until methods of testing, comparing, and monitoring this symptom are developed and used, PEM is and has to be suspect. That’s not a dig at ME/CFS or ME/CFS experts, and they’re not saying it may be not be valid. It’s a warranted conclusion and caution based on past medical findings that later found the “experts” were wrong. (They also suggested that future studies should include findings on PEM, neurocognitive status, and autonomic functioning.)
It also shows why the attempt to institute the Canadian Consensus Criteria as the definition for ME/CFS was doomed to failure – at least at this level. Consensus definitions are simply not rigorous enough to “make the grade.”
Importantly, they noted that the many problems with ME/CFS diagnostic procedures and studies make it difficult for clinicians to draw any firm conclusions from the report. Some advocates will scoff at that conclusion and say the report was a waste of taxpayer dollars, but I disagree. This report simply documents how poorly ME/CFS treatment, definition, and diagnostic studies meet the standards of excellence in the medical community. What carries weight in any scientific community are rigorous peer-reviewed studies, and the AHRQ is representative of the “peers” in the medical community.
Welcome to the big leagues! AHRQ finds ME/CFS wanting in effective diagnostic and treatment studies
That may be painful to hear, but if you ask yourself why ME/CFS is so poorly funded and has such a poor reputation this is one reason. (It’s also an inevitable outcome of very poor funding over time. ). Some mistakes were surely made in a review of this size, but I think the report is probably right on in its finding that there’s not much “there” there with regard to defining, diagnosing, and treating ME/CFS.
Negative reviews by AHRQ like what occurred here with ME/CFS may not be particularly unusual either. A September AHRQ review of Chronic Urinary Treatments found:
“Evidence was insufficient due to risk of bias and imprecision, and we were not able to evaluate consistency of results across studies. Further research should address conceptual issues in studying CUR as well as strengthening the evidence base with adequately powered controlled trials or prospective cohort studies for populations and interventions common in practice.”
A Review of Imagining Efficacy in Colorectal Cancer that was more positive indicated how far the AHRQ can go when it has the material to work with.
“Low-strength evidence suggests ERUS is more accurate than CT… and MRI is similar in accuracy to ERUS. Moderate-strength evidence suggests MRI is more likely to detect colorectal liver metastases than CT. Insufficient evidence was available to allow us reach any evidence-based conclusions about the use of PET/CT. Low-strength evidence suggests that CT, MRI, and ERUS are comparable … but all are limited in sensitivity … . Long-range harm from radiation exposure with repeat examinations is particularly of concern with PET/CT.”
Again, welcome to the big leagues! We said we wanted to be treated like other major disorders are treated, and now we are. It’s not going to be pretty at times. Growing pains are to be expected.
Guidelines and Gaps
The most important part of the report may be its outline of the gaps the ME/CFS field needs to fill with regards to diagnosis, definitions, and treatments, and the guidelines it can use to fill them.
Future Diagnostic studies:
- Should include a broad range of people, including people with similar symptoms such as Fibromyalgia and depression.
- Should use concordance and net classification index data to determine how well a particular test distinguishes ME/CFS from other disorders.
- Should use comparative groups rather than healthy controls in diagnostic biomarker studies.
Treatment Studies:
- Should settle on one definition.
- Should include multiple treatments (to mirror what doctors do), have larger sample sizes with calculated power calculations, and provide more rigorous adherence to methodological standards for clinical trials.
- Follow-up periods of greater than a year would be preferred to capture the cyclical nature of ME/CFS.
- Develop and use a set of core outcomes, including more patient-centered outcomes such as quality of life, employment, and time spent supine versus time active.
- Report about co-interventions, the timing of interventions in relation to other interventions, and adherence to interventions.
- Stratify findings based on patient characteristics (e.g., baseline severity, comorbidities, demographics).
- In particular, future studies should report findings that reflect the cardinal features of ME/CFS such as PEM, neurocognitive status, and autonomic function.
- Harms should be more clearly reported, particularly for exercise studies.
Positive Outcomes
Changes – primarily better funding from the federal government – are needed for the ME/CFS to get to the next level
Some positive outcomes were
- They searched for myalgic encephalomyelitis. (That would certainly not have happened before Dennis Mangan came to town).
- They acknowledged the Oxford criteria may select out people who don’t have CFS.
- They acknowledged that ME/CFS and ME criteria select out patients with more severe symptoms.
- They acknowledged considerable harms may be occurring in GET studies
- They had only moderate confidence in the ability of CBT to improve fatigue
- They had low confidence CBT was able to improve overall functioning, enhance quality of life, increase working hours, or reduce work impairment. Nowhere was CBT/GET suggested to be a cure.
More importantly, any report that concludes that almost 30 years of research and study have failed to provide the evidentiary basis to provide any firm conclusions regarding diagnosis, diagnostic biomarkers, and treatment efficacy says something about the state of ME/CFS in our medical system. That AHRQ could find only 64 studies that fit its parameters says something very important.
To say almost 30 years later that you cannot say anything firm about the effectiveness of treatments is a damning indictment of the medical research establishment’s support of the Chronic Fatigue Syndrome community. To have only 39 treatment trials qualify for inclusion in this report – most of which were of poor to fair quality – from almost 30 years of study is astonishing. To have only one drug or alternative treatment provide other than “insufficient” evidence of it’s effectiveness in ME/CFS is tragic in a disorder effecting a million people in the U.S. and millions more across the world.
Read between the lines and this report says that ME/CFS has been spectacularly poorly studied, that it lacks fundamental tools that other disorders have, and until it has those tools not much effective work is going to be done in the realm of diagnosis and treatment. A definition that works must be developed (and validated), validated outcome measures must be developed and used consistently, study sizes need to be larger and rigorous study designs must be used.
That’s the real message from this report and that’s how this report will make a difference. For all the hubbub about the reports failings advocates will be able to use this government sanctioned report as a powerful cudgel to slam the NIH, CDC and FDA for their neglect of this disease. It’s going to one of the most be powerful tools we have for insisting on adequate research funding, for support in building up the infrastructure ME/CFS needs to be researched properly, and for more flexibility at the FDA regarding treatment trials.
The very fact that it’s an independent analysis – something some advocates railed against – will enable them to use it’s conclusions – that decades of research have largely been useless with regards to diagnosis and treatment – that much more effectively.
Pathways To Prevention
The Pathways to Prevention (P2P) program is an NIH-based program that host workshops which identify research gaps and methodological and scientific weaknesses in a field using “unbiased, evidence-based assessments”. The research gaps the AHRQ report outlines are plain – no validated definitions, few properly designed diagnostic biomarker studies, little basis for determining effective treatments, etc.
The Pathways to Prevention program is supposed to identify research gaps need to move the field forward. That should not be a problem…
The federal government often requires overviews be done before it begins major funding efforts. The Neuroimmune Conference in the early 2000s provided the basis for what was probably the first and only major grant effort by the NIH for ME/CFS. The P2P report should provide an evidentiary basis the federal government needs to embark on a major funding effort to assist with defining, diagnosing, and treating ME/CFS. (Why else would do a report like this?)
The AHRQ report provides the most comprehensive overview of those facets of ME/CFS that the P2P panel will receive. Given its rigorous standards and its officially sanctioned nature, they will rely heavily on it in making their recommendations. Hopefully, some of the recommendations should lay the groundwork for a major grant designed to solve these problems.
Comments
Comments to the report are allowed until Oct. 20th here.
I say don’t gut the report. Don’t argue that ‘x’ treatment approach works. Any federal report that says we still don’t know how to diagnose or treat ME/CFS is perfect as it is. Leave it as it is.
For me, I’m going to look into the Oxford definition and behavioral studies and if most of them use it, I’m going to request a warning that the results may not apply to many people with ME/CFS. I’m also going to ask the panel to reconsider their assessment of the PACE trial given the many questions raised about it.
The main focus of my comments, though, will be on how the report’s findings reflect a tragic and decades long lack of support from, in particular, the federal government – for a million chronically ill people in the U.S. My comments will also reflect the enormous disparity between the number of clinical trials assessing CBT and GET and any other treatment approach.
Cort, what a solid view and analysis and what a solid stance on your part this is! Great work, in my humble opinion (now I just have to get over the idea of PEM as an example of an expert bias… This just freaks me out!…No matter the rationale or how one looks at that statement…I think I get it, but still… )
🙂 It freaked me out too. I thought what were they thinking? But they need statistical validation – something we’re not apparently too good at – for everything. I think of it as a “possible” example of expert bias that will surely turn out not to be expert bias. I can’t imagine it won’t turn out as they attempt to validate it.
Thanks!
Thanks Cort
A ray of sunshine as usual. I totally agree with you although without out your brilliant analysis I would be in the forest. PEM – expert bias …well now we can ask to have out PEM measured…. Out illness poorly defined yep we agree so where to now guys!!
Tx
Thanks Adrienne.. Dr. Unger has been asserting the need to validate PEM for quite some time. I’ve been thinking she’s been kind of full of it 🙂 – but apparently others agree…
Thanks for this very interesting review, Cort – plenty of food for thought there. Just skimming this before I go to bed and I didn’t see you mention the Norwegian Rituximab trial, which I would have thought would have done fairly well in the review. I wondered how the reviewers treated it?
It didn’t make the cut for analysis….I’ll be looking at the studies that didn’t make the cut in an upcoming blog and we’ll see where the panel felt it fell short…
Cort, as Christian G. said above, Great Work! As a former nurse practitioner I know how recommendations from the AHRQ are viewed in the medical community and you are correct in your statements. I LOVE that this independent body couldn’t find good research on hardly ANYTHING having to do with this horrible disease. Even the analysis of UK studies were damning in their faint praise.
As you correctly ID’d, this finding and report can be used to advocate for rigorous studies, which will in time lead to better outcomes for all of us.
You know we can say to the NIH
Thanks for your thirty years of putrid funding
Thanks for not funding Centers of Excellence.
Thanks for 1 major grant opportunity in 30 years.
Thanks for throwing us into a dark corner of the NIH with no clout and no money 15 years ago
Thanks for spending more on ME/CFS (adjusted for inflation) twenty years ago than you do today
Thanks for closing down the CFS Research Centers
This is what it’s got us. Is anyone surprised?
I think all is shows is that the govt hasnt put enough money into serious research.
Also are they only using research done by American scientist?
I just find it hard to believe they havent used the Australian work by Griffith University on natural killer cells which included 100 cfs patients and i think 50 plus controls and followed them for 18months. Its a significant sized amount of patients in the study over a significant time period.
They looked at everything…..One of the inclusion was that research not be focused on etiology but have something to do with treatment – I actually couldn’t figure out what the sentence meant – but I assume that its part of their standard criteria.
That must have knocked out many studies…including that one…
They also want – for diagnostic biomarkers- studies that include people with illnesses close to ME/CFS – such as Fibromyalgia. The CDC has done this using people with idiopathic fatigue who do not meet criteria for ME/CFS. Studies including depression might fit that criteria as well. Other than that only a very few studies do this – Natelson’s ME/CFS and FM studies come to mind.
If this does lead to a serious increase in federal funding for proper research into ME/CFS, that will be a good thing – unless it turns out to be more like the PACE Trial. I expect it won’t be news to anyone here that the PACE researchers, having used a tool for making physical measurement of activity (an actometer? By now we should all have watches that have this feature!) in the pilot study, did not continue its use in the final study. Their justification was that research shows that self-reports (on which the study now relies) differ from physical measurements, according to Wiborg. They do. Self reports exaggerate rates of improvement compared to physical measurements.
It has also been widely reported that the original criteria for recovery was going to be a score of 85 on the Chalder fatigue scale. In order to quality as sick enough to enter the trial, patients had to have a score of 65 or lower. During the course of the research, the decision was taken to lower the ‘recovery’ criterion to a score of 60. The implication of this is that a patient could enter a treatment arm with a fatigue scale of 65, engage with treatment, deteriorate to a score of 60, and be recorded as having recovered. These are not the only criticisms which have been made of the PACE Trial, but it does seem hard to understand that this study could be given any credence by this group of people. So I hope you are successful in your invitation to them to reconsider it.
The point about a much longer follow-up seems very important. I am an NLP Master practitioner, and have been accredited for more than 20 years by the United Kingdom Council for Psychotherapy, and I have ME. I am very aware of the power of well-delivered NLP techniques to fire up motivation, be truly inspirational, and literally get people to the point where they can do a fire-walk successfully (usually successfully!). Techniques like the Lightning Process, and any other uses of NLP to persuade patients that they can reverse their illness just by changing their beliefs and then ignoring their symptoms (does that sound like anything else we know and love…) can result in what appears to be immediate improvement, and which is very likely to lead to a delayed and protracted crash in a few weeks time. I regard these programs as potentially dangerous, and am appalled that they are going to be experimenting using the Lightning Process with adolescents in Bristol. This is just one argument for long-term follow-up.
Other than the study by Jason et al showing that activity levels improved if patients followed a regime based on estimating their energy levels each day and then keeping their activity level within it, I am not aware that anyone has considered a trial of complete rest from an early diagnosis as against other treatments. My thoughts are that if rest were the standard prescription, no one stands to make any money from it, so no one is even going to think about it. In 1996, Wessely and Co put a couple of paragraphs in their report warning of the dangers of rest and stating that under no circumstances should it be suggested to patients except in very limited periods. It is probably a coincidence that many patients treated by the doctors who made this recommendation (Ramsay, Acheson) appear to have recovered, while now that we know that exercise cures everything, people with ME so often get worse, and so seldom recover. My personal view is that patients should be provided with a list of suggestions similar to a list available for MS patients who suffer from fatigue – how to manage everyday tasks while conserving physical exertion as much as possible. If muscular energy were treated as a suddenly very limited resource (which it is, if we don’t want to get worse), and every assistance were given to help people continue their domestic/professional/educational lives by conserving every bit of unnecessary physical exertion, I think many of us would be able, as I was, to continue at least a semblance of a normal life. (Working in the Social Services Department, I was informed that acquiring and using a wheelchair would be ‘career suicide’, so I guess a few attitudes would have to change. So I wish someone would do a really good project investigating exertion-conservation as a treatment strategy. Most ME patients do this as a matter of necessity, often against opposition….
The AHRQ report is critical, regardless of what the P2P panel comes up with. According to the AHRQ summay, the report can be used for things like clinical practice guidelines and insurance coverage. Unfortunately, the report doesn’t state the conclusions that Cort has above. Yes if you know the illness and can read between the lines you might make those same conclusions, but if you think doctors or insurance company medical directors reading the 2 sentance summary at the beginning and end of this report will come to any other conclusion than what they already believe–that we are all just crazy and need CBT and excerise–then please let me know what you are drinking or smoking, cause I want some.
???? I don’t see how you can say this report states CBT and exercise is a cure for ME/CFS.
Show me that I’m wrong by citing the report and I’ll change the blog.
Sorry ‘doctors who made this recommendation’ refers to Ramsay and Acheson’s unequivocal statements that complete rest from the beginning gave the best prognosis, not to Wessely and co’s subsequent claims about the dangers of rest.
Thanks, Cort.
Seems like a lot of the solid work is being done overseas, in Japan, for example, as the pet scan results seemed very definitive. I’m hopeful that good research is being done in Norway and Germany (as per the EBV findings) as well. Since CFS/ME is worldwide, many other countries should be getting involved in the future.
I hope you are right, Cort. In 2002 the AHRQ did an analysis of CFS research. They have tight controls on what studies survive the final cut. Back then, the conclusion was that CFS research was of poor quality and quantity. Funding went into the tanker from 2002 forward, so it appears that the review was not taken seriously.
Anyway, the bulk of US CFS research with significant findings has not been replicated to the standards of the AHRQ. Didn’t we know that already? MONEY…we follow the money.
However, the year is 2014 and I hope for much better. We have new advocates and the ability to reach a larger audience. OUTCRY
The thought process is odd. Without paper and pens, the very basic tools of writing, a academic can’t come back to a student and expect a novel.
THIS must be clear by now. No money, no replication, nothing to show here. THIS does not negate the tireless efforts of our researchers. They work with what they have and squeeze a buck out of a quarter.
Thanks Pat. Yes, we have really good researchers and they do really good research with the money they have and a few of them have good enough grants to do really good research..There is good research being done.
I think we’ll do better this time. We have better advocacy and this report was focused on diagnosis and treatment and I think that will enable us to push harder for more funding in these areas.
Great article Cort.
I’d be interested in any comments they may have made regarding the CDC empirical case definition since it appears to avoid the problems associated with expert bias/consensus criteria and would include those with similar symptoms that don’t meet ME/CFS ‘caseness’.
I didn’t see a single mention of the Empirical Definition and I don’t know if any studies using it made the cut.
My guess is that so little work has been done using it lately that it was not considered important enough to report on. (I’m sure more studies have used it than have used the CCC.)
As you note, though, the CDC did regularly include people that were severely fatigued but didn’t meet the criteria for ME/CFS in their studies….
The ED’s position via the AHRQ panel is a mystery to me.
Reeves Empirical is mentioned on pages 12 and 15. In the Appendices at G1-10, G1-12, G2-1, H1-4, I-4 to I-6.
It was explicitly included among the eight case definitions they considered. They did not address Jason’s work proving how unreliable it was.
Thanks…I don’t think they made any statements about it’s reliability or unreliability in the body of the report?
No they did not. The examined one of Jason’s studies that assessed it focused on the diagnostic methods (SF-36, etc) that were used, not on the accuracy of the definition. The review failed to note Jason’s comment that the 2005 Reeves has sensitivity and specificity problems.
Thanks for a really thoughtful piece, Cort. I think you may be right but we need to see the final version. Do you know if research funding subsequently increased for those other diseases you quoted that also had their research panned?
The other negative assessment I noted was done this September actually – so no time to see if anything changes there.
I think it will take strong advocacy….
Thanks Cort,
Excellent analysis, yet again. This just really shows why this disease does not get much respect in the medical and research communities yet. Despite thinking that there are thousands of studies that have been done proving that we really are sick, the reality is that only a handful of these studies are “valid” in the eyes of the scientific/medical world. This is a good thing, by the way, as painful as it is for me to say. I agree with most of the rigorous rules that were applied to weed out so many of the studies, even if they were deemed excellent in our eyes and will most likely turn out to be correct in the long run.
I am confident some of the future studies coming along will fit the stringent rules and will change how we define ME/CFS, hopefully by providing a diagnostic test(s) and then things will take a turn for the better. I think we are seeing that change happening now as more and more prestigious researchers jump on board. As they conduct larger studies, with better technology, more stringent rules and a better cohort of patients and controls, we are bound to get better results, and finally more respect!
Yes. Please note too that this review was focused only on diagnostic and treatment studies.
I agree that the appearance of Ron Davis and the other Stanford researchers and Ian Lipkin and Dr. Klimas’s growing institute at NSU and is very good news for the research field. Most importantly I believe it will eventually translate to better funding which will translate to better studies…
We have these great Biobanks set up with their superb identification of patients that will make that task so much easier…
The infrastructure for success is slowly (actually fairly rapidly) being built – hopefully as its built the money will come.
Studies were not excluded from the evidence review because they were invalid or low quality. They were excluded because the studies did not target the specific questions asked in this review, failed to meet inclusion and exclusion criteria or were deemed to be etiological studies not diagnostic methods studies. That’s why NK-Cell Function, CPET, autonomic testing, etc were excluded.
There were very good quality studies excluded from this review that I believe would have resulted in different overall conclusions if included
I agree that the inclusion/exclusion criteria were incredibly tough…and many otherwise good quality studies were excluded because of them.
thank you for spending the time and effort to write this. I suspect this blog took a very long time to compose. I will make a donation to your site as soon as I can figure out how to do it within the contraints of your new website “look”.
uh oh….we didn’t mean to have a new “look” ….Something has gone wrong…I’ll get back to you.
I had the same problem, the website looking different. Then I found a button on the very bottom of the page saying: iPad version on/off. You may be viewing it on a tablet device that changes the look of the webpage for some strange reason. Anyway I changed the button to off and now it looks like the original. I hope this helps.
One thing that I can say for sure is that this paper’s classification of the PACE study as “Good” is dead wrong.
Following is a critique of that paper.
https://www.dropbox.com/s/lj3fyoqi2qxxkt4/Problems%20with%20the%20PACE%20Study.pdf?dl=0
Best,
Lisa Petrison, Ph.D.
Executive Director
Paradigm Change
http://www.paradigmchange.me
I also am rather shocked that this article about the Tahoe epidemic – written by respected researchers and published in a very good journal – was omitted entirely from the review.
http://www.ncbi.nlm.nih.gov/pubmed/?term=1309285
The authors did not omit it for cause. It is not listed in the document at all.
Insofar as the few papers that have studied severely ill M.E. patients are omitted from research reviews, then it is no wonder that the disease is dismissed as something not very important and thus receives inadequate funding.
I’ll bet their search did not bring it up. Note that the title does not say “Chronic Fatigue Syndrome” (even though the term was in use at the time) it says
“A chronic illness characterized by fatigue, neurologic and immunologic disorders, and active human herpesvirus type 6 infection.”
The only thing to indirectly connect it with ME/CFS (even though it about CFS) is the word “fatigue”. Even in the abstract they didn’t say CFS.
I recently stumbled on a bunch of Japanese studies on ME/CFS that I’d missed because they didn’t use CFS in the title. Ditto with “post-viral” studies done on ME/CFS…
I agree with you and I’m not surprised that it was stated to have good quality. It took some digging to unearth the problems with the PACE report and I think they would not have been readily visible to this group or to a review like this.
That said, the “good” rating is something I’m going to mention in the comments.
Thank you very much for providing your critique of the PACE study particularly the GET arm. My local pain clinic (diagnosed with CFS/ME in 1994 and fibro in 2012) can only offer their own pain management programme (Kingdom Pain Management Programme) if it is searchable, it concentrates on telling you that chronic pain is all in your head and is not real pain like arthritis – which I also have).
In my past life before I had to stop working in 2009, I worked in Public Health in UK so am well able to understand the reasons for the outcome of the review on ME/CFS research. We have NICE in some parts of the UK and SIGN in Scotland where I was forced to return to after I lost everything due to this illness.
To the point at last (rambling is a big part of my illness)
The programme I am now engaged in includes exercise therapy. It is delivered by a physio who was the one to allude to the fact that chronic pain does not exist, a psychologist and an OT. During our first real induction to exercise, we were asked to do a set of very basic stretching exercises. One group member pointed out that we were not given any instructions about posture etc. She was completelyy ignored as we all are when we comment at all. The exercises were very easy for me except for those involving the hip and leg where my pain currently is.
I have confronted the pain consultant on several occassions of the lack of evidence regarding the efficacy of GET and other such programmes but as usual he remains deaf to my complaints and blind to the reams of evidence I bring to him.
Did you say you had a blog I could subscribe to? If not I would love to stay in touch. I have an MSc in Public Health but haven’t got the brain power now to continue to a doctorate but I am constantly searching for research so maybe I might have found something that could be of use to your cause. Chrisine B., MSc.
Dear Cort,
Thanks for your thorough analysis of this report and its meaning.
Unfortunately, now, in addition to being exhausted and brain fogged and in pain, I’m very thoroughly depressed.
On the bright side, I haven’t wasted years and dollars on things that have not worked well enough to be proved.
I pace myself very seriously, nap 3-6 times a day, use B1 in massive quantities, bike and walk a tiny bit – and use the rest of me to write. So many people have to spend their whole life pursuing and ameliorating symptoms – at least I get a little bit out of mine.
I hope, along with you, that it will result in something sometime. Meanwhile, I have a life to get through.
Alicia
Forgot to mention – what I’m writing is a novel with a protagonist who lives with CFS – and is a writer, and a former physician.
It’s my little bit of education. Many of my readers (I’m serializing on my blog) have mentioned they never had even heard about CFS until they read Pride’s Children; that is also incredibly discouraging.
Alicia
Thanks Alicia – can you leave your website link?
re: Canadian Consensus v. other diagnostic criteria
Exactly how is the CCC any different than Fukuda; or any other criteria? I thought they have all come from people sitting around a table determining by fiat the criteria for “illness”. Or does this committee have evidence to suggest that science magically emanates from individuals sitting around tables in government offices located in Washington, DC and London, UK?
I have a question which is probably a little late… but I’m stunned that they set up this review in such a way as to not include studies of etiology. If they wanted a review for the P2P which will be guiding the NIH’s future research priorities in the ME/CFS field – why didn’t they do a review of etiology studies? Also, if they wanted to review possible diagnostic markers in a field which is obviously not very mature, wouldn’t they need to start with reviewing the etiology studies?
Any thoughts on this, Cort?
They didn’t really focus on research priorities per se (unfortunately) but on diagnostics and that changed everything. My impression is that the scope of the project narrowed considerably over time. It appears they didn’t take field maturity into account at all- which is probably why some researchers are rather upset at the findings. I’ll bet the AHRQ just doesn’t do that – they have their standard exclusion/inclusion criteria and they apply them to every disorder.
Dear Cort,
Thanks a million for a great overview of the report!
Has anyone tried Low Dose Naltrexone (LDN) for CFS/ME?
I’ve been on 1.5 mg LDN now for 11 days now and I’m beginning to feel some good effects from it.
I’ll keep you informed.
Cheers,
Prof. Myron A. Hoffman (Age 84, and suffering with CFS for the past 10 years)