Cochrane Asks for Field’s Input on Draft Reporting Standards

Series - Alzheimer's Association International Conference 2012:

Most researchers agree that clinical dementia papers need reporting standards, especially when it comes to diagnostic tests. Say you are interested in this question: How well does a given biomarker predict Alzheimer’s disease (AD), or distinguish one form of dementia from another? Researchers wanting to do meta-analyses of the literature to answer questions such as these find it nearly impossible to determine the field’s knowledge because individual papers vary greatly in what and how they report. Enter STARDdem, which stands for “standards for reporting studies of diagnostic accuracy in dementia.” STARDdem is an initiative by the nonprofit Cochrane Dementia and Cognitive Improvement Group in Oxford, UK, to guide researchers in exactly how to report on diagnostic tests in the field of dementia. Compliance would help ensure that researchers control all the necessary factors and, in essence, make studies more meta-analyzable by the standards of evidence-based medicine. At the Alzheimer’s Association International Conference, held 14-19 July 2012 in Vancouver, Canada, Cochrane researchers introduced a STARDdem working document and invited the field to provide feedback during an open comment period lasting until August 31.

“There is a recognition that the literature at the moment needs a degree of standardization,” said Rupert McShane, University of Oxford, UK, who led the study. McShane heads the Cochrane dementia group, which reviews studies in the prevention, treatment, and management of cognitive impairment. “Our hope is that the STARDdem recommendations will be widely accepted and implemented.”

Modeled after the Consolidated Standards of Reporting Trials (CONSORT) project for randomized clinical trials reports (see Moher et al., 2001), the original STARD document aimed to standardize reporting in general studies of diagnostic accuracy (see Bossuyt et al., 2003). However, dementia researchers have been lax about putting its principles into practice, in part because generic standards do not apply to the unique reporting needs of the dementia field. “We are trying to make it easier for people in this field to use those criteria and understand what they mean,” said Leon Flicker, Western Australian Centre for Health & Ageing, Crawley. Flicker presented the document, now open for public comment, at the conference.

STARD is intended for studies that report the sensitivity or specificity of a given dementia test and hold that test up to the “gold standard” of AD diagnosis—usually autopsy confirmation or conversion from MCI to AD. The gold standards come some time after the diagnostic test, and this constitutes one aspect of the dementia field’s unique needs. STARDdem gives guidance on how to write up those studies, defining which aspects need to be reported. For instance, it recommends that studies report the reference standard and cite the study that validates it. It also details how to report missing data, divulge patient exclusion criteria, and include reasons for dropout, among other things. The guidelines intend to ensure that researchers report their findings thoroughly, but also that they think about and control all the important factors, said McShane. Without proper adherence to the guidelines, readers don’t always know how old a study population is or if results are from patients in a memory clinic versus the general population. “There’s rather a lot of scope in the literature for bias to creep in because the reporting suggestions of STARDdem aren't being applied,” said McShane. There are some rare exceptions; For example, a widely cited study comparing CSF diagnostic results across centers did use STARDdem criteria (Mattsson et al., 2010).

The STARDdem draft will stay open for public comment until 31 August 2012. After that, the Cochrane scientists will compile the comments and prepare a new draft, to be presented at the Clinical Trials Conference on Alzheimer’s Disease (CTAD) this October in Monte Carlo, Monaco. Soon after that, the authors hope to publish the guidelines in a scientific journal. It will then depend on journal editors and reviewers to adopt these measures and require compliance to assure widespread adoption by research groups across the field, said McShane.

Standardization is becoming especially important for biomarker studies. These are exploding in the literature and becoming increasingly important for AD diagnosis, said Henrik Zetterberg of Sahlgrenska University Hospital in Mölndal, Sweden. However, Zetterberg considered the first STARDdem draft to be insufficiently informed in terms of biomarker reporting. Zetterberg has critiqued the draft online and stressed that others should follow his example. “I think it is very important that specialists read and comment on the draft to make the first public version as good as possible,” he told Alzforum (see full comment below).

McShane’s team plans to conduct 15 literature reviews on potential diagnostic tests by September of 2013—about half on biomarker tests and half on cognitive tests for dementia. Craig Ritchie, Imperial College London, also in the Cochrane dementia group, presented a poster at the conference on the first of these reviews. He claimed that, based on the 13 studies in the literature that met his review criteria, cerebrospinal fluid (CSF) Aβ42 is neither sensitive nor specific enough to be confidently used as a diagnostic test for progression from mild cognitive impairment to AD. The poster’s conclusion met with considerable skepticism from Alzheimer’s scientists; however, AD scientists do agree that the literature for CSF tests needs to be better standardized.

“The field is at a place now where the next logical step is to bring biomarkers into the clinical realm,” said Anne Fagan, Washington University School of Medicine, St. Louis, Missouri. Before that happens, several committees will decide when and for whom such testing would be appropriate, and they will use published studies to determine those answers. While the STARDdem will not improve previous studies, it may help standardize future ones. For example, it would be helpful if authors provided simple definitions for "cognitively normal," or other qualifiers that are often neglected, to enhance comparability. “I think there has to be a common ground,” she told Alzforum.

Separately, another poster pointed to a perhaps even greater need for standardization, and that is in the still-emerging field of blood-based biomarkers. Unlike CSF, the plasma field has not begun to converge around a few markers that generate similar results among centers and studies; it is considered wide open for discovery. Andrew Watt, from the lab of Kevin Barnham at the University of Melbourne, Australia, reviewed 87 blood biomarker papers. He found no uniform way to collect and store the Aβ samples. Since the biomarker measures vary depending on the time of day samples were collected, speed of centrifugation, storage temperature, and other factors, the field really needs a standard way to collect and analyze samples and report its procedures so that studies become comparable, he told Alzforum. In toto, scientists called for a combination of quality control, assay, and sample handling standardization on the one hand, and reporting standards as developed by STARDdem on the other, to move dementia diagnosis to the next level.—Gwyneth Dickey Zakaib.

Comments

  1. When looking in PubMed, one sees that there is an exponential increase in the number of studies on biomarkers for AD, and the opportunities should be excellent for making meta-analyses on the topic. However, if one sets out to curate the literature, it will rapidly become clear that there is no standardized way of reporting how tests were performed, on what patients, how samples were handled, how cut-points were determined, etc. This is a well-known problem in other fields of medicine, which stimulated the creation of the STARD (STAndards for the Reporting of Diagnostic accuracy studies) criteria some years ago.

    The work now to adapt the STARD criteria to studies on neurodegenerative diseases is very important. In my mind, the first draft of the STARDdem document was rather naïve in respect to what is important to know for fluid biomarkers, which is my specialty. For imaging biomarkers and other measures to help in making the diagnosis, the situation may be the same. I think that it is very important that specialists respond to the invitation by the STARDdem authors to read and comment on the draft to make the first public version as good as possible.

    View all comments by Henrik Zetterberg
    • User Profile Image Lei Feng
      National University of Singapore
    • Posted:

    This is an exciting and timely project that will benefit scientific research and clinical practice in the long run.

    One difficulty I can foresee is that dementia is a long, pathological process rather than an event. So it would be very hard to define a "standard" point of "conversion" from "MCI" or "pre-AD" to dementia. The distinction between MCI and dementia depends on an individual's "function" compared to a previous level. The comparison is often subject to clinical judgment, and, in certain contexts, making a good judgment can be very difficult.

    "MCI" also suffers from lack of a standard "operational" definition. For example, one person could be labeled as amnestic MCI, non-amnestic MCI/single- or multiple-domain MCI, based on poor performance on neuropsychological tests, but in reality, it is almost impossible to standardize the number of tests, types of tests, and which norm to be compared to in defining MCI.

    View all comments by Lei Feng

  3. There is a similar open project entitled Use of Biomarkers for Neurodegenerative Disorders in the Clinical Setting. It is devoted to all neurodegenerative diseases, including those with dementia. It is funded by the Neurosciences Foundation, a Spanish not-for-profit organization. Anyone with interest in the field is invited to join.

    View all comments by Carlos Vázquez

Make a Comment

To make a comment you must login or register.

References

Paper Citations

  1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001 Apr 14;357(9263):1191-4. PubMed.
  2. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC, . Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003 Jan;49(1):1-6. PubMed.
  3. Mattsson N, Zetterberg H, Hansson O, Andreasen N, Parnetti L, Jonsson M, Herukka SK, van der Flier WM, Blankenstein MA, Ewers M, Rich K, Kaiser E, Verbeek M, Tsolaki M, Mulugeta E, Rosén E, Aarsland D, Visser PJ, Schröder J, Marcusson J, de Leon M, Hampel H, Scheltens P, Pirttilä T, Wallin A, Jönhagen ME, Minthon L, Winblad B, Blennow K. CSF biomarkers and incipient Alzheimer disease in patients with mild cognitive impairment. JAMA. 2009 Jul 22;302(4):385-93. PubMed.

External Citations

  1. Cochrane Dementia and Cognitive Improvement Group
  2. public comment

Further Reading

Papers

  1. Westman E, Muehlboeck JS, Simmons A. Combining MRI and CSF measures for classification of Alzheimer's disease and prediction of mild cognitive impairment conversion. Neuroimage. 2012 Aug 1;62(1):229-38. PubMed.
  2. Lu Y, Riddell D, Hajos-Korcsok E, Bales K, Wood KM, Nolan CE, Robshaw AE, Zhang L, Leung L, Becker SL, Tseng E, Barricklow J, Miller EH, Osgood S, O'Neill BT, Brodney MA, Johnson DS, Pettersson M. Cerebrospinal Fluid Amyloid-β (Aβ) as an Effect Biomarker for Brain Aβ Lowering Verified by Quantitative Preclinical Analyses. J Pharmacol Exp Ther. 2012 Aug;342(2):366-75. PubMed.
  3. Doecke JD, Laws SM, Faux NG, Wilson W, Burnham SC, Lam CP, Mondal A, Bedo J, Bush AI, Brown B, De Ruyck K, Ellis KA, Fowler C, Gupta VB, Head R, Macaulay SL, Pertile K, Rowe CC, Rembach A, Rodrigues M, Rumble R, Szoeke C, Taddei K, Taddei T, Trounson B, Ames D, Masters CL, Martins RN, . Blood-Based Protein Biomarkers for Diagnosis of Alzheimer Disease. Arch Neurol. 2012 Jul 16;:1-8. PubMed.
  4. Soares HD, Potter WZ, Pickering E, Kuhn M, Immermann FW, Shera DM, Ferm M, Dean RA, Simon AJ, Swenson F, Siuciak JA, Kaplow J, Thambisetty M, Zagouras P, Koroshetz WJ, Wan HI, Trojanowski JQ, Shaw LM, . Plasma Biomarkers Associated With the Apolipoprotein E Genotype and Alzheimer Disease. Arch Neurol. 2012 Jul 16;:1-8. PubMed.

Annotate

To make an annotation you must Login or Register.