Ioannidis JP, Panagiotou OA.
Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses.
JAMA. 2011 Jun 1;305(21):2200-10.
PubMed.
This JAMA paper subjects a number of blood-based tests, thought to have diagnostic discrimination in initial publications, to meta-analysis, and found greatly reduced discrimination. Does this mean that current enthusiasm for AD biomarkers should be curbed?
Some principles in the JAMA paper apply to AD biomarkers, namely, the hazards of overinterpreting diagnostic (or prognostic) discrimination from a single study. However, the biomarkers reviewed in the JAMA paper are all blood tests, and imaging biomarkers (an area of great relevance to AD) are not represented or discussed. Many of the blood tests in the JAMA paper are genetic polymorphisms or mutations, which have different connotations and applicability than biomarkers—a gene test indicates a trait, present from birth, whereas a biochemical biomarker represents an acquired state, and may change over time or be modifiable.
Some take-home messages for AD:
Understanding what aspects of biology the biomarker is measuring is often helpful. For example, the JAMA paper discusses markers such as homocysteine and CRP for heart disease—these are indirectly related to the pathology of heart disease. By comparison, CSF biomarkers that have been developed for AD are related to the biochemistry of the defining lesions of AD.
Initial studies need to be large enough to provide clearly interpretable data, with good principles of study design (e.g., gold standard diagnoses; separate groups of subjects for the discovery and the replication analyses; comparison with a non-AD dementia group to determine diagnostic specificity of the biomarker). For the biomarkers that have been proposed in the new AD criteria, there are many large-scale studies that define diagnostic sensitivity and specificity. ADNI is the largest prognostic biomarker study of mild cognitive impairment to date, and there are few large-scale studies of cognitively healthy controls who progress to AD and have had biomarker measurements. So the data on effect size/predictive power of biomarkers for prognosis in these settings will need further study in different populations.
Effect sizes are likely to decrease when biomarkers are studied in a less selected population. AD biomarker studies have generally involved carefully selected volunteers, sometimes within a limited age range. Effect sizes of biomarkers proposed for AD may be lower, for example, in a group of people aged 85 or older than in younger patients with dementia. Pathology studies representative of older adults living in the community (e.g., the Rush Memory and Aging Project study by David Bennett and collaborators; see Negash et al., 2011) has identified that mixed pathology often underlies dementia; biomarkers related to AD pathology may have lower predictive value in this type of setting than in a group of patients referred to a clinic.
The JAMA paper does not discuss issues surrounding measurement/assays for biomarkers. The AD field has been highly aware of these questions, and is addressing QC for imaging and biofluid biomarkers appropriately.
References:
Negash S, Bennett DA, Wilson RS, Schneider JA, Arnold SE.
Cognition and neuropathology in aging: multidimensional perspectives from the Rush Religious Orders Study and Rush Memory And Aging Project.
Curr Alzheimer Res. 2011 Jun;8(4):336-40.
PubMed.
The remarkable, if not entirely surprising, findings reported in this paper should be of interest to the readers of Alzforum because of the current excitement in the field about "AD biomarkers." As summarized above, the study by Ioannidis and Panagiotou indicates that most of the highly cited associations (in which a particular biomarker was associated as a risk factor for a given disease) were exaggerated, and that many associations were valid but had relatively modest effects, and may have only nominal value for clinical use.
To understand the potential implications of these findings for the AD research field, it is helpful to reflect on two points. First, why does this phenomenon (initial inflation of effect size) occur, and second, if this phenomenon is unavoidable, then how should we incorporate this knowledge while going forward?
The summary by Landhuis addresses the first issue, and I agree that the current practice of publication (endowing an enormously great "reward" on highly cited papers), coupled with funding realities, may be generally responsible for this trend. Previous works of Ioannidis and colleagues (1-3) present a compelling argument that the publishing practices in biomedical research, especially in the last two to three decades, can be best explained by applying the economic principle of “winner’s curse” (a mathematical model which explains the tendency of a winning bidder to have overpaid for an asset whose intrinsic value is uncertain). Ioannidis has forcefully argued how this practice seems to distort science (1) and could contribute to the phenomenon reported in the JAMA paper. (Incidentally, inflated associations are not restricted to the biomarker field or a particular disease, but are also observed in clinical trials [2,3] and genomewide association studies [4]). Since it is inconceivable that the current system of endowing great rewards to highly cited papers will change in the foreseeable future, it follows that current publication practices and the above phenomenon will continue to occur.
This leads to the second point: If this phenomenon is true and cannot be avoided, then how should we protect ourselves from the winner’s curse? Game theory and mathematical models give us tools to protect buyers (that is, us, the consumers of the scientific information) from overpaying (5), and thus from misallocating the resources (6). However, one prudent and practical way going forward will be for the AD field to evaluate the biomarker data critically and not to subscribe only to the most optimistic view. Formation of ADNI shows that the AD field as such is aware of the issues of reproducibility and effect size. We all share in the excitement brought to the AD field by biomarkers and understand the necessity of this approach. However, high endowment (price) and limited feedback have been shown to lead to extreme curses (5), and I hope that these comments are viewed as constructive. The conclusions of the JAMA studies have significant implications for the AD field, since biomarkers form an important component of new sets of criteria proposed by NIA/Alzheimer’s Association for diagnosing AD (7). The Webinar organized by Alzforum to discuss the new biomarker-based criteria could not have been more timely (8). It will be interesting to hear what the panel members have to say about the findings of this paper.
Comments
University of California, San Diego
This JAMA paper subjects a number of blood-based tests, thought to have diagnostic discrimination in initial publications, to meta-analysis, and found greatly reduced discrimination. Does this mean that current enthusiasm for AD biomarkers should be curbed?
Some principles in the JAMA paper apply to AD biomarkers, namely, the hazards of overinterpreting diagnostic (or prognostic) discrimination from a single study. However, the biomarkers reviewed in the JAMA paper are all blood tests, and imaging biomarkers (an area of great relevance to AD) are not represented or discussed. Many of the blood tests in the JAMA paper are genetic polymorphisms or mutations, which have different connotations and applicability than biomarkers—a gene test indicates a trait, present from birth, whereas a biochemical biomarker represents an acquired state, and may change over time or be modifiable.
Some take-home messages for AD:
Understanding what aspects of biology the biomarker is measuring is often helpful. For example, the JAMA paper discusses markers such as homocysteine and CRP for heart disease—these are indirectly related to the pathology of heart disease. By comparison, CSF biomarkers that have been developed for AD are related to the biochemistry of the defining lesions of AD.
Initial studies need to be large enough to provide clearly interpretable data, with good principles of study design (e.g., gold standard diagnoses; separate groups of subjects for the discovery and the replication analyses; comparison with a non-AD dementia group to determine diagnostic specificity of the biomarker). For the biomarkers that have been proposed in the new AD criteria, there are many large-scale studies that define diagnostic sensitivity and specificity. ADNI is the largest prognostic biomarker study of mild cognitive impairment to date, and there are few large-scale studies of cognitively healthy controls who progress to AD and have had biomarker measurements. So the data on effect size/predictive power of biomarkers for prognosis in these settings will need further study in different populations.
Effect sizes are likely to decrease when biomarkers are studied in a less selected population. AD biomarker studies have generally involved carefully selected volunteers, sometimes within a limited age range. Effect sizes of biomarkers proposed for AD may be lower, for example, in a group of people aged 85 or older than in younger patients with dementia. Pathology studies representative of older adults living in the community (e.g., the Rush Memory and Aging Project study by David Bennett and collaborators; see Negash et al., 2011) has identified that mixed pathology often underlies dementia; biomarkers related to AD pathology may have lower predictive value in this type of setting than in a group of patients referred to a clinic.
The JAMA paper does not discuss issues surrounding measurement/assays for biomarkers. The AD field has been highly aware of these questions, and is addressing QC for imaging and biofluid biomarkers appropriately.
References:
Negash S, Bennett DA, Wilson RS, Schneider JA, Arnold SE. Cognition and neuropathology in aging: multidimensional perspectives from the Rush Religious Orders Study and Rush Memory And Aging Project. Curr Alzheimer Res. 2011 Jun;8(4):336-40. PubMed.
Case Western Reserve University
The remarkable, if not entirely surprising, findings reported in this paper should be of interest to the readers of Alzforum because of the current excitement in the field about "AD biomarkers." As summarized above, the study by Ioannidis and Panagiotou indicates that most of the highly cited associations (in which a particular biomarker was associated as a risk factor for a given disease) were exaggerated, and that many associations were valid but had relatively modest effects, and may have only nominal value for clinical use.
To understand the potential implications of these findings for the AD research field, it is helpful to reflect on two points. First, why does this phenomenon (initial inflation of effect size) occur, and second, if this phenomenon is unavoidable, then how should we incorporate this knowledge while going forward?
The summary by Landhuis addresses the first issue, and I agree that the current practice of publication (endowing an enormously great "reward" on highly cited papers), coupled with funding realities, may be generally responsible for this trend. Previous works of Ioannidis and colleagues (1-3) present a compelling argument that the publishing practices in biomedical research, especially in the last two to three decades, can be best explained by applying the economic principle of “winner’s curse” (a mathematical model which explains the tendency of a winning bidder to have overpaid for an asset whose intrinsic value is uncertain). Ioannidis has forcefully argued how this practice seems to distort science (1) and could contribute to the phenomenon reported in the JAMA paper. (Incidentally, inflated associations are not restricted to the biomarker field or a particular disease, but are also observed in clinical trials [2,3] and genomewide association studies [4]). Since it is inconceivable that the current system of endowing great rewards to highly cited papers will change in the foreseeable future, it follows that current publication practices and the above phenomenon will continue to occur.
This leads to the second point: If this phenomenon is true and cannot be avoided, then how should we protect ourselves from the winner’s curse? Game theory and mathematical models give us tools to protect buyers (that is, us, the consumers of the scientific information) from overpaying (5), and thus from misallocating the resources (6). However, one prudent and practical way going forward will be for the AD field to evaluate the biomarker data critically and not to subscribe only to the most optimistic view. Formation of ADNI shows that the AD field as such is aware of the issues of reproducibility and effect size. We all share in the excitement brought to the AD field by biomarkers and understand the necessity of this approach. However, high endowment (price) and limited feedback have been shown to lead to extreme curses (5), and I hope that these comments are viewed as constructive. The conclusions of the JAMA studies have significant implications for the AD field, since biomarkers form an important component of new sets of criteria proposed by NIA/Alzheimer’s Association for diagnosing AD (7). The Webinar organized by Alzforum to discuss the new biomarker-based criteria could not have been more timely (8). It will be interesting to hear what the panel members have to say about the findings of this paper.
References:
See also New guidelines for diagnosing Alzheimer’s: What do they mean for you? Retrieved on 5 June 2011; and Alzforum Webinar: Two New Sets of Diagnostic Criteria.
References:
Young NS, Ioannidis JP, Al-Ubaydli O. Why current publication practices may distort science. PLoS Med. 2008 Oct 7;5(10):e201. PubMed.
Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005 Jul 13;294(2):218-28. PubMed.
Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008 Sep;19(5):640-8. PubMed.
Kraft P. Curses--winner's and otherwise--in genetic epidemiology. Epidemiology. 2008 Sep;19(5):649-51; discussion 657-8. PubMed.
Foreman P, Murnighan JK. How to avoid the Winner’s curse. Org. Behav. Human Decision Process. 1996 Aug;67(2):170-80.
Bezprozvanny I. The rise and fall of Dimebon. Drug News Perspect. 2010 Oct;23(8):518-23. PubMed.
Make a Comment
To make a comment you must login or register.