Assessment of Improved Prediction Beyond Traditional Risk Factors

• editorial
• prediction
• family history
• risk assessment
• cardiovascular disease
• area under receiver operating characteristic curve
• reclassification

The moderate predictive performance of cardiovascular disease risk models necessitates more studies that investigate the incremental value of novel biomarkers. In recent years, many new biomarkers have been evaluated for their ability to improve prediction of cardiovascular disease beyond traditional risk factors, including C-reactive protein, coronary artery calcium, and single-nucleotide polymorphisms in the 9p21 region.¹ The interest in novel biomarkers is propelled, in part, by emerging discoveries from genomewide association studies of genetic variants associated with risk for many common diseases.² Nonetheless, family history, an “old” tool in clinical practice—crucial for the diagnosis and management of genetic disorders—has not been adequately explored for its value in risk assessment and prevention of common diseases.³ Positive family history is a strong risk factor for cardiovascular diseases,^4,5 reflecting the consequences of genetic and nongenetic risk factors that are shared among relatives. From a practical perspective, family history is a strong determinant of disease that is relatively easy to assess. For this reason, family history is advocated as a useful tool for identifying individuals at increased risk of disease and for tailoring preventive interventions.^4,6,7 However, the challenge is to show that this knowledge has clinical utility for improving health.⁸

Article see p 97

In this issue of Circulation: Cardiovascular Genetics, Scheuner et al⁹ report the incremental value of adding family history information to the General Cardiovascular Risk Profile to improve identification of individuals with advanced coronary artery calcification. The incremental value of family history in disease prediction is assessed in several questions¹⁰:

1. Does family history improve prediction beyond traditional risk factors?

2. Does improved prediction change treatment decisions or health recommendations?

3. Do these changes improve health outcomes or have other benefits?

4. Do the incremental benefits outweigh the extra costs?

This evaluation is a stepwise process: subsequent questions are only relevant when preceding questions have a positive answer. For example, the question of whether improved predictions change medical decision making is only relevant when the added biomarker improves prediction.

Statistical methods for assessing incremental value of a risk factor for prediction and risk reclassification are evolving.^11–16 One of the most frequently used measures for assessing the incremental value of a new risk factor is the c-statistic or area under the receiver operating characteristic curve (AUC),^1,17 but this measure is receiving much criticism. Opponents of the AUC argue that it is not a clinically relevant way of summarizing predictive performance, that it has no intuitive clinical interpretation, and that it is naive and insensitive to change.^12–14 As an alternative, measures of reclassification have been proposed, and these are now increasingly used.^12,15 Yet, AUC and reclassification address different questions in the evaluation of the incremental value.

The AUC assesses the discriminative accuracy of a prediction model and indicates the probability that the predicted risk of a random “patient” is higher than that of a random “nonpatient.” When the AUC of a prediction model is 1.0, predicted risks of individuals who will develop the disease are always higher than the risks of those who will not develop the disease. When the AUC is 0.5, this probability is 50%, which equals the predictive performance of tossing a coin. The AUC is particularly useful for investigating the extent to which a biomarker improves the predictive performance of a risk model. An updated model has better predictive performance than the initial model when the AUC value is higher. Also, when there is no existing risk model, the AUC can be interpreted as the improvement of the predictive performance because it is then implicitly compared with tossing a coin. The Figure shows that for every possible cut-off value, the risk model including all risk factors yielded more optimal combinations of sensitivity and specificity than the model based on genes only. When both models have similar AUCs, as also shown in the Figure, the receiver operating characteristic curves show substantial overlap implying that the models give the same combinations of sensitivity and specificity. When both models also were well calibrated, there would be no difference in the predictive performance between the 2 models.

• Download figure
• Open in new tab
• Download powerpoint

Figure. Example of receiver operating characteristic curves and AUC for the prediction of type 2 diabetes based on previously published data.^18,19 The curves indicate the sensitivity and 1 specificity for every possible cut-off value. The dashed line indicates a hypothetical random predictor, which AUC=0.50. BMI indicates body mass index; FPG, fasting plasma glucose.

Reclassification assesses the extent to which improvements of risk prediction models influence medical decisions.¹² When prediction models are used for making treatment decisions, predicted risks often are categorized. Through updating a risk model, individuals may move to a category different from that of the initial model. The percentage of individuals that change risk categories is referred to as the percentage of reclassification. Because not every reclassification needs to be correct, alternative measures such as percentage of correct reclassification and net reclassification improvement have been developed.^15,20,21 These alternative measures differ predominantly in their definition of correct reclassification. Reclassification can be considered correct not only when the observed risk in a reclassified group falls within, or is closer to, the new risk category^20,21 but also when individuals who will develop the disease change to a higher risk category and those who will not, change to a lower risk category.¹⁵ Assessment of reclassification is only meaningful when clinically relevant risk thresholds are available because the percentage of reclassification depends on the values and numbers of the thresholds chosen.

Scheuner et al examined 3 models for coronary artery calcium prediction, one based on the General Cardiovascular Risk Profile only (model 1) and the other 2 where family history either was added as a simple variable (model 2) or was a complex assessment (model 3). To investigate the incremental value of family history, they assessed and compared the AUC, Akaike Information Criteria, likelihood ratios, and net reclassification improvement. They found that the AUCs of the 3 models were 0.752, 0.756, and 0.759 in women and 0.718, 0.721, and 0.725 in men. These minimal differences were statistically significant according to comparisons of the likelihood ratios and Akaike Information Criteria, but these comparisons demonstrate that the updated models have a better model fit, not that they show improved prediction. In addition to the AUC, Scheuner et al examined net reclassification improvement as a measure of reclassification. They observed that model 2 reclassified 6.9% of the participants from model 1 and that model 3 reclassified 10.5% of them. The investigators concluded that incorporation of family history provides incremental prognostic value.

How should we interpret the different conclusions that follow from the assessment of AUC and reclassification? The similar AUC values show that prediction is not practically improved by adding family history, but reclassification suggests that family history changes medical decisions. The answer is given by Scheuner et al. They additionally investigated the extent to which reclassification was correct—whether individuals moved in the right direction—and found that a little more than half of the individuals who were reclassified using model 2 and 3 (55% and 57.7%, respectively) were correct reclassifications. In other words, reclassification yielded approximately as many good moves as bad moves. This result is typically observed when reclassification is demonstrated in the absence of an improvement in AUC. When a risk factor does not improve prediction, reclassification simply indicates that the updated model makes different errors.

Evaluating the incremental value of new risk factors is a sequential process for which AUC is a very suitable measure for a first impression about the improvement of the predictive performance of a model. When the AUC of an updated model does not differ markedly from the original model, further evaluations of reclassification, health impact, and cost-effectiveness are not warranted. Reclassification only demonstrates clinical utility if this is supported by improved predictive performance. Measures of reclassification should follow rather than replace analysis of AUC.

In conclusion, it is important to assess the clinical validity and utility of family history, along with emerging biomarkers and other established risk factors for risk assessment and disease prediction. Scheuner et al are to be commended for assessing the incremental value of family history beyond other risk factors. As shown here, methods for such evaluation and their applications are still evolving. For future research, we need a strong methodologic foundation for assessing claims for improved prediction, and for answering the important question, “When does a difference make a difference?”