Assessing the discriminative ability of risk models for more than two outcome categories

Eur J Epidemiol. 2012 Oct;27(10):761-70. doi: 10.1007/s10654-012-9733-3. Epub 2012 Oct 7.

Abstract

The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Data Interpretation, Statistical
  • Discriminant Analysis*
  • Female
  • Humans
  • Logistic Models
  • Male
  • Models, Statistical*
  • Ovarian Neoplasms / diagnosis
  • Ovarian Neoplasms / epidemiology
  • Prevalence
  • Prognosis
  • ROC Curve
  • Risk Assessment*
  • Testicular Neoplasms / diagnosis
  • Testicular Neoplasms / epidemiology