Skip to main navigation Skip to search Skip to main content

A critical perspective on finite sample conformal prediction theory in medical applications

  • Klaus-Rudolf Kladny*
  • , Bernhard Schölkopf
  • , Lisa Koch
  • , Christian F. Baumgartner
  • , Michael Muehlebach
  • *Corresponding author for this work

Publication: Scientific journal › Journal article › peer-review

Abstract

Machine learning (ML) is transforming healthcare, but safe clinical decisions demand reliable uncertainty estimates that standard ML models fail to provide. Conformal prediction (CP) is a popular tool that allows users to turn heuristic uncertainty estimates into uncertainty estimates with statistical guarantees. CP works by converting predictions of a ML model, together with a calibration sample, into prediction sets that are guaranteed to contain the true label with any desired probability. An often cited advantage is that CP theory holds for calibration samples of arbitrary size, suggesting that uncertainty estimates with practically meaningful statistical guarantees can be achieved even if only small calibration sets are available. We question this promise by showing that, although the statistical guarantees hold for calibration sets of arbitrary size, the practical utility of these guarantees does highly depend on the size of the calibration set. This observation is relevant in medical domains because data is often scarce and obtaining large calibration sets is therefore infeasible. We corroborate our critique in an empirical demonstration on a medical image classification task.

Original languageEnglish
Article number103462
JournalArtificial Intelligence in Medicine
Volume180
DOIs
Publication statusPublished - Oct 2026
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2026 The Authors.

Keywords

  • Calibration
  • Conformal prediction
  • Finite-sample guarantees
  • Medical AI
  • Uncertainty quantification

Cite this