View source on GitHub
|
Decompose the Brier score into uncertainty, resolution, and reliability.
tfp.substrates.jax.stats.brier_decomposition(
labels, logits, name=None
)
[Proper scoring rules][1] measure the quality of probabilistic predictions;
any proper scoring rule admits a [unique decomposition][2] as
Score = Uncertainty - Resolution + Reliability, where:
Uncertainty, is a generalized entropy of the average predictive distribution; it can both be positive or negative.Resolution, is a generalized variance of individual predictive distributions; it is always non-negative. Difference in predictions reveal information, that is why a larger resolution improves the predictive score.Reliability, a measure of calibration of predictions against the true frequency of events. It is always non-negative and a lower value here indicates better calibration.
This method estimates the above decomposition for the case of the Brier
scoring rule for discrete outcomes. For this, we need to discretize the space
of probability distributions; we choose a simple partition of the space into
nlabels events: given a distribution p over nlabels outcomes, the index
k for which p_k > p_i for all i != k determines the discretization
outcome; that is, p in M_k, where M_k is the set of all distributions for
which p_k is the largest value among all probabilities.
The estimation error of each component is O(k/n), where n is the number
of instances and k is the number of labels. There may be an error of this
order when compared to brier_score.
References
[1]: Tilmann Gneiting, Adrian E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, Vol. 102, 2007. https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf [2]: Jochen Broecker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, Vol. 135, 2009. https://rmets.onlinelibrary.wiley.com/doi/epdf/10.1002/qj.456