tfma.metrics.to_label_prediction_example_weight

Yields label, prediction, and example weights for use in calculations.

tfma.metrics.to_label_prediction_example_weight(
    inputs: tfma.metrics.StandardMetricInputs,
    eval_config: Optional[tfma.EvalConfig] = None,
    model_name: str = '',
    output_name: str = '',
    sub_key: Optional[tfma.metrics.SubKey] = None,
    aggregation_type: Optional[metric_types.AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    fractional_labels: bool = False,
    flatten: bool = True,
    squeeze: bool = True,
    allow_none: bool = False,
    require_single_example_weight: bool = False
) -> Iterator[Tuple[np.ndarray, np.ndarray, np.ndarray]]

Where applicable this function will perform model and output name lookups as well as any required class ID, top K, etc conversions. It will also apply prediction keys and label vocabularies given the necessary information is provided as part of the EvalConfig (or standard estimator based naming is used). The sparseness of labels will be inferred from the shapes of the labels and predictions (i.e. if the shapes are different then the labels will be assumed to be sparse).

If successful, the final output of calling this function will be a tuple of numpy arrays representing the label, prediction, and example weight respectively. Labels and predictions will be returned in the same shape provided (default behavior) unless (1) flatten is True in which case a series of values (one per class ID) will be returned with last dimension of size 1 or (2) a sub_key is used in which case the last dimension may be re-shaped to match the new number of outputs (1 for class_id or k, top_k for top k with aggregation).

Note that for top_k without aggregation, the non-top_k prediction values will be set to float('-inf'), but for top_k with aggregation the values will be truncated to only return the top k values.

Examples
default behavior # Binary classification
Input	labels=[1] predictions=[0.6]
Output	(np.array([1]), np.array([0.6]), np.array([1.0])) Multi-class classification w/ sparse labels
Input	labels=[2] predictions=[0.3, 0.6, 0.1]
Output	(np.array([2]), np.array([0.3, 0.6, 0.1]), np.array([1.0])) Multi-class / multi-label classification w/ dense labels
Input	labels=[0, 1, 1] predictions=[0.3, 0.6, 0.1]
Output	(np.array([0, 1, 1]), np.array([0.3, 0.6, 0.1]), np.array([1.0])) flatten=True # Multi-class classification w/ sparse labels
Input	labels=[2], predictions=[0.3, 0.6, 0.1]
Output	(np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0])) Multi-class/multi-label classification w/ dense labels
Input	labels=[0, 0, 1], predictions=[0.3, 0.6, 0.1]
Output	(np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0])) sub_key.class_id=[2] # Multi-class classification w/ sparse labels
Input	labels=[2] predictions=[0.3, 0.6, 0.1]
Output	(np.array([1]), np.array([0.1]), np.array([1.0])) Multi-class classification w/ dense labels
Input	labels=[0, 0, 1] predictions=[0.3, 0.6, 0.1]
Output	(np.array([1]), np.array([0.1]), np.array([1.0])) sub_key.top_k=2 and aggregation_type is None (i.e. binarization of top 2). # Multi-class classification w/ sparse labels
Input	labels=[2] predictions=[0.3, 0.6, 0.1]
Output	(np.array([0, 0, 1]), np.array([0.3, 0.6, -inf]), np.array([1.0])) Multi-class classification w/ dense labels
Input	labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6]
Output	(np.array([0, 0, 1]), np.array([0.3, -inf, 0.6]), np.array([1.0])) sub_key.top_k=2 and aggregation_type is not None (i.e. aggregate top 2). # Multi-class classification w/ sparse labels
Input	labels=[2] predictions=[0.3, 0.6, 0.1]
Output	(np.array([0, 1]), np.array([0.3, 0.6]), np.array([1.0])) Multi-class classification w/ dense labels
Input	labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6]
Output	(np.array([0, 0]), np.array([0.3, 0.6]), np.array([1.0])) sub_key.k=2 (i.e. binarization by choosing 2nd largest predicted value). # Multi-class classification w/ sparse labels
Input	labels=[0] predictions=[0.3, 0.6, 0.1]
Output	(np.array([1]), np.array([0.3]), np.array([1.0])) Multi-class classification w/ dense labels
Input	labels=[0] predictions=[0.3]
Output	(np.array([0]), np.array([0.3]), np.array([1.0]))

Args
inputs	Standard metric inputs.
eval_config	Eval config
model_name	Optional model name (if multi-model evaluation).
output_name	Optional output name (if multi-output model type).
sub_key	Optional sub key.
aggregation_type	Optional aggregation type.
class_weights	Optional class weights to apply to multi-class / multi-label labels and predictions. If used, flatten must also be True.
example_weighted	True if example weights should be applied.
fractional_labels	If true, each incoming tuple of (label, prediction, and example weight) will be split into two tuples as follows (where l, p, w represent the resulting label, prediction, and example weight values): (1) l = 0.0, p = prediction, and w = example_weight * (1.0 - label) (2) l = 1.0, p = prediction, and w = example_weight * label If enabled, an exception will be raised if labels are not within [0, 1]. The implementation is such that tuples associated with a weight of zero are not yielded. This means it is safe to enable fractional_labels even when the labels only take on the values of 0.0 or 1.0.
flatten	True to flatten the final label and prediction outputs so that the yielded values are always arrays of size 1. For example, multi-class / multi-label outputs would be converted into label and prediction pairs that could then be processed by a binary classification metric in order to compute a micro average over all classes. If the example weight is not a scalar, then they will be flattened as well, otherwise the same example weight value will be output for each pair of labels and predictions.
squeeze	True to squeeze any outputs that have rank > 1. This transforms outputs such as np.array([[1]]) to np.array([1]).
allow_none	True to allow labels or predictions with None values to be returned. When used, the values will be returned as empty np.ndarrays. The example weight will always be non-empty.
require_single_example_weight	True to require that the example_weight be a single value.

Yields
Tuple of (label, prediction, example_weight).