tfma.ExtractEvaluateAndWriteResults

PTransform for performing extraction, evaluation, and writing results.

tfma.ExtractEvaluateAndWriteResults(
    eval_shared_model: Optional[tfma.types.EvalSharedModel] = None,
    eval_config: Optional[tfma.EvalConfig] = None,
    extractors: Optional[List[extractor.Extractor]] = None,
    evaluators: Optional[List[evaluator.Evaluator]] = None,
    writers: Optional[List[writer.Writer]] = None,
    output_path: Optional[str] = None,
    display_only_data_location: Optional[str] = None,
    display_only_file_format: Optional[str] = None,
    slice_spec: Optional[List[slicer.SingleSliceSpec]] = None,
    write_config: Optional[bool] = True,
    compute_confidence_intervals: Optional[bool] = False,
    min_slice_size: int = 1,
    random_seed_for_testing: Optional[int] = None,
    tensor_adapter_config: Optional[tensor_adapter.TensorAdapterConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    config_version: Optional[int] = None
) -> Dict[str, beam.PCollection]

Used in the notebooks

Used in the tutorials
FaceSSD Fairness Indicators Example Colab Introduction to Fairness Indicators TensorFlow Constrained Optimization Example Using CelebA Dataset Wiki Talk Comments Toxicity Prediction

Users who want to construct their own Beam pipelines instead of using the lightweight run_model_analysis functions should use this PTransform.

Example usage:

eval_config = tfma.EvalConfig(model_specs=[...], metrics_specs=[...],
                              slicing_specs=[...])
eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=model_location, eval_config=eval_config)
tfx_io = tf_example_record.TFExampleRecord(
    file_pattern=data_location,
    raw_record_column_name=tfma.ARROW_INPUT_COLUMN)
with beam.Pipeline(runner=...) as p:
  _ = (p
       | 'ReadData' >> tfx_io.BeamSource()
       | 'ExtractEvaluateAndWriteResults' >>
       tfma.ExtractEvaluateAndWriteResults(
           eval_shared_model=eval_shared_model,
           eval_config=eval_config,
           ...))
result = tfma.load_eval_result(output_path=output_path)
tfma.view.render_slicing_metrics(result)

Note: If running with an EvalSavedModel (i.e. the ModelSpec has signature_name
"eval"), then instead of using the tfxio.BeamSource() code use the following
beam.io.ReadFromTFRecord(data_location)

Note that the exact serialization format is an internal implementation detail and subject to change. Users should only use the TFMA functions to write and read the results.

Args
examples	PCollection of input examples or Arrow Record batches. Examples can be any format the model accepts (e.g. string containing CSV row, TensorFlow.Example, etc). If the examples are in the form of a dict it will be assumed that input is already in the form of tfma.Extracts with examples stored under tfma.INPUT_KEY (any other keys will be passed along unchanged to downstream extractors and evaluators).
eval_shared_model	Optional shared model (single-model evaluation) or list of shared models (multi-model evaluation). Only required if needed by default extractors, evaluators, or writers and for display purposes of the model path.
eval_config	Eval config.
extractors	Optional list of Extractors to apply to Extracts. Typically these will be added by calling the default_extractors function. If no extractors are provided, default_extractors (non-materialized) will be used.
evaluators	Optional list of Evaluators for evaluating Extracts. Typically these will be added by calling the default_evaluators function. If no evaluators are provided, default_evaluators will be used.
writers	Optional list of Writers for writing Evaluation output. Typically these will be added by calling the default_writers function. If no writers are provided, default_writers will be used.
output_path	Path to output results to (config file, metrics, plots, etc).
display_only_data_location	Optional path indicating where the examples were read from. This is used only for display purposes - data will not actually be read from this path.
display_only_file_format	Optional format of the examples. This is used only for display purposes.
slice_spec	Deprecated (use EvalConfig).
write_config	Deprecated (use EvalConfig).
compute_confidence_intervals	Deprecated (use EvalConfig).
min_slice_size	Deprecated (use EvalConfig).
random_seed_for_testing	Provide for deterministic tests only.
tensor_adapter_config	Tensor adapter config which specifies how to obtain tensors from the Arrow RecordBatch. If None, an attempt will be made to create the tensors using default TensorRepresentations.
schema	A schema to use for customizing evaluators.
config_version	Optional config version for this evaluation. This should not be explicitly set by users. It is only intended to be used in cases where the provided eval_config was generated internally, and thus not a reliable indicator of user intent.

Raises
ValueError	If EvalConfig invalid or matching Extractor not found for an Evaluator.

Returns
A dict of writer results keyed by the writer stage name.