tfx.v1.orchestration.experimental.KubeflowV2DagRunner

Kubeflow V2 pipeline runner (currently for managed pipelines).

tfx.v1.orchestration.experimental.KubeflowV2DagRunner(
    config: tfx.v1.orchestration.experimental.KubeflowV2DagRunnerConfig,
    output_dir: Optional[str] = None,
    output_filename: Optional[str] = None
)

Used in the notebooks

Used in the tutorials

Builds a pipeline job spec in json format based on TFX pipeline DSL object.

config An KubeflowV2DagRunnerConfig object to specify runtime configuration when running the pipeline in Kubeflow.
output_dir An optional output directory into which to output the pipeline definition files. Defaults to the current working directory.
output_filename An optional output file name for the pipeline definition file. The file output format will be a JSON-serialized or YAML-serialized PipelineJob pb message. Defaults to 'pipeline.json'.

Methods

run

run(
    pipeline: tfx.v1.dsl.Pipeline,
    parameter_values: Optional[Dict[str, Any]] = None,
    write_out: Optional[bool] = True
) -> Dict[str, Any]

Compiles a pipeline DSL object into pipeline file.

Args
pipeline TFX pipeline object.
parameter_values mapping from runtime parameter names to its values.
write_out set to True to actually write out the file to the place designated by output_dir and output_filename. Otherwise return the JSON-serialized pipeline job spec.

Returns
Returns the JSON/YAML pipeline job spec.

Raises
RuntimeError if trying to write out to a place occupied by an existing file.

set_exit_handler

set_exit_handler(
    exit_handler: tfx.v1.types.BaseNode
)

Set exit handler components for the Kubeflow V2(Vertex AI) dag runner.

This feature is currently experimental without backward compatibility gaurantee.

Args
exit_handler exit handler component.