Join the SIG TFX-Addons community and help make TFX even better!

tfx.orchestration.kubeflow.kubeflow_dag_runner.KubeflowDagRunner

Kubeflow Pipelines runner.

Inherits From: TfxRunner

tfx.orchestration.kubeflow.kubeflow_dag_runner.KubeflowDagRunner(
    output_dir: Optional[Text] = None,
    output_filename: Optional[Text] = None,
    config: Optional[tfx.orchestration.kubeflow.kubeflow_dag_runner.KubeflowDagRunnerConfig] = None,
    pod_labels_to_attach: Optional[Dict[Text, Text]] = None
)

Constructs a pipeline definition YAML file based on the TFX logical pipeline.

output_dir An optional output directory into which to output the pipeline definition files. Defaults to the current working directory.
output_filename An optional output file name for the pipeline definition file. Defaults to pipeline_name.tar.gz when compiling a TFX pipeline. Currently supports .tar.gz, .tgz, .zip, .yaml, .yml formats. See https://github.com/kubeflow/pipelines/blob/181de66cf9fa87bcd0fe9291926790c400140783/sdk/python/kfp/compiler/compiler.py#L851 for format restriction.
config An optional KubeflowDagRunnerConfig object to specify runtime configuration when running the pipeline under Kubeflow.
pod_labels_to_attach Optional set of pod labels to attach to GKE pod spinned up for this pipeline. Default to the 3 labels:

  1. add-pod-env: true,
  2. pipeline SDK type,
  3. pipeline unique ID, where 2 and 3 are instrumentation of usage tracking.

Methods

run

View source

run(
    pipeline: tfx.orchestration.pipeline.Pipeline
)

Compiles and outputs a Kubeflow Pipeline YAML definition file.

Args
pipeline The logical TFX pipeline to use when building the Kubeflow pipeline.