Join the SIG TFX-Addons community and help make TFX even better!

tfx.orchestration.kubeflow.v2.step_builder.StepBuilder

Kubeflow pipelines task builder.

tfx.orchestration.kubeflow.v2.step_builder.StepBuilder(
    node: tfx.dsl.components.base.base_node.BaseNode,
    deployment_config: pipeline_pb2.PipelineDeploymentConfig,
    image: Optional[Text] = None,
    image_cmds: Optional[List[Text]] = None,
    beam_pipeline_args: Optional[List[Text]] = None,
    enable_cache: bool = False,
    pipeline_info: Optional[tfx.orchestration.data_types.PipelineInfo] = None,
    channel_redirect_map: Optional[Dict[Tuple[Text, Text], Text]] = None
)

Constructs a pipeline task spec based on the TFX node information. Meanwhile, augments the deployment config associated with the node.

node A TFX node. The logical unit of a step. Note, currently for resolver node we only support two types of resolver policies, including: 1) latest blessed model, and 2) latest model artifact.
deployment_config The deployment config in Kubeflow IR to be populated.
image TFX image used in the underlying container spec. Required if node is a TFX component.
image_cmds Optional. If not specified the default ENTRYPOINT defined in the docker image will be used. Note: the commands here refers to the K8S container command, which maps to Docker entrypoint field. If one supplies command but no args are provided for the container, the container will be invoked with the provided command, ignoring the ENTRYPOINT and CMD defined in the Dockerfile. One can find more details regarding the difference between K8S and Docker conventions at https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#notes
beam_pipeline_args Pipeline arguments for Beam powered Components.
enable_cache If true, enables cache lookup for this pipeline step. Defaults to False.
pipeline_info Optionally, the pipeline info associated with current pipeline. The pipeline context is required if the current node is a resolver. Defaults to None.
channel_redirect_map Map from (producer component id, output key) to (new producer component id, output key). This is needed for cases where one DSL node is splitted into multiple tasks in pipeline API proto. For example, latest blessed model resolver.

ValueError On the following two cases:

  1. The node being built is an instance of BaseComponent but image was not provided.
  2. The node being built is a Resolver but the associated pipeline info was not provided.

Methods

build

View source

build() -> List[pipeline_pb2.PipelineTaskSpec]

Builds a pipeline StepSpec given the node information.

Returns
A list of PipelineTaskSpec messages corresponding to the node. For most of the cases, the list contains a single element. The only exception is when compiling latest blessed model resolver. One DSL node will be split to two resolver specs to reflect the two-phased query execution.

Raises
NotImplementedError When the node being built is an InfraValidator.