tfx.orchestration.kubeflow.v2.step_builder.StepBuilder

Kubeflow pipelines task builder.

tfx.orchestration.kubeflow.v2.step_builder.StepBuilder(
    node: tfx.dsl.components.base.base_node.BaseNode,
    deployment_config: pipeline_pb2.PipelineDeploymentConfig,
    image: Optional[Text] = None,
    image_cmds: Optional[List[Text]] = None,
    beam_pipeline_args: Optional[List[Text]] = None,
    enable_cache: bool = False,
    pipeline_info: Optional[tfx.orchestration.data_types.PipelineInfo] = None,
    channel_redirect_map: Optional[Dict[Tuple[Text, Text], Text]] = None
)

Constructs a pipeline task spec based on the TFX node information. Meanwhile, augments the deployment config associated with the node.

Args
node	A TFX node. The logical unit of a step. Note, currently for resolver node we only support two types of resolver policies, including: 1) latest blessed model, and 2) latest model artifact.
deployment_config	The deployment config in Kubeflow IR to be populated.
image	TFX image used in the underlying container spec. Required if node is a TFX component.
image_cmds	Optional. If not specified the default ENTRYPOINT defined in the docker image will be used. Note: the commands here refers to the K8S container command, which maps to Docker entrypoint field. If one supplies command but no args are provided for the container, the container will be invoked with the provided command, ignoring the ENTRYPOINT and CMD defined in the Dockerfile. One can find more details regarding the difference between K8S and Docker conventions at https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#notes
beam_pipeline_args	Pipeline arguments for Beam powered Components.
enable_cache	If true, enables cache lookup for this pipeline step. Defaults to False.
pipeline_info	Optionally, the pipeline info associated with current pipeline. The pipeline context is required if the current node is a resolver. Defaults to None.
channel_redirect_map	Map from (producer component id, output key) to (new producer component id, output key). This is needed for cases where one DSL node is splitted into multiple tasks in pipeline API proto. For example, latest blessed model resolver.

Raises
ValueError	On the following two cases: The node being built is an instance of BaseComponent but image was not provided. The node being built is a Resolver but the associated pipeline info was not provided.

Methods

build

View source

build() -> List[pipeline_pb2.PipelineTaskSpec]

Builds a pipeline StepSpec given the node information.

Returns
A list of PipelineTaskSpec messages corresponding to the node. For most of the cases, the list contains a single element. The only exception is when compiling latest blessed model resolver. One DSL node will be split to two resolver specs to reflect the two-phased query execution.

Raises
NotImplementedError	When the node being built is an InfraValidator.