Join the SIG TFX-Addons community and help make TFX even better!

tfx.orchestration.experimental.core.service_jobs.ServiceJobManager

Interface for service job manager.

Service jobs are long-running jobs associated with a node or a pipeline that persist across executions (eg: worker pools, Tensorboard, etc). Service jobs should be started before the nodes that depend on them can be run.

Methods

ensure_node_services

View source

Ensures necessary service jobs are started and healthy for the node.

ensure_node_services will be called in the orchestration loop periodically and is expected to:

  • Start any service jobs required by the pipeline node.
  • Probe job health, handle failure and return appropriate status.

Note that this method will only be called if either is_pure_service_node or is_mixed_service_node return True for the node.

Args
pipeline_state A PipelineState object for an active pipeline.
node_id Id of the node to ensure services.

Returns
Status of the service job(s) for the node.

is_mixed_service_node

View source

Returns True if the given node has a mix of executor and service jobs.

Args
pipeline_state A PipelineState object for an active pipeline.
node_id Id of the node in the pipeline to be checked.

Returns
True if the node has a mix of executor and service jobs.

is_pure_service_node

View source

Returns True if the given node only has service job(s).

Args
pipeline_state A PipelineState object for an active pipeline.
node_id Id of the node in the pipeline to be checked.

Returns
True if the node only has service job(s).

stop_node_services

View source

Stops service jobs (if any) associated with the node.

Note that this method will only be called if either is_pure_service_node or is_mixed_service_node return True for the node.

Args
pipeline_state A PipelineState object for an active pipeline.
node_id Id of the node to stop services.