tft_beam.Context

Context manager for tensorflow-transform.

tft_beam.Context(
    temp_dir: Optional[str] = None,
    desired_batch_size: Optional[int] = None,
    passthrough_keys: Optional[Iterable[str]] = None,
    use_deep_copy_optimization: Optional[bool] = None,
    force_tf_compat_v1: Optional[bool] = None,
    save_options: Optional[tf.saved_model.SaveOptions] = None
)

All the attributes in this context are kept on a thread local state.

Attributes

temp_dir: (Optional) The temporary directory used within in this block. desired_batch_size: (Optional) A batch size to batch elements by. If not provided, a batch size will be computed automatically. passthrough_keys: (Optional) A set of strings that are keys to instances that should pass through the pipeline and be hidden from the preprocessing_fn. This should only be used in cases where additional information should be attached to instances in the pipeline which should not be part of the transformation graph, instance keys is one such example. use_deep_copy_optimization: (Optional) If True, makes deep copies of PCollections that are used in multiple TFT phases. force_tf_compat_v1: (Optional) If True, TFT's public APIs (e.g. AnalyzeDataset) will use Tensorflow in compat.v1 mode irrespective of installed version of Tensorflow. Defaults to False. save_options: (Optional) If set, the tf.saved_model.SaveOptions to save the transform_fn with. Only applies for TF2.

Note that the temp dir should be accessible to worker jobs, e.g. if running with the Cloud Dataflow runner, the temp dir should be on GCS and should have permissions that allow both launcher and workers to access it.

Methods

create_base_temp_dir

View source

@classmethod
create_base_temp_dir() -> str

Generate a temporary location.

get_desired_batch_size

View source

@classmethod
get_desired_batch_size() -> Optional[int]

Retrieves a user set fixed batch size, None if not set.

get_passthrough_keys

View source

@classmethod
get_passthrough_keys() -> Iterable[str]

Retrieves a user set passthrough_keys, None if not set.

get_save_options

View source

@classmethod
get_save_options() -> Optional[tf.saved_model.SaveOptions]

Retrieves a user set save_options, None if not set.

get_use_deep_copy_optimization

View source

@classmethod
get_use_deep_copy_optimization() -> bool

Retrieves a user set use_deep_copy_optimization, None if not set.

get_use_tf_compat_v1

View source

@classmethod
get_use_tf_compat_v1() -> bool

Computes use_tf_compat_v1 from TF environment and force_tf_compat_v1.

__enter__

View source

__enter__()

__exit__

View source

__exit__(
    *exn_info
)