ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more


Experimental API for building input pipelines.

This module contains experimental Dataset sources and transformations that can be used in conjunction with the API. Note that the API is not subject to the same backwards compatibility guarantees as, but we will provide deprecation advice in advance of removing existing functionality.

See Importing Data for an overview.


service module: API for using the service.


class AutoShardPolicy: Represents the type of auto-sharding to use.

class CheckpointInputPipelineHook: Checkpoints input pipeline state every N steps or seconds.

class CsvDataset: A Dataset comprising lines from one or more CSV files.

class DatasetInitializer: Creates a table initializer from a

class DistributeOptions: Represents options for distributed data processing.

class ExternalStatePolicy: Represents how to handle external state during serialization.

class OptimizationOptions: Represents options for dataset optimizations.

class Optional: Represents a value that may or may not be present.

class RandomDataset: A Dataset of pseudorandom values. (deprecated)

class Reducer: A reducer is used for reducing a set of elements.

class SqlDataset: A Dataset consisting of the results from a SQL query.

class TFRecordWriter: Writes a dataset to a TFRecord file. (deprecated)

class ThreadingOptions: Represents options for dataset threading.


Counter(...): Creates a Dataset that counts from start in steps of size step.

assert_cardinality(...): Asserts the cardinality of the input dataset.

bucket_by_sequence_length(...): A transformation that buckets elements in a Dataset by length. (deprecated)

cardinality(...): Returns the cardinality of dataset, if known.

choose_from_datasets(...): Creates a dataset that deterministically chooses elements from datasets.

copy_to_device(...): A transformation that copies dataset elements to the given target_device.

dense_to_ragged_batch(...): A transformation that batches ragged elements into tf.RaggedTensors.

dense_to_sparse_batch(...): A transformation that batches ragged elements into tf.sparse.SparseTensors.

enable_debug_mode(...): Enables debug mode for

enumerate_dataset(...): A transformation that enumerates the elements of a dataset. (deprecated)

from_variant(...): Constructs a dataset from the given variant and (nested) structure.

get_next_as_optional(...): Returns a tf.experimental.Optional with the next element of the iterator. (deprecated)

get_single_element(...): Returns the single element of the dataset as a nested structure of tensors. (deprecated)

get_structure(...): Returns the type signature for elements of the input dataset / iterator.