tfds.core.SplitInfo

Wraps proto.SplitInfo with an additional property.

tfds.core.SplitInfo(
    name: str,
    shard_lengths: list[int],
    num_bytes: int,
    filename_template: (naming.ShardedFileTemplate | None) = None,
    statistics: statistics_pb2.DatasetFeatureStatistics = dataclasses.field(default_factory=statistics_pb2.DatasetFeatureStatistics)
)

name Name of the split (e.g. train, test,...)
shard_lengths List of length containing the number of examples stored in each file.
filename_template The template used to create sharded filenames.
num_examples Total number of examples (sum(shard_lengths))
num_shards Number of files (len(shard_lengths))
num_bytes Size of the files (in bytes)
statistics Additional statistics of the split.
file_instructions Returns the list of dict(filename, take, skip).

This allows for creating your own tf.data.Dataset using the low-level TFDS values.

file_instructions = info.splits['train[75%:]'].file_instructions
instruction_ds = tf.data.Dataset.from_generator(
    lambda: file_instructions,
    output_types={
        'filename': tf.string,
        'take': tf.int64,
        'skip': tf.int64,
    },
)
ds = instruction_ds.interleave(
    lambda f: tf.data.TFRecordDataset(
        f['filename']).skip(f['skip']).take(f['take'])
)

When skip=0 and take=-1, the full shard will be read, so the ds.skip and ds.take could be skipped.

filenames Returns the list of filenames.
filepaths All the paths for all the files that are part of this split.

Methods

file_spec

View source

file_spec(
    file_format: (str | file_adapters.FileFormat)
) -> (str | None)

Returns the file spec of the split for the given file format.

A file spec is the full path with sharded notation, e.g., /data/ds/cfg/1.2.3/ds-train@10.

Args
file_format the file format for which to create the file spec for.

from_proto

View source

@classmethod
from_proto(
    proto: proto_lib.SplitInfo, filename_template: naming.ShardedFileTemplate
) -> SplitInfo

Returns a SplitInfo class instance from a SplitInfo proto.

get_available_shards

View source

get_available_shards(
    data_dir: (epath.Path | None) = None,
    file_format: (str | file_adapters.FileFormat | None) = None,
    strict_matching: bool = True
) -> list[epath.Path]

Returns the list of shards that are present in the data dir.

Args
data_dir The data directory to look for shards in. If not provided, the data directory from the filename template is used.
file_format The file format to look for shards in. If not provided, the file format from the filename template is used.
strict_matching If True, only shards that match the filename template exactly are returned taking into account the number of shards. Otherwise, shards that match the template with a wildcard for the shard number are returned.

replace

View source

replace(
    **kwargs
) -> SplitInfo

Returns a copy of the SplitInfo with updated attributes.

to_proto

View source

to_proto() -> proto_lib.SplitInfo

filename_template None