TFDS now supports the Croissant 🥐 format! Read the documentation to know more.

tfds.features.Text

FeatureConnector for text, encoding to integers with a TextEncoder.

Inherits From: Tensor, FeatureConnector

tfds.features.Text(
    encoder=None,
    encoder_config=None,
    *,
    doc: tfds.features.DocArg = None
)

Args
`encoder`	`tfds.deprecated.text.TextEncoder`, an encoder that can convert text to integers. If None, the text will be utf-8 byte-encoded.
`encoder_config`	`tfds.deprecated.text.TextEncoderConfig`, needed if restoring from a file with `load_metadata`.
`doc`	Documentation of this feature (e.g. description).

Attributes
`doc`
`dtype`	Return the dtype (or dict of dtype) of this FeatureConnector.
`encoder`
`np_dtype`
`numpy_dtype`
`shape`	Return the shape (or dict of shape) of this FeatureConnector.
`tf_dtype`
`vocab_size`

Methods

`catalog_documentation`

View source

catalog_documentation() -> List[CatalogFeatureDocumentation]

Returns the feature documentation to be shown in the catalog.

`cls_from_name`

View source

@classmethod
cls_from_name(
    python_class_name: str
) -> Type['FeatureConnector']

Returns the feature class for the given Python class.

`decode_batch_example`

View source

decode_batch_example(
    example_data
)

See base class for details.

`decode_example`

View source

decode_example(
    tfexample_data
)

See base class for details.

`decode_example_np`

View source

decode_example_np(
    example_data
)

Encode the feature dict into NumPy-compatible input.

Args
`example_data`	Value to convert to NumPy.

Returns
`np_data`	Data as NumPy-compatible type: either a Python primitive (bytes, int, etc) or a NumPy array.

`decode_ragged_example`

View source

decode_ragged_example(
    example_data
)

See base class for details.

`encode_example`

View source

encode_example(
    example_data
)

See base class for details.

`from_config`

View source

@classmethod
from_config(
    root_dir: str
) -> FeatureConnector

Reconstructs the FeatureConnector from the config file.

Usage:

features = FeatureConnector.from_config('path/to/dir')

Args
`root_dir`	Directory containing the features.json file.

Returns
The reconstructed feature instance.

`from_json`

View source

@classmethod
from_json(
    value: Json
) -> FeatureConnector

FeatureConnector factory.

This function should be called from the tfds.features.FeatureConnector base class. Subclass should implement the from_json_content.

Example:

feature = tfds.features.FeatureConnector.from_json(
    {'type': 'Image', 'content': {'shape': [32, 32, 3], 'dtype': 'uint8'} }
)
assert isinstance(feature, tfds.features.Image)

Args
`value`	`dict(type=, content=)` containing the feature to restore. Match dict returned by `to_json`.

Returns
The reconstructed FeatureConnector.

`from_json_content`

View source

@classmethod
from_json_content(
    value: Union[tfds.typing.Json, feature_pb2.TextFeature]
) -> 'Text'

FeatureConnector factory (to overwrite).

Subclasses should overwrite this method. This method is used when importing the feature connector from the config.

This function should not be called directly. FeatureConnector.from_json should be called instead.

See existing FeatureConnectors for implementation examples.

Args
`value`	FeatureConnector information represented as either Json or a Feature proto. The content must match what is returned by `to_json_content`.
`doc`	Documentation of this feature (e.g. description).

Returns
The reconstructed FeatureConnector.

`from_proto`

View source

@classmethod
from_proto(
    feature_proto: feature_pb2.Feature
) -> T

Instantiates a feature from its proto representation.

`get_serialized_info`

View source

get_serialized_info()

See base class for details.

`get_tensor_info`

View source

get_tensor_info() -> feature_lib.TensorInfo

See base class for details.

`get_tensor_spec`

View source

get_tensor_spec() -> TreeDict[tf.TensorSpec]

Returns the tf.TensorSpec of this feature (not the element spec!).

Note that the output of this method may not correspond to the element spec of the dataset. For example, currently this method does not support RaggedTensorSpec.

`ints2str`

View source

ints2str(
    int_values
)

Conversion list[int] => decoded string.

`load_metadata`

View source

load_metadata(
    data_dir, feature_name: str
) -> None

Restore the feature metadata from disk.

If a dataset is re-loaded and generated files exists on disk, this function will restore the feature metadata from the saved file.

Args
`data_dir`	path to the dataset folder to which save the info (ex: `~/datasets/cifar10/1.2.0/`)
`feature_name`	the name of the feature (from the FeaturesDict key)

`maybe_build_from_corpus`

View source

maybe_build_from_corpus(
    corpus_generator, **kwargs
)

Call SubwordTextEncoder.build_from_corpus is encoder_cls is such.

If self.encoder is None and self._encoder_cls is of type SubwordTextEncoder, the method instantiates self.encoder as returned by SubwordTextEncoder.build_from_corpus().

Args
`corpus_generator`	generator yielding `str`, from which subwords will be constructed.
`**kwargs`	kwargs forwarded to `SubwordTextEncoder.build_from_corpus()`

`maybe_set_encoder`

View source

maybe_set_encoder(
    new_encoder
)

Set encoder, but no-op if encoder is already set.

`repr_html`

View source

repr_html(
    ex: bytes
) -> str

Text are decoded.

`repr_html_batch`

View source

repr_html_batch(
    ex: np.ndarray
) -> str

Returns the HTML str representation of the object (Sequence).

`repr_html_ragged`

View source

repr_html_ragged(
    ex: np.ndarray
) -> str

Returns the HTML str representation of the object (Nested sequence).

`save_config`

View source

save_config(
    root_dir: str
) -> None

Exports the FeatureConnector to a file.

Args
`root_dir`	`path/to/dir` containing the `features.json`

`save_metadata`

View source

save_metadata(
    data_dir, feature_name: str
) -> None

Save the feature metadata on disk.

This function is called after the data has been generated (by _download_and_prepare) to save the feature connector info with the generated dataset.

Some dataset/features dynamically compute info during _download_and_prepare. For instance:

Labels are loaded from the downloaded data
Vocabulary is created from the downloaded data
ImageLabelFolder compute the image dtypes/shape from the manual_dir

After the info have been added to the feature, this function allow to save those additional info to be restored the next time the data is loaded.

By default, this function do not save anything, but sub-classes can overwrite the function.

Args
`data_dir`	path to the dataset folder to which save the info (ex: `~/datasets/cifar10/1.2.0/`)
`feature_name`	the name of the feature (from the FeaturesDict key)

`str2ints`

View source

str2ints(
    str_value
)

Conversion string => encoded list[int].

`to_json`

View source

to_json() -> Json

Exports the FeatureConnector to Json.

Each feature is serialized as a dict(type=..., content=...).

type: The cannonical name of the feature (module.FeatureName).
content: is specific to each feature connector and defined in to_json_content. Can contain nested sub-features (like for tfds.features.FeaturesDict and tfds.features.Sequence).

For example:

tfds.features.FeaturesDict({
    'input': tfds.features.Image(),
    'target': tfds.features.ClassLabel(num_classes=10),
})

Is serialized as:

{
    "type": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
    "content": {
        "input": {
            "type": "tensorflow_datasets.core.features.image_feature.Image",
            "content": {
                "shape": [null, null, 3],
                "dtype": "uint8",
                "encoding_format": "png"
            }
        },
        "target": {
            "type":
            "tensorflow_datasets.core.features.class_label_feature.ClassLabel",
            "content": {
              "num_classes": 10
            }
        }
    }
}

Returns
A `dict(type=, content=)`. Will be forwarded to `from_json` when reconstructing the feature.

`to_json_content`

View source

to_json_content() -> Union[tfds.typing.Json, feature_pb2.TextFeature]

FeatureConnector factory (to overwrite).

This function should be overwritten by the subclass to allow re-importing the feature connector from the config. See existing FeatureConnector for example of implementation.

Returns
The FeatureConnector metadata in either a dict, or a Feature proto. This output is used in `from_json_content` when reconstructing the feature.

`to_proto`

View source

to_proto() -> feature_pb2.Feature

Exports the FeatureConnector to the Feature proto.

For features that have a specific schema defined in a proto, this function needs to be overriden. If there's no specific proto schema, then the feature will be represented using JSON.

Returns
The feature proto describing this feature.

Class Variables
ALIASES	`['tensorflow_datasets.core.features.feature.Tensor']`

tfds.features.Text

Args

Attributes

Methods

catalog_documentation

cls_from_name

decode_batch_example

decode_example

decode_example_np

decode_ragged_example

encode_example

from_config

Usage:

from_json

Example:

from_json_content

from_proto

get_serialized_info

get_tensor_info

get_tensor_spec

ints2str

load_metadata

maybe_build_from_corpus

maybe_set_encoder

repr_html

repr_html_batch

repr_html_ragged

save_config

save_metadata

str2ints

to_json

For example:

Is serialized as:

to_json_content

to_proto

Class Variables

`catalog_documentation`

`cls_from_name`

`decode_batch_example`

`decode_example`

`decode_example_np`

`decode_ragged_example`

`encode_example`

`from_config`

`from_json`

`from_json_content`

`from_proto`

`get_serialized_info`

`get_tensor_info`

`get_tensor_spec`

`ints2str`

`load_metadata`

`maybe_build_from_corpus`

`maybe_set_encoder`

`repr_html`

`repr_html_batch`

`repr_html_ragged`

`save_config`

`save_metadata`

`str2ints`

`to_json`

`to_json_content`

`to_proto`