tfds.deprecated.text.TextEncoder
Stay organized with collections
Save and categorize content based on your preferences.
Abstract base class for converting between text and integers.
A note on padding:
Because text data is typically variable length and nearly always requires
padding during training, ID 0 is always reserved for padding. To accommodate
this, all TextEncoders behave in certain ways:
Attributes |
|---|
vocab_size
|
Size of the vocabulary. Decode produces ints [1, vocab_size).
|
Methods
decode
View source
@abc.abstractmethod
decode(
ids
)
Decodes a list of integers into text.
encode
View source
@abc.abstractmethod
encode(
s
)
Encodes text into a list of integers.
load_from_file
View source
@classmethod
@abc.abstractmethod
load_from_file(
filename_prefix
)
Load from file. Inverse of save_to_file.
save_to_file
View source
@abc.abstractmethod
save_to_file(
filename_prefix
)
Store to file. Inverse of load_from_file.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-04-26 UTC.
[null,null,["Last updated 2024-04-26 UTC."],[],[],null,["# tfds.deprecated.text.TextEncoder\n\n\u003cbr /\u003e\n\n|-----------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/datasets/blob/v4.9.3/tensorflow_datasets/core/deprecated/text/text_encoder.py#L61-L113) |\n\nAbstract base class for converting between text and integers.\n\n**A note on padding**:\n\nBecause text data is typically variable length and nearly always requires\npadding during training, ID 0 is always reserved for padding. To accommodate\nthis, all `TextEncoder`s behave in certain ways:\n\n- `encode`: never returns id 0 (all ids are 1+)\n- `decode`: drops 0 in the input ids\n- `vocab_size`: includes ID 0\n\n New subclasses should be careful to match this behavior.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Attributes ---------- ||\n|--------------|----------------------------------------------------------------|\n| `vocab_size` | Size of the vocabulary. Decode produces ints \\[1, vocab_size). |\n\n\u003cbr /\u003e\n\nMethods\n-------\n\n### `decode`\n\n[View source](https://github.com/tensorflow/datasets/blob/v4.9.3/tensorflow_datasets/core/deprecated/text/text_encoder.py#L82-L85) \n\n @abc.abstractmethod\n decode(\n ids\n )\n\nDecodes a list of integers into text.\n\n### `encode`\n\n[View source](https://github.com/tensorflow/datasets/blob/v4.9.3/tensorflow_datasets/core/deprecated/text/text_encoder.py#L77-L80) \n\n @abc.abstractmethod\n encode(\n s\n )\n\nEncodes text into a list of integers.\n\n### `load_from_file`\n\n[View source](https://github.com/tensorflow/datasets/blob/v4.9.3/tensorflow_datasets/core/deprecated/text/text_encoder.py#L97-L101) \n\n @classmethod\n @abc.abstractmethod\n load_from_file(\n filename_prefix\n )\n\nLoad from file. Inverse of save_to_file.\n\n### `save_to_file`\n\n[View source](https://github.com/tensorflow/datasets/blob/v4.9.3/tensorflow_datasets/core/deprecated/text/text_encoder.py#L92-L95) \n\n @abc.abstractmethod\n save_to_file(\n filename_prefix\n )\n\nStore to file. Inverse of load_from_file."]]
View source on GitHub