|View source on GitHub|
A preprocessing layer which maps text features to integer sequences.
tf.keras.layers.TextVectorization( max_tokens=None, standardize='lower_and_strip_punctuation', split='whitespace', ngrams=None, output_mode='int', output_sequence_length=None, pad_to_max_tokens=False, vocabulary=None, idf_weights=None, sparse=False, ragged=False, **kwargs )
Used in the notebooks
|Used in the guide||Used in the tutorials|
This layer has basic options for managing text in a Keras model. It transforms a batch of strings (one example = one string) into either a list of token indices (one example = 1D tensor of integer token indices) or a dense representation (one example = 1D tensor of float values representing data about the example's tokens).
The vocabulary for the layer must be either supplied on construction or
adapt(). When this layer is adapted, it will analyze the
dataset, determine the frequency of individual string values, and create a
vocabulary from them. This vocabulary can have unlimited size or be capped,
depending on the configuration options for this layer; if there are more
unique values in the input than the maximum vocabulary size, the most frequent
terms will be used to create the vocabulary.
The processing of each example contains the following steps:
- Standardize each example (usually lowercasing + punctuation stripping)
- Split each example into substrings (usually words)
- Recombine substrings into tokens (usually ngrams)
- Index tokens (associate a unique int value with each token)
- Transform each example using this index, either into a vector of ints or a dense float vector.
Some notes on passing callables to customize splitting and normalization for this layer:
- Any callable can be passed to this Layer, but if you want to serialize
this object you should only pass functions that are registered Keras
tf.keras.utils.register_keras_serializablefor more details).
- When using a custom callable for
standardize, the data received by the callable will be exactly as passed to this layer. The callable should return a tensor of the same shape as the input.
- When using a custom callable for
split, the data received by the callable will have the 1st dimension squeezed out - instead of
[["string to split"], ["another string to split"]], the Callable will see
["string to split", "another string to split"]. The callable should return a Tensor with the first dimension containing the split tokens - in this example, we should see something like
[["string", "to", "split"], ["another", "string", "to", "split"]]. This makes the callable site natively compatible with
For an overview and full list of preprocessing layers, see the preprocessing guide.
Maximum size of the vocabulary for this layer. This should only
be specified when adapting a vocabulary or when setting
Optional specification for standardization to apply to the
input text. Values can be None (no standardization),
Optional specification for splitting the input text. Values can be
None (no splitting),
||Optional specification for ngrams to create from the possibly-split input text. Values can be None, an integer or tuple of integers; passing an integer will create ngrams up to that integer, and passing a tuple of integers will create ngrams for the specified values in the tuple. Passing None means that no ngrams will be created.|
Optional specification for the output of the layer. Values can
Only valid in INT mode. If set, the output will have
its time dimension padded or truncated to exactly
Only valid in
Optional. Either an array of strings or a string path to a text
file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D
tensor containing the string vocbulary terms. If passing a file path, the
file should contain one line per term in the vocabulary. If this argument
is set, there is no need to