|View source on GitHub|
Reindex integer inputs to be in a contiguous range, via a dict lookup.
tf.keras.layers.IntegerLookup( max_tokens=None, num_oov_indices=1, mask_token=None, oov_token=-1, vocabulary=None, invert=False, output_mode='int', sparse=False, pad_to_max_tokens=False, **kwargs )
Used in the notebooks
|Used in the guide||Used in the tutorials|
This layer maps a set of arbitrary integer input tokens into indexed
integer output via a table-based vocabulary lookup. The layer's output indices
will be contiguously arranged up to the maximum vocab size, even if the input
tokens are non-continguous or unbounded. The layer supports multiple options
for encoding the output via
output_mode, and has optional support for
out-of-vocabulary (OOV) tokens and masking.
The vocabulary for the layer can be supplied on construction or learned via
adapt(), the layer will analyze a data set, determine the
frequency of individual integer tokens, and create a vocabulary from them. If
the vocabulary is capped in size, the most frequent tokens will be used to
create the vocabulary and all others will be treated as OOV.
There are two possible output modes for the layer.
input integers are converted to their index in the vocabulary (an integer).
"tf_idf", input integers
are encoded into an array where each dimension corresponds to an element in
The vocabulary can optionally contain a mask token as well as an OOV token
(which can optionally occupy multiple indices in the vocabulary, as set
The position of these tokens in the vocabulary is fixed. When
"int", the vocabulary will begin with the mask token at index 0, followed by
OOV indices, followed by the rest of the vocabulary. When
"tf_idf" the vocabulary will begin with OOV
indices and instances of the mask token will be dropped.
||The maximum size of the vocabulary for this layer. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. Default to None.|
||The number of out-of-vocabulary tokens to use. If this value is more than 1, OOV inputs are modulated to determine their OOV value. If this value is 0, OOV inputs will cause an error when calling the layer. Defaults to 1.|
An integer token that represents masked inputs. When
Only used when
Optional. Either an array of integers or a string path to a text
file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D
tensor containing the integer vocbulary terms. If passing a file path, the
file should contain one line per term in the vocabulary. If this argument
is set, there is no need to
Only valid when
||Specification for the output of the lay|