Reindex integer inputs to be in a contiguous range, via a dict lookup.

Inherits From: PreprocessingLayer, Layer, Module

Used in the notebooks

Used in the guide Used in the tutorials

This layer maps a set of arbitrary integer input tokens into indexed integer output via a table-based vocabulary lookup. The layer's output indices will be contiguously arranged up to the maximum vocab size, even if the input tokens are non-continguous or unbounded. The layer supports multiple options for encoding the output via output_mode, and has optional support for out-of-vocabulary (OOV) tokens and masking.

The vocabulary for the layer can be supplied on construction or learned via adapt(). During adapt(), the layer will analyze a data set, determine the frequency of individual integer tokens, and create a vocabulary from them. If the vocabulary is capped in size, the most frequent tokens will be used to create the vocabulary and all others will be treated as OOV.

There are two possible output modes for the layer. When output_mode is "int", input integers are converted to their index in the vocabulary (an integer). When output_mode is "binary", "count", or "tf-idf", input integers are encoded into an array where each dimension corresponds to an element in the vocabulary.

The vocabulary can optionally contain a mask token as well as an OOV token (which can optionally occupy multiple indices in the vocabulary, as set by num_oov_indices). The position of these tokens in the vocabulary is fixed. When output_mode is "int", the vocabulary will begin with the mask token at index 0, followed by OOV indices, followed by the rest of the vocabulary. When output_mode is "binary", "count", or "tf-idf" the vocabulary will begin with OOV indices and instances of the mask token will be dropped.

max_tokens The maximum size of the vocabulary for this layer. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. Default to None.
num_oov_indices The number of out-of-vocabulary tokens to use. If this value is more than 1, OOV inputs are modulated to determine their OOV value. If this value is 0, OOV inputs will map to -1 when output_mode is "int" and are dropped otherwise. Defaults to 1.
mask_token An integer token that represents masked inputs. When output_mode is "int", the token is included in vocabulary and mapped to index 0. In other output modes, the token will not appear in the vocabulary and instances of the mask token in the input will be dropped. If set to None, no mask term will be added. Defaults to 0.
oov_token Only used when invert is True. The token to return for OOV indices. Defaults to -1.
vocabulary An optional list of integer tokens, or a path to a text file containing a vocabulary to load into this layer. The file should contain one integer token per line. If the list or file contains the same token multiple times, an error will be thrown.
invert Only valid when output_mode is "int". If True, this layer will map indices to vocabulary items instead of mapping vocabulary items to indices. Default to False.
output_mode Specification for the output of the layer. Defaults to "int". Values can be "int", "binary", "count", or "tf-idf" configuring the layer as follows: "int": Return the vocabulary indices of the input tokens. "binary": Outputs a single int array per sample, of either vocabulary size or max_tokens size, containing 1s in all elements where the token mapped to that index exists at least once in the sample. "count": Like "binary", but the int array contains a count of the number of times the token at that index appeared in the sample. "tf-idf": As "binary", but the TF-IDF algorithm is applied to find the value in each token slot.
pad_to_max_tokens Only applicable when output_mode is "binary", "count", or "tf-idf". If True, the output will have its feature axis padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens, resulting in a tensor of shape [batch_size, max_tokens] regardless of vocabulary size. Defaults to False.
sparse Boolean. Only applicable when output_mode is "binary", "count", or "tf-idf". If True, returns a SparseTensor instead of a dense Tensor. Defaults to False.


Creating a lookup layer with a known vocabulary

This example creates a lookup layer with a pre-existing vocabulary.

vocab = [12, 36, 1138, 42]
data = tf.constant([[12, 1138, 42], [42, 1000, 36]])  # Note OOV tokens
layer = IntegerLookup(vocabulary=vocab)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[2, 4, 5],
       [5, 1, 3]])>

Creating a lookup layer with an adapted vocabulary

This example creates a lookup layer and generates the vocabulary by analyzing the dataset.

data = tf.constant([[12, 1138, 42], [42, 1000, 36]])
layer = IntegerLookup()
[0, -1, 42,