tfm.nlp.models.Seq2SeqTransformer

Transformer model with Keras.

tfm.nlp.models.Seq2SeqTransformer(
    vocab_size=33708,
    embedding_width=512,
    dropout_rate=0.0,
    padded_decode=False,
    decode_max_length=None,
    extra_decode_length=0,
    beam_size=4,
    alpha=0.6,
    encoder_layer=None,
    decoder_layer=None,
    eos_id=EOS_ID,
    **kwargs
)

Implemented as described in: https://arxiv.org/pdf/1706.03762.pdf

The Transformer model consists of an encoder and decoder. The input is an int sequence (or a batch of sequences). The encoder produces a continuous representation, and the decoder uses the encoder output to generate probabilities for the output sequence.

Args
`vocab_size`	Size of vocabulary.
`embedding_width`	Size of hidden layer for embedding.
`dropout_rate`	Dropout probability.
`padded_decode`	Whether to max_sequence_length padding is used. If set False, max_sequence_length padding is not used.
`decode_max_length`	maximum number of steps to decode a sequence.
`extra_decode_length`	Beam search will run extra steps to decode.
`beam_size`	Number of beams for beam search
`alpha`	The strength of length normalization for beam search.
`encoder_layer`	An initialized encoder layer.
`decoder_layer`	An initialized decoder layer.
`eos_id`	Id of end of sentence token.
`**kwargs`	other keyword arguments.

Methods

`call`

View source

call(
    inputs
)

Calculate target logits or inferred target sequences.

Args

inputs a dictionary of tensors. Feature inputs (optional): int tensor with shape [batch_size, input_length]. Feature embedded_inputs (optional): float tensor with shape [batch_size, input_length, embedding_width]. Feature targets (optional): None or int tensor with shape [batch_size, target_length]. Feature input_masks (optional): When providing the embedded_inputs, the dictionary must provide a boolean mask marking the filled time steps. The shape of the tensor is [batch_size, input_length]. Either inputs or embedded_inputs and input_masks must be present in the input dictionary. In the second case the projection of the integer tokens to the transformer embedding space is skipped and input_masks is expected to be present.

Args
`inputs`	a dictionary of tensors. Feature `inputs` (optional): int tensor with shape `[batch_size, input_length]`. Feature `embedded_inputs` (optional): float tensor with shape `[batch_size, input_length, embedding_width]`. Feature `targets` (optional): None or int tensor with shape `[batch_size, target_length]`. Feature `input_masks` (optional): When providing the `embedded_inputs`, the dictionary must provide a boolean mask marking the filled time steps. The shape of the tensor is `[batch_size, input_length]`. Either `inputs` or `embedded_inputs` and `input_masks` must be present in the input dictionary. In the second case the projection of the integer tokens to the transformer embedding space is skipped and `input_masks` is expected to be present.

Returns
If targets is defined, then return logits for each word in the target sequence, which is a float tensor with shape `(batch_size, target_length, vocab_size)`. If target is `None`, then generate output sequence one token at a time and returns a dictionary { outputs: `(batch_size, decoded_length)` scores: `(batch_size, 1)`} Even when `float16` is used, the output tensor(s) are always `float32`.

Raises
`NotImplementedError`	If try to use padded decode method on CPU/GPUs.