 View on TensorFlow.org
|
 Run in Google Colab
|
 View source on GitHub
|
 Download notebook
|
This text classification tutorial trains a recurrent neural network on the IMDB large movie review dataset for sentiment analysis.
Setup
!pip install -q -U tf-keras
import os
os.environ['TF_USE_LEGACY_KERAS'] = '1'
import numpy as np
import tensorflow_datasets as tfds
import tensorflow as tf
tfds.disable_progress_bar()
Import matplotlib and create a helper function to plot graphs:
import matplotlib.pyplot as plt
def plot_graphs(history, metric):
plt.plot(history.history[metric])
plt.plot(history.history['val_'+metric], '')
plt.xlabel("Epochs")
plt.ylabel(metric)
plt.legend([metric, 'val_'+metric])
Setup input pipeline
The IMDB large movie review dataset is a binary classification dataset—all the reviews have either a positive or negative sentiment.
Download the dataset using TFDS. See the loading text tutorial for details on how to load this sort of data manually.
dataset, info = tfds.load('imdb_reviews', with_info=True,
as_supervised=True)
train_dataset, test_dataset = dataset['train'], dataset['test']
train_dataset.element_spec
(TensorSpec(shape=(), dtype=tf.string, name=None), TensorSpec(shape=(), dtype=tf.int64, name=None))
Initially this returns a dataset of (text, label pairs):
for example, label in train_dataset.take(1):
print('text: ', example.numpy())
print('label: ', label.numpy())
text: b"This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like Christopher Walken's good name. I could barely sit through it." label: 0
Next shuffle the data for training and create batches of these (text, label) pairs:
BUFFER_SIZE = 10000
BATCH_SIZE = 64
train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
test_dataset = test_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
for example, label in train_dataset.take(1):
print('texts: ', example.numpy()[:3])
print()
print('labels: ', label.numpy()[:3])
texts: [b'America. A land of freedom, of hope and of dreams. This is the nation that, since its independence, has striven to bring democracy, prosperity, and peace to the entire world, for the good of all mankind. There are times, however, when one cannot help but wish that the American\'s would just stay on their side of the Atlantic. <br /><br />This \'movie\' (and I use that word with some reservations) evokes these feelings with an intense purity. This vision of hell follows the adventures of Calvin, a freakish jewel thief who was created by attaching the severed head of Marlon Wayan onto the body of a two foot-high dwarf. After inadvertently dropping a large diamond into the handbag of Vanessa, a career-woman who is reluctant to have children, Calvin realises that in order to recover the diamond he must ingratiate himself with her. So, as any normal man would, Calvin dresses himself up as a 2 year-old and parks himself upon the poor woman\'s doorstep, where he is discovered by Darryl, the broody husband of Vanessa. <br /><br />Darryl incongruously falls for Calvin\'s disguise despite the fact that the \'baby\' has a full set of teeth, stubble, a tattoo, a knife-scar, and the sex-drive of a 16-year-old. Even more absurdly, Vanessa doesn\'t see past Calvin\'s baby-wear either and actually attempts to breastfeed the diminutive pervert. This wretched assault upon the soul of mankind attempts, and fails, to find humour in rape, scatology, sexual assault, and paedophilia, however, in a dishonest attempt to transform itself into a piece of \'family-entertainment\' the Wayan brothers stir in a sickening amount of sentiment and flawed morality. <br /><br />The brothers dim attempt a Freudian rehabilitation of their thieving rapist by revealing that he "had a bad father". Repeatedly hitting Darryl in the crotch enables Calvin to develop the loving father-son relationship that both he and Darryl have always wished for. As if this wasn\'t ridiculous enough, Calvin\'s attempts to sexually assault Vanessa somehow convince her that it is selfish for a woman to indulge herself with a successful career, and that instead she should spend her life playing the role of the housebound little-woman, who spends her time alternatively squeezing out babies and cooking for her husband. <br /><br />In this movie the Wayan brothers have mixed their crass and twisted form of humour together with the clich\xc3\xa9d sentimentality that has infected much of Hollywood\'s recent body of work. Additionally, they are endemic of the current generation of black comedians who are responsible for transforming African-American humour into a poor and wretched shadow of itself that over-indulges in fart-jokes and crude sexual gags. By rights these two should be legally barred from picking up anything even remotely resembling a camera ever again. <br /><br />Unfortunately the current artistic and moral bankruptcy of American cinema means that by this time next month they will undoubtedly have filmed two sequels and be making millions of dollars from tacky merchandising deals.' b'This film has a special place in my heart, as when I caught it the first time, I was teaching adult literacy. It rang very true to me and even an outstanding student I had at the time. There are scenes which make you gulp with sudden emotion, and those which even put a smile on your face through sheer identification with the characters and their situation. <br /><br />Excellent performances by Jane Fonda and Robert DeNiro that rank with their best work, a great turn by a young Martha Plimpton, an inspiring story line, and a haunting musical score makes for a most enjoyable and rewarding experience.' b'There is a scene in Dan in Real Life where the family is competing to see which sex can finish the crossword puzzle first. The answer to one of the clues is Murphy\'s Law: anything that can go wrong, will go wrong. This is exactly the case for Dan Burns (Steve Carell, the Office) a columnist for the local newspaper. Dan is an expert at giving advice for everyday life, yet he comes to realize that things aren\'t so picture perfect in his own. Dan in Real Life is amazing at capturing these ironies of everyday life and is successful at embracing the comedy, tragedy, and beauty of them all. Besides that this movie is pretty damn hilarious.<br /><br />The death of his wife forces Dan to raise his three daughters all on his own... each daughter in their own pivotal stages in life: the first one anxious to try out her drivers license, the middle one well into her teenage angst phase, and the youngest one drifting away from early childhood. Things take a turn for Dan when he goes to Rhode Island for a family reunion and stumbles across an intriguing woman in a bookstore.<br /><br />Her name is Marie (Juliette Binoche, Chocolat) and she is looking for a book to help her avoid awkward situations... which is precisely whats in store when they get thrown into the Burns Family household.<br /><br />If you\'ve seen Steve Carell in The Office or Little Miss Sunshine, you\'d know that he is incomparable with comedic timing and a tremendously dynamic actor as well. Steve Carell is awesome at capturing all the emotions that come with family life: the frustration and sincere compassion. The family as well as the house itself provides a warm environment for the movie that contrasts the inner turmoil that builds throughout the movie and finally bursts out in a pretty suspenseful climax. The movie only falls short in some of the predictable outcomes, yet at the same time life is made up of both irony and predictability: which is an irony within itself.<br /><br />Dan in Real Life is definitely worth seeing, for the sole enjoyment of watching all the funny subtleties we often miss in everyday life, and I\'ll most likely enjoy it a second time, or even a third. Just "put it on my tab."'] labels: [0 1 1]
Create the text encoder
The raw text loaded by tfds needs to be processed before it can be used in a model. The simplest way to process text for training is using the TextVectorization layer. This layer has many capabilities, but this tutorial sticks to the default behavior.
Create the layer, and pass the dataset's text to the layer's .adapt method:
VOCAB_SIZE = 1000
encoder = tf.keras.layers.TextVectorization(
max_tokens=VOCAB_SIZE)
encoder.adapt(train_dataset.map(lambda text, label: text))
The .adapt method sets the layer's vocabulary. Here are the first 20 tokens. After the padding and unknown tokens they're sorted by frequency:
vocab = np.array(encoder.get_vocabulary())
vocab[:20]
array(['', '[UNK]', 'the', 'and', 'a', 'of', 'to', 'is', 'in', 'it', 'i',
'this', 'that', 'br', 'was', 'as', 'for', 'with', 'movie', 'but'],
dtype='<U14')
Once the vocabulary is set, the layer can encode text into indices. The tensors of indices are 0-padded to the longest sequence in the batch (unless you set a fixed output_sequence_length):
encoded_example = encoder(example)[:3].numpy()
encoded_example
array([[939, 4, 1, ..., 0, 0, 0],
[ 11, 20, 44, ..., 0, 0, 0],
[ 48, 7, 4, ..., 0, 0, 0]], shape=(3, 706))
With the default settings, the process is not completely reversible. There are three main reasons for that:
- The default value for
preprocessing.TextVectorization'sstandardizeargument is"lower_and_strip_punctuation". - The limited vocabulary size and lack of character-based fallback results in some unknown tokens.
for n in range(3):
print("Original: ", example[n].numpy())
print("Round-trip: ", " ".join(vocab[encoded_example[n]]))
print()
Original: b'America. A land of freedom, of hope and of dreams. This is the nation that, since its independence, has striven to bring democracy, prosperity, and peace to the entire world, for the good of all mankind. There are times, however, when one cannot help but wish that the American\'s would just stay on their side of the Atlantic. <br /><br />This \'movie\' (and I use that word with some reservations) evokes these feelings with an intense purity. This vision of hell follows the adventures of Calvin, a freakish jewel thief who was created by attaching the severed head of Marlon Wayan onto the body of a two foot-high dwarf. After inadvertently dropping a large diamond into the handbag of Vanessa, a career-woman who is reluctant to have children, Calvin realises that in order to recover the diamond he must ingratiate himself with her. So, as any normal man would, Calvin dresses himself up as a 2 year-old and parks himself upon the poor woman\'s doorstep, where he is discovered by Darryl, the broody husband of Vanessa. <br /><br />Darryl incongruously falls for Calvin\'s disguise despite the fact that the \'baby\' has a full set of teeth, stubble, a tattoo, a knife-scar, and the sex-drive of a 16-year-old. Even more absurdly, Vanessa doesn\'t see past Calvin\'s baby-wear either and actually attempts to breastfeed the diminutive pervert. This wretched assault upon the soul of mankind attempts, and fails, to find humour in rape, scatology, sexual assault, and paedophilia, however, in a dishonest attempt to transform itself into a piece of \'family-entertainment\' the Wayan brothers stir in a sickening amount of sentiment and flawed morality. <br /><br />The brothers dim attempt a Freudian rehabilitation of their thieving rapist by revealing that he "had a bad father". Repeatedly hitting Darryl in the crotch enables Calvin to develop the loving father-son relationship that both he and Darryl have always wished for. As if this wasn\'t ridiculous enough, Calvin\'s attempts to sexually assault Vanessa somehow convince her that it is selfish for a woman to indulge herself with a successful career, and that instead she should spend her life playing the role of the housebound little-woman, who spends her time alternatively squeezing out babies and cooking for her husband. <br /><br />In this movie the Wayan brothers have mixed their crass and twisted form of humour together with the clich\xc3\xa9d sentimentality that has infected much of Hollywood\'s recent body of work. Additionally, they are endemic of the current generation of black comedians who are responsible for transforming African-American humour into a poor and wretched shadow of itself that over-indulges in fart-jokes and crude sexual gags. By rights these two should be legally barred from picking up anything even remotely resembling a camera ever again. <br /><br />Unfortunately the current artistic and moral bankruptcy of American cinema means that by this time next month they will undoubtedly have filmed two sequels and be making millions of dollars from tacky merchandising deals.' Round-trip: america a [UNK] of [UNK] of hope and of [UNK] this is the [UNK] that since its [UNK] has [UNK] to bring [UNK] [UNK] and [UNK] to the entire world for the good of all [UNK] there are times however when one cannot help but wish that the [UNK] would just stay on their side of the [UNK] br br this movie and i use that word with some [UNK] [UNK] these [UNK] with an [UNK] [UNK] this [UNK] of hell [UNK] the [UNK] of [UNK] a [UNK] [UNK] [UNK] who was [UNK] by [UNK] the [UNK] head of [UNK] [UNK] [UNK] the body of a two [UNK] [UNK] after [UNK] [UNK] a [UNK] [UNK] into the [UNK] of [UNK] a [UNK] who is [UNK] to have children [UNK] [UNK] that in order to [UNK] the [UNK] he must [UNK] himself with her so as any [UNK] man would [UNK] [UNK] himself up as a 2 [UNK] and [UNK] himself upon the poor [UNK] [UNK] where he is [UNK] by [UNK] the [UNK] husband of [UNK] br br [UNK] [UNK] falls for [UNK] [UNK] despite the fact that the baby has a full set of [UNK] [UNK] a [UNK] a [UNK] and the [UNK] of a [UNK] even more [UNK] [UNK] doesnt see past [UNK] [UNK] either and actually attempts to [UNK] the [UNK] [UNK] this [UNK] [UNK] upon the [UNK] of [UNK] attempts and fails to find [UNK] in [UNK] [UNK] sexual [UNK] and [UNK] however in a [UNK] attempt to [UNK] itself into a piece of [UNK] the [UNK] brothers [UNK] in a [UNK] [UNK] of [UNK] and [UNK] [UNK] br br the brothers [UNK] attempt a [UNK] [UNK] of their [UNK] [UNK] by [UNK] that he had a bad father [UNK] [UNK] [UNK] in the [UNK] [UNK] [UNK] to [UNK] the [UNK] [UNK] relationship that both he and [UNK] have always [UNK] for as if this wasnt ridiculous enough [UNK] attempts to [UNK] [UNK] [UNK] somehow [UNK] her that it is [UNK] for a woman to [UNK] herself with a [UNK] career and that instead she should [UNK] her life playing the role of the [UNK] [UNK] who [UNK] her time [UNK] [UNK] out [UNK] and [UNK] for her husband br br in this movie the [UNK] brothers have [UNK] their [UNK] and [UNK] form of [UNK] together with the [UNK] [UNK] that has [UNK] much of [UNK] [UNK] body of work [UNK] they are [UNK] of the [UNK] [UNK] of black [UNK] who are [UNK] for [UNK] [UNK] [UNK] into a poor and [UNK] [UNK] of itself that [UNK] in [UNK] and [UNK] sexual [UNK] by [UNK] these two should be [UNK] [UNK] from [UNK] up anything even [UNK] [UNK] a camera ever again br br unfortunately the [UNK] [UNK] and [UNK] [UNK] of american cinema means that by this time next [UNK] they will [UNK] have filmed two [UNK] and be making [UNK] of [UNK] from [UNK] [UNK] [UNK] Original: b'This film has a special place in my heart, as when I caught it the first time, I was teaching adult literacy. It rang very true to me and even an outstanding student I had at the time. There are scenes which make you gulp with sudden emotion, and those which even put a smile on your face through sheer identification with the characters and their situation. <br /><br />Excellent performances by Jane Fonda and Robert DeNiro that rank with their best work, a great turn by a young Martha Plimpton, an inspiring story line, and a haunting musical score makes for a most enjoyable and rewarding experience.' Round-trip: this film has a special place in my heart as when i [UNK] it the first time i was [UNK] [UNK] [UNK] it [UNK] very true to me and even an [UNK] [UNK] i had at the time there are scenes which make you [UNK] with [UNK] [UNK] and those which even put a [UNK] on your face through [UNK] [UNK] with the characters and their situation br br excellent performances by jane [UNK] and robert [UNK] that [UNK] with their best work a great turn by a young [UNK] [UNK] an [UNK] story line and a [UNK] musical score makes for a most enjoyable and [UNK] experience Original: b'There is a scene in Dan in Real Life where the family is competing to see which sex can finish the crossword puzzle first. The answer to one of the clues is Murphy\'s Law: anything that can go wrong, will go wrong. This is exactly the case for Dan Burns (Steve Carell, the Office) a columnist for the local newspaper. Dan is an expert at giving advice for everyday life, yet he comes to realize that things aren\'t so picture perfect in his own. Dan in Real Life is amazing at capturing these ironies of everyday life and is successful at embracing the comedy, tragedy, and beauty of them all. Besides that this movie is pretty damn hilarious.<br /><br />The death of his wife forces Dan to raise his three daughters all on his own... each daughter in their own pivotal stages in life: the first one anxious to try out her drivers license, the middle one well into her teenage angst phase, and the youngest one drifting away from early childhood. Things take a turn for Dan when he goes to Rhode Island for a family reunion and stumbles across an intriguing woman in a bookstore.<br /><br />Her name is Marie (Juliette Binoche, Chocolat) and she is looking for a book to help her avoid awkward situations... which is precisely whats in store when they get thrown into the Burns Family household.<br /><br />If you\'ve seen Steve Carell in The Office or Little Miss Sunshine, you\'d know that he is incomparable with comedic timing and a tremendously dynamic actor as well. Steve Carell is awesome at capturing all the emotions that come with family life: the frustration and sincere compassion. The family as well as the house itself provides a warm environment for the movie that contrasts the inner turmoil that builds throughout the movie and finally bursts out in a pretty suspenseful climax. The movie only falls short in some of the predictable outcomes, yet at the same time life is made up of both irony and predictability: which is an irony within itself.<br /><br />Dan in Real Life is definitely worth seeing, for the sole enjoyment of watching all the funny subtleties we often miss in everyday life, and I\'ll most likely enjoy it a second time, or even a third. Just "put it on my tab."' Round-trip: there is a scene in [UNK] in real life where the family is [UNK] to see which sex can [UNK] the [UNK] [UNK] first the [UNK] to one of the [UNK] is [UNK] [UNK] anything that can go wrong will go wrong this is exactly the case for [UNK] [UNK] [UNK] [UNK] the [UNK] a [UNK] for the local [UNK] [UNK] is an [UNK] at giving [UNK] for [UNK] life yet he comes to realize that things arent so picture perfect in his own [UNK] in real life is amazing at [UNK] these [UNK] of [UNK] life and is [UNK] at [UNK] the comedy [UNK] and beauty of them all [UNK] that this movie is pretty [UNK] [UNK] br the death of his wife [UNK] [UNK] to [UNK] his three [UNK] all on his own each daughter in their own [UNK] [UNK] in life the first one [UNK] to try out her [UNK] [UNK] the middle one well into her [UNK] [UNK] [UNK] and the [UNK] one [UNK] away from early [UNK] things take a turn for [UNK] when he goes to [UNK] [UNK] for a family [UNK] and [UNK] across an [UNK] woman in a [UNK] br her name is [UNK] [UNK] [UNK] [UNK] and she is looking for a book to help her avoid [UNK] [UNK] which is [UNK] whats in [UNK] when they get [UNK] into the [UNK] family [UNK] br if youve seen [UNK] [UNK] in the [UNK] or little miss [UNK] [UNK] know that he is [UNK] with [UNK] [UNK] and a [UNK] [UNK] actor as well [UNK] [UNK] is [UNK] at [UNK] all the [UNK] that come with family life the [UNK] and [UNK] [UNK] the family as well as the house itself [UNK] a [UNK] [UNK] for the movie that [UNK] the [UNK] [UNK] that [UNK] throughout the movie and finally [UNK] out in a pretty [UNK] [UNK] the movie only falls short in some of the predictable [UNK] yet at the same time life is made up of both [UNK] and [UNK] which is an [UNK] within [UNK] br [UNK] in real life is definitely worth seeing for the [UNK] [UNK] of watching all the funny [UNK] we often miss in [UNK] life and ill most [UNK] enjoy it a second time or even a third just put it on my [UNK]
Create the model
Above is a diagram of the model.
This model can be build as a
tf.keras.Sequential.The first layer is the
encoder, which converts the text to a sequence of token indices.After the encoder is an embedding layer. An embedding layer stores one vector per word. When called, it converts the sequences of word indices to sequences of vectors. These vectors are trainable. After training (on enough data), words with similar meanings often have similar vectors.
This index-lookup is much more efficient than the equivalent operation of passing a one-hot encoded vector through a
tf.keras.layers.Denselayer.A recurrent neural network (RNN) processes sequence input by iterating through the elements. RNNs pass the outputs from one timestep to their input on the next timestep.
The
tf.keras.layers.Bidirectionalwrapper can also be used with an RNN layer. This propagates the input forward and backwards through the RNN layer and then concatenates the final output.The main advantage of a bidirectional RNN is that the signal from the beginning of the input doesn't need to be processed all the way through every timestep to affect the output.
The main disadvantage of a bidirectional RNN is that you can't efficiently stream predictions as words are being added to the end.
After the RNN has converted the sequence to a single vector the two
layers.Densedo some final processing, and convert from this vector representation to a single logit as the classification output.
The code to implement this is below:
model = tf.keras.Sequential([
encoder,
tf.keras.layers.Embedding(
input_dim=len(encoder.get_vocabulary()),
output_dim=64,
# Use masking to handle the variable sequence lengths
mask_zero=True),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1)
])
Please note that Keras sequential model is used here since all the layers in the model only have single input and produce single output. In case you want to use stateful RNN layer, you might want to build your model with Keras functional API or model subclassing so that you can retrieve and reuse the RNN layer states. Please check Keras RNN guide for more details.
The embedding layer uses masking to handle the varying sequence-lengths. All the layers after the Embedding support masking:
print([layer.supports_masking for layer in model.layers])
[False, True, True, True, True]
To confirm that this works as expected, evaluate a sentence twice. First, alone so there's no padding to mask:
# predict on a sample text without padding.
sample_text = ('The movie was cool. The animation and the graphics '
'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))
print(predictions[0])
1/1 [==============================] - 3s 3s/step [-0.00213648]
Now, evaluate it again in a batch with a longer sentence. The result should be identical:
# predict on a sample text with padding
padding = "the " * 2000
predictions = model.predict(np.array([sample_text, padding]))
print(predictions[0])
1/1 [==============================] - 0s 97ms/step [-0.00213648]
Compile the Keras model to configure the training process:
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(1e-4),
metrics=['accuracy'])
Train the model
history = model.fit(train_dataset, epochs=10,
validation_data=test_dataset,
validation_steps=30)
Epoch 1/10
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1793451621.376103 79145 type_inference.cc:340] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
goo.gle/debugstr
type_id: TFT_OPTIONAL
args {
type_id: TFT_PRODUCT
args {
type_id: TFT_TENSOR
args {
type_id: TFT_INT32
}
}
}
is neither a subtype nor a supertype of the combined inputs preceding it:
goo.gle/debugstr
type_id: TFT_OPTIONAL
args {
type_id: TFT_PRODUCT
args {
type_id: TFT_TENSOR
args {
type_id: TFT_FLOAT
}
}
}
for Tuple type infernce function 0
while inferring type of node 'cond_36/output/_23'
391/391 [==============================] - 44s 90ms/step - loss: 0.6570 - accuracy: 0.5431 - val_loss: 0.5542 - val_accuracy: 0.6927
Epoch 2/10
391/391 [==============================] - 23s 58ms/step - loss: 0.4562 - accuracy: 0.7741 - val_loss: 0.4157 - val_accuracy: 0.8167
Epoch 3/10
391/391 [==============================] - 22s 55ms/step - loss: 0.3763 - accuracy: 0.8332 - val_loss: 0.3694 - val_accuracy: 0.8422
Epoch 4/10
391/391 [==============================] - 22s 56ms/step - loss: 0.3394 - accuracy: 0.8536 - val_loss: 0.3466 - val_accuracy: 0.8438
Epoch 5/10
391/391 [==============================] - 22s 55ms/step - loss: 0.3219 - accuracy: 0.8605 - val_loss: 0.3305 - val_accuracy: 0.8552
Epoch 6/10
391/391 [==============================] - 23s 57ms/step - loss: 0.3127 - accuracy: 0.8666 - val_loss: 0.3255 - val_accuracy: 0.8625
Epoch 7/10
391/391 [==============================] - 23s 56ms/step - loss: 0.3081 - accuracy: 0.8694 - val_loss: 0.3281 - val_accuracy: 0.8536
Epoch 8/10
391/391 [==============================] - 22s 54ms/step - loss: 0.3026 - accuracy: 0.8718 - val_loss: 0.3228 - val_accuracy: 0.8620
Epoch 9/10
391/391 [==============================] - 22s 55ms/step - loss: 0.3001 - accuracy: 0.8715 - val_loss: 0.3306 - val_accuracy: 0.8474
Epoch 10/10
391/391 [==============================] - 22s 55ms/step - loss: 0.2981 - accuracy: 0.8731 - val_loss: 0.3220 - val_accuracy: 0.8552
test_loss, test_acc = model.evaluate(test_dataset)
print('Test Loss:', test_loss)
print('Test Accuracy:', test_acc)
391/391 [==============================] - 10s 25ms/step - loss: 0.3182 - accuracy: 0.8590 Test Loss: 0.3181871771812439 Test Accuracy: 0.8590400218963623
plt.figure(figsize=(16, 8))
plt.subplot(1, 2, 1)
plot_graphs(history, 'accuracy')
plt.ylim(None, 1)
plt.subplot(1, 2, 2)
plot_graphs(history, 'loss')
plt.ylim(0, None)
(0.0, 0.6749902337789535)
Run a prediction on a new sentence:
If the prediction is >= 0.0, it is positive else it is negative.
sample_text = ('The movie was cool. The animation and the graphics '
'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))
1/1 [==============================] - 3s 3s/step
Stack two or more LSTM layers
Keras recurrent layers have two available modes that are controlled by the return_sequences constructor argument:
If
Falseit returns only the last output for each input sequence (a 2D tensor of shape (batch_size, output_features)). This is the default, used in the previous model.If
Truethe full sequences of successive outputs for each timestep is returned (a 3D tensor of shape(batch_size, timesteps, output_features)).
Here is what the flow of information looks like with return_sequences=True:
The interesting thing about using an RNN with return_sequences=True is that the output still has 3-axes, like the input, so it can be passed to another RNN layer, like this:
model = tf.keras.Sequential([
encoder,
tf.keras.layers.Embedding(len(encoder.get_vocabulary()), 64, mask_zero=True),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1)
])
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(1e-4),
metrics=['accuracy'])
history = model.fit(train_dataset, epochs=10,
validation_data=test_dataset,
validation_steps=30)
Epoch 1/10 391/391 [==============================] - 73s 145ms/step - loss: 0.6134 - accuracy: 0.6121 - val_loss: 0.4123 - val_accuracy: 0.8177 Epoch 2/10 391/391 [==============================] - 43s 108ms/step - loss: 0.3776 - accuracy: 0.8345 - val_loss: 0.3478 - val_accuracy: 0.8453 Epoch 3/10 391/391 [==============================] - 42s 105ms/step - loss: 0.3392 - accuracy: 0.8534 - val_loss: 0.3342 - val_accuracy: 0.8568 Epoch 4/10 391/391 [==============================] - 41s 104ms/step - loss: 0.3195 - accuracy: 0.8638 - val_loss: 0.3240 - val_accuracy: 0.8505 Epoch 5/10 391/391 [==============================] - 41s 104ms/step - loss: 0.3127 - accuracy: 0.8673 - val_loss: 0.3305 - val_accuracy: 0.8411 Epoch 6/10 391/391 [==============================] - 41s 104ms/step - loss: 0.3079 - accuracy: 0.8681 - val_loss: 0.3190 - val_accuracy: 0.8583 Epoch 7/10 391/391 [==============================] - 42s 105ms/step - loss: 0.3025 - accuracy: 0.8704 - val_loss: 0.3202 - val_accuracy: 0.8573 Epoch 8/10 391/391 [==============================] - 41s 104ms/step - loss: 0.2985 - accuracy: 0.8714 - val_loss: 0.3301 - val_accuracy: 0.8443 Epoch 9/10 391/391 [==============================] - 41s 104ms/step - loss: 0.2964 - accuracy: 0.8730 - val_loss: 0.3225 - val_accuracy: 0.8594 Epoch 10/10 391/391 [==============================] - 41s 104ms/step - loss: 0.2927 - accuracy: 0.8762 - val_loss: 0.3276 - val_accuracy: 0.8620
test_loss, test_acc = model.evaluate(test_dataset)
print('Test Loss:', test_loss)
print('Test Accuracy:', test_acc)
391/391 [==============================] - 18s 45ms/step - loss: 0.3235 - accuracy: 0.8635 Test Loss: 0.32354024052619934 Test Accuracy: 0.8635200262069702
# predict on a sample text without padding.
sample_text = ('The movie was not good. The animation and the graphics '
'were terrible. I would not recommend this movie.')
predictions = model.predict(np.array([sample_text]))
print(predictions)
1/1 [==============================] - 5s 5s/step [[-1.8561211]]
plt.figure(figsize=(16, 6))
plt.subplot(1, 2, 1)
plot_graphs(history, 'accuracy')
plt.subplot(1, 2, 2)
plot_graphs(history, 'loss')
Check out other existing recurrent layers such as GRU layers.
If you're interested in building custom RNNs, see the Keras RNN Guide.