 Lihat di TensorFlow.org |  Jalankan di Google Colab |  Lihat sumber di GitHub |  Unduh buku catatan |
Diberikan gambar seperti contoh di bawah ini, tujuan Anda adalah membuat teks seperti "seorang peselancar yang menunggangi ombak".
Sumber Gambar ; Lisensi: Domain Publik
Untuk mencapai ini, Anda akan menggunakan model berbasis perhatian, yang memungkinkan kami melihat bagian gambar mana yang menjadi fokus model saat menghasilkan keterangan.
Arsitektur modelnya mirip dengan Show, Attend and Tell: Neural Image Caption Generation dengan Visual Attention .
Notebook ini adalah contoh ujung ke ujung. Saat Anda menjalankan notebook, notebook mengunduh kumpulan data MS-COCO , mempraproses dan menyimpan subset gambar menggunakan Inception V3, melatih model encoder-decoder, dan menghasilkan teks pada gambar baru menggunakan model terlatih.
Dalam contoh ini, Anda akan melatih model pada jumlah data yang relatif kecil—30.000 teks pertama untuk sekitar 20.000 gambar (karena ada beberapa teks per gambar dalam kumpulan data).
import tensorflow as tf
# You'll generate plots of attention in order to see which parts of an image
# your model focuses on during captioning
import matplotlib.pyplot as plt
import collections
import random
import numpy as np
import os
import time
import json
from PIL import Image
Unduh dan siapkan kumpulan data MS-COCO
Anda akan menggunakan kumpulan data MS-COCO untuk melatih model Anda. Kumpulan data berisi lebih dari 82.000 gambar, yang masing-masing memiliki setidaknya 5 anotasi teks yang berbeda. Kode di bawah ini mengunduh dan mengekstrak dataset secara otomatis.
# Download caption annotation files
annotation_folder = '/annotations/'
if not os.path.exists(os.path.abspath('.') + annotation_folder):
annotation_zip = tf.keras.utils.get_file('captions.zip',
cache_subdir=os.path.abspath('.'),
origin='http://images.cocodataset.org/annotations/annotations_trainval2014.zip',
extract=True)
annotation_file = os.path.dirname(annotation_zip)+'/annotations/captions_train2014.json'
os.remove(annotation_zip)
# Download image files
image_folder = '/train2014/'
if not os.path.exists(os.path.abspath('.') + image_folder):
image_zip = tf.keras.utils.get_file('train2014.zip',
cache_subdir=os.path.abspath('.'),
origin='http://images.cocodataset.org/zips/train2014.zip',
extract=True)
PATH = os.path.dirname(image_zip) + image_folder
os.remove(image_zip)
else:
PATH = os.path.abspath('.') + image_folder
Downloading data from http://images.cocodataset.org/annotations/annotations_trainval2014.zip 252878848/252872794 [==============================] - 16s 0us/step 252887040/252872794 [==============================] - 16s 0us/step Downloading data from http://images.cocodataset.org/zips/train2014.zip 13510574080/13510573713 [==============================] - 784s 0us/step 13510582272/13510573713 [==============================] - 784s 0us/step
Opsional: batasi ukuran set pelatihan
Untuk mempercepat pelatihan untuk tutorial ini, Anda akan menggunakan subset dari 30.000 teks dan gambar yang sesuai untuk melatih model Anda. Memilih untuk menggunakan lebih banyak data akan menghasilkan peningkatan kualitas teks.
with open(annotation_file, 'r') as f:
annotations = json.load(f)
# Group all captions together having the same image ID.
image_path_to_caption = collections.defaultdict(list)
for val in annotations['annotations']:
caption = f"<start> {val['caption']} <end>"
image_path = PATH + 'COCO_train2014_' + '%012d.jpg' % (val['image_id'])
image_path_to_caption[image_path].append(caption)
image_paths = list(image_path_to_caption.keys())
random.shuffle(image_paths)
# Select the first 6000 image_paths from the shuffled set.
# Approximately each image id has 5 captions associated with it, so that will
# lead to 30,000 examples.
train_image_paths = image_paths[:6000]
print(len(train_image_paths))
6000
train_captions = []
img_name_vector = []
for image_path in train_image_paths:
caption_list = image_path_to_caption[image_path]
train_captions.extend(caption_list)
img_name_vector.extend([image_path] * len(caption_list))
print(train_captions[0])
Image.open(img_name_vector[0])
<start> a person trying to get a cat out of a suitcase <end>
Praproses gambar menggunakan InceptionV3
Selanjutnya, Anda akan menggunakan InceptionV3 (yang telah dilatih sebelumnya di Imagenet) untuk mengklasifikasikan setiap gambar. Anda akan mengekstrak fitur dari lapisan konvolusi terakhir.
Pertama, Anda akan mengonversi gambar menjadi format yang diharapkan dari InceptionV3 dengan:
- Mengubah ukuran gambar menjadi 299px kali 299px
- Praproses citra menggunakan metode preprocess_input untuk menormalkan citra sehingga mengandung piksel dalam rentang -1 hingga 1, yang sesuai dengan format citra yang digunakan untuk melatih InceptionV3.
def load_image(image_path):
img = tf.io.read_file(image_path)
img = tf.io.decode_jpeg(img, channels=3)
img = tf.keras.layers.Resizing(299, 299)(img)
img = tf.keras.applications.inception_v3.preprocess_input(img)
return img, image_path
Inisialisasi InceptionV3 dan muat bobot Imagenet yang telah dilatih sebelumnya
Sekarang Anda akan membuat model tf.keras di mana lapisan keluaran adalah lapisan konvolusi terakhir dalam arsitektur InceptionV3. Bentuk output dari layer ini adalah 8x8x2048 . Anda menggunakan lapisan konvolusi terakhir karena Anda menggunakan perhatian dalam contoh ini. Anda tidak melakukan inisialisasi ini selama pelatihan karena dapat menjadi hambatan.
- Anda meneruskan setiap gambar melalui jaringan dan menyimpan vektor yang dihasilkan dalam kamus (nama_gambar --> vektor_fitur).
- Setelah semua gambar melewati jaringan, Anda menyimpan kamus ke disk.
image_model = tf.keras.applications.InceptionV3(include_top=False,
weights='imagenet')
new_input = image_model.input
hidden_layer = image_model.layers[-1].output
image_features_extract_model = tf.keras.Model(new_input, hidden_layer)
Caching fitur yang diekstraksi dari InceptionV3
Anda akan melakukan pra-proses setiap gambar dengan InceptionV3 dan cache output ke disk. Caching output dalam RAM akan lebih cepat tetapi juga intensif memori, membutuhkan 8 * 8 * 2048 float per gambar. Pada saat penulisan, ini melebihi batasan memori Colab (saat ini memori 12 GB).
Performa dapat ditingkatkan dengan strategi caching yang lebih canggih (misalnya, dengan men-sharding gambar untuk mengurangi I/O disk akses acak), tetapi itu akan membutuhkan lebih banyak kode.
Caching akan memakan waktu sekitar 10 menit untuk berjalan di Colab dengan GPU. Jika Anda ingin melihat bilah kemajuan, Anda dapat:
Instal tqdm :
!pip install tqdmimpor tqm:
from tqdm import tqdmUbah baris berikut:
for img, path in image_dataset:ke:
for img, path in tqdm(image_dataset):
# Get unique images
encode_train = sorted(set(img_name_vector))
# Feel free to change batch_size according to your system configuration
image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)
image_dataset = image_dataset.map(
load_image, num_parallel_calls=tf.data.AUTOTUNE).batch(16)
for img, path in image_dataset:
batch_features = image_features_extract_model(img)
batch_features = tf.reshape(batch_features,
(batch_features.shape[0], -1, batch_features.shape[3]))
for bf, p in zip(batch_features, path):
path_of_feature = p.numpy().decode("utf-8")
np.save(path_of_feature, bf.numpy())
Praproses dan tokenize teks
Anda akan mengubah keterangan teks menjadi urutan bilangan bulat menggunakan lapisan TextVectorization , dengan langkah-langkah berikut:
- Gunakan adaptasi untuk mengulangi semua teks, pisahkan teks menjadi kata-kata, dan hitung kosakata dari 5.000 kata teratas (untuk menghemat memori).
- Token semua teks dengan memetakan setiap kata ke indeksnya dalam kosa kata. Semua urutan output akan diisi dengan panjang 50.
- Buat pemetaan kata-ke-indeks dan indeks-ke-kata untuk menampilkan hasil.
caption_dataset = tf.data.Dataset.from_tensor_slices(train_captions)
# We will override the default standardization of TextVectorization to preserve
# "<>" characters, so we preserve the tokens for the <start> and <end>.
def standardize(inputs):
inputs = tf.strings.lower(inputs)
return tf.strings.regex_replace(inputs,
r"!\"#$%&\(\)\*\+.,-/:;=?@\[\\\]^_`{|}~", "")
# Max word count for a caption.
max_length = 50
# Use the top 5000 words for a vocabulary.
vocabulary_size = 5000
tokenizer = tf.keras.layers.TextVectorization(
max_tokens=vocabulary_size,
standardize=standardize,
output_sequence_length=max_length)
# Learn the vocabulary from the caption data.
tokenizer.adapt(caption_dataset)
# Create the tokenized vectors
cap_vector = caption_dataset.map(lambda x: tokenizer(x))
# Create mappings for words to indices and indicies to words.
word_to_index = tf.keras.layers.StringLookup(
mask_token="",
vocabulary=tokenizer.get_vocabulary())
index_to_word = tf.keras.layers.StringLookup(
mask_token="",
vocabulary=tokenizer.get_vocabulary(),
invert=True)
Bagi data menjadi pelatihan dan pengujian
img_to_cap_vector = collections.defaultdict(list)
for img, cap in zip(img_name_vector, cap_vector):
img_to_cap_vector[img].append(cap)
# Create training and validation sets using an 80-20 split randomly.
img_keys = list(img_to_cap_vector.keys())
random.shuffle(img_keys)
slice_index = int(len(img_keys)*0.8)
img_name_train_keys, img_name_val_keys = img_keys[:slice_index], img_keys[slice_index:]
img_name_train = []
cap_train = []
for imgt in img_name_train_keys:
capt_len = len(img_to_cap_vector[imgt])
img_name_train.extend([imgt] * capt_len)
cap_train.extend(img_to_cap_vector[imgt])
img_name_val = []
cap_val = []
for imgv in img_name_val_keys:
capv_len = len(img_to_cap_vector[imgv])
img_name_val.extend([imgv] * capv_len)
cap_val.extend(img_to_cap_vector[imgv])
len(img_name_train), len(cap_train), len(img_name_val), len(cap_val)
(24012, 24012, 6004, 6004)
Buat kumpulan data tf.data untuk pelatihan
Gambar dan keterangan Anda sudah siap! Selanjutnya, mari buat kumpulan data tf.data yang akan digunakan untuk melatih model Anda.
# Feel free to change these parameters according to your system's configuration
BATCH_SIZE = 64
BUFFER_SIZE = 1000
embedding_dim = 256
units = 512
num_steps = len(img_name_train) // BATCH_SIZE
# Shape of the vector extracted from InceptionV3 is (64, 2048)
# These two variables represent that vector shape
features_shape = 2048
attention_features_shape = 64
# Load the numpy files
def map_func(img_name, cap):
img_tensor = np.load(img_name.decode('utf-8')+'.npy')
return img_tensor, cap
dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))
# Use map to load the numpy files in parallel
dataset = dataset.map(lambda item1, item2: tf.numpy_function(
map_func, [item1, item2], [tf.float32, tf.int64]),
num_parallel_calls=tf.data.AUTOTUNE)
# Shuffle and batch
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)
Model
Fakta menyenangkan: decoder di bawah ini identik dengan yang ada di contoh untuk Neural Machine Translation with Attention .
Arsitektur model terinspirasi oleh kertas Show, Attend and Tell .
- Dalam contoh ini, Anda mengekstrak fitur dari lapisan konvolusi bawah InceptionV3 yang memberi kita vektor bentuk (8, 8, 2048).
- Anda meremasnya menjadi bentuk (64, 2048).
- Vektor ini kemudian dilewatkan melalui CNN Encoder (yang terdiri dari satu lapisan yang sepenuhnya terhubung).
- RNN (di sini GRU) hadir di atas gambar untuk memprediksi kata berikutnya.
class BahdanauAttention(tf.keras.Model):
def __init__(self, units):
super(BahdanauAttention, self).__init__()
self.W1 = tf.keras.layers.Dense(units)
self.W2 = tf.keras.layers.Dense(units)
self.V = tf.keras.layers.Dense(1)
def call(self, features, hidden):
# features(CNN_encoder output) shape == (batch_size, 64, embedding_dim)
# hidden shape == (batch_size, hidden_size)
# hidden_with_time_axis shape == (batch_size, 1, hidden_size)
hidden_with_time_axis = tf.expand_dims(hidden, 1)
# attention_hidden_layer shape == (batch_size, 64, units)
attention_hidden_layer = (tf.nn.tanh(self.W1(features) +
self.W2(hidden_with_time_axis)))
# score shape == (batch_size, 64, 1)
# This gives you an unnormalized score for each image feature.
score = self.V(attention_hidden_layer)
# attention_weights shape == (batch_size, 64, 1)
attention_weights = tf.nn.softmax(score, axis=1)
# context_vector shape after sum == (batch_size, hidden_size)
context_vector = attention_weights * features
context_vector = tf.reduce_sum(context_vector, axis=1)
return context_vector, attention_weights
class CNN_Encoder(tf.keras.Model):
# Since you have already extracted the features and dumped it
# This encoder passes those features through a Fully connected layer
def __init__(self, embedding_dim):
super(CNN_Encoder, self).__init__()
# shape after fc == (batch_size, 64, embedding_dim)
self.fc = tf.keras.layers.Dense(embedding_dim)
def call(self, x):
x = self.fc(x)
x = tf.nn.relu(x)
return x
class RNN_Decoder(tf.keras.Model):
def __init__(self, embedding_dim, units, vocab_size):
super(RNN_Decoder, self).__init__()
self.units = units
self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
self.gru = tf.keras.layers.GRU(self.units,
return_sequences=True,
return_state=True,
recurrent_initializer='glorot_uniform')
self.fc1 = tf.keras.layers.Dense(self.units)
self.fc2 = tf.keras.layers.Dense(vocab_size)
self.attention = BahdanauAttention(self.units)
def call(self, x, features, hidden):
# defining attention as a separate model
context_vector, attention_weights = self.attention(features, hidden)
# x shape after passing through embedding == (batch_size, 1, embedding_dim)
x = self.embedding(x)
# x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
# passing the concatenated vector to the GRU
output, state = self.gru(x)
# shape == (batch_size, max_length, hidden_size)
x = self.fc1(output)
# x shape == (batch_size * max_length, hidden_size)
x = tf.reshape(x, (-1, x.shape[2]))
# output shape == (batch_size * max_length, vocab)
x = self.fc2(x)
return x, state, attention_weights
def reset_state(self, batch_size):
return tf.zeros((batch_size, self.units))
encoder = CNN_Encoder(embedding_dim)
decoder = RNN_Decoder(embedding_dim, units, tokenizer.vocabulary_size())
optimizer = tf.keras.optimizers.Adam()
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction='none')
def loss_function(real, pred):
mask = tf.math.logical_not(tf.math.equal(real, 0))
loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
Pos pemeriksaan
checkpoint_path = "./checkpoints/train"
ckpt = tf.train.Checkpoint(encoder=encoder,
decoder=decoder,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
# restoring the latest checkpoint in checkpoint_path
ckpt.restore(ckpt_manager.latest_checkpoint)
Pelatihan
- Anda mengekstrak fitur yang disimpan dalam file
.npymasing-masing dan kemudian meneruskan fitur tersebut melalui encoder. - Output encoder, status tersembunyi (diinisialisasi ke 0) dan input dekoder (yang merupakan token awal) diteruskan ke dekoder.
- Dekoder mengembalikan prediksi dan status tersembunyi dekoder.
- Decoder hidden state kemudian diteruskan kembali ke model dan prediksi digunakan untuk menghitung kerugian.
- Gunakan pemaksaan guru untuk memutuskan input berikutnya ke dekoder.
- Pemaksaan guru adalah teknik di mana kata target dilewatkan sebagai input berikutnya ke dekoder.
- Langkah terakhir adalah menghitung gradien dan menerapkannya ke pengoptimal dan backpropagate.
# adding this in a separate cell because if you run the training cell
# many times, the loss_plot array will be reset
loss_plot = []
@tf.function
def train_step(img_tensor, target):
loss = 0
# initializing the hidden state for each batch
# because the captions are not related from image to image
hidden = decoder.reset_state(batch_size=target.shape[0])
dec_input = tf.expand_dims([word_to_index('<start>')] * target.shape[0], 1)
with tf.GradientTape() as tape:
features = encoder(img_tensor)
for i in range(1, target.shape[1]):
# passing the features through the decoder
predictions, hidden, _ = decoder(dec_input, features, hidden)
loss += loss_function(target[:, i], predictions)
# using teacher forcing
dec_input = tf.expand_dims(target[:, i], 1)
total_loss = (loss / int(target.shape[1]))
trainable_variables = encoder.trainable_variables + decoder.trainable_variables
gradients = tape.gradient(loss, trainable_variables)
optimizer.apply_gradients(zip(gradients, trainable_variables))
return loss, total_loss
EPOCHS = 20
for epoch in range(start_epoch, EPOCHS):
start = time.time()
total_loss = 0
for (batch, (img_tensor, target)) in enumerate(dataset):
batch_loss, t_loss = train_step(img_tensor, target)
total_loss += t_loss
if batch % 100 == 0:
average_batch_loss = batch_loss.numpy()/int(target.shape[1])
print(f'Epoch {epoch+1} Batch {batch} Loss {average_batch_loss:.4f}')
# storing the epoch end loss value to plot later
loss_plot.append(total_loss / num_steps)
if epoch % 5 == 0:
ckpt_manager.save()
print(f'Epoch {epoch+1} Loss {total_loss/num_steps:.6f}')
print(f'Time taken for 1 epoch {time.time()-start:.2f} sec\n')
Epoch 1 Batch 0 Loss 1.9157 Epoch 1 Batch 100 Loss 1.1384 Epoch 1 Batch 200 Loss 0.9826 Epoch 1 Batch 300 Loss 0.8792 Epoch 1 Loss 1.025084 Time taken for 1 epoch 153.68 sec Epoch 2 Batch 0 Loss 0.8554 Epoch 2 Batch 100 Loss 0.8062 Epoch 2 Batch 200 Loss 0.7998 Epoch 2 Batch 300 Loss 0.6949 Epoch 2 Loss 0.775522 Time taken for 1 epoch 47.44 sec Epoch 3 Batch 0 Loss 0.7251 Epoch 3 Batch 100 Loss 0.6746 Epoch 3 Batch 200 Loss 0.7269 Epoch 3 Batch 300 Loss 0.7025 Epoch 3 Loss 0.699518 Time taken for 1 epoch 47.78 sec Epoch 4 Batch 0 Loss 0.6970 Epoch 4 Batch 100 Loss 0.6150 Epoch 4 Batch 200 Loss 0.6196 Epoch 4 Batch 300 Loss 0.6131 Epoch 4 Loss 0.650994 Time taken for 1 epoch 46.87 sec Epoch 5 Batch 0 Loss 0.6139 Epoch 5 Batch 100 Loss 0.6305 Epoch 5 Batch 200 Loss 0.6493 Epoch 5 Batch 300 Loss 0.5535 Epoch 5 Loss 0.611642 Time taken for 1 epoch 45.06 sec Epoch 6 Batch 0 Loss 0.6755 Epoch 6 Batch 100 Loss 0.5603 Epoch 6 Batch 200 Loss 0.5161 Epoch 6 Batch 300 Loss 0.5671 Epoch 6 Loss 0.578854 Time taken for 1 epoch 45.25 sec Epoch 7 Batch 0 Loss 0.5575 Epoch 7 Batch 100 Loss 0.4937 Epoch 7 Batch 200 Loss 0.5625 Epoch 7 Batch 300 Loss 0.5456 Epoch 7 Loss 0.549154 Time taken for 1 epoch 44.85 sec Epoch 8 Batch 0 Loss 0.5555 Epoch 8 Batch 100 Loss 0.5142 Epoch 8 Batch 200 Loss 0.4842 Epoch 8 Batch 300 Loss 0.5119 Epoch 8 Loss 0.519941 Time taken for 1 epoch 44.78 sec Epoch 9 Batch 0 Loss 0.4790 Epoch 9 Batch 100 Loss 0.4654 Epoch 9 Batch 200 Loss 0.4568 Epoch 9 Batch 300 Loss 0.4468 Epoch 9 Loss 0.494242 Time taken for 1 epoch 44.99 sec Epoch 10 Batch 0 Loss 0.4740 Epoch 10 Batch 100 Loss 0.4592 Epoch 10 Batch 200 Loss 0.4380 Epoch 10 Batch 300 Loss 0.4556 Epoch 10 Loss 0.468823 Time taken for 1 epoch 44.89 sec Epoch 11 Batch 0 Loss 0.4488 Epoch 11 Batch 100 Loss 0.4423 Epoch 11 Batch 200 Loss 0.4317 Epoch 11 Batch 300 Loss 0.4371 Epoch 11 Loss 0.444164 Time taken for 1 epoch 45.02 sec Epoch 12 Batch 0 Loss 0.4335 Epoch 12 Batch 100 Loss 0.4473 Epoch 12 Batch 200 Loss 0.3770 Epoch 12 Batch 300 Loss 0.4506 Epoch 12 Loss 0.421234 Time taken for 1 epoch 44.95 sec Epoch 13 Batch 0 Loss 0.4289 Epoch 13 Batch 100 Loss 0.4215 Epoch 13 Batch 200 Loss 0.3689 Epoch 13 Batch 300 Loss 0.3864 Epoch 13 Loss 0.399234 Time taken for 1 epoch 45.16 sec Epoch 14 Batch 0 Loss 0.4013 Epoch 14 Batch 100 Loss 0.3571 Epoch 14 Batch 200 Loss 0.3847 Epoch 14 Batch 300 Loss 0.3722 Epoch 14 Loss 0.379495 Time taken for 1 epoch 44.99 sec Epoch 15 Batch 0 Loss 0.3879 Epoch 15 Batch 100 Loss 0.3652 Epoch 15 Batch 200 Loss 0.3025 Epoch 15 Batch 300 Loss 0.3522 Epoch 15 Loss 0.360756 Time taken for 1 epoch 44.96 sec Epoch 16 Batch 0 Loss 0.3542 Epoch 16 Batch 100 Loss 0.3199 Epoch 16 Batch 200 Loss 0.3565 Epoch 16 Batch 300 Loss 0.3352 Epoch 16 Loss 0.344851 Time taken for 1 epoch 44.96 sec Epoch 17 Batch 0 Loss 0.3681 Epoch 17 Batch 100 Loss 0.3477 Epoch 17 Batch 200 Loss 0.3025 Epoch 17 Batch 300 Loss 0.3349 Epoch 17 Loss 0.326141 Time taken for 1 epoch 44.89 sec Epoch 18 Batch 0 Loss 0.3286 Epoch 18 Batch 100 Loss 0.3203 Epoch 18 Batch 200 Loss 0.3029 Epoch 18 Batch 300 Loss 0.2952 Epoch 18 Loss 0.309969 Time taken for 1 epoch 44.89 sec Epoch 19 Batch 0 Loss 0.2942 Epoch 19 Batch 100 Loss 0.2920 Epoch 19 Batch 200 Loss 0.2899 Epoch 19 Batch 300 Loss 0.2875 Epoch 19 Loss 0.295664 Time taken for 1 epoch 46.18 sec Epoch 20 Batch 0 Loss 0.2843 Epoch 20 Batch 100 Loss 0.2907 Epoch 20 Batch 200 Loss 0.2813 Epoch 20 Batch 300 Loss 0.2554 Epoch 20 Loss 0.283829 Time taken for 1 epoch 45.51 sec
plt.plot(loss_plot)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Loss Plot')
plt.show()
Keterangan!
- Fungsi evaluasi mirip dengan loop pelatihan, kecuali Anda tidak menggunakan paksaan guru di sini. Input ke decoder pada setiap langkah waktu adalah prediksi sebelumnya bersama dengan keadaan tersembunyi dan output encoder.
- Berhenti memprediksi saat model memprediksi token akhir.
- Dan simpan bobot perhatian untuk setiap langkah waktu.
def evaluate(image):
attention_plot = np.zeros((max_length, attention_features_shape))
hidden = decoder.reset_state(batch_size=1)
temp_input = tf.expand_dims(load_image(image)[0], 0)
img_tensor_val = image_features_extract_model(temp_input)
img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0],
-1,
img_tensor_val.shape[3]))
features = encoder(img_tensor_val)
dec_input = tf.expand_dims([word_to_index('<start>')], 0)
result = []
for i in range(max_length):
predictions, hidden, attention_weights = decoder(dec_input,
features,
hidden)
attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()
predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()
predicted_word = tf.compat.as_text(index_to_word(predicted_id).numpy())
result.append(predicted_word)
if predicted_word == '<end>':
return result, attention_plot
dec_input = tf.expand_dims([predicted_id], 0)
attention_plot = attention_plot[:len(result), :]
return result, attention_plot
def plot_attention(image, result, attention_plot):
temp_image = np.array(Image.open(image))
fig = plt.figure(figsize=(10, 10))
len_result = len(result)
for i in range(len_result):
temp_att = np.resize(attention_plot[i], (8, 8))
grid_size = max(int(np.ceil(len_result/2)), 2)
ax = fig.add_subplot(grid_size, grid_size, i+1)
ax.set_title(result[i])
img = ax.imshow(temp_image)
ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())
plt.tight_layout()
plt.show()
# captions on the validation set
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
real_caption = ' '.join([tf.compat.as_text(index_to_word(i).numpy())
for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image)
print('Real Caption:', real_caption)
print('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
Real Caption: <start> the bus is driving down the busy street. <end> Prediction Caption: a bus is on the street <end>
Cobalah pada gambar Anda sendiri
Untuk bersenang-senang, di bawah ini Anda menyediakan metode yang dapat Anda gunakan untuk memberi keterangan gambar Anda sendiri dengan model yang baru saja Anda latih. Perlu diingat, ini dilatih pada jumlah data yang relatif kecil, dan gambar Anda mungkin berbeda dari data pelatihan (jadi bersiaplah untuk hasil yang aneh!)
image_url = 'https://tensorflow.org/images/surf.jpg'
image_extension = image_url[-4:]
image_path = tf.keras.utils.get_file('image'+image_extension, origin=image_url)
result, attention_plot = evaluate(image_path)
print('Prediction Caption:', ' '.join(result))
plot_attention(image_path, result, attention_plot)
# opening the image
Image.open(image_path)
Prediction Caption: an image of a man with man standing wearing a [UNK] into the [UNK] <end>
Langkah selanjutnya
Selamat! Anda baru saja melatih model teks gambar dengan perhatian. Selanjutnya, lihat contoh ini Neural Machine Translation with Attention . Ini menggunakan arsitektur serupa untuk menerjemahkan antara kalimat Spanyol dan Inggris. Anda juga dapat bereksperimen dengan melatih kode di buku catatan ini pada kumpulan data yang berbeda.