 Lihat di TensorFlow.org |  Lihat di GitHub |  Unduh buku catatan |  Lihat model TF Hub |
Tutorial ini mengilustrasikan cara membuat penyematan dari modul TensorFlow Hub (TF-Hub) dengan data masukan, dan membuat indeks perkiraan tetangga terdekat (ANN) menggunakan penyematan yang diekstrak. Indeks tersebut kemudian dapat digunakan untuk pencocokan dan pengambilan kesamaan secara real-time.
Saat menangani kumpulan data yang besar, melakukan pencocokan tepat dengan memindai seluruh repositori untuk menemukan item yang paling mirip dengan kueri tertentu secara real-time tidaklah efisien. Oleh karena itu, kami menggunakan algoritme pencocokan kemiripan perkiraan yang memungkinkan kami mengorbankan sedikit keakuratan dalam menemukan kecocokan tetangga terdekat yang tepat demi peningkatan kecepatan yang signifikan.
Dalam tutorial ini, kami menampilkan contoh penelusuran teks real-time pada kumpulan judul berita untuk menemukan judul yang paling mirip dengan kueri. Tidak seperti pencarian kata kunci, pencarian ini menangkap kesamaan semantik yang dikodekan dalam penyematan teks.
Langkah-langkah tutorial ini adalah:
- Unduh data sampel.
- Hasilkan penyematan untuk data menggunakan modul TF-Hub
- Buat indeks ANN untuk penyematan
- Gunakan indeks untuk pencocokan kesamaan
Kami menggunakan Apache Beam untuk menghasilkan embeddings dari modul TF-Hub. Kami juga menggunakan perpustakaan ANNOY Spotify untuk membuat perkiraan indeks tetangga terdekat.
Lebih banyak model
Untuk model yang memiliki arsitektur yang sama namun dilatih dalam bahasa yang berbeda, lihat koleksi ini . Di sini Anda dapat menemukan semua penyematan teks yang saat ini dihosting di tfhub.dev .
Pengaturan
Instal perpustakaan yang diperlukan.
pip install -q apache_beampip install -q 'scikit_learn~=0.23.0' # For gaussian_random_matrix.pip install -q annoy
Impor perpustakaan yang diperlukan
import os
import sys
import pickle
from collections import namedtuple
from datetime import datetime
import numpy as np
import apache_beam as beam
from apache_beam.transforms import util
import tensorflow as tf
import tensorflow_hub as hub
import annoy
from sklearn.random_projection import gaussian_random_matrix
print('TF version: {}'.format(tf.__version__))
print('TF-Hub version: {}'.format(hub.__version__))
print('Apache Beam version: {}'.format(beam.__version__))
TF version: 2.4.0 TF-Hub version: 0.11.0 Apache Beam version: 2.26.0
1. Unduh Contoh Data
Kumpulan data Sejuta Berita Utama berisi judul berita yang diterbitkan selama periode 15 tahun yang bersumber dari Australian Broadcasting Corp. (ABC) yang memiliki reputasi baik. Kumpulan data berita ini memiliki ringkasan catatan sejarah peristiwa-peristiwa penting di dunia dari awal tahun 2003 hingga akhir tahun 2017 dengan fokus yang lebih terperinci di Australia.
Format : Data dua kolom yang dipisahkan tab: 1) tanggal publikasi dan 2) teks judul. Kami hanya tertarik pada teks judul.
wget 'https://dataverse.harvard.edu/api/access/datafile/3450625?format=tab&gbrecs=true' -O raw.tsvwc -l raw.tsvhead raw.tsv
--2021-01-07 12:50:08-- https://dataverse.harvard.edu/api/access/datafile/3450625?format=tab&gbrecs=true Resolving dataverse.harvard.edu (dataverse.harvard.edu)... 206.191.184.198 Connecting to dataverse.harvard.edu (dataverse.harvard.edu)|206.191.184.198|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 57600231 (55M) [text/tab-separated-values] Saving to: ‘raw.tsv’ raw.tsv 100%[===================>] 54.93M 14.7MB/s in 4.4s 2021-01-07 12:50:14 (12.4 MB/s) - ‘raw.tsv’ saved [57600231/57600231] 1103664 raw.tsv publish_date headline_text 20030219 "aba decides against community broadcasting licence" 20030219 "act fire witnesses must be aware of defamation" 20030219 "a g calls for infrastructure protection summit" 20030219 "air nz staff in aust strike for pay rise" 20030219 "air nz strike to affect australian travellers" 20030219 "ambitious olsson wins triple jump" 20030219 "antic delighted with record breaking barca" 20030219 "aussie qualifier stosur wastes four memphis match" 20030219 "aust addresses un security council over iraq"
Untuk mempermudah, kami hanya menyimpan teks judul dan menghapus tanggal publikasi
!rm -r corpus
!mkdir corpus
with open('corpus/text.txt', 'w') as out_file:
with open('raw.tsv', 'r') as in_file:
for line in in_file:
headline = line.split('\t')[1].strip().strip('"')
out_file.write(headline+"\n")
rm: cannot remove 'corpus': No such file or directory
tail corpus/text.txtsevere storms forecast for nye in south east queensland snake catcher pleads for people not to kill reptiles south australia prepares for party to welcome new year strikers cool off the heat with big win in adelaide stunning images from the sydney to hobart yacht the ashes smiths warners near miss liven up boxing day test timelapse: brisbanes new year fireworks what 2017 meant to the kids of australia what the papodopoulos meeting may mean for ausus who is george papadopoulos the former trump campaign aide
2. Hasilkan Penyematan untuk Data.
Dalam tutorial ini, kami menggunakan Neural Network Language Model (NNLM) untuk menghasilkan penyematan untuk data judul. Penyematan kalimat kemudian dapat dengan mudah digunakan untuk menghitung kesamaan makna tingkat kalimat. Kami menjalankan proses pembuatan penyematan menggunakan Apache Beam.
Menanamkan metode ekstraksi
embed_fn = None
def generate_embeddings(text, module_url, random_projection_matrix=None):
# Beam will run this function in different processes that need to
# import hub and load embed_fn (if not previously loaded)
global embed_fn
if embed_fn is None:
embed_fn = hub.load(module_url)
embedding = embed_fn(text).numpy()
if random_projection_matrix is not None:
embedding = embedding.dot(random_projection_matrix)
return text, embedding
Konversikan ke metode tf.Contoh
def to_tf_example(entries):
examples = []
text_list, embedding_list = entries
for i in range(len(text_list)):
text = text_list[i]
embedding = embedding_list[i]
features = {
'text': tf.train.Feature(
bytes_list=tf.train.BytesList(value=[text.encode('utf-8')])),
'embedding': tf.train.Feature(
float_list=tf.train.FloatList(value=embedding.tolist()))
}
example = tf.train.Example(
features=tf.train.Features(
feature=features)).SerializeToString(deterministic=True)
examples.append(example)
return examples
Pipa balok
def run_hub2emb(args):
'''Runs the embedding generation pipeline'''
options = beam.options.pipeline_options.PipelineOptions(**args)
args = namedtuple("options", args.keys())(*args.values())
with beam.Pipeline(args.runner, options=options) as pipeline:
(
pipeline
| 'Read sentences from files' >> beam.io.ReadFromText(
file_pattern=args.data_dir)
| 'Batch elements' >> util.BatchElements(
min_batch_size=args.batch_size, max_batch_size=args.batch_size)
| 'Generate embeddings' >> beam.Map(
generate_embeddings, args.module_url, args.random_projection_matrix)
| 'Encode to tf example' >> beam.FlatMap(to_tf_example)
| 'Write to TFRecords files' >> beam.io.WriteToTFRecord(
file_path_prefix='{}/emb'.format(args.output_dir),
file_name_suffix='.tfrecords')
)
Menghasilkan Matriks Bobot Proyeksi Acak
Proyeksi acak adalah teknik sederhana namun ampuh yang digunakan untuk mereduksi dimensi sekumpulan titik yang terletak di ruang Euclidean. Untuk latar belakang teoretis, lihat lemma Johnson-Lindenstrauss .
Mengurangi dimensi penyematan dengan proyeksi acak berarti lebih sedikit waktu yang diperlukan untuk membuat dan mengkueri indeks ANN.
Dalam tutorial ini kami menggunakan Proyeksi Acak Gaussian dari perpustakaan Scikit-learn .
def generate_random_projection_weights(original_dim, projected_dim):
random_projection_matrix = None
random_projection_matrix = gaussian_random_matrix(
n_components=projected_dim, n_features=original_dim).T
print("A Gaussian random weight matrix was creates with shape of {}".format(random_projection_matrix.shape))
print('Storing random projection matrix to disk...')
with open('random_projection_matrix', 'wb') as handle:
pickle.dump(random_projection_matrix,
handle, protocol=pickle.HIGHEST_PROTOCOL)
return random_projection_matrix
Tetapkan parameter
Jika Anda ingin membuat indeks menggunakan ruang penyematan asli tanpa proyeksi acak, atur parameter projected_dim ke None . Perhatikan bahwa ini akan memperlambat langkah pengindeksan untuk penyematan berdimensi tinggi.
module_url = 'https://tfhub.dev/google/nnlm-en-dim128/2'
projected_dim = 64
Jalankan saluran pipa
import tempfile
output_dir = tempfile.mkdtemp()
original_dim = hub.load(module_url)(['']).shape[1]
random_projection_matrix = None
if projected_dim:
random_projection_matrix = generate_random_projection_weights(
original_dim, projected_dim)
args = {
'job_name': 'hub2emb-{}'.format(datetime.utcnow().strftime('%y%m%d-%H%M%S')),
'runner': 'DirectRunner',
'batch_size': 1024,
'data_dir': 'corpus/*.txt',
'output_dir': output_dir,
'module_url': module_url,
'random_projection_matrix': random_projection_matrix,
}
print("Pipeline args are set.")
args
A Gaussian random weight matrix was creates with shape of (128, 64)
Storing random projection matrix to disk...
Pipeline args are set.
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/sklearn/utils/deprecation.py:86: FutureWarning: Function gaussian_random_matrix is deprecated; gaussian_random_matrix is deprecated in 0.22 and will be removed in version 0.24.
warnings.warn(msg, category=FutureWarning)
{'job_name': 'hub2emb-210107-125029',
'runner': 'DirectRunner',
'batch_size': 1024,
'data_dir': 'corpus/*.txt',
'output_dir': '/tmp/tmp0g361gzp',
'module_url': 'https://tfhub.dev/google/nnlm-en-dim128/2',
'random_projection_matrix': array([[-0.1349755 , -0.12082699, 0.07092581, ..., -0.02680793,
-0.0459312 , -0.20462361],
[-0.06197901, 0.01832142, 0.21362496, ..., 0.06641898,
0.14553738, -0.117217 ],
[ 0.03452009, 0.14239163, 0.01371371, ..., 0.10422342,
0.02966668, -0.07094185],
...,
[ 0.03384223, 0.05102025, 0.01941788, ..., -0.07500625,
0.09584965, -0.08593636],
[ 0.11010087, -0.10597793, 0.06668758, ..., -0.0518654 ,
-0.14681441, 0.08449293],
[ 0.26909502, -0.0291555 , 0.04305639, ..., -0.02295843,
0.1164921 , -0.04828371]])}
print("Running pipeline...")
%time run_hub2emb(args)
print("Pipeline is done.")
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. Running pipeline... Warning:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7efcac3599d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Warning:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7efcac3599d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Warning:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7efcac475598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Warning:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7efcac475598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. CPU times: user 9min 4s, sys: 10min 14s, total: 19min 19s Wall time: 2min 30s Pipeline is done.
ls {output_dir}emb-00000-of-00001.tfrecords
Baca beberapa embeddings yang dihasilkan...
embed_file = os.path.join(output_dir, 'emb-00000-of-00001.tfrecords')
sample = 5
# Create a description of the features.
feature_description = {
'text': tf.io.FixedLenFeature([], tf.string),
'embedding': tf.io.FixedLenFeature([projected_dim], tf.float32)
}
def _parse_example(example):
# Parse the input `tf.Example` proto using the dictionary above.
return tf.io.parse_single_example(example, feature_description)
dataset = tf.data.TFRecordDataset(embed_file)
for record in dataset.take(sample).map(_parse_example):
print("{}: {}".format(record['text'].numpy().decode('utf-8'), record['embedding'].numpy()[:10]))
headline_text: [ 0.07743962 -0.10065071 -0.03604915 0.03902601 0.02538098 -0.01991337 -0.11972483 0.03102058 0.16498186 -0.04299153] aba decides against community broadcasting licence: [ 0.02420221 -0.07736929 0.05655728 -0.18739551 0.11344934 0.12652674 -0.18189304 0.00422473 0.13149698 0.01910412] act fire witnesses must be aware of defamation: [-0.17413895 -0.05418579 0.07769868 0.05096476 0.08622053 0.33112594 0.04067763 0.00448784 0.15882017 0.33829722] a g calls for infrastructure protection summit: [ 0.16939437 -0.18585566 -0.14201084 -0.21779229 -0.1374832 0.14933842 -0.19583155 0.12921487 0.09811856 0.099967 ] air nz staff in aust strike for pay rise: [ 0.0230642 -0.03269081 0.18271443 0.23761444 -0.01575144 0.06109515 -0.01963143 -0.05211507 0.06050447 -0.20023327]
3. Buat Indeks ANN untuk Embeddings
ANNOY (Approximate Nearest Neighbors Oh Yeah) adalah pustaka C++ dengan binding Python untuk mencari titik dalam ruang yang dekat dengan titik kueri tertentu. Ini juga menciptakan struktur data berbasis file read-only yang besar yang dipetakan ke dalam memori. Itu dibuat dan digunakan oleh Spotify untuk rekomendasi musik. Jika tertarik, Anda bisa bermain bersama alternatif lain selain ANNOY seperti NGT , FAISS , dll.
def build_index(embedding_files_pattern, index_filename, vector_length,
metric='angular', num_trees=100):
'''Builds an ANNOY index'''
annoy_index = annoy.AnnoyIndex(vector_length, metric=metric)
# Mapping between the item and its identifier in the index
mapping = {}
embed_files = tf.io.gfile.glob(embedding_files_pattern)
num_files = len(embed_files)
print('Found {} embedding file(s).'.format(num_files))
item_counter = 0
for i, embed_file in enumerate(embed_files):
print('Loading embeddings in file {} of {}...'.format(i+1, num_files))
dataset = tf.data.TFRecordDataset(embed_file)
for record in dataset.map(_parse_example):
text = record['text'].numpy().decode("utf-8")
embedding = record['embedding'].numpy()
mapping[item_counter] = text
annoy_index.add_item(item_counter, embedding)
item_counter += 1
if item_counter % 100000 == 0:
print('{} items loaded to the index'.format(item_counter))
print('A total of {} items added to the index'.format(item_counter))
print('Building the index with {} trees...'.format(num_trees))
annoy_index.build(n_trees=num_trees)
print('Index is successfully built.')
print('Saving index to disk...')
annoy_index.save(index_filename)
print('Index is saved to disk.')
print("Index file size: {} GB".format(
round(os.path.getsize(index_filename) / float(1024 ** 3), 2)))
annoy_index.unload()
print('Saving mapping to disk...')
with open(index_filename + '.mapping', 'wb') as handle:
pickle.dump(mapping, handle, protocol=pickle.HIGHEST_PROTOCOL)
print('Mapping is saved to disk.')
print("Mapping file size: {} MB".format(
round(os.path.getsize(index_filename + '.mapping') / float(1024 ** 2), 2)))
embedding_files = "{}/emb-*.tfrecords".format(output_dir)
embedding_dimension = projected_dim
index_filename = "index"
!rm {index_filename}
!rm {index_filename}.mapping
%time build_index(embedding_files, index_filename, embedding_dimension)
rm: cannot remove 'index': No such file or directory rm: cannot remove 'index.mapping': No such file or directory Found 1 embedding file(s). Loading embeddings in file 1 of 1... 100000 items loaded to the index 200000 items loaded to the index 300000 items loaded to the index 400000 items loaded to the index 500000 items loaded to the index 600000 items loaded to the index 700000 items loaded to the index 800000 items loaded to the index 900000 items loaded to the index 1000000 items loaded to the index 1100000 items loaded to the index A total of 1103664 items added to the index Building the index with 100 trees... Index is successfully built. Saving index to disk... Index is saved to disk. Index file size: 1.61 GB Saving mapping to disk... Mapping is saved to disk. Mapping file size: 50.61 MB CPU times: user 9min 54s, sys: 53.9 s, total: 10min 48s Wall time: 5min 5s
lscorpus random_projection_matrix index raw.tsv index.mapping tf2_semantic_approximate_nearest_neighbors.ipynb
4. Gunakan Indeks untuk Pencocokan Kesamaan
Sekarang kita dapat menggunakan indeks ANN untuk menemukan judul berita yang secara semantik dekat dengan kueri masukan.
Muat indeks dan file pemetaan
index = annoy.AnnoyIndex(embedding_dimension)
index.load(index_filename, prefault=True)
print('Annoy index is loaded.')
with open(index_filename + '.mapping', 'rb') as handle:
mapping = pickle.load(handle)
print('Mapping file is loaded.')
Annoy index is loaded. /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: The default argument for metric will be removed in future version of Annoy. Please pass metric='angular' explicitly. """Entry point for launching an IPython kernel. Mapping file is loaded.
Metode pencocokan kesamaan
def find_similar_items(embedding, num_matches=5):
'''Finds similar items to a given embedding in the ANN index'''
ids = index.get_nns_by_vector(
embedding, num_matches, search_k=-1, include_distances=False)
items = [mapping[i] for i in ids]
return items
Ekstrak penyematan dari kueri tertentu
# Load the TF-Hub module
print("Loading the TF-Hub module...")
%time embed_fn = hub.load(module_url)
print("TF-Hub module is loaded.")
random_projection_matrix = None
if os.path.exists('random_projection_matrix'):
print("Loading random projection matrix...")
with open('random_projection_matrix', 'rb') as handle:
random_projection_matrix = pickle.load(handle)
print('random projection matrix is loaded.')
def extract_embeddings(query):
'''Generates the embedding for the query'''
query_embedding = embed_fn([query])[0].numpy()
if random_projection_matrix is not None:
query_embedding = query_embedding.dot(random_projection_matrix)
return query_embedding
Loading the TF-Hub module... CPU times: user 757 ms, sys: 619 ms, total: 1.38 s Wall time: 1.37 s TF-Hub module is loaded. Loading random projection matrix... random projection matrix is loaded.
extract_embeddings("Hello Machine Learning!")[:10]
array([ 0.12164804, 0.0162079 , -0.15466002, -0.14580576, 0.03926325,
-0.10124508, -0.1333948 , 0.0515029 , -0.14688903, -0.09971556])
Masukkan kueri untuk menemukan item yang paling mirip
query = "confronting global challenges"
print("Generating embedding for the query...")
%time query_embedding = extract_embeddings(query)
print("")
print("Finding relevant items in the index...")
%time items = find_similar_items(query_embedding, 10)
print("")
print("Results:")
print("=========")
for item in items:
print(item)
Generating embedding for the query... CPU times: user 5.18 ms, sys: 596 µs, total: 5.77 ms Wall time: 2.19 ms Finding relevant items in the index... CPU times: user 555 µs, sys: 327 µs, total: 882 µs Wall time: 601 µs Results: ========= confronting global challenges emerging nations to help struggling global economy g7 warns of increasing global economic crisis world struggling to cope with global terrorism companies health to struggle amid global crisis external risks biggest threat to economy asian giants unite to tackle global crisis g7 ministers warn of slowing global growth experts to discuss global warming threat scientists warn of growing natural disasters
Ingin mempelajari lebih lanjut?
Anda dapat mempelajari TensorFlow lebih lanjut di tensorflow.org dan melihat dokumentasi TF-Hub API di tensorflow.org/hub . Temukan modul TensorFlow Hub yang tersedia di tfhub.dev termasuk lebih banyak modul penyematan teks dan modul vektor fitur gambar.
Lihat juga Kursus Singkat Machine Learning yang merupakan pengenalan praktis dan cepat dari Google tentang pembelajaran mesin.