 TensorFlow.org で表示
|
 Google Colabで実行
|
 GitHub でソースを表示 |
 ノートブックをダウンロード |
 TF Hub モデルを参照 |
この Colab は、文の埋め込みの品質を測定するためのライブラリである SentEval ツールキットを使用したユニバーサルセンテンスエンコーダ CMLM モデルを説明します。 SentEval ツールキットには、埋め込みモデルの一般化能力を評価し、エンコードされた言語プロパティを評価できる、さまざまな下流タスクのセットが含まれています。
最初の 2 つのコードブロックを実行して環境をセットアップします。3 番目のコードブロックでは、SentEval タスクを選択してモデルを評価できます。この Colab を実行するには、GPU ランタイムをお勧めします。
ユニバーサルセンテンスエンコーダ CMLM モデルの詳細については、https://openreview.net/forum?id=WDVD4lUCTzU を参照してください。
Install dependencies
pip install --quiet "tensorflow-text==2.11.*"pip install --quiet torch==1.8.1
SentEvalとタスクデータをダウンロードする
このステップでは、github から SentEval をダウンロードし、データスクリプトを実行してタスクデータをダウンロードします。完了するまでに最大 5 分かかる場合があります。
Install SentEval and download task data
rm -rf ./SentEvalgit clone https://github.com/facebookresearch/SentEval.gitcd $PWD/SentEval/data/downstream && bash get_transfer_data.bash > /dev/null 2>&1
Cloning into 'SentEval'... remote: Enumerating objects: 691, done. remote: Counting objects: 50% (1/2) remote: Counting objects: 100% (2/2) remote: Counting objects: 100% (2/2), done. remote: Compressing objects: 50% (1/2) remote: Compressing objects: 100% (2/2) remote: Compressing objects: 100% (2/2), done. Receiving objects: 0% (1/691) Receiving objects: 1% (7/691) Receiving objects: 2% (14/691) Receiving objects: 3% (21/691) Receiving objects: 4% (28/691) Receiving objects: 5% (35/691) Receiving objects: 6% (42/691) Receiving objects: 7% (49/691) Receiving objects: 8% (56/691) Receiving objects: 9% (63/691) Receiving objects: 10% (70/691) Receiving objects: 11% (77/691) Receiving objects: 12% (83/691) Receiving objects: 13% (90/691) Receiving objects: 14% (97/691) Receiving objects: 15% (104/691) Receiving objects: 16% (111/691) Receiving objects: 17% (118/691) Receiving objects: 18% (125/691) Receiving objects: 19% (132/691) Receiving objects: 20% (139/691) Receiving objects: 21% (146/691) Receiving objects: 22% (153/691) Receiving objects: 23% (159/691) Receiving objects: 24% (166/691), 11.57 MiB | 23.12 MiB/s Receiving objects: 24% (168/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 25% (173/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 26% (180/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 27% (187/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 28% (194/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 29% (201/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 30% (208/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 31% (215/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 32% (222/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 33% (229/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 34% (235/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 35% (242/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 36% (249/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 37% (256/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 38% (263/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 39% (270/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 40% (277/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 41% (284/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 42% (291/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 43% (298/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 44% (305/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 45% (311/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 46% (318/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 47% (325/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 48% (332/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 49% (339/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 50% (346/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 51% (353/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 52% (360/691), 26.68 MiB | 26.67 MiB/s remote: Total 691 (delta 0), reused 2 (delta 0), pack-reused 689 Receiving objects: 53% (367/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 54% (374/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 55% (381/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 56% (387/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 57% (394/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 58% (401/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 59% (408/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 60% (415/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 61% (422/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 62% (429/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 63% (436/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 64% (443/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 65% (450/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 66% (457/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 67% (463/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 68% (470/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 69% (477/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 70% (484/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 71% (491/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 72% (498/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 73% (505/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 74% (512/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 75% (519/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 76% (526/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 77% (533/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 78% (539/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 79% (546/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 80% (553/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 81% (560/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 82% (567/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 83% (574/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 84% (581/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 85% (588/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 86% (595/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 87% (602/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 88% (609/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 89% (615/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 90% (622/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 91% (629/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 92% (636/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 93% (643/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 94% (650/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 95% (657/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 96% (664/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 97% (671/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 98% (678/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 99% (685/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 100% (691/691), 26.68 MiB | 26.67 MiB/s Receiving objects: 100% (691/691), 33.25 MiB | 27.39 MiB/s, done. Resolving deltas: 0% (0/434) Resolving deltas: 1% (6/434) Resolving deltas: 2% (11/434) Resolving deltas: 3% (17/434) Resolving deltas: 7% (32/434) Resolving deltas: 9% (40/434) Resolving deltas: 11% (52/434) Resolving deltas: 12% (55/434) Resolving deltas: 15% (67/434) Resolving deltas: 17% (77/434) Resolving deltas: 18% (79/434) Resolving deltas: 19% (85/434) Resolving deltas: 21% (93/434) Resolving deltas: 23% (100/434) Resolving deltas: 27% (121/434) Resolving deltas: 35% (155/434) Resolving deltas: 36% (157/434) Resolving deltas: 38% (166/434) Resolving deltas: 42% (183/434) Resolving deltas: 43% (187/434) Resolving deltas: 44% (194/434) Resolving deltas: 45% (196/434) Resolving deltas: 47% (205/434) Resolving deltas: 51% (224/434) Resolving deltas: 53% (231/434) Resolving deltas: 54% (238/434) Resolving deltas: 55% (239/434) Resolving deltas: 57% (248/434) Resolving deltas: 59% (259/434) Resolving deltas: 60% (264/434) Resolving deltas: 61% (267/434) Resolving deltas: 62% (270/434) Resolving deltas: 63% (277/434) Resolving deltas: 64% (278/434) Resolving deltas: 65% (285/434) Resolving deltas: 67% (291/434) Resolving deltas: 68% (298/434) Resolving deltas: 69% (300/434) Resolving deltas: 70% (304/434) Resolving deltas: 73% (320/434) Resolving deltas: 77% (336/434) Resolving deltas: 79% (347/434) Resolving deltas: 80% (350/434) Resolving deltas: 86% (377/434) Resolving deltas: 87% (378/434) Resolving deltas: 88% (383/434) Resolving deltas: 90% (392/434) Resolving deltas: 91% (399/434) Resolving deltas: 92% (402/434) Resolving deltas: 94% (412/434) Resolving deltas: 97% (424/434) Resolving deltas: 98% (429/434) Resolving deltas: 100% (434/434) Resolving deltas: 100% (434/434), done.
＃SentEval 評価タスクの実行。次のコードブロックは SentEval タスクを実行して結果を出力し、次のタスクのいずれかを選択して USE CMLM モデルを評価します。
MR CR SUBJ MPQA SST TREC MRPC SICK-E
実行するモデル、パラメーター、およびタスクを選択します。ラピッドプロトタイピングパラメータは、計算時間を短縮するために使用でき、より速く結果を得ることができます。
通常、'rapid prototyping' パラメータを使用してタスクを完了するには 5〜15 分かかり、'slower, best performance' パラメータでは最大 1 時間かかります。
params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
'tenacity': 3, 'epoch_size': 2}
より良い結果を得るには、より遅い 'slower, best performance' パラメータを使用してください。計算には最大 1 時間かかる場合があります。
params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
'tenacity': 5, 'epoch_size': 6}
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import sys
sys.path.append(f'{os.getcwd()}/SentEval')
import tensorflow as tf
# Prevent TF from claiming all GPU memory so there is some left for pytorch.
gpus = tf.config.list_physical_devices('GPU')
if gpus:
# Memory growth needs to be the same across GPUs.
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
import tensorflow_hub as hub
import tensorflow_text
import senteval
import time
PATH_TO_DATA = f'{os.getcwd()}/SentEval/data'
MODEL = 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1'
PARAMS = 'rapid prototyping'
TASK = 'CR'
params_prototyping = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params_prototyping['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
'tenacity': 3, 'epoch_size': 2}
params_best = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params_best['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
'tenacity': 5, 'epoch_size': 6}
params = params_best if PARAMS == 'slower, best performance' else params_prototyping
preprocessor = hub.KerasLayer(
"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder = hub.KerasLayer(
"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1")
inputs = tf.keras.Input(shape=tf.shape(''), dtype=tf.string)
outputs = encoder(preprocessor(inputs))
model = tf.keras.Model(inputs=inputs, outputs=outputs)
def prepare(params, samples):
return
def batcher(_, batch):
batch = [' '.join(sent) if sent else '.' for sent in batch]
return model.predict(tf.constant(batch))["default"]
se = senteval.engine.SE(params, batcher, prepare)
print("Evaluating task %s with %s parameters" % (TASK, PARAMS))
start = time.time()
results = se.eval(TASK)
end = time.time()
print('Time took on task %s : %.1f. seconds' % (TASK, end - start))
print(results)
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089
Evaluating task CR with rapid prototyping parameters
4/4 [==============================] - 8s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 7s 2s/step
4/4 [==============================] - 6s 2s/step
4/4 [==============================] - 6s 2s/step
2/2 [==============================] - 4s 2s/step
Time took on task CR : 223.1. seconds
{'devacc': 90.42, 'acc': 88.98, 'ndev': 3775, 'ntest': 3775}
詳細情報
- その他のテキスト埋め込みモデルは、TensorFlow Hub をご覧ください。
- 多言語ユニバーサルセンテンスエンコーダ CMLM モデルもご覧ください。
- 他のユニバーサルセンテンスエンコーダモデルを確認してください
リファレンス
- Ziyi Yang、Yinfei Yang、Daniel Cer、Jax Law、Eric Darve。Universal Sentence Representations Learning with Conditional Masked Language Model. November 2020