Universal Sentence Encoder SentEval 演示

View on TensorFlow.org 在 Google Colab 中运行 在 GitHub 上查看源代码 下载笔记本 查看 TF Hub 模型

本 Colab 使用 SentEval 工具套件演示 Universal Sentence Encoder CMLM 模型，该工具套件是用于测量句子嵌入质量的库。SentEval 工具套件包括一组多样化的下游任务，能够评估嵌入模型的泛化能力并评估编码的语言属性。

运行前两个代码块设置环境，在第三个代码块中，可以选择一个 SentEval 任务来评估模型。建议使用 GPU 运行时来运行本 Colab。

要了解有关 Universal Sentence Encoder CMLM 模型的更多信息，请参阅 https://openreview.net/forum?id=WDVD4lUCTzU。

Install dependencies

下载 SentEval 和任务数据

本步骤从 github 下载 SentEval 并执行数据脚本下载任务数据。可能需要长达 5 分钟的时间才能完成。

Install SentEval and download task data

Cloning into 'SentEval'...
remote: Enumerating objects: 691, done.
remote: Counting objects:  50% (1/2)
remote: Counting objects: 100% (2/2)
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects:  50% (1/2)
remote: Compressing objects: 100% (2/2)
remote: Compressing objects: 100% (2/2), done.
Receiving objects:   0% (1/691)
Receiving objects:   1% (7/691)
Receiving objects:   2% (14/691)
Receiving objects:   3% (21/691)
Receiving objects:   4% (28/691)
Receiving objects:   5% (35/691)
Receiving objects:   6% (42/691)
Receiving objects:   7% (49/691)
Receiving objects:   8% (56/691)
Receiving objects:   9% (63/691)
Receiving objects:  10% (70/691)
Receiving objects:  11% (77/691)
Receiving objects:  12% (83/691)
Receiving objects:  13% (90/691)
Receiving objects:  14% (97/691)
Receiving objects:  15% (104/691)
Receiving objects:  16% (111/691)
Receiving objects:  17% (118/691)
Receiving objects:  18% (125/691)
Receiving objects:  19% (132/691)
Receiving objects:  20% (139/691)
Receiving objects:  21% (146/691)
Receiving objects:  22% (153/691)
Receiving objects:  23% (159/691)
Receiving objects:  24% (166/691), 11.67 MiB | 23.33 MiB/s
Receiving objects:  24% (168/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  25% (173/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  26% (180/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  27% (187/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  28% (194/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  29% (201/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  30% (208/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  31% (215/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  32% (222/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  33% (229/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  34% (235/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  35% (242/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  36% (249/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  37% (256/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  38% (263/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  39% (270/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  40% (277/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  41% (284/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  42% (291/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  43% (298/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  44% (305/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  45% (311/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  46% (318/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  47% (325/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  48% (332/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  49% (339/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  50% (346/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  51% (353/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  52% (360/691), 27.54 MiB | 27.54 MiB/s
remote: Total 691 (delta 0), reused 2 (delta 0), pack-reused 689
Receiving objects:  53% (367/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  54% (374/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  55% (381/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  56% (387/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  57% (394/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  58% (401/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  59% (408/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  60% (415/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  61% (422/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  62% (429/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  63% (436/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  64% (443/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  65% (450/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  66% (457/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  67% (463/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  68% (470/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  69% (477/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  70% (484/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  71% (491/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  72% (498/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  73% (505/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  74% (512/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  75% (519/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  76% (526/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  77% (533/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  78% (539/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  79% (546/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  80% (553/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  81% (560/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  82% (567/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  83% (574/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  84% (581/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  85% (588/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  86% (595/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  87% (602/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  88% (609/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  89% (615/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  90% (622/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  91% (629/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  92% (636/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  93% (643/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  94% (650/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  95% (657/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  96% (664/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  97% (671/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  98% (678/691), 27.54 MiB | 27.54 MiB/s
Receiving objects:  99% (685/691), 27.54 MiB | 27.54 MiB/s
Receiving objects: 100% (691/691), 27.54 MiB | 27.54 MiB/s
Receiving objects: 100% (691/691), 33.25 MiB | 28.21 MiB/s, done.
Resolving deltas:   0% (0/434)
Resolving deltas:   1% (7/434)
Resolving deltas:   2% (12/434)
Resolving deltas:   4% (18/434)
Resolving deltas:   7% (32/434)
Resolving deltas:   9% (40/434)
Resolving deltas:  11% (52/434)
Resolving deltas:  12% (55/434)
Resolving deltas:  15% (66/434)
Resolving deltas:  17% (77/434)
Resolving deltas:  18% (81/434)
Resolving deltas:  19% (85/434)
Resolving deltas:  21% (92/434)
Resolving deltas:  22% (96/434)
Resolving deltas:  23% (101/434)
Resolving deltas:  28% (124/434)
Resolving deltas:  35% (154/434)
Resolving deltas:  36% (157/434)
Resolving deltas:  38% (165/434)
Resolving deltas:  42% (183/434)
Resolving deltas:  43% (187/434)
Resolving deltas:  44% (191/434)
Resolving deltas:  45% (197/434)
Resolving deltas:  47% (208/434)
Resolving deltas:  52% (228/434)
Resolving deltas:  54% (235/434)
Resolving deltas:  55% (242/434)
Resolving deltas:  57% (250/434)
Resolving deltas:  59% (259/434)
Resolving deltas:  61% (265/434)
Resolving deltas:  63% (276/434)
Resolving deltas:  65% (284/434)
Resolving deltas:  66% (289/434)
Resolving deltas:  67% (292/434)
Resolving deltas:  68% (297/434)
Resolving deltas:  69% (300/434)
Resolving deltas:  70% (304/434)
Resolving deltas:  73% (319/434)
Resolving deltas:  76% (334/434)
Resolving deltas:  79% (346/434)
Resolving deltas:  80% (351/434)
Resolving deltas:  81% (354/434)
Resolving deltas:  87% (379/434)
Resolving deltas:  88% (384/434)
Resolving deltas:  89% (390/434)
Resolving deltas:  91% (397/434)
Resolving deltas:  92% (401/434)
Resolving deltas:  95% (413/434)
Resolving deltas:  97% (423/434)
Resolving deltas:  99% (430/434)
Resolving deltas: 100% (434/434)
Resolving deltas: 100% (434/434), done.

执行 SentEval 评估任务 以下代码块执行 SentEval 任务并输出结果，选择以下任务之一来评估 USE CMLM 模型：

MR  CR  SUBJ    MPQA    SST TREC    MRPC    SICK-E

选择要运行的模型、参数和任务。可以使用 rapid prototyping 参数减少计算时间以更快获得结果。

使用 'rapid prototyping' 参数完成任务通常需要 5-15 分钟，使用 'slower, best performance' 参数最多需要一个小时。

params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
                                 'tenacity': 3, 'epoch_size': 2}

要获得更好的结果，请使用较慢的 'slower, best performance' 参数，计算时间可能长达 1 小时：

params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
                                 'tenacity': 5, 'epoch_size': 6}

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

import sys
sys.path.append(f'{os.getcwd()}/SentEval')

import tensorflow as tf

# Prevent TF from claiming all GPU memory so there is some left for pytorch.
gpus = tf.config.list_physical_devices('GPU')
if gpus:
  # Memory growth needs to be the same across GPUs.
  for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

import tensorflow_hub as hub
import tensorflow_text
import senteval
import time

PATH_TO_DATA = f'{os.getcwd()}/SentEval/data'
MODEL = 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1'
PARAMS = 'rapid prototyping'
TASK = 'CR'

params_prototyping = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params_prototyping['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
                                 'tenacity': 3, 'epoch_size': 2}

params_best = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params_best['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
                                 'tenacity': 5, 'epoch_size': 6}

params = params_best if PARAMS == 'slower, best performance' else params_prototyping

preprocessor = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder = hub.KerasLayer(
    "https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1")

inputs = tf.keras.Input(shape=tf.shape(''), dtype=tf.string)
outputs = encoder(preprocessor(inputs))

model = tf.keras.Model(inputs=inputs, outputs=outputs)

def prepare(params, samples):
    return

def batcher(_, batch):
    batch = [' '.join(sent) if sent else '.' for sent in batch]
    return model.predict(tf.constant(batch))["default"]


se = senteval.engine.SE(params, batcher, prepare)
print("Evaluating task %s with %s parameters" % (TASK, PARAMS))
start = time.time()
results = se.eval(TASK)
end = time.time()
print('Time took on task %s : %.1f. seconds' % (TASK, end - start))
print(results)

Evaluating task CR with rapid prototyping parameters
Time took on task CR : 53.0. seconds
{'devacc': 90.42, 'acc': 88.98, 'ndev': 3775, 'ntest': 3775}

了解更多

参考