 TensorFlow.org에서 보기 |
 Google Colab에서 실행하기 |
 GitHub에서 소스 보기 |
 노트북 다운로드하기 |
 TF Hub 모델 보기 |
이 colab에서는 문장 임베딩의 품질을 측정하기 위한 라이브러리인 SentEval 툴킷을 사용하여 범용 문장 인코더 CMLM 모델을 시연합니다. SentEval 툴킷에는 임베딩 모델의 일반화 능력을 평가하고 인코딩된 언어 속성을 평가할 수 있는 다양한 다운스트림 작업 세트가 포함되어 있습니다.
처음 두 코드 블록을 실행하여 환경을 설정하고 세 번째 코드 블록에서 SentEval 작업을 선택하여 모델을 평가할 수 있습니다. 이 Colab을 실행하려면 GPU 런타임을 사용하는 것이 좋습니다.
범용 문장 인코더 CMLM 모델에 대한 자세한 내용은 https://openreview.net/forum?id=WDVD4lUCTzU를 참조하십시오.
Install dependencies
pip install --quiet "tensorflow-text==2.8.*"pip install --quiet torch==1.8.1
SentEval 및 작업 데이터 다운로드
이 단계는 github에서 SentEval을 다운로드하고 데이터 스크립트를 실행하여 작업 데이터를 다운로드합니다. 완료하는 데 최대 5분이 소요될 수 있습니다.
Install SentEval and download task data
rm -rf ./SentEvalgit clone https://github.com/facebookresearch/SentEval.gitcd $PWD/SentEval/data/downstream && bash get_transfer_data.bash > /dev/null 2>&1
Cloning into 'SentEval'... remote: Enumerating objects: 691, done. remote: Counting objects: 50% (1/2) remote: Counting objects: 100% (2/2) remote: Counting objects: 100% (2/2), done. remote: Compressing objects: 50% (1/2) remote: Compressing objects: 100% (2/2) remote: Compressing objects: 100% (2/2), done. Receiving objects: 0% (1/691) Receiving objects: 1% (7/691) Receiving objects: 2% (14/691) Receiving objects: 3% (21/691) Receiving objects: 4% (28/691) Receiving objects: 5% (35/691) Receiving objects: 6% (42/691) Receiving objects: 7% (49/691) Receiving objects: 8% (56/691) Receiving objects: 9% (63/691) Receiving objects: 10% (70/691) Receiving objects: 11% (77/691) Receiving objects: 12% (83/691) Receiving objects: 13% (90/691) Receiving objects: 14% (97/691) Receiving objects: 15% (104/691) Receiving objects: 16% (111/691) Receiving objects: 17% (118/691) Receiving objects: 18% (125/691) Receiving objects: 19% (132/691) Receiving objects: 20% (139/691) Receiving objects: 21% (146/691) Receiving objects: 22% (153/691) Receiving objects: 23% (159/691) Receiving objects: 24% (166/691), 11.26 MiB | 22.51 MiB/s Receiving objects: 24% (168/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 25% (173/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 26% (180/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 27% (187/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 28% (194/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 29% (201/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 30% (208/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 31% (215/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 32% (222/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 33% (229/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 34% (235/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 35% (242/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 36% (249/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 37% (256/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 38% (263/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 39% (270/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 40% (277/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 41% (284/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 42% (291/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 43% (298/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 44% (305/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 45% (311/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 46% (318/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 47% (325/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 48% (332/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 49% (339/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 50% (346/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 51% (353/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 52% (360/691), 26.61 MiB | 26.60 MiB/s remote: Total 691 (delta 0), reused 2 (delta 0), pack-reused 689 Receiving objects: 53% (367/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 54% (374/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 55% (381/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 56% (387/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 57% (394/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 58% (401/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 59% (408/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 60% (415/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 61% (422/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 62% (429/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 63% (436/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 64% (443/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 65% (450/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 66% (457/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 67% (463/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 68% (470/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 69% (477/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 70% (484/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 71% (491/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 72% (498/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 73% (505/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 74% (512/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 75% (519/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 76% (526/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 77% (533/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 78% (539/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 79% (546/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 80% (553/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 81% (560/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 82% (567/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 83% (574/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 84% (581/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 85% (588/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 86% (595/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 87% (602/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 88% (609/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 89% (615/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 90% (622/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 91% (629/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 92% (636/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 93% (643/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 94% (650/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 95% (657/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 96% (664/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 97% (671/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 98% (678/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 99% (685/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 100% (691/691), 26.61 MiB | 26.60 MiB/s Receiving objects: 100% (691/691), 33.25 MiB | 27.52 MiB/s, done. Resolving deltas: 0% (0/434) Resolving deltas: 2% (9/434) Resolving deltas: 3% (16/434) Resolving deltas: 8% (36/434) Resolving deltas: 11% (48/434) Resolving deltas: 12% (55/434) Resolving deltas: 15% (68/434) Resolving deltas: 17% (78/434) Resolving deltas: 18% (79/434) Resolving deltas: 19% (86/434) Resolving deltas: 21% (93/434) Resolving deltas: 22% (96/434) Resolving deltas: 23% (101/434) Resolving deltas: 28% (124/434) Resolving deltas: 35% (154/434) Resolving deltas: 36% (157/434) Resolving deltas: 38% (166/434) Resolving deltas: 42% (183/434) Resolving deltas: 43% (187/434) Resolving deltas: 44% (191/434) Resolving deltas: 45% (198/434) Resolving deltas: 48% (209/434) Resolving deltas: 52% (230/434) Resolving deltas: 54% (235/434) Resolving deltas: 55% (243/434) Resolving deltas: 57% (251/434) Resolving deltas: 58% (254/434) Resolving deltas: 59% (260/434) Resolving deltas: 60% (261/434) Resolving deltas: 61% (267/434) Resolving deltas: 63% (276/434) Resolving deltas: 64% (278/434) Resolving deltas: 65% (285/434) Resolving deltas: 66% (290/434) Resolving deltas: 67% (293/434) Resolving deltas: 68% (298/434) Resolving deltas: 69% (300/434) Resolving deltas: 70% (305/434) Resolving deltas: 73% (320/434) Resolving deltas: 74% (323/434) Resolving deltas: 75% (329/434) Resolving deltas: 76% (330/434) Resolving deltas: 79% (344/434) Resolving deltas: 83% (363/434) Resolving deltas: 87% (380/434) Resolving deltas: 89% (387/434) Resolving deltas: 90% (391/434) Resolving deltas: 91% (396/434) Resolving deltas: 93% (404/434) Resolving deltas: 95% (413/434) Resolving deltas: 97% (422/434) Resolving deltas: 99% (431/434) Resolving deltas: 100% (434/434) Resolving deltas: 100% (434/434), done.
SentEval 평가 작업 실행 다음 코드 블록은 SentEval 작업을 실행하고, 결과를 출력하며, 다음 작업 중 하나를 선택하여 USE CMLM 모델을 평가합니다.
MR CR SUBJ MPQA SST TREC MRPC SICK-E
실행할 모델, 매개변수 및 작업을 선택합니다. 더 빠른 결과를 도출하기 위해 신속한 프로토타이핑 매개변수를 계산 시간을 줄이는 데 사용할 수 있습니다.
일반적으로 '빠른 프로토타이핑' 매개변수로 작업을 완료하는 데 5-15분이 소요되고 '더 느리고 최고의 성능' 매개변수로 작업을 완료하는 데 최대 1시간이 걸립니다.
params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
'tenacity': 3, 'epoch_size': 2}
더 나은 결과를 얻으려면 더 느린 '더 느리고 최고의 성능' 매개변수를 사용하십시오. 계산에 최대 1시간이 소요될 수 있습니다.
params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
'tenacity': 5, 'epoch_size': 6}
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import sys
sys.path.append(f'{os.getcwd()}/SentEval')
import tensorflow as tf
# Prevent TF from claiming all GPU memory so there is some left for pytorch.
gpus = tf.config.list_physical_devices('GPU')
if gpus:
# Memory growth needs to be the same across GPUs.
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
import tensorflow_hub as hub
import tensorflow_text
import senteval
import time
PATH_TO_DATA = f'{os.getcwd()}/SentEval/data'
MODEL = 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1'
PARAMS = 'rapid prototyping'
TASK = 'CR'
params_prototyping = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}
params_prototyping['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,
'tenacity': 3, 'epoch_size': 2}
params_best = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}
params_best['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,
'tenacity': 5, 'epoch_size': 6}
params = params_best if PARAMS == 'slower, best performance' else params_prototyping
preprocessor = hub.KerasLayer(
"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder = hub.KerasLayer(
"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1")
inputs = tf.keras.Input(shape=tf.shape(''), dtype=tf.string)
outputs = encoder(preprocessor(inputs))
model = tf.keras.Model(inputs=inputs, outputs=outputs)
def prepare(params, samples):
return
def batcher(_, batch):
batch = [' '.join(sent) if sent else '.' for sent in batch]
return model.predict(tf.constant(batch))["default"]
se = senteval.engine.SE(params, batcher, prepare)
print("Evaluating task %s with %s parameters" % (TASK, PARAMS))
start = time.time()
results = se.eval(TASK)
end = time.time()
print('Time took on task %s : %.1f. seconds' % (TASK, end - start))
print(results)
Evaluating task CR with rapid prototyping parameters
Time took on task CR : 52.8. seconds
{'devacc': 90.42, 'acc': 88.98, 'ndev': 3775, 'ntest': 3775}
자세히 알아보기
- TensorFlow Hub에서 더 많은 텍스트 임베딩 모델 찾기
- 다국어 범용 문장 인코더 CMLM 모델 참조
- 다른 범용 문장 인코더 모델 확인
참고
- Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, Eric Darve.
조건부 마스크 언어 모델을 사용한 범용 문장 표현 학습. 2020년 11월