TF-Hub'dan BERT Uzmanları

TensorFlow.org'da görüntüleyin

Google Colab'da çalıştırın

GitHub'da görüntüle

Not defterini indir

TF Hub modellerine bakın

Bu ortak çalışma şunların nasıl yapılacağını gösterir:

Yük Bert modelleri TensorFlow Hub MNLI, Squad ve PubMed dahil olmak üzere farklı görevler üzerinde eğitim verildi
Ham metni belirtmek ve kimliklere dönüştürmek için eşleşen bir ön işleme modeli kullanın
Yüklenen modeli kullanarak belirteç giriş kimliklerinden havuzlanmış ve sıralı çıktı oluşturun
Farklı cümlelerin birleştirilmiş çıktılarının anlamsal benzerliğine bakın

Not: Bu ortak çalışma, bir GPU çalışma zamanı ile çalıştırılmalıdır

Kurulum ve içe aktarma

pip3 install --quiet tensorflow
pip3 install --quiet tensorflow_text

import seaborn as sns
from sklearn.metrics import pairwise

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  # Imports TF ops for preprocessing.

Modeli yapılandırın

BERT_MODEL = "https://tfhub.dev/google/experts/bert/wiki_books/2" # @param {type: "string"} ["https://tfhub.dev/google/experts/bert/wiki_books/2", "https://tfhub.dev/google/experts/bert/wiki_books/mnli/2", "https://tfhub.dev/google/experts/bert/wiki_books/qnli/2", "https://tfhub.dev/google/experts/bert/wiki_books/qqp/2", "https://tfhub.dev/google/experts/bert/wiki_books/squad2/2", "https://tfhub.dev/google/experts/bert/wiki_books/sst2/2",  "https://tfhub.dev/google/experts/bert/pubmed/2", "https://tfhub.dev/google/experts/bert/pubmed/squad2/2"]
# Preprocessing must match the model, but all the above use the same.
PREPROCESS_MODEL = "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3"

Cümleler

Modelden geçmek için Wikipedia'dan bazı cümleler alalım

sentences = [
  "Here We Go Then, You And I is a 1999 album by Norwegian pop artist Morten Abel. It was Abel's second CD as a solo artist.",
  "The album went straight to number one on the Norwegian album chart, and sold to double platinum.",
  "Among the singles released from the album were the songs \"Be My Lover\" and \"Hard To Stay Awake\".",
  "Riccardo Zegna is an Italian jazz musician.",
  "Rajko Maksimović is a composer, writer, and music pedagogue.",
  "One of the most significant Serbian composers of our time, Maksimović has been and remains active in creating works for different ensembles.",
  "Ceylon spinach is a common name for several plants and may refer to: Basella alba Talinum fruticosum",
  "A solar eclipse occurs when the Moon passes between Earth and the Sun, thereby totally or partly obscuring the image of the Sun for a viewer on Earth.",
  "A partial solar eclipse occurs in the polar regions of the Earth when the center of the Moon's shadow misses the Earth.",
]

modeli çalıştır

TF-Hub'dan BERT modelini yükleyeceğiz, TF-Hub'dan eşleşen ön işleme modelini kullanarak cümlelerimizi tokenleştireceğiz, sonra tokenize edilmiş cümleleri modele besleyeceğiz. Bu ortak çalışmayı hızlı ve basit tutmak için GPU üzerinde çalıştırmanızı öneririz.

GPU seçili olduğundan emin olmak için Süre → Değişim zamanı tipi git

preprocess = hub.load(PREPROCESS_MODEL)
bert = hub.load(BERT_MODEL)
inputs = preprocess(sentences)
outputs = bert(inputs)

print("Sentences:")
print(sentences)

print("\nBERT inputs:")
print(inputs)

print("\nPooled embeddings:")
print(outputs["pooled_output"])

print("\nPer token embeddings:")
print(outputs["sequence_output"])

Sentences:
["Here We Go Then, You And I is a 1999 album by Norwegian pop artist Morten Abel. It was Abel's second CD as a solo artist.", 'The album went straight to number one on the Norwegian album chart, and sold to double platinum.', 'Among the singles released from the album were the songs "Be My Lover" and "Hard To Stay Awake".', 'Riccardo Zegna is an Italian jazz musician.', 'Rajko Maksimović is a composer, writer, and music pedagogue.', 'One of the most significant Serbian composers of our time, Maksimović has been and remains active in creating works for different ensembles.', 'Ceylon spinach is a common name for several plants and may refer to: Basella alba Talinum fruticosum', 'A solar eclipse occurs when the Moon passes between Earth and the Sun, thereby totally or partly obscuring the image of the Sun for a viewer on Earth.', "A partial solar eclipse occurs in the polar regions of the Earth when the center of the Moon's shadow misses the Earth."]

BERT inputs:
{'input_word_ids': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[  101,  2182,  2057, ...,     0,     0,     0],
       [  101,  1996,  2201, ...,     0,     0,     0],
       [  101,  2426,  1996, ...,     0,     0,     0],
       ...,
       [  101, 16447,  6714, ...,     0,     0,     0],
       [  101,  1037,  5943, ...,     0,     0,     0],
       [  101,  1037,  7704, ...,     0,     0,     0]], dtype=int32)>, 'input_type_ids': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>, 'input_mask': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>}

Pooled embeddings:
tf.Tensor(
[[ 0.7975967  -0.48580563  0.49781477 ... -0.3448825   0.3972752
  -0.2063976 ]
 [ 0.57120323 -0.41205275  0.7048914  ... -0.35185075  0.19032307
  -0.4041895 ]
 [-0.699383    0.1586691   0.06569938 ... -0.0623244  -0.81550187
  -0.07923658]
 ...
 [-0.35727128  0.7708977   0.1575658  ...  0.44185698 -0.8644815
   0.04504769]
 [ 0.91077     0.41501352  0.5606345  ... -0.49263868  0.39640594
  -0.05036103]
 [ 0.90502906 -0.15505145  0.72672117 ... -0.34734493  0.5052651
  -0.19543159]], shape=(9, 768), dtype=float32)

Per token embeddings:
tf.Tensor(
[[[ 1.0919718e+00 -5.3055555e-01  5.4639673e-01 ... -3.5962367e-01
    4.2040938e-01 -2.0940571e-01]
  [ 1.0143853e+00  7.8079259e-01  8.5375798e-01 ...  5.5282074e-01
   -1.1245787e+00  5.6027526e-01]
  [ 7.8862888e-01  7.7776514e-02  9.5150793e-01 ... -1.9075295e-01
    5.9206045e-01  6.1910731e-01]
  ...
  [-3.2203159e-01 -4.2521179e-01 -1.2823829e-01 ... -3.9094865e-01
   -7.9097575e-01  4.2236605e-01]
  [-3.1039350e-02  2.3985808e-01 -2.1994556e-01 ... -1.1440065e-01
   -1.2680519e+00 -1.6136172e-01]
  [-4.2063516e-01  5.4972863e-01 -3.2444897e-01 ... -1.8478543e-01
   -1.1342984e+00 -5.8974154e-02]]

 [[ 6.4930701e-01 -4.3808129e-01  8.7695646e-01 ... -3.6755449e-01
    1.9267237e-01 -4.2864648e-01]
  [-1.1248719e+00  2.9931602e-01  1.1799662e+00 ...  4.8729455e-01
    5.3400528e-01  2.2836192e-01]
  [-2.7057338e-01  3.2351881e-02  1.0425698e+00 ...  5.8993816e-01
    1.5367918e+00  5.8425623e-01]
  ...
  [-1.4762508e+00  1.8239072e-01  5.5875197e-02 ... -1.6733241e+00
   -6.7398834e-01 -7.2449744e-01]
  [-1.5138135e+00  5.8184558e-01  1.6141933e-01 ... -1.2640834e+00
   -4.0272138e-01 -9.7197199e-01]
  [-4.7153085e-01  2.2817247e-01  5.2776134e-01 ... -7.5483751e-01
   -9.0903056e-01 -1.6954714e-01]]

 [[-8.6609173e-01  1.6002113e-01  6.5794155e-02 ... -6.2405296e-02
   -1.1432388e+00 -7.9403043e-02]
  [ 7.7117836e-01  7.0804822e-01  1.1350115e-01 ...  7.8831035e-01
   -3.1438148e-01 -9.7487110e-01]
  [-4.4002479e-01 -3.0059522e-01  3.5479453e-01 ...  7.9739094e-02
   -4.7393662e-01 -1.1001848e+00]
  ...
  [-1.0205302e+00  2.6938522e-01 -4.7310370e-01 ... -6.6319543e-01
   -1.4579915e+00 -3.4665459e-01]
  [-9.7003460e-01 -4.5014530e-02 -5.9779549e-01 ... -3.0526626e-01
   -1.2744237e+00 -2.8051588e-01]
  [-7.3144108e-01  1.7699355e-01 -4.6257967e-01 ... -1.6062307e-01
   -1.6346070e+00 -3.2060605e-01]]

 ...

 [[-3.7375441e-01  1.0225365e+00  1.5888955e-01 ...  4.7453594e-01
   -1.3108152e+00  4.5078207e-02]
  [-4.1589144e-01  5.0019276e-01 -4.5844245e-01 ...  4.1482472e-01
   -6.2065876e-01 -7.1555024e-01]
  [-1.2504390e+00  5.0936425e-01 -5.7103634e-01 ...  3.5491806e-01
    2.4368477e-01 -2.0577228e+00]
  ...
  [ 1.3393667e-01  1.1859171e+00 -2.2169831e-01 ... -8.1946820e-01
   -1.6737309e+00 -3.9692628e-01]
  [-3.3662504e-01  1.6556220e+00 -3.7812781e-01 ... -9.6745497e-01
   -1.4801039e+00 -8.3330971e-01]
  [-2.2649485e-01  1.6178465e+00 -6.7044652e-01 ... -4.9078423e-01
   -1.4535751e+00 -7.1707505e-01]]

 [[ 1.5320227e+00  4.4165283e-01  6.3375801e-01 ... -5.3953874e-01
    4.1937760e-01 -5.0403677e-02]
  [ 8.9377600e-01  8.9395344e-01  3.0626178e-02 ...  5.9039176e-02
   -2.0649448e-01 -8.4811246e-01]
  [-1.8557828e-02  1.0479081e+00 -1.3329606e+00 ... -1.3869843e-01
   -3.7879568e-01 -4.9068305e-01]
  ...
  [ 1.4275622e+00  1.0696816e-01 -4.0635362e-02 ... -3.1778324e-02
   -4.1460156e-01  7.0036823e-01]
  [ 1.1286633e+00  1.4547651e-01 -6.1372471e-01 ...  4.7491628e-01
   -3.9852056e-01  4.3124324e-01]
  [ 1.4393284e+00  1.8030575e-01 -4.2854339e-01 ... -2.5022790e-01
   -1.0000544e+00  3.5985461e-01]]

 [[ 1.4993407e+00 -1.5631223e-01  9.2174333e-01 ... -3.6242130e-01
    5.5635113e-01 -1.9797830e-01]
  [ 1.1110539e+00  3.6651433e-01  3.5505858e-01 ... -5.4297698e-01
    1.4471304e-01 -3.1675813e-01]
  [ 2.4048802e-01  3.8115788e-01 -5.9182465e-01 ...  3.7410852e-01
   -5.9829473e-01 -1.0166264e+00]
  ...
  [ 1.0158644e+00  5.0260526e-01  1.0737082e-01 ... -9.5642781e-01
   -4.1039532e-01 -2.6760197e-01]
  [ 1.1848929e+00  6.5479934e-01  1.0166168e-03 ... -8.6154389e-01
   -8.8036627e-02 -3.0636966e-01]
  [ 1.2669108e+00  4.7768092e-01  6.6289604e-03 ... -1.1585802e+00
   -7.0675731e-02 -1.8678737e-01]]], shape=(9, 128, 768), dtype=float32)

anlamsal benzerlik

Şimdi de tabloya bakalım pooled_output bizim cümlelerin içelik ve onlar cümlelerin karşısında ne kadar benzer karşılaştırın.

yardımcı fonksiyonlar

def plot_similarity(features, labels):
  """Plot a similarity matrix of the embeddings."""
  cos_sim = pairwise.cosine_similarity(features)
  sns.set(font_scale=1.2)
  cbar_kws=dict(use_gridspec=False, location="left")
  g = sns.heatmap(
      cos_sim, xticklabels=labels, yticklabels=labels,
      vmin=0, vmax=1, cmap="Blues", cbar_kws=cbar_kws)
  g.tick_params(labelright=True, labelleft=False)
  g.set_yticklabels(labels, rotation=0)
  g.set_title("Semantic Textual Similarity")

plot_similarity(outputs["pooled_output"], sentences)

png

Daha fazla bilgi edin

Daha Bert modellerini bul TensorFlow Hub
Bu defter bert ile basit çıkarım gösteriyor, en ince ayar ya Bert daha gelişmiş bir öğretici bulabilirsiniz tensorflow.org/official_models/fine_tuning_bert
Size de tf.distribute kullanarak nasıl yük modellerine hakkında daha fazla bilgi edinebilir, modelini çalıştırmak için sadece bir GPU çipi kullanılan tensorflow.org/tutorials/distribute/save_and_load