TF-Hub로 간단한 텍스트 분류자를 빌드하는 방법

참고: 이 튜토리얼에서는 지원 중단된 TensorFlow 1 기능을 사용합니다. 이 작업에 대한 최신 접근 방식은 TensorFlow 2 버전을 참조하세요.

Google Colab에서 실행

GitHub 소스 보기

TF Hub 모델보기

TF-Hub는 재사용 가능한 리소스, 특히 사전 훈련된 모듈 형태로 머신러닝 전문 지식을 공유하는 플랫폼입니다. 이 튜토리얼은 크게 두 부분으로 구성되어 있습니다.

소개: TF-Hub로 텍스트 분류자 훈련하기

TF-Hub 텍스트 임베딩 모듈을 사용하여 합리적인 기준 정확성으로 간단한 감상 분류자를 훈련합니다. 그런 다음, 예측을 분석하여 모델이 합리적인지 확인하고 정확성을 높이기 위한 개선점을 제안합니다.

고급: 전이 학습 분석

이 섹션에서는 다양한 TF-Hub 모듈을 사용하여 추정기(estimator)의 정확성에 미치는 영향을 비교하고 전이 학습의 장점과 함정을 보여줍니다.

선택적 전제 조건

Tensorflow 사전 제작된 추정기 프레임워크에 대한 기본적 이해
Pandas 라이브러리에 익숙

설정

# Install TF-Hub.
pip install -q seaborn

Tensorflow 설치에 대한 자세한 정보는 https://www.tensorflow.org/install/에서 찾을 수 있습니다.

from absl import logging

import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import re
import seaborn as sns

시작하기

데이터

Large Movie Review Dataset v1.0 과제(Mass 등, 2011)의 해결을 시도할 것입니다. 이 데이터세트는 긍정성을 기준으로 1부터 10까지 분류된 IMDB 영화 리뷰로 구성됩니다. 리뷰를 부정적 또는 긍정적으로 분류하는 것이 과제입니다.

# Load all files from a directory in a DataFrame.
def load_directory_data(directory):
  data = {}
  data["sentence"] = []
  data["sentiment"] = []
  for file_path in os.listdir(directory):
    with tf.io.gfile.GFile(os.path.join(directory, file_path), "r") as f:
      data["sentence"].append(f.read())
      data["sentiment"].append(re.match("\d+_(\d+)\.txt", file_path).group(1))
  return pd.DataFrame.from_dict(data)

# Merge positive and negative examples, add a polarity column and shuffle.
def load_dataset(directory):
  pos_df = load_directory_data(os.path.join(directory, "pos"))
  neg_df = load_directory_data(os.path.join(directory, "neg"))
  pos_df["polarity"] = 1
  neg_df["polarity"] = 0
  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)

# Download and process the dataset files.
def download_and_load_datasets(force_download=False):
  dataset = tf.keras.utils.get_file(
      fname="aclImdb.tar.gz", 
      origin="http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz", 
      extract=True)

  train_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                       "aclImdb", "train"))
  test_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                      "aclImdb", "test"))

  return train_df, test_df

# Reduce logging output.
logging.set_verbosity(logging.ERROR)

train_df, test_df = download_and_load_datasets()
train_df.head()

Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
84131840/84125825 [==============================] - 7s 0us/step

모델

입력 함수

Estimator 프레임워크는 Pandas 데이터프레임을 래핑하는 입력 함수를 제공합니다.

# Training input on the whole training set with no limit on training epochs.
train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    train_df, train_df["polarity"], num_epochs=None, shuffle=True)

# Prediction on the whole training set.
predict_train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    train_df, train_df["polarity"], shuffle=False)
# Prediction on the test set.
predict_test_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
    test_df, test_df["polarity"], shuffle=False)

특성 열

TF-Hub는 주어진 텍스트 특성에 모듈을 적용하고 모듈의 출력을 추가적으로 전달하는 특성 열을 제공합니다. 이 튜토리얼에서는 nnlm-en-dim128 모듈을 사용합니다. 이 튜토리얼의 목적에 비추어 가장 중요한 사실은 다음과 같습니다.

모듈은 1-D 문자열 텐서의 문장 배치를 입력으로 사용합니다.
모듈은 문장의 전처리(예: 구두점 제거 및 공백 분할)를 담당합니다.
모듈은 모든 입력에서 동작합니다(예: nnlm-en-dim128은 어휘에 없는 단어를 ~20.000개 버킷으로 해시 처리함).

embedded_text_feature_column = hub.text_embedding_column(
    key="sentence", 
    module_spec="https://tfhub.dev/google/nnlm-en-dim128/1")

추정기

분류를 위해 DNN 분류자를 사용할 수 있습니다(튜토리얼 마지막에 있는 레이블 함수의 다른 모델링에 대한 추가 설명 참고).

estimator = tf.estimator.DNNClassifier(
    hidden_units=[500, 100],
    feature_columns=[embedded_text_feature_column],
    n_classes=2,
    optimizer=tf.keras.optimizers.Adagrad(lr=0.003))

INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp25p8qgdi
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp25p8qgdi
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp25p8qgdi', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp25p8qgdi', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

훈련

적절한 스텝 수만큼 추정기를 훈련합니다.

# Training for 5,000 steps means 640,000 training examples with the default
# batch size. This is roughly equivalent to 25 epochs since the training dataset
# contains 25,000 examples.
estimator.train(input_fn=train_input_fn, steps=5000);

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:65: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:65: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:491: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:491: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py:906: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py:906: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp25p8qgdi/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp25p8qgdi/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 0.7182708, step = 0
INFO:tensorflow:loss = 0.7182708, step = 0
INFO:tensorflow:global_step/sec: 127.617
INFO:tensorflow:global_step/sec: 127.617
INFO:tensorflow:loss = 0.6844012, step = 100 (0.785 sec)
INFO:tensorflow:loss = 0.6844012, step = 100 (0.785 sec)
INFO:tensorflow:global_step/sec: 131.405
INFO:tensorflow:global_step/sec: 131.405
INFO:tensorflow:loss = 0.6741289, step = 200 (0.761 sec)
INFO:tensorflow:loss = 0.6741289, step = 200 (0.761 sec)
INFO:tensorflow:global_step/sec: 136.965
INFO:tensorflow:global_step/sec: 136.965
INFO:tensorflow:loss = 0.6673078, step = 300 (0.730 sec)
INFO:tensorflow:loss = 0.6673078, step = 300 (0.730 sec)
INFO:tensorflow:global_step/sec: 136.714
INFO:tensorflow:global_step/sec: 136.714
INFO:tensorflow:loss = 0.6552642, step = 400 (0.731 sec)
INFO:tensorflow:loss = 0.6552642, step = 400 (0.731 sec)
INFO:tensorflow:global_step/sec: 136.346
INFO:tensorflow:global_step/sec: 136.346
INFO:tensorflow:loss = 0.6353569, step = 500 (0.733 sec)
INFO:tensorflow:loss = 0.6353569, step = 500 (0.733 sec)
INFO:tensorflow:global_step/sec: 137.661
INFO:tensorflow:global_step/sec: 137.661
INFO:tensorflow:loss = 0.64040166, step = 600 (0.727 sec)
INFO:tensorflow:loss = 0.64040166, step = 600 (0.727 sec)
INFO:tensorflow:global_step/sec: 137.399
INFO:tensorflow:global_step/sec: 137.399
INFO:tensorflow:loss = 0.6415657, step = 700 (0.728 sec)
INFO:tensorflow:loss = 0.6415657, step = 700 (0.728 sec)
INFO:tensorflow:global_step/sec: 137.674
INFO:tensorflow:global_step/sec: 137.674
INFO:tensorflow:loss = 0.57805824, step = 800 (0.726 sec)
INFO:tensorflow:loss = 0.57805824, step = 800 (0.726 sec)
INFO:tensorflow:global_step/sec: 135.23
INFO:tensorflow:global_step/sec: 135.23
INFO:tensorflow:loss = 0.6090031, step = 900 (0.740 sec)
INFO:tensorflow:loss = 0.6090031, step = 900 (0.740 sec)
INFO:tensorflow:global_step/sec: 135.593
INFO:tensorflow:global_step/sec: 135.593
INFO:tensorflow:loss = 0.6005795, step = 1000 (0.737 sec)
INFO:tensorflow:loss = 0.6005795, step = 1000 (0.737 sec)
INFO:tensorflow:global_step/sec: 138.183
INFO:tensorflow:global_step/sec: 138.183
INFO:tensorflow:loss = 0.5641889, step = 1100 (0.723 sec)
INFO:tensorflow:loss = 0.5641889, step = 1100 (0.723 sec)
INFO:tensorflow:global_step/sec: 134.507
INFO:tensorflow:global_step/sec: 134.507
INFO:tensorflow:loss = 0.5737369, step = 1200 (0.744 sec)
INFO:tensorflow:loss = 0.5737369, step = 1200 (0.744 sec)
INFO:tensorflow:global_step/sec: 123.168
INFO:tensorflow:global_step/sec: 123.168
INFO:tensorflow:loss = 0.5868721, step = 1300 (0.812 sec)
INFO:tensorflow:loss = 0.5868721, step = 1300 (0.812 sec)
INFO:tensorflow:global_step/sec: 128.158
INFO:tensorflow:global_step/sec: 128.158
INFO:tensorflow:loss = 0.49987888, step = 1400 (0.780 sec)
INFO:tensorflow:loss = 0.49987888, step = 1400 (0.780 sec)
INFO:tensorflow:global_step/sec: 129.335
INFO:tensorflow:global_step/sec: 129.335
INFO:tensorflow:loss = 0.5376605, step = 1500 (0.773 sec)
INFO:tensorflow:loss = 0.5376605, step = 1500 (0.773 sec)
INFO:tensorflow:global_step/sec: 125.539
INFO:tensorflow:global_step/sec: 125.539
INFO:tensorflow:loss = 0.49874097, step = 1600 (0.797 sec)
INFO:tensorflow:loss = 0.49874097, step = 1600 (0.797 sec)
INFO:tensorflow:global_step/sec: 120.396
INFO:tensorflow:global_step/sec: 120.396
INFO:tensorflow:loss = 0.48102918, step = 1700 (0.830 sec)
INFO:tensorflow:loss = 0.48102918, step = 1700 (0.830 sec)
INFO:tensorflow:global_step/sec: 127.267
INFO:tensorflow:global_step/sec: 127.267
INFO:tensorflow:loss = 0.5827813, step = 1800 (0.786 sec)
INFO:tensorflow:loss = 0.5827813, step = 1800 (0.786 sec)
INFO:tensorflow:global_step/sec: 125.768
INFO:tensorflow:global_step/sec: 125.768
INFO:tensorflow:loss = 0.49500355, step = 1900 (0.795 sec)
INFO:tensorflow:loss = 0.49500355, step = 1900 (0.795 sec)
INFO:tensorflow:global_step/sec: 125.109
INFO:tensorflow:global_step/sec: 125.109
INFO:tensorflow:loss = 0.49588934, step = 2000 (0.800 sec)
INFO:tensorflow:loss = 0.49588934, step = 2000 (0.800 sec)
INFO:tensorflow:global_step/sec: 124.604
INFO:tensorflow:global_step/sec: 124.604
INFO:tensorflow:loss = 0.5236983, step = 2100 (0.802 sec)
INFO:tensorflow:loss = 0.5236983, step = 2100 (0.802 sec)
INFO:tensorflow:global_step/sec: 126.21
INFO:tensorflow:global_step/sec: 126.21
INFO:tensorflow:loss = 0.5347569, step = 2200 (0.792 sec)
INFO:tensorflow:loss = 0.5347569, step = 2200 (0.792 sec)
INFO:tensorflow:global_step/sec: 124.531
INFO:tensorflow:global_step/sec: 124.531
INFO:tensorflow:loss = 0.5205074, step = 2300 (0.803 sec)
INFO:tensorflow:loss = 0.5205074, step = 2300 (0.803 sec)
INFO:tensorflow:global_step/sec: 123.366
INFO:tensorflow:global_step/sec: 123.366
INFO:tensorflow:loss = 0.49978924, step = 2400 (0.811 sec)
INFO:tensorflow:loss = 0.49978924, step = 2400 (0.811 sec)
INFO:tensorflow:global_step/sec: 137.342
INFO:tensorflow:global_step/sec: 137.342
INFO:tensorflow:loss = 0.48663512, step = 2500 (0.728 sec)
INFO:tensorflow:loss = 0.48663512, step = 2500 (0.728 sec)
INFO:tensorflow:global_step/sec: 136.938
INFO:tensorflow:global_step/sec: 136.938
INFO:tensorflow:loss = 0.48076186, step = 2600 (0.730 sec)
INFO:tensorflow:loss = 0.48076186, step = 2600 (0.730 sec)
INFO:tensorflow:global_step/sec: 137.33
INFO:tensorflow:global_step/sec: 137.33
INFO:tensorflow:loss = 0.49339342, step = 2700 (0.728 sec)
INFO:tensorflow:loss = 0.49339342, step = 2700 (0.728 sec)
INFO:tensorflow:global_step/sec: 134.956
INFO:tensorflow:global_step/sec: 134.956
INFO:tensorflow:loss = 0.43466532, step = 2800 (0.741 sec)
INFO:tensorflow:loss = 0.43466532, step = 2800 (0.741 sec)
INFO:tensorflow:global_step/sec: 135.938
INFO:tensorflow:global_step/sec: 135.938
INFO:tensorflow:loss = 0.44667464, step = 2900 (0.735 sec)
INFO:tensorflow:loss = 0.44667464, step = 2900 (0.735 sec)
INFO:tensorflow:global_step/sec: 137.858
INFO:tensorflow:global_step/sec: 137.858
INFO:tensorflow:loss = 0.55256486, step = 3000 (0.725 sec)
INFO:tensorflow:loss = 0.55256486, step = 3000 (0.725 sec)
INFO:tensorflow:global_step/sec: 134.744
INFO:tensorflow:global_step/sec: 134.744
INFO:tensorflow:loss = 0.43467033, step = 3100 (0.742 sec)
INFO:tensorflow:loss = 0.43467033, step = 3100 (0.742 sec)
INFO:tensorflow:global_step/sec: 130.146
INFO:tensorflow:global_step/sec: 130.146
INFO:tensorflow:loss = 0.37364233, step = 3200 (0.769 sec)
INFO:tensorflow:loss = 0.37364233, step = 3200 (0.769 sec)
INFO:tensorflow:global_step/sec: 133.063
INFO:tensorflow:global_step/sec: 133.063
INFO:tensorflow:loss = 0.5110926, step = 3300 (0.751 sec)
INFO:tensorflow:loss = 0.5110926, step = 3300 (0.751 sec)
INFO:tensorflow:global_step/sec: 135.087
INFO:tensorflow:global_step/sec: 135.087
INFO:tensorflow:loss = 0.39641047, step = 3400 (0.740 sec)
INFO:tensorflow:loss = 0.39641047, step = 3400 (0.740 sec)
INFO:tensorflow:global_step/sec: 138.62
INFO:tensorflow:global_step/sec: 138.62
INFO:tensorflow:loss = 0.46425578, step = 3500 (0.722 sec)
INFO:tensorflow:loss = 0.46425578, step = 3500 (0.722 sec)
INFO:tensorflow:global_step/sec: 137.898
INFO:tensorflow:global_step/sec: 137.898
INFO:tensorflow:loss = 0.44300562, step = 3600 (0.725 sec)
INFO:tensorflow:loss = 0.44300562, step = 3600 (0.725 sec)
INFO:tensorflow:global_step/sec: 137.631
INFO:tensorflow:global_step/sec: 137.631
INFO:tensorflow:loss = 0.42918092, step = 3700 (0.727 sec)
INFO:tensorflow:loss = 0.42918092, step = 3700 (0.727 sec)
INFO:tensorflow:global_step/sec: 137.969
INFO:tensorflow:global_step/sec: 137.969
INFO:tensorflow:loss = 0.43171275, step = 3800 (0.725 sec)
INFO:tensorflow:loss = 0.43171275, step = 3800 (0.725 sec)
INFO:tensorflow:global_step/sec: 138.155
INFO:tensorflow:global_step/sec: 138.155
INFO:tensorflow:loss = 0.4509111, step = 3900 (0.724 sec)
INFO:tensorflow:loss = 0.4509111, step = 3900 (0.724 sec)
INFO:tensorflow:global_step/sec: 140.929
INFO:tensorflow:global_step/sec: 140.929
INFO:tensorflow:loss = 0.46139807, step = 4000 (0.709 sec)
INFO:tensorflow:loss = 0.46139807, step = 4000 (0.709 sec)
INFO:tensorflow:global_step/sec: 134.56
INFO:tensorflow:global_step/sec: 134.56
INFO:tensorflow:loss = 0.468181, step = 4100 (0.743 sec)
INFO:tensorflow:loss = 0.468181, step = 4100 (0.743 sec)
INFO:tensorflow:global_step/sec: 132.27
INFO:tensorflow:global_step/sec: 132.27
INFO:tensorflow:loss = 0.41919896, step = 4200 (0.756 sec)
INFO:tensorflow:loss = 0.41919896, step = 4200 (0.756 sec)
INFO:tensorflow:global_step/sec: 131.276
INFO:tensorflow:global_step/sec: 131.276
INFO:tensorflow:loss = 0.48834482, step = 4300 (0.762 sec)
INFO:tensorflow:loss = 0.48834482, step = 4300 (0.762 sec)
INFO:tensorflow:global_step/sec: 139.647
INFO:tensorflow:global_step/sec: 139.647
INFO:tensorflow:loss = 0.4555532, step = 4400 (0.716 sec)
INFO:tensorflow:loss = 0.4555532, step = 4400 (0.716 sec)
INFO:tensorflow:global_step/sec: 138.475
INFO:tensorflow:global_step/sec: 138.475
INFO:tensorflow:loss = 0.39587742, step = 4500 (0.722 sec)
INFO:tensorflow:loss = 0.39587742, step = 4500 (0.722 sec)
INFO:tensorflow:global_step/sec: 137.51
INFO:tensorflow:global_step/sec: 137.51
INFO:tensorflow:loss = 0.39814836, step = 4600 (0.727 sec)
INFO:tensorflow:loss = 0.39814836, step = 4600 (0.727 sec)
INFO:tensorflow:global_step/sec: 137.762
INFO:tensorflow:global_step/sec: 137.762
INFO:tensorflow:loss = 0.4029638, step = 4700 (0.726 sec)
INFO:tensorflow:loss = 0.4029638, step = 4700 (0.726 sec)
INFO:tensorflow:global_step/sec: 137.791
INFO:tensorflow:global_step/sec: 137.791
INFO:tensorflow:loss = 0.4339965, step = 4800 (0.726 sec)
INFO:tensorflow:loss = 0.4339965, step = 4800 (0.726 sec)
INFO:tensorflow:global_step/sec: 136.404
INFO:tensorflow:global_step/sec: 136.404
INFO:tensorflow:loss = 0.40314075, step = 4900 (0.733 sec)
INFO:tensorflow:loss = 0.40314075, step = 4900 (0.733 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5000...
INFO:tensorflow:Saving checkpoints for 5000 into /tmp/tmp25p8qgdi/model.ckpt.
INFO:tensorflow:Saving checkpoints for 5000 into /tmp/tmp25p8qgdi/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5000...
INFO:tensorflow:Loss for final step: 0.48744428.
INFO:tensorflow:Loss for final step: 0.48744428.
<tensorflow_estimator.python.estimator.canned.dnn.DNNClassifierV2 at 0x7f8e2fd17c50>

예측

훈련 및 테스트 세트 모두에 대한 예측을 실행합니다.

train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)

print("Training set accuracy: {accuracy}".format(**train_eval_result))
print("Test set accuracy: {accuracy}".format(**test_eval_result))

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:03Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:03Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.81372s
INFO:tensorflow:Inference Time : 2.81372s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:06
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:06
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.796, accuracy_baseline = 0.5, auc = 0.87709147, auc_precision_recall = 0.8776079, average_loss = 0.44140813, global_step = 5000, label/mean = 0.5, loss = 0.4414658, precision = 0.80342793, prediction/mean = 0.4937591, recall = 0.78376
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.796, accuracy_baseline = 0.5, auc = 0.87709147, auc_precision_recall = 0.8776079, average_loss = 0.44140813, global_step = 5000, label/mean = 0.5, loss = 0.4414658, precision = 0.80342793, prediction/mean = 0.4937591, recall = 0.78376
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:07Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:07Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.73390s
INFO:tensorflow:Inference Time : 2.73390s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:10
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:10
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78724, accuracy_baseline = 0.5, auc = 0.87125826, auc_precision_recall = 0.8732454, average_loss = 0.45014507, global_step = 5000, label/mean = 0.5, loss = 0.4506318, precision = 0.7999332, prediction/mean = 0.48966122, recall = 0.76608
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78724, accuracy_baseline = 0.5, auc = 0.87125826, auc_precision_recall = 0.8732454, average_loss = 0.45014507, global_step = 5000, label/mean = 0.5, loss = 0.4506318, precision = 0.7999332, prediction/mean = 0.48966122, recall = 0.76608
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
Training set accuracy: 0.7960000038146973
Test set accuracy: 0.7872400283813477

혼동 행렬

오분류 분포를 이해하기 위해 혼동 행렬을 시각적으로 확인할 수 있습니다.

def get_predictions(estimator, input_fn):
  return [x["class_ids"][0] for x in estimator.predict(input_fn=input_fn)]

LABELS = [
    "negative", "positive"
]

# Create a confusion matrix on training data.
cm = tf.math.confusion_matrix(train_df["polarity"], 
                              get_predictions(estimator, predict_train_input_fn))

# Normalize the confusion matrix so that each row sums to 1.
cm = tf.cast(cm, dtype=tf.float32)
cm = cm / tf.math.reduce_sum(cm, axis=1)[:, np.newaxis]

sns.heatmap(cm, annot=True, xticklabels=LABELS, yticklabels=LABELS);
plt.xlabel("Predicted");
plt.ylabel("True");

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
Text(33.0, 0.5, 'True')

png

추가 개선

감상 회귀: 분류자를 사용하여 각 예를 극성 클래스에 할당했습니다. 그러나 우리에게는 이용할 수 있는 또 다른 범주 특성인 감상이 있습니다. 여기서 클래스는 실제로 척도를 나타내며 기본 값(긍정적/부정적)은 연속 범위로 잘 매핑될 수 있습니다. 분류(DNN Classifier) 대신 회귀(DNN Regressor)를 계산하여 이 속성을 사용할 수 있습니다.
더 큰 모듈: 이 튜토리얼에서는 메모리 사용을 제한하기 위해 작은 모듈을 사용했습니다. 더 큰 어휘와 더 큰 임베딩 공간을 가진 모듈의 경우에는 정확성과 관련해 추가적으로 고려할 사항이 있을 수 있습니다.
매개변수 조정: 특히 다른 모듈을 사용하는 경우에 학습률 또는 스텝 수와 같은 메타 매개변수를 조정하여 정확성을 향상할 수 있습니다. 테스트 세트로 잘 일반화하지 않고 훈련 데이터를 예측하는 방법을 배우는 모델을 설정하는 것이 매우 쉽기 때문에 검증 세트는 합리적인 결과를 얻으려는 경우에 매우 중요합니다.
더 복잡한 모델: 여기서는 각 개별 단어를 포함한 다음 평균과 결합하여 문장 임베딩을 계산하는 모듈을 사용했습니다. 문장의 본질을 더 잘 포착하기 위해 순차 모듈(예: Universal Sentence Encoder 모듈)을 사용할 수도 있습니다. 또는 두 개 이상의 TF-Hub 모듈을 조화롭게 사용할 수도 있습니다.
정규화: 과대적합을 방지하기 위해 일종의 정규화를 수행하는 옵티마이저(예: Proximal Adagrad Optimizer)를 사용할 수 있습니다.

고급: 전이 학습 분석

전이 학습을 사용하면 훈련 리소스를 절약하고 작은 데이터세트에서 훈련하는 경우에도 우수한 모델 일반화를 달성할 수 있습니다. 여기서는 두 가지 다른 TF-Hub 모듈로 훈련하는 예를 설명합니다.

nnlm-en-dim128 - 사전 훈련된 텍스트 임베딩 모듈
random-nnlm-en-dim128 - nnlm-KO-dim128과 같은 어휘와 네트워크를 가진 텍스트 임베딩 모듈이지만 가중치는 무작위로 초기화되고 실제 데이터에서 훈련되지 않습니다.

그리고 두 가지 모드로 훈련합니다.

분류자만 훈련(예: 모듈 동결)
모듈과 함께 분류자 훈련

다양한 모듈을 사용할 때 정확성에 어떤 영향을 미치는지 알아보기 위해 몇 가지 훈련과 평가를 실행해 보겠습니다.

def train_and_evaluate_with_module(hub_module, train_module=False):
  embedded_text_feature_column = hub.text_embedding_column(
      key="sentence", module_spec=hub_module, trainable=train_module)

  estimator = tf.estimator.DNNClassifier(
      hidden_units=[500, 100],
      feature_columns=[embedded_text_feature_column],
      n_classes=2,
      optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003))

  estimator.train(input_fn=train_input_fn, steps=1000)

  train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
  test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)

  training_set_accuracy = train_eval_result["accuracy"]
  test_set_accuracy = test_eval_result["accuracy"]

  return {
      "Training accuracy": training_set_accuracy,
      "Test accuracy": test_set_accuracy
  }


results = {}
results["nnlm-en-dim128"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/nnlm-en-dim128/1")
results["nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/nnlm-en-dim128/1", True)
results["random-nnlm-en-dim128"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/random-nnlm-en-dim128/1")
results["random-nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
    "https://tfhub.dev/google/random-nnlm-en-dim128/1", True)

INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpg5ilvn11
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpg5ilvn11
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpg5ilvn11', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpg5ilvn11', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpg5ilvn11/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpg5ilvn11/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 0.6966276, step = 0
INFO:tensorflow:loss = 0.6966276, step = 0
INFO:tensorflow:global_step/sec: 129.442
INFO:tensorflow:global_step/sec: 129.442
INFO:tensorflow:loss = 0.68398184, step = 100 (0.774 sec)
INFO:tensorflow:loss = 0.68398184, step = 100 (0.774 sec)
INFO:tensorflow:global_step/sec: 138.189
INFO:tensorflow:global_step/sec: 138.189
INFO:tensorflow:loss = 0.6671169, step = 200 (0.724 sec)
INFO:tensorflow:loss = 0.6671169, step = 200 (0.724 sec)
INFO:tensorflow:global_step/sec: 138.989
INFO:tensorflow:global_step/sec: 138.989
INFO:tensorflow:loss = 0.6603984, step = 300 (0.720 sec)
INFO:tensorflow:loss = 0.6603984, step = 300 (0.720 sec)
INFO:tensorflow:global_step/sec: 138.812
INFO:tensorflow:global_step/sec: 138.812
INFO:tensorflow:loss = 0.6468728, step = 400 (0.720 sec)
INFO:tensorflow:loss = 0.6468728, step = 400 (0.720 sec)
INFO:tensorflow:global_step/sec: 137.371
INFO:tensorflow:global_step/sec: 137.371
INFO:tensorflow:loss = 0.64965165, step = 500 (0.728 sec)
INFO:tensorflow:loss = 0.64965165, step = 500 (0.728 sec)
INFO:tensorflow:global_step/sec: 133.92
INFO:tensorflow:global_step/sec: 133.92
INFO:tensorflow:loss = 0.63341486, step = 600 (0.747 sec)
INFO:tensorflow:loss = 0.63341486, step = 600 (0.747 sec)
INFO:tensorflow:global_step/sec: 135.683
INFO:tensorflow:global_step/sec: 135.683
INFO:tensorflow:loss = 0.6093527, step = 700 (0.737 sec)
INFO:tensorflow:loss = 0.6093527, step = 700 (0.737 sec)
INFO:tensorflow:global_step/sec: 137.77
INFO:tensorflow:global_step/sec: 137.77
INFO:tensorflow:loss = 0.62642115, step = 800 (0.726 sec)
INFO:tensorflow:loss = 0.62642115, step = 800 (0.726 sec)
INFO:tensorflow:global_step/sec: 137.524
INFO:tensorflow:global_step/sec: 137.524
INFO:tensorflow:loss = 0.58040327, step = 900 (0.727 sec)
INFO:tensorflow:loss = 0.58040327, step = 900 (0.727 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpg5ilvn11/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpg5ilvn11/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 0.6100675.
INFO:tensorflow:Loss for final step: 0.6100675.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:23Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:23Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.74168s
INFO:tensorflow:Inference Time : 2.74168s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:26
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:26
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.73172, accuracy_baseline = 0.5, auc = 0.80655324, auc_precision_recall = 0.8065964, average_loss = 0.5852609, global_step = 1000, label/mean = 0.5, loss = 0.5851797, precision = 0.74657357, prediction/mean = 0.49543127, recall = 0.7016
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.73172, accuracy_baseline = 0.5, auc = 0.80655324, auc_precision_recall = 0.8065964, average_loss = 0.5852609, global_step = 1000, label/mean = 0.5, loss = 0.5851797, precision = 0.74657357, prediction/mean = 0.49543127, recall = 0.7016
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:27Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:27Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.76805s
INFO:tensorflow:Inference Time : 2.76805s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:30
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:30
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72592, accuracy_baseline = 0.5, auc = 0.80071825, auc_precision_recall = 0.8001092, average_loss = 0.58876866, global_step = 1000, label/mean = 0.5, loss = 0.5888964, precision = 0.7444598, prediction/mean = 0.49247152, recall = 0.688
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.72592, accuracy_baseline = 0.5, auc = 0.80071825, auc_precision_recall = 0.8001092, average_loss = 0.58876866, global_step = 1000, label/mean = 0.5, loss = 0.5888964, precision = 0.7444598, prediction/mean = 0.49247152, recall = 0.688
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpg5ilvn11/model.ckpt-1000
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpo5y2_exn
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpo5y2_exn
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpo5y2_exn', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpo5y2_exn', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpo5y2_exn/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpo5y2_exn/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 0.69066644, step = 0
INFO:tensorflow:loss = 0.69066644, step = 0
INFO:tensorflow:global_step/sec: 129.24
INFO:tensorflow:global_step/sec: 129.24
INFO:tensorflow:loss = 0.68900317, step = 100 (0.775 sec)
INFO:tensorflow:loss = 0.68900317, step = 100 (0.775 sec)
INFO:tensorflow:global_step/sec: 131.149
INFO:tensorflow:global_step/sec: 131.149
INFO:tensorflow:loss = 0.6723022, step = 200 (0.763 sec)
INFO:tensorflow:loss = 0.6723022, step = 200 (0.763 sec)
INFO:tensorflow:global_step/sec: 131.029
INFO:tensorflow:global_step/sec: 131.029
INFO:tensorflow:loss = 0.66016746, step = 300 (0.763 sec)
INFO:tensorflow:loss = 0.66016746, step = 300 (0.763 sec)
INFO:tensorflow:global_step/sec: 131.959
INFO:tensorflow:global_step/sec: 131.959
INFO:tensorflow:loss = 0.64996684, step = 400 (0.758 sec)
INFO:tensorflow:loss = 0.64996684, step = 400 (0.758 sec)
INFO:tensorflow:global_step/sec: 138.118
INFO:tensorflow:global_step/sec: 138.118
INFO:tensorflow:loss = 0.64283985, step = 500 (0.724 sec)
INFO:tensorflow:loss = 0.64283985, step = 500 (0.724 sec)
INFO:tensorflow:global_step/sec: 138.638
INFO:tensorflow:global_step/sec: 138.638
INFO:tensorflow:loss = 0.6282129, step = 600 (0.721 sec)
INFO:tensorflow:loss = 0.6282129, step = 600 (0.721 sec)
INFO:tensorflow:global_step/sec: 137.23
INFO:tensorflow:global_step/sec: 137.23
INFO:tensorflow:loss = 0.6134773, step = 700 (0.729 sec)
INFO:tensorflow:loss = 0.6134773, step = 700 (0.729 sec)
INFO:tensorflow:global_step/sec: 137.735
INFO:tensorflow:global_step/sec: 137.735
INFO:tensorflow:loss = 0.60235065, step = 800 (0.726 sec)
INFO:tensorflow:loss = 0.60235065, step = 800 (0.726 sec)
INFO:tensorflow:global_step/sec: 139.686
INFO:tensorflow:global_step/sec: 139.686
INFO:tensorflow:loss = 0.6195818, step = 900 (0.716 sec)
INFO:tensorflow:loss = 0.6195818, step = 900 (0.716 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpo5y2_exn/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpo5y2_exn/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 0.58300316.
INFO:tensorflow:Loss for final step: 0.58300316.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:41Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:41Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.74517s
INFO:tensorflow:Inference Time : 2.74517s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:43
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:43
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.727, accuracy_baseline = 0.5, auc = 0.8026381, auc_precision_recall = 0.8027697, average_loss = 0.5896255, global_step = 1000, label/mean = 0.5, loss = 0.5895052, precision = 0.7226014, prediction/mean = 0.50736445, recall = 0.73688
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.727, accuracy_baseline = 0.5, auc = 0.8026381, auc_precision_recall = 0.8027697, average_loss = 0.5896255, global_step = 1000, label/mean = 0.5, loss = 0.5895052, precision = 0.7226014, prediction/mean = 0.50736445, recall = 0.73688
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:44Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:24:44Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 3.01804s
INFO:tensorflow:Inference Time : 3.01804s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:47
INFO:tensorflow:Finished evaluation at 2021-04-07-18:24:47
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7194, accuracy_baseline = 0.5, auc = 0.79430765, auc_precision_recall = 0.7919129, average_loss = 0.5941327, global_step = 1000, label/mean = 0.5, loss = 0.5942076, precision = 0.72001606, prediction/mean = 0.50422966, recall = 0.718
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.7194, accuracy_baseline = 0.5, auc = 0.79430765, auc_precision_recall = 0.7919129, average_loss = 0.5941327, global_step = 1000, label/mean = 0.5, loss = 0.5942076, precision = 0.72001606, prediction/mean = 0.50422966, recall = 0.718
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpo5y2_exn/model.ckpt-1000
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpycuo12_w
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpycuo12_w
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpycuo12_w', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpycuo12_w', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpycuo12_w/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpycuo12_w/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 0.7800672, step = 0
INFO:tensorflow:loss = 0.7800672, step = 0
INFO:tensorflow:global_step/sec: 128.367
INFO:tensorflow:global_step/sec: 128.367
INFO:tensorflow:loss = 0.6719709, step = 100 (0.780 sec)
INFO:tensorflow:loss = 0.6719709, step = 100 (0.780 sec)
INFO:tensorflow:global_step/sec: 135.353
INFO:tensorflow:global_step/sec: 135.353
INFO:tensorflow:loss = 0.64645493, step = 200 (0.739 sec)
INFO:tensorflow:loss = 0.64645493, step = 200 (0.739 sec)
INFO:tensorflow:global_step/sec: 136.13
INFO:tensorflow:global_step/sec: 136.13
INFO:tensorflow:loss = 0.63022035, step = 300 (0.735 sec)
INFO:tensorflow:loss = 0.63022035, step = 300 (0.735 sec)
INFO:tensorflow:global_step/sec: 137.056
INFO:tensorflow:global_step/sec: 137.056
INFO:tensorflow:loss = 0.60145795, step = 400 (0.729 sec)
INFO:tensorflow:loss = 0.60145795, step = 400 (0.729 sec)
INFO:tensorflow:global_step/sec: 136.609
INFO:tensorflow:global_step/sec: 136.609
INFO:tensorflow:loss = 0.62151605, step = 500 (0.732 sec)
INFO:tensorflow:loss = 0.62151605, step = 500 (0.732 sec)
INFO:tensorflow:global_step/sec: 130.587
INFO:tensorflow:global_step/sec: 130.587
INFO:tensorflow:loss = 0.611493, step = 600 (0.766 sec)
INFO:tensorflow:loss = 0.611493, step = 600 (0.766 sec)
INFO:tensorflow:global_step/sec: 131.329
INFO:tensorflow:global_step/sec: 131.329
INFO:tensorflow:loss = 0.59871376, step = 700 (0.761 sec)
INFO:tensorflow:loss = 0.59871376, step = 700 (0.761 sec)
INFO:tensorflow:global_step/sec: 128.472
INFO:tensorflow:global_step/sec: 128.472
INFO:tensorflow:loss = 0.63158166, step = 800 (0.778 sec)
INFO:tensorflow:loss = 0.63158166, step = 800 (0.778 sec)
INFO:tensorflow:global_step/sec: 138.698
INFO:tensorflow:global_step/sec: 138.698
INFO:tensorflow:loss = 0.6442197, step = 900 (0.721 sec)
INFO:tensorflow:loss = 0.6442197, step = 900 (0.721 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpycuo12_w/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmpycuo12_w/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 0.64912486.
INFO:tensorflow:Loss for final step: 0.64912486.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:09Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:09Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.80920s
INFO:tensorflow:Inference Time : 2.80920s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:11
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:11
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.67964, accuracy_baseline = 0.5, auc = 0.74468064, auc_precision_recall = 0.7348812, average_loss = 0.5957832, global_step = 1000, label/mean = 0.5, loss = 0.59526575, precision = 0.6788815, prediction/mean = 0.5004618, recall = 0.68176
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.67964, accuracy_baseline = 0.5, auc = 0.74468064, auc_precision_recall = 0.7348812, average_loss = 0.5957832, global_step = 1000, label/mean = 0.5, loss = 0.59526575, precision = 0.6788815, prediction/mean = 0.5004618, recall = 0.68176
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:12Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:12Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.76283s
INFO:tensorflow:Inference Time : 2.76283s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:15
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:15
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66804, accuracy_baseline = 0.5, auc = 0.72762626, auc_precision_recall = 0.71798956, average_loss = 0.6092877, global_step = 1000, label/mean = 0.5, loss = 0.60936695, precision = 0.6662972, prediction/mean = 0.50048774, recall = 0.67328
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66804, accuracy_baseline = 0.5, auc = 0.72762626, auc_precision_recall = 0.71798956, average_loss = 0.6092877, global_step = 1000, label/mean = 0.5, loss = 0.60936695, precision = 0.6662972, prediction/mean = 0.50048774, recall = 0.67328
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmpycuo12_w/model.ckpt-1000
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmphf6qqll4
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmphf6qqll4
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmphf6qqll4', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmphf6qqll4', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmphf6qqll4/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmphf6qqll4/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 0.7174974, step = 0
INFO:tensorflow:loss = 0.7174974, step = 0
INFO:tensorflow:global_step/sec: 128.54
INFO:tensorflow:global_step/sec: 128.54
INFO:tensorflow:loss = 0.6398541, step = 100 (0.779 sec)
INFO:tensorflow:loss = 0.6398541, step = 100 (0.779 sec)
INFO:tensorflow:global_step/sec: 140.093
INFO:tensorflow:global_step/sec: 140.093
INFO:tensorflow:loss = 0.62839, step = 200 (0.714 sec)
INFO:tensorflow:loss = 0.62839, step = 200 (0.714 sec)
INFO:tensorflow:global_step/sec: 138.617
INFO:tensorflow:global_step/sec: 138.617
INFO:tensorflow:loss = 0.5835835, step = 300 (0.721 sec)
INFO:tensorflow:loss = 0.5835835, step = 300 (0.721 sec)
INFO:tensorflow:global_step/sec: 138.696
INFO:tensorflow:global_step/sec: 138.696
INFO:tensorflow:loss = 0.6841439, step = 400 (0.721 sec)
INFO:tensorflow:loss = 0.6841439, step = 400 (0.721 sec)
INFO:tensorflow:global_step/sec: 138.534
INFO:tensorflow:global_step/sec: 138.534
INFO:tensorflow:loss = 0.5705359, step = 500 (0.722 sec)
INFO:tensorflow:loss = 0.5705359, step = 500 (0.722 sec)
INFO:tensorflow:global_step/sec: 137.218
INFO:tensorflow:global_step/sec: 137.218
INFO:tensorflow:loss = 0.60557395, step = 600 (0.729 sec)
INFO:tensorflow:loss = 0.60557395, step = 600 (0.729 sec)
INFO:tensorflow:global_step/sec: 137.413
INFO:tensorflow:global_step/sec: 137.413
INFO:tensorflow:loss = 0.61547655, step = 700 (0.728 sec)
INFO:tensorflow:loss = 0.61547655, step = 700 (0.728 sec)
INFO:tensorflow:global_step/sec: 133.924
INFO:tensorflow:global_step/sec: 133.924
INFO:tensorflow:loss = 0.6278802, step = 800 (0.747 sec)
INFO:tensorflow:loss = 0.6278802, step = 800 (0.747 sec)
INFO:tensorflow:global_step/sec: 130.213
INFO:tensorflow:global_step/sec: 130.213
INFO:tensorflow:loss = 0.5850229, step = 900 (0.768 sec)
INFO:tensorflow:loss = 0.5850229, step = 900 (0.768 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmphf6qqll4/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmphf6qqll4/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 0.57805085.
INFO:tensorflow:Loss for final step: 0.57805085.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:26Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:26Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.72223s
INFO:tensorflow:Inference Time : 2.72223s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:29
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:29
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.6776, accuracy_baseline = 0.5, auc = 0.747035, auc_precision_recall = 0.73735934, average_loss = 0.5952772, global_step = 1000, label/mean = 0.5, loss = 0.59471047, precision = 0.6625421, prediction/mean = 0.5247151, recall = 0.72392
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.6776, accuracy_baseline = 0.5, auc = 0.747035, auc_precision_recall = 0.73735934, average_loss = 0.5952772, global_step = 1000, label/mean = 0.5, loss = 0.59471047, precision = 0.6625421, prediction/mean = 0.5247151, recall = 0.72392
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:30Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:30Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.78489s
INFO:tensorflow:Inference Time : 2.78489s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:33
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:33
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66536, accuracy_baseline = 0.5, auc = 0.7290196, auc_precision_recall = 0.7190014, average_loss = 0.6100916, global_step = 1000, label/mean = 0.5, loss = 0.6101327, precision = 0.6523213, prediction/mean = 0.5244902, recall = 0.70816
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.66536, accuracy_baseline = 0.5, auc = 0.7290196, auc_precision_recall = 0.7190014, average_loss = 0.6100916, global_step = 1000, label/mean = 0.5, loss = 0.6101327, precision = 0.6523213, prediction/mean = 0.5244902, recall = 0.70816
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmphf6qqll4/model.ckpt-1000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmphf6qqll4/model.ckpt-1000

결과를 살펴보겠습니다.

pd.DataFrame.from_dict(results, orient="index")

이미 일부 패턴을 볼 수 있지만 먼저 테스트 세트의 기준 정확성(가장 대표적인 클래스의 레이블만 출력하여 얻을 수 있는 하한)을 수립해야 합니다.

estimator.evaluate(input_fn=predict_test_input_fn)["accuracy_baseline"]

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:33Z
INFO:tensorflow:Starting evaluation at 2021-04-07T18:25:33Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Restoring parameters from /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 2.77839s
INFO:tensorflow:Inference Time : 2.77839s
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:36
INFO:tensorflow:Finished evaluation at 2021-04-07-18:25:36
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78724, accuracy_baseline = 0.5, auc = 0.87125826, auc_precision_recall = 0.8732454, average_loss = 0.45014507, global_step = 5000, label/mean = 0.5, loss = 0.4506318, precision = 0.7999332, prediction/mean = 0.48966122, recall = 0.76608
INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.78724, accuracy_baseline = 0.5, auc = 0.87125826, auc_precision_recall = 0.8732454, average_loss = 0.45014507, global_step = 5000, label/mean = 0.5, loss = 0.4506318, precision = 0.7999332, prediction/mean = 0.48966122, recall = 0.76608
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/tmp25p8qgdi/model.ckpt-5000
0.5

가장 대표적인 클래스를 지정하면 50%의 정확성을 얻을 수 있습니다. 여기서 몇 가지 주목해야 할 사항이 있습니다.

놀라울 수도 있겠지만, 고정된 무작위 임베딩이 아니어도 모델을 훈련할 수 있습니다. 그 이유는 사전의 모든 단어가 무작위 벡터에 매핑되더라도 추정기는 완전히 연결된 레이어를 사용하여 공간을 분리할 수 있기 때문입니다.
무작위 임베딩을 사용하여 모듈을 훈련할 수 있게 하면 분류자만 훈련할 때와 달리 훈련 및 테스트 정확성이 모두 향상됩니다.
사전 훈련된 임베딩을 사용하여 모듈을 훈련하면 두 정확성이 모두 향상됩니다. 그러나 훈련 세트의 과대적합에 유의해야 합니다. 사전 훈련된 모듈을 훈련하는 것은 정규화를 적용하더라도 위험할 수 있는데, 그 이유는 임베딩 가중치가 더 이상 다양한 데이터에서 훈련된 언어 모델을 나타내지 않고, 대신 새 데이터세트의 이상적인 표현으로 수렴하기 때문입니다.