सहायता Kaggle पर TensorFlow साथ ग्रेट बैरियर रीफ की रक्षा चैलेंज में शामिल हों

TF- हब टेक्स्ट एंबेडिंग पर निष्पक्षता संकेतक

TensorFlow.org पर देखें गिटहब पर देखें नोटबुक डाउनलोड करें TF हब मॉडल देखें

इस ट्यूटोरियल में, आप सीखेंगे कि TF हब से एम्बेडिंग के मूल्यांकन के लिए फेयरनेस इंडिकेटर्स का उपयोग कैसे करें। यह नोटबुक सिविल टिप्पणियाँ डेटासेट का उपयोग करता है।

सेट अप

आवश्यक लाइब्रेरी स्थापित करें।

!pip install -q -U pip==20.2

!pip install fairness-indicators \
  "absl-py==0.12.0" \
  "pyarrow==2.0.0" \
  "apache-beam==2.28.0" \
  "avro-python3==1.9.1"

अन्य आवश्यक पुस्तकालयों को आयात करें।

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util

डेटासेट

इस नोटबुक में, आप सिविल टिप्पणियाँ डेटासेट के साथ काम करते हैं जिसमें लगभग 2 मिलियन सार्वजनिक टिप्पणियां हैं जो चल रहे शोध के लिए 2017 में सिविल टिप्पणियाँ प्लेटफ़ॉर्म द्वारा सार्वजनिक की गई थीं। यह प्रयास आरा द्वारा प्रायोजित किया गया था, जिन्होंने विषाक्त टिप्पणियों को वर्गीकृत करने में मदद करने के लिए केगल पर प्रतियोगिताओं की मेजबानी की है और साथ ही अनपेक्षित मॉडल पूर्वाग्रह को कम से कम किया है।

डेटासेट में प्रत्येक व्यक्तिगत पाठ टिप्पणी में एक विषाक्तता लेबल होता है, जिसमें लेबल 1 होता है यदि टिप्पणी विषाक्त है और 0 यदि टिप्पणी गैर विषैले है। डेटा के भीतर, टिप्पणियों के एक सबसेट को लिंग, यौन अभिविन्यास, धर्म और नस्ल या जातीयता के लिए श्रेणियों सहित विभिन्न पहचान विशेषताओं के साथ लेबल किया जाता है।

डेटा तैयार करें

TensorFlow पार्स tf.io.FixedLenFeature और tf.io.VarLenFeature का उपयोग करके डेटा से सुविधाएँ। इनपुट फीचर, आउटपुट फीचर और अन्य सभी स्लाइसिंग विशेषताओं को मैप करें।

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

डिफ़ॉल्ट रूप से, नोटबुक इस डेटासेट का एक प्रीप्रोस्ड संस्करण डाउनलोड करता है, लेकिन आप मूल डेटासेट का उपयोग कर सकते हैं और यदि चाहें तो प्रोसेसिंग चरणों को फिर से चला सकते हैं।

मूल डेटासेट में, प्रत्येक टिप्पणी को उन रिटेनरों के प्रतिशत के साथ लेबल किया जाता है जो मानते थे कि एक टिप्पणी एक विशेष पहचान से मेल खाती है। उदाहरण के लिए, एक टिप्पणी को निम्नलिखित के साथ लेबल किया जा सकता है: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 }

प्रोसेसिंग स्टेप ग्रुप्स आइडेंटिटी बाय कैटेगरी (जेंडर, सेक्शुअल_रिएशन इत्यादि) और 0.5 से कम स्कोर वाले आइडेंटिटी को हटा देता है। तो ऊपर दिए गए उदाहरण को निम्नलिखित में बदल दिया जाएगा: रैटर्स का मानना ​​है कि एक टिप्पणी एक विशेष पहचान से मेल खाती है। उदाहरण के लिए, ऊपर की टिप्पणी को निम्नलिखित के साथ लेबल किया जाएगा: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }

डेटासेट डाउनलोड करें।

download_original_data = False

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
  # text column and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')

एक TensorFlow मॉडल विश्लेषण पाइपलाइन बनाएँ

फेयरनेस इंडिकेटर्स लाइब्रेरी TensorFlow Model Analysis (TFMA) मॉडल पर काम करती है । TFMA मॉडल अपने परिणामों का मूल्यांकन और कल्पना करने के लिए अतिरिक्त कार्यक्षमता के साथ TensorFlow मॉडल लपेटते हैं। वास्तविक मूल्यांकन एक अपाचे बीम पाइपलाइन के अंदर होता है।

TFMA पाइपलाइन बनाने के लिए आप जिन चरणों का पालन करते हैं, वे हैं:

  1. एक TensorFlow मॉडल बनाएँ
  2. TensorFlow मॉडल के शीर्ष पर TFMA मॉडल बनाएँ
  3. एक आर्केस्ट्रा में मॉडल विश्लेषण चलाएं। इस नोटबुक में उदाहरण मॉडल अपाचे बीम को ऑर्केस्ट्रेटर के रूप में उपयोग करता है।
def embedding_fairness_result(embedding, identity_term='gender'):

  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # Create a unique path to store the results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

TFMA और निष्पक्षता संकेतक चलाएँ

निष्पक्षता संकेतक मेट्रिक्स

निष्पक्षता संकेतक के साथ उपलब्ध कुछ मैट्रिक्स हैं:

पाठ एंबेडिंग

TF-Hub कई टेक्स्ट एम्बेडिंग प्रदान करता है। ये एम्बेडिंग विभिन्न मॉडलों के लिए फीचर कॉलम के रूप में काम करेंगे। यह ट्यूटोरियल निम्नलिखित एम्बेडिंग का उपयोग करता है:

निष्पक्षता संकेतक परिणाम

embedding_fairness_result पाइपलाइन के साथ निष्पक्षता संकेतकों की गणना करें, और फिर उपरोक्त सभी एम्बेडिंग के लिए निष्पक्षता सूचक UI विजेट के साथ widget_view.render_fairness_indicator में परिणाम प्रस्तुत करें।

रैंडम NNLM

eval_result_random_nnlm = embedding_fairness_result('https://hub.tensorflow.google.cn/google/random-nnlm-en-dim128/1')
Training classifier for https://hub.tensorflow.google.cn/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094321', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094321', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 60.320545, step = 0
INFO:tensorflow:loss = 60.320545, step = 0
INFO:tensorflow:global_step/sec: 24.4256
INFO:tensorflow:global_step/sec: 24.4256
INFO:tensorflow:loss = 68.23323, step = 100 (4.097 sec)
INFO:tensorflow:loss = 68.23323, step = 100 (4.097 sec)
INFO:tensorflow:global_step/sec: 26.1011
INFO:tensorflow:global_step/sec: 26.1011
INFO:tensorflow:loss = 58.783485, step = 200 (3.830 sec)
INFO:tensorflow:loss = 58.783485, step = 200 (3.830 sec)
INFO:tensorflow:global_step/sec: 25.7955
INFO:tensorflow:global_step/sec: 25.7955
INFO:tensorflow:loss = 59.22664, step = 300 (3.876 sec)
INFO:tensorflow:loss = 59.22664, step = 300 (3.876 sec)
INFO:tensorflow:global_step/sec: 26.2729
INFO:tensorflow:global_step/sec: 26.2729
INFO:tensorflow:loss = 61.110313, step = 400 (3.806 sec)
INFO:tensorflow:loss = 61.110313, step = 400 (3.806 sec)
INFO:tensorflow:global_step/sec: 25.7911
INFO:tensorflow:global_step/sec: 25.7911
INFO:tensorflow:loss = 57.710922, step = 500 (3.878 sec)
INFO:tensorflow:loss = 57.710922, step = 500 (3.878 sec)
INFO:tensorflow:global_step/sec: 25.7231
INFO:tensorflow:global_step/sec: 25.7231
INFO:tensorflow:loss = 56.67649, step = 600 (3.887 sec)
INFO:tensorflow:loss = 56.67649, step = 600 (3.887 sec)
INFO:tensorflow:global_step/sec: 26.233
INFO:tensorflow:global_step/sec: 26.233
INFO:tensorflow:loss = 60.52134, step = 700 (3.812 sec)
INFO:tensorflow:loss = 60.52134, step = 700 (3.812 sec)
INFO:tensorflow:global_step/sec: 26.0555
INFO:tensorflow:global_step/sec: 26.0555
INFO:tensorflow:loss = 58.405262, step = 800 (3.839 sec)
INFO:tensorflow:loss = 58.405262, step = 800 (3.839 sec)
INFO:tensorflow:global_step/sec: 24.9941
INFO:tensorflow:global_step/sec: 24.9941
INFO:tensorflow:loss = 56.718044, step = 900 (4.000 sec)
INFO:tensorflow:loss = 56.718044, step = 900 (4.000 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 59.395683.
INFO:tensorflow:Loss for final step: 59.395683.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094321/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094321/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961453/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961453/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961453/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961453/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961453/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961453/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://hub.tensorflow.google.cn/google/nnlm-en-dim128/1')
Training classifier for https://hub.tensorflow.google.cn/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094614', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094614', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.234177, step = 0
INFO:tensorflow:loss = 59.234177, step = 0
INFO:tensorflow:global_step/sec: 23.9624
INFO:tensorflow:global_step/sec: 23.9624
INFO:tensorflow:loss = 55.79068, step = 100 (4.175 sec)
INFO:tensorflow:loss = 55.79068, step = 100 (4.175 sec)
INFO:tensorflow:global_step/sec: 24.7819
INFO:tensorflow:global_step/sec: 24.7819
INFO:tensorflow:loss = 47.50544, step = 200 (4.035 sec)
INFO:tensorflow:loss = 47.50544, step = 200 (4.035 sec)
INFO:tensorflow:global_step/sec: 22.4029
INFO:tensorflow:global_step/sec: 22.4029
INFO:tensorflow:loss = 55.776627, step = 300 (4.464 sec)
INFO:tensorflow:loss = 55.776627, step = 300 (4.464 sec)
INFO:tensorflow:global_step/sec: 21.9616
INFO:tensorflow:global_step/sec: 21.9616
INFO:tensorflow:loss = 55.773647, step = 400 (4.553 sec)
INFO:tensorflow:loss = 55.773647, step = 400 (4.553 sec)
INFO:tensorflow:global_step/sec: 22.3908
INFO:tensorflow:global_step/sec: 22.3908
INFO:tensorflow:loss = 41.876427, step = 500 (4.466 sec)
INFO:tensorflow:loss = 41.876427, step = 500 (4.466 sec)
INFO:tensorflow:global_step/sec: 24.1889
INFO:tensorflow:global_step/sec: 24.1889
INFO:tensorflow:loss = 45.470932, step = 600 (4.134 sec)
INFO:tensorflow:loss = 45.470932, step = 600 (4.134 sec)
INFO:tensorflow:global_step/sec: 24.5477
INFO:tensorflow:global_step/sec: 24.5477
INFO:tensorflow:loss = 51.15878, step = 700 (4.073 sec)
INFO:tensorflow:loss = 51.15878, step = 700 (4.073 sec)
INFO:tensorflow:global_step/sec: 23.6935
INFO:tensorflow:global_step/sec: 23.6935
INFO:tensorflow:loss = 47.61961, step = 800 (4.221 sec)
INFO:tensorflow:loss = 47.61961, step = 800 (4.221 sec)
INFO:tensorflow:global_step/sec: 23.1377
INFO:tensorflow:global_step/sec: 23.1377
INFO:tensorflow:loss = 48.06709, step = 900 (4.322 sec)
INFO:tensorflow:loss = 48.06709, step = 900 (4.322 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 51.064987.
INFO:tensorflow:Loss for final step: 51.064987.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094614/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094614/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961619/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961619/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961619/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961619/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961619/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961619/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

यूनिवर्सल सेंटेंस एनकोडर

eval_result_use = embedding_fairness_result('https://hub.tensorflow.google.cn/google/universal-sentence-encoder/2')
Training classifier for https://hub.tensorflow.google.cn/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.33705, step = 0
INFO:tensorflow:loss = 59.33705, step = 0
INFO:tensorflow:global_step/sec: 9.17094
INFO:tensorflow:global_step/sec: 9.17094
INFO:tensorflow:loss = 49.904232, step = 100 (10.906 sec)
INFO:tensorflow:loss = 49.904232, step = 100 (10.906 sec)
INFO:tensorflow:global_step/sec: 9.5816
INFO:tensorflow:global_step/sec: 9.5816
INFO:tensorflow:loss = 45.914093, step = 200 (10.437 sec)
INFO:tensorflow:loss = 45.914093, step = 200 (10.437 sec)
INFO:tensorflow:global_step/sec: 9.42711
INFO:tensorflow:global_step/sec: 9.42711
INFO:tensorflow:loss = 48.298733, step = 300 (10.608 sec)
INFO:tensorflow:loss = 48.298733, step = 300 (10.608 sec)
INFO:tensorflow:global_step/sec: 9.5081
INFO:tensorflow:global_step/sec: 9.5081
INFO:tensorflow:loss = 44.76107, step = 400 (10.517 sec)
INFO:tensorflow:loss = 44.76107, step = 400 (10.517 sec)
INFO:tensorflow:global_step/sec: 9.58888
INFO:tensorflow:global_step/sec: 9.58888
INFO:tensorflow:loss = 35.319324, step = 500 (10.429 sec)
INFO:tensorflow:loss = 35.319324, step = 500 (10.429 sec)
INFO:tensorflow:global_step/sec: 9.51424
INFO:tensorflow:global_step/sec: 9.51424
INFO:tensorflow:loss = 41.977276, step = 600 (10.511 sec)
INFO:tensorflow:loss = 41.977276, step = 600 (10.511 sec)
INFO:tensorflow:global_step/sec: 9.55503
INFO:tensorflow:global_step/sec: 9.55503
INFO:tensorflow:loss = 40.810833, step = 700 (10.465 sec)
INFO:tensorflow:loss = 40.810833, step = 700 (10.465 sec)
INFO:tensorflow:global_step/sec: 9.57973
INFO:tensorflow:global_step/sec: 9.57973
INFO:tensorflow:loss = 37.127766, step = 800 (10.439 sec)
INFO:tensorflow:loss = 37.127766, step = 800 (10.439 sec)
INFO:tensorflow:global_step/sec: 9.51487
INFO:tensorflow:global_step/sec: 9.51487
INFO:tensorflow:loss = 32.48211, step = 900 (10.510 sec)
INFO:tensorflow:loss = 32.48211, step = 900 (10.510 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 46.977016.
INFO:tensorflow:Loss for final step: 46.977016.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094902/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094902/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961882/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961882/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961882/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961882/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

तुलना करना

आप सीधे एम्बेडिंग की तुलना करने के लिए निष्पक्षता संकेतक का उपयोग भी कर सकते हैं। उदाहरण के लिए, NNLM और USE एम्बेडिंग से उत्पन्न मॉडल की तुलना करें।

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…