Ta strona została przetłumaczona przez Cloud Translation API.
Switch to English

Wstępne przetwarzanie danych za pomocą transformacji TensorFlow

Komponent Feature Engineering TensorFlow Extended (TFX)

Ten przykładowy notatnik colab stanowi nieco bardziej zaawansowany przykład tego, jak można wykorzystać transformację TensorFlow ( tf.Transform ) do wstępnego przetwarzania danych przy użyciu dokładnie tego samego kodu zarówno do uczenia modelu, jak i do obsługi wniosków w środowisku produkcyjnym.

TensorFlow Transform to biblioteka do wstępnego przetwarzania danych wejściowych dla TensorFlow, w tym do tworzenia funkcji wymagających pełnego przejścia przez zestaw danych szkoleniowych. Na przykład, używając TensorFlow Transform możesz:

  • Normalizuj wartość wejściową, używając średniej i odchylenia standardowego
  • Konwertuj ciągi znaków na liczby całkowite, generując słownik dla wszystkich wartości wejściowych
  • Konwertuj liczby zmiennoprzecinkowe na liczby całkowite, przypisując je do zasobników na podstawie obserwowanego rozkładu danych

TensorFlow ma wbudowaną obsługę manipulacji na pojedynczym przykładzie lub grupie przykładów. tf.Transform rozszerza te możliwości, aby obsługiwać pełne przejścia w całym zestawie danych szkoleniowych.

Dane wyjściowe tf.Transform są eksportowane jako wykres TensorFlow, którego można używać zarówno do uczenia, jak i udostępniania. Używanie tego samego wykresu zarówno do treningu, jak i do serwowania, może zapobiec wypaczeniu, ponieważ te same transformacje są stosowane na obu etapach.

Co robimy w tym przykładzie

W tym przykładzie będziemy przetwarzać powszechnie używany zbiór danych zawierający dane ze spisu powszechnego i trenować model do klasyfikacji. Po drodze będziemy przekształcać dane za pomocą tf.Transform .

Upgrade Pip

Aby uniknąć aktualizacji Pipa w systemie podczas pracy lokalnie, upewnij się, że działamy w Colab. Systemy lokalne można oczywiście aktualizować oddzielnie.

try:
  import colab
  !pip install --upgrade pip
except:
  pass

Zainstaluj TensorFlow

pip install tensorflow==2.2.0
Collecting tensorflow==2.2.0
  Using cached tensorflow-2.2.0-cp36-cp36m-manylinux2010_x86_64.whl (516.2 MB)
Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (3.3.0)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.1.2)
Requirement already satisfied: absl-py>=0.7.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorflow==2.2.0) (0.9.0)
Requirement already satisfied: six>=1.12.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.15.0)
Requirement already satisfied: protobuf>=3.8.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorflow==2.2.0) (3.13.0)
Collecting tensorflow-estimator<2.3.0,>=2.2.0
  Using cached tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB)
Requirement already satisfied: wrapt>=1.11.1 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.12.1)
Requirement already satisfied: termcolor>=1.1.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.1.0)
Requirement already satisfied: numpy<2.0,>=1.16.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.18.5)
Requirement already satisfied: scipy==1.4.1; python_version >= "3" in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.4.1)
Collecting tensorboard<2.3.0,>=2.2.0
  Using cached tensorboard-2.2.2-py3-none-any.whl (3.0 MB)
Requirement already satisfied: google-pasta>=0.1.8 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (0.2.0)
Requirement already satisfied: grpcio>=1.8.6 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.31.0)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (0.35.1)
Requirement already satisfied: astunparse==1.6.3 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (1.6.3)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (2.10.0)
Requirement already satisfied: gast==0.3.3 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow==2.2.0) (0.3.3)
Requirement already satisfied: setuptools in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from protobuf>=3.8.0->tensorflow==2.2.0) (49.6.0)
Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.2.2)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.7.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.20.1)
Requirement already satisfied: werkzeug>=0.11.15 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.0.1)
Requirement already satisfied: requests<3,>=2.21.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.24.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/kbuilder/.local/lib/python3.6/site-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.7.0)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.3.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.2.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.6)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.1.1)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.6)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /home/kbuilder/.local/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2020.6.20)
Requirement already satisfied: zipp>=0.5 in /home/kbuilder/.local/lib/python3.6/site-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.1.0)
Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.1.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4; python_version >= "3.5"->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.2)
Installing collected packages: tensorflow-estimator, tensorboard, tensorflow
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.3.0
    Uninstalling tensorflow-estimator-2.3.0:
      Successfully uninstalled tensorflow-estimator-2.3.0
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.3.0
    Uninstalling tensorboard-2.3.0:
      Successfully uninstalled tensorboard-2.3.0
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.3.0
    Uninstalling tensorflow-2.3.0:
      Successfully uninstalled tensorflow-2.3.0
Successfully installed tensorboard-2.2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0

Sprawdzanie, importowanie i globalne w Pythonie

Najpierw upewnimy się, że używamy Pythona 3, a następnie zainstalujemy i zaimportujemy potrzebne nam rzeczy.

import sys

# Confirm that we're using Python 3
assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime > Change runtime type'
import os
import pprint

import tensorflow as tf
print('TF: {}'.format(tf.__version__))

print('Installing Apache Beam')
!pip install -Uq apache_beam==2.21.0
import apache_beam as beam
print('Beam: {}'.format(beam.__version__))

print('Installing TensorFlow Transform')
!pip install -q tensorflow-transform==0.22.0
import tensorflow_transform as tft
print('Transform: {}'.format(tft.__version__))

import tensorflow_transform.beam as tft_beam

!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test

train = './adult.data'
test = './adult.test'
TF: 2.2.0
Installing Apache Beam
Beam: 2.21.0
Installing TensorFlow Transform
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

tensorflow-serving-api 2.3.0 requires tensorflow<3,>=2.3, but you'll have tensorflow 2.2.0 which is incompatible.
Transform: 0.22.0
--2020-08-18 09:24:09--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
Resolving storage.googleapis.com (storage.googleapis.com)... 64.233.189.128, 108.177.97.128, 108.177.125.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|64.233.189.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3974305 (3.8M) [application/octet-stream]
Saving to: ‘adult.data’

adult.data          100%[===================>]   3.79M  --.-KB/s    in 0.02s   

2020-08-18 09:24:10 (216 MB/s) - ‘adult.data’ saved [3974305/3974305]

--2020-08-18 09:24:10--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test
Resolving storage.googleapis.com (storage.googleapis.com)... 64.233.188.128, 74.125.203.128, 108.177.125.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|64.233.188.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2003153 (1.9M) [application/octet-stream]
Saving to: ‘adult.test’

adult.test          100%[===================>]   1.91M  --.-KB/s    in 0.02s   

2020-08-18 09:24:11 (96.5 MB/s) - ‘adult.test’ saved [2003153/2003153]


Nazwij nasze kolumny

Stworzymy kilka przydatnych list do odwoływania się do kolumn w naszym zbiorze danych.

CATEGORICAL_FEATURE_KEYS = [
    'workclass',
    'education',
    'marital-status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'native-country',
]
NUMERIC_FEATURE_KEYS = [
    'age',
    'capital-gain',
    'capital-loss',
    'hours-per-week',
]
OPTIONAL_NUMERIC_FEATURE_KEYS = [
    'education-num',
]
LABEL_KEY = 'label'

Zdefiniuj nasze funkcje i schemat

Zdefiniujmy schemat na podstawie typów kolumn w naszym wejściu. Pomoże to między innymi w prawidłowym zaimportowaniu ich.

RAW_DATA_FEATURE_SPEC = dict(
    [(name, tf.io.FixedLenFeature([], tf.string))
     for name in CATEGORICAL_FEATURE_KEYS] +
    [(name, tf.io.FixedLenFeature([], tf.float32))
     for name in NUMERIC_FEATURE_KEYS] +
    [(name, tf.io.VarLenFeature(tf.float32))
     for name in OPTIONAL_NUMERIC_FEATURE_KEYS] +
    [(LABEL_KEY, tf.io.FixedLenFeature([], tf.string))]
)

RAW_DATA_METADATA = tft.tf_metadata.dataset_metadata.DatasetMetadata(
    tft.tf_metadata.dataset_schema.schema_utils.schema_from_feature_spec(RAW_DATA_FEATURE_SPEC))

Ustawianie hiperparametrów i podstawowych czynności porządkowych

Stałe i hiperparametry używane do treningu. Rozmiar zasobnika obejmuje wszystkie wymienione kategorie w opisie zbioru danych, a także jedną dodatkową dla „?” który reprezentuje nieznane.

testing = os.getenv("WEB_TEST_BROWSER", False)
if testing:
  TRAIN_NUM_EPOCHS = 1
  NUM_TRAIN_INSTANCES = 1
  TRAIN_BATCH_SIZE = 1
  NUM_TEST_INSTANCES = 1
else:
  TRAIN_NUM_EPOCHS = 16
  NUM_TRAIN_INSTANCES = 32561
  TRAIN_BATCH_SIZE = 128
  NUM_TEST_INSTANCES = 16281

# Names of temp files
TRANSFORMED_TRAIN_DATA_FILEBASE = 'train_transformed'
TRANSFORMED_TEST_DATA_FILEBASE = 'test_transformed'
EXPORTED_MODEL_DIR = 'exported_model_dir'

Czyszczenie

Utwórz transformację wiązki do czyszczenia naszych danych wejściowych

PTransform transformację wiązki , tworząc podklasę klasy PTransform Apache Beam i zastępując metodę expand , aby określić rzeczywistą logikę przetwarzania. PTransform reprezentuje operację przetwarzania danych lub krok w potoku. Każdy PTransform przyjmuje jeden lub więcej obiektów PCollection jako dane wejściowe, wykonuje funkcję przetwarzania, którą zapewniasz na elementach tego PCollection i wytwarza zero lub więcej wyjściowych obiektów PCollection.

Nasza klasa transformacji zastosuje parametr Beam's ParDo na wejściowym PCollection zawierającym nasz zbiór danych spisu, tworząc czyste dane w wyjściowym PCollection .

class MapAndFilterErrors(beam.PTransform):
  """Like beam.Map but filters out errors in the map_fn."""

  class _MapAndFilterErrorsDoFn(beam.DoFn):
    """Count the bad examples using a beam metric."""

    def __init__(self, fn):
      self._fn = fn
      # Create a counter to measure number of bad elements.
      self._bad_elements_counter = beam.metrics.Metrics.counter(
          'census_example', 'bad_elements')

    def process(self, element):
      try:
        yield self._fn(element)
      except Exception:  # pylint: disable=broad-except
        # Catch any exception the above call.
        self._bad_elements_counter.inc(1)

  def __init__(self, fn):
    self._fn = fn

  def expand(self, pcoll):
    return pcoll | beam.ParDo(self._MapAndFilterErrorsDoFn(self._fn))

Przetwarzanie tf.Transform z tf.Transform

Utwórz plik tf.Transform preprocessing_fn

Funkcja przetwarzania wstępnego jest najważniejszą koncepcją tf.Transform. Funkcja przetwarzania wstępnego to miejsce, w którym naprawdę zachodzi transformacja zbioru danych. Przyjmuje i zwraca słownik tensorów, gdzie tensor oznacza Tensor lub SparseTensor . Istnieją dwie główne grupy wywołań API, które zwykle stanowią serce funkcji przetwarzania wstępnego:

  1. TensorFlow Ops: każda funkcja, która akceptuje i zwraca tensory, co zwykle oznacza operacje TensorFlow. Dodają one operacje TensorFlow do wykresu, który przekształca surowe dane w przekształcone dane po jednym wektorze cech na raz. Będą działać na każdym przykładzie, zarówno podczas treningu, jak i serwowania.
  2. Analizatory transformacji TensorFlow: dowolny z analizatorów dostarczonych przez tf.Transform. Analizatory również akceptują i zwracają tensory, ale w przeciwieństwie do operacji TensorFlow działają tylko raz, podczas treningu i zazwyczaj wykonują pełny przebieg całego zbioru danych szkoleniowych. Tworzą stałe tensorowe , które są dodawane do twojego wykresu. Na przykład tft.min oblicza minimum tensora w zbiorze danych tft.min . tf.Transform zapewnia stały zestaw analizatorów, ale zostanie on rozszerzony w przyszłych wersjach.
def preprocessing_fn(inputs):
  """Preprocess input columns into transformed columns."""
  # Since we are modifying some features and leaving others unchanged, we
  # start by setting `outputs` to a copy of `inputs.
  outputs = inputs.copy()

  # Scale numeric columns to have range [0, 1].
  for key in NUMERIC_FEATURE_KEYS:
    outputs[key] = tft.scale_to_0_1(outputs[key])

  for key in OPTIONAL_NUMERIC_FEATURE_KEYS:
    # This is a SparseTensor because it is optional. Here we fill in a default
    # value when it is missing.
    sparse = tf.sparse.SparseTensor(outputs[key].indices, outputs[key].values,
                                    [outputs[key].dense_shape[0], 1])
    dense = tf.sparse.to_dense(sp_input=sparse, default_value=0.)
    # Reshaping from a batch of vectors of size 1 to a batch to scalars.
    dense = tf.squeeze(dense, axis=1)
    outputs[key] = tft.scale_to_0_1(dense)

  # For all categorical columns except the label column, we generate a
  # vocabulary but do not modify the feature.  This vocabulary is instead
  # used in the trainer, by means of a feature column, to convert the feature
  # from a string to an integer id.
  for key in CATEGORICAL_FEATURE_KEYS:
    tft.vocabulary(inputs[key], vocab_filename=key)

  # For the label column we provide the mapping from string to index.
  table_keys = ['>50K', '<=50K']
  initializer = tf.lookup.KeyValueTensorInitializer(
      keys=table_keys,
      values=tf.cast(tf.range(len(table_keys)), tf.int64),
      key_dtype=tf.string,
      value_dtype=tf.int64)
  table = tf.lookup.StaticHashTable(initializer, default_value=-1)
  outputs[LABEL_KEY] = table.lookup(outputs[LABEL_KEY])

  return outputs

Przekształć dane

Teraz jesteśmy gotowi, aby rozpocząć przekształcanie naszych danych w potoku Apache Beam.

  1. Wczytaj dane za pomocą czytnika CSV
  2. Wyczyść go za pomocą naszej nowej transformacji MapAndFilterErrors
  3. Przekształć go za pomocą potoku przetwarzania wstępnego, który skaluje dane liczbowe i konwertuje dane kategoryczne z ciągów na indeksy wartości int64, tworząc słownictwo dla każdej kategorii
  4. Zapisz wynik jako TFRecord of Example protos, którego użyjemy do późniejszego trenowania modelu
def transform_data(train_data_file, test_data_file, working_dir):
  """Transform the data and write out as a TFRecord of Example protos.

  Read in the data using the CSV reader, and transform it using a
  preprocessing pipeline that scales numeric data and converts categorical data
  from strings to int64 values indices, by creating a vocabulary for each
  category.

  Args:
    train_data_file: File containing training data
    test_data_file: File containing test data
    working_dir: Directory to write transformed data and metadata to
  """

  # The "with" block will create a pipeline, and run that pipeline at the exit
  # of the block.
  with beam.Pipeline() as pipeline:
    with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
      # Create a coder to read the census data with the schema.  To do this we
      # need to list all columns in order since the schema doesn't specify the
      # order of columns in the csv.
      ordered_columns = [
          'age', 'workclass', 'fnlwgt', 'education', 'education-num',
          'marital-status', 'occupation', 'relationship', 'race', 'sex',
          'capital-gain', 'capital-loss', 'hours-per-week', 'native-country',
          'label'
      ]
      converter = tft.coders.CsvCoder(ordered_columns, RAW_DATA_METADATA.schema)

      # Read in raw data and convert using CSV converter.  Note that we apply
      # some Beam transformations here, which will not be encoded in the TF
      # graph since we don't do them from within tf.Transform's methods
      # (AnalyzeDataset, TransformDataset etc.).  These transformations are just
      # to get data into a format that the CSV converter can read, in particular
      # removing spaces after commas.
      #
      # We use MapAndFilterErrors instead of Map to filter out decode errors in
      # convert.decode which should only occur for the trailing blank line.
      raw_data = (
          pipeline
          | 'ReadTrainData' >> beam.io.ReadFromText(train_data_file)
          | 'FixCommasTrainData' >> beam.Map(
              lambda line: line.replace(', ', ','))
          | 'DecodeTrainData' >> MapAndFilterErrors(converter.decode))

      # Combine data and schema into a dataset tuple.  Note that we already used
      # the schema to read the CSV data, but we also need it to interpret
      # raw_data.
      raw_dataset = (raw_data, RAW_DATA_METADATA)
      transformed_dataset, transform_fn = (
          raw_dataset | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))
      transformed_data, transformed_metadata = transformed_dataset
      transformed_data_coder = tft.coders.ExampleProtoCoder(
          transformed_metadata.schema)

      _ = (
          transformed_data
          | 'EncodeTrainData' >> beam.Map(transformed_data_coder.encode)
          | 'WriteTrainData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE)))

      # Now apply transform function to test data.  In this case we remove the
      # trailing period at the end of each line, and also ignore the header line
      # that is present in the test data file.
      raw_test_data = (
          pipeline
          | 'ReadTestData' >> beam.io.ReadFromText(test_data_file,
                                                   skip_header_lines=1)
          | 'FixCommasTestData' >> beam.Map(
              lambda line: line.replace(', ', ','))
          | 'RemoveTrailingPeriodsTestData' >> beam.Map(lambda line: line[:-1])
          | 'DecodeTestData' >> MapAndFilterErrors(converter.decode))

      raw_test_dataset = (raw_test_data, RAW_DATA_METADATA)

      transformed_test_dataset = (
          (raw_test_dataset, transform_fn) | tft_beam.TransformDataset())
      # Don't need transformed data schema, it's the same as before.
      transformed_test_data, _ = transformed_test_dataset

      _ = (
          transformed_test_data
          | 'EncodeTestData' >> beam.Map(transformed_data_coder.encode)
          | 'WriteTestData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE)))

      # Will write a SavedModel and metadata to working_dir, which can then
      # be read by the tft.TFTransformOutput class.
      _ = (
          transform_fn
          | 'WriteTransformFn' >> tft_beam.WriteTransformFn(working_dir))

Korzystanie z naszych wstępnie przetworzonych danych do trenowania modelu

Aby pokazać, w jaki sposób tf.Transform umożliwia nam używanie tego samego kodu zarówno do uczenia, jak i udostępniania, a tym samym zapobiega wypaczaniu, będziemy trenować model. Aby wytrenować nasz model i przygotować nasz wytrenowany model do produkcji, musimy stworzyć funkcje wejściowe. Główna różnica między naszą funkcją danych treningowych a funkcją wejściową obsługującą polega na tym, że dane uczące zawierają etykiety, a dane produkcyjne nie. Argumenty i zwroty również są nieco inne.

Utwórz funkcję wejściową do treningu

def _make_training_input_fn(tf_transform_output, transformed_examples,
                            batch_size):
  """Creates an input function reading from transformed data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    transformed_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input function for training or eval.
  """
  def input_fn():
    """Input function for training and eval."""
    dataset = tf.data.experimental.make_batched_features_dataset(
        file_pattern=transformed_examples,
        batch_size=batch_size,
        features=tf_transform_output.transformed_feature_spec(),
        reader=tf.data.TFRecordDataset,
        shuffle=True)

    transformed_features = tf.compat.v1.data.make_one_shot_iterator(
        dataset).get_next()

    # Extract features and label from the transformed tensors.
    transformed_labels = transformed_features.pop(LABEL_KEY)

    return transformed_features, transformed_labels

  return input_fn

Utwórz funkcję wejściową do obsługi

Stwórzmy funkcję wejściową, której moglibyśmy użyć w produkcji, i przygotujmy nasz wyszkolony model do obsługi.

def _make_serving_input_fn(tf_transform_output):
  """Creates an input function reading from raw data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.

  Returns:
    The serving input function.
  """
  raw_feature_spec = RAW_DATA_FEATURE_SPEC.copy()
  # Remove label since it is not available during serving.
  raw_feature_spec.pop(LABEL_KEY)

  def serving_input_fn():
    """Input function for serving."""
    # Get raw features by generating the basic serving input_fn and calling it.
    # Here we generate an input_fn that expects a parsed Example proto to be fed
    # to the model at serving time.  See also
    # tf.estimator.export.build_raw_serving_input_receiver_fn.
    raw_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(
        raw_feature_spec, default_batch_size=None)
    serving_input_receiver = raw_input_fn()

    # Apply the transform function that was used to generate the materialized
    # data.
    raw_features = serving_input_receiver.features
    transformed_features = tf_transform_output.transform_raw_features(
        raw_features)

    return tf.estimator.export.ServingInputReceiver(
        transformed_features, serving_input_receiver.receiver_tensors)

  return serving_input_fn

Owiń nasze dane wejściowe w FeatureColumns

Nasz model będzie oczekiwał naszych danych w TensorFlow FeatureColumns.

def get_feature_columns(tf_transform_output):
  """Returns the FeatureColumns for the model.

  Args:
    tf_transform_output: A `TFTransformOutput` object.

  Returns:
    A list of FeatureColumns.
  """
  # Wrap scalars as real valued columns.
  real_valued_columns = [tf.feature_column.numeric_column(key, shape=())
                         for key in NUMERIC_FEATURE_KEYS]

  # Wrap categorical columns.
  one_hot_columns = [
      tf.feature_column.categorical_column_with_vocabulary_file(
          key=key,
          vocabulary_file=tf_transform_output.vocabulary_file_by_name(
              vocab_filename=key))
      for key in CATEGORICAL_FEATURE_KEYS]

  return real_valued_columns + one_hot_columns

Trenuj, oceniaj i eksportuj nasz model

def train_and_evaluate(working_dir, num_train_instances=NUM_TRAIN_INSTANCES,
                       num_test_instances=NUM_TEST_INSTANCES):
  """Train the model on training data and evaluate on test data.

  Args:
    working_dir: Directory to read transformed data and metadata from and to
        write exported model to.
    num_train_instances: Number of instances in train set
    num_test_instances: Number of instances in test set

  Returns:
    The results from the estimator's 'evaluate' method
  """
  tf_transform_output = tft.TFTransformOutput(working_dir)

  run_config = tf.estimator.RunConfig()

  estimator = tf.estimator.LinearClassifier(
      feature_columns=get_feature_columns(tf_transform_output),
      config=run_config,
      loss_reduction=tf.losses.Reduction.SUM)

  # Fit the model using the default optimizer.
  train_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE + '*'),
      batch_size=TRAIN_BATCH_SIZE)
  estimator.train(
      input_fn=train_input_fn,
      max_steps=TRAIN_NUM_EPOCHS * num_train_instances / TRAIN_BATCH_SIZE)

  # Evaluate model on test dataset.
  eval_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE + '*'),
      batch_size=1)

  # Export the model.
  serving_input_fn = _make_serving_input_fn(tf_transform_output)
  exported_model_dir = os.path.join(working_dir, EXPORTED_MODEL_DIR)
  estimator.export_saved_model(exported_model_dir, serving_input_fn)

  return estimator.evaluate(input_fn=eval_input_fn, steps=num_test_instances)

Złóż to wszystko w całość

Stworzyliśmy wszystko, czego potrzebujemy, aby wstępnie przetworzyć dane ze spisu, wytrenować model i przygotować go do podania. Do tej pory tylko się przygotowywaliśmy. Czas zacząć biec!

import tempfile
temp = tempfile.gettempdir()

transform_data(train, test, temp)
results = train_and_evaluate(temp)
pprint.pprint(results)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_transform/tf_utils.py:220: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_transform/tf_utils.py:220: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

Warning:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_analyzer_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_analyzer_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/6807e6f02233499db5d910c34462bba9/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/6807e6f02233499db5d910c34462bba9/saved_model.pb

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

Warning:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_mapper_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_analyzer_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

Warning:tensorflow:Issue encountered when serializing tft_analyzer_use.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'Counter' object has no attribute 'name'

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/d350b2f9afca405f80c7f6069b138eb0/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/d350b2f9afca405f80c7f6069b138eb0/saved_model.pb

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 

Warning:tensorflow:Tensorflow version (2.2.0) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 
WARNING:apache_beam.utils.interactive_utils:Failed to alter the label of a transform with the ipython prompt metadata. Cannot figure out the pipeline that the given pvalueish ((<PCollection[DecodeTestData/ParDo(_MapAndFilterErrorsDoFn).None] at 0x7f042c2f0be0>, {'_schema': feature {
  name: "age"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "capital-gain"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "capital-loss"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "education"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "education-num"
  type: FLOAT
}
feature {
  name: "hours-per-week"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "label"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "marital-status"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "native-country"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "occupation"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "race"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "relationship"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "sex"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "workclass"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
}), (<PCollection[AnalyzeAndTransformDataset/AnalyzeDataset/CreateSavedModel/BindTensors/ReplaceWithConstants.None] at 0x7f042c318550>, BeamDatasetMetadata(dataset_metadata={'_schema': feature {
  name: "age"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "capital-gain"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "capital-loss"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "education"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "education-num"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "hours-per-week"
  type: FLOAT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "label"
  type: INT
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "marital-status"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "native-country"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "occupation"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "race"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "relationship"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "sex"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
feature {
  name: "workclass"
  type: BYTES
  presence {
    min_fraction: 1.0
  }
  shape {
  }
}
}, deferred_metadata=<PCollection[AnalyzeAndTransformDataset/AnalyzeDataset/ComputeDeferredMetadata.None] at 0x7f042c2fce48>))) belongs to. Thus noop.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tmptntmkgtc/tftransform_tmp/a838335ddf08440faef27b83689d05cf/assets

INFO:tensorflow:Assets written to: /tmp/tmptntmkgtc/tftransform_tmp/a838335ddf08440faef27b83689d05cf/assets

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/a838335ddf08440faef27b83689d05cf/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tmptntmkgtc/tftransform_tmp/a838335ddf08440faef27b83689d05cf/saved_model.pb

Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.

Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:vocabulary_size = 9 in workclass is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/workclass.

INFO:tensorflow:vocabulary_size = 9 in workclass is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/workclass.

INFO:tensorflow:vocabulary_size = 16 in education is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/education.

INFO:tensorflow:vocabulary_size = 16 in education is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/education.

INFO:tensorflow:vocabulary_size = 7 in marital-status is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/marital-status.

INFO:tensorflow:vocabulary_size = 7 in marital-status is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/marital-status.

INFO:tensorflow:vocabulary_size = 15 in occupation is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/occupation.

INFO:tensorflow:vocabulary_size = 15 in occupation is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/occupation.

INFO:tensorflow:vocabulary_size = 6 in relationship is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/relationship.

INFO:tensorflow:vocabulary_size = 6 in relationship is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/relationship.

INFO:tensorflow:vocabulary_size = 5 in race is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/race.

INFO:tensorflow:vocabulary_size = 5 in race is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/race.

INFO:tensorflow:vocabulary_size = 2 in sex is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/sex.

INFO:tensorflow:vocabulary_size = 2 in sex is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/sex.

INFO:tensorflow:vocabulary_size = 42 in native-country is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/native-country.

INFO:tensorflow:vocabulary_size = 42 in native-country is inferred from the number of elements in the vocabulary_file /tmp/transform_fn/assets/native-country.

Warning:tensorflow:Using temporary folder as model directory: /tmp/tmputnc4riv

Warning:tensorflow:Using temporary folder as model directory: /tmp/tmputnc4riv

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmputnc4riv', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmputnc4riv', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py:540: Layer.add_variable (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py:540: Layer.add_variable (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:144: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:144: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmputnc4riv/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmputnc4riv/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 88.72284, step = 0

INFO:tensorflow:loss = 88.72284, step = 0

INFO:tensorflow:global_step/sec: 76.9862

INFO:tensorflow:global_step/sec: 76.9862

INFO:tensorflow:loss = 38.703667, step = 100 (1.301 sec)

INFO:tensorflow:loss = 38.703667, step = 100 (1.301 sec)

INFO:tensorflow:global_step/sec: 99.9878

INFO:tensorflow:global_step/sec: 99.9878

INFO:tensorflow:loss = 45.19542, step = 200 (1.000 sec)

INFO:tensorflow:loss = 45.19542, step = 200 (1.000 sec)

INFO:tensorflow:global_step/sec: 99.859

INFO:tensorflow:global_step/sec: 99.859

INFO:tensorflow:loss = 45.45166, step = 300 (1.001 sec)

INFO:tensorflow:loss = 45.45166, step = 300 (1.001 sec)

INFO:tensorflow:global_step/sec: 100.861

INFO:tensorflow:global_step/sec: 100.861

INFO:tensorflow:loss = 43.172836, step = 400 (0.991 sec)

INFO:tensorflow:loss = 43.172836, step = 400 (0.991 sec)

INFO:tensorflow:global_step/sec: 100.344

INFO:tensorflow:global_step/sec: 100.344

INFO:tensorflow:loss = 47.976437, step = 500 (0.997 sec)

INFO:tensorflow:loss = 47.976437, step = 500 (0.997 sec)

INFO:tensorflow:global_step/sec: 100.028

INFO:tensorflow:global_step/sec: 100.028

INFO:tensorflow:loss = 36.716545, step = 600 (0.999 sec)

INFO:tensorflow:loss = 36.716545, step = 600 (0.999 sec)

INFO:tensorflow:global_step/sec: 101.002

INFO:tensorflow:global_step/sec: 101.002

INFO:tensorflow:loss = 49.73072, step = 700 (0.990 sec)

INFO:tensorflow:loss = 49.73072, step = 700 (0.990 sec)

INFO:tensorflow:global_step/sec: 100.215

INFO:tensorflow:global_step/sec: 100.215

INFO:tensorflow:loss = 51.18441, step = 800 (0.998 sec)

INFO:tensorflow:loss = 51.18441, step = 800 (0.998 sec)

INFO:tensorflow:global_step/sec: 100.284

INFO:tensorflow:global_step/sec: 100.284

INFO:tensorflow:loss = 43.9189, step = 900 (0.997 sec)

INFO:tensorflow:loss = 43.9189, step = 900 (0.997 sec)

INFO:tensorflow:global_step/sec: 100.351

INFO:tensorflow:global_step/sec: 100.351

INFO:tensorflow:loss = 42.52917, step = 1000 (0.997 sec)

INFO:tensorflow:loss = 42.52917, step = 1000 (0.997 sec)

INFO:tensorflow:global_step/sec: 112.56

INFO:tensorflow:global_step/sec: 112.56

INFO:tensorflow:loss = 44.90046, step = 1100 (0.888 sec)

INFO:tensorflow:loss = 44.90046, step = 1100 (0.888 sec)

INFO:tensorflow:global_step/sec: 119.925

INFO:tensorflow:global_step/sec: 119.925

INFO:tensorflow:loss = 43.706593, step = 1200 (0.834 sec)

INFO:tensorflow:loss = 43.706593, step = 1200 (0.834 sec)

INFO:tensorflow:global_step/sec: 120.519

INFO:tensorflow:global_step/sec: 120.519

INFO:tensorflow:loss = 44.676178, step = 1300 (0.829 sec)

INFO:tensorflow:loss = 44.676178, step = 1300 (0.829 sec)

INFO:tensorflow:global_step/sec: 119.963

INFO:tensorflow:global_step/sec: 119.963

INFO:tensorflow:loss = 43.093307, step = 1400 (0.834 sec)

INFO:tensorflow:loss = 43.093307, step = 1400 (0.834 sec)

INFO:tensorflow:global_step/sec: 119.763

INFO:tensorflow:global_step/sec: 119.763

INFO:tensorflow:loss = 42.738785, step = 1500 (0.835 sec)

INFO:tensorflow:loss = 42.738785, step = 1500 (0.835 sec)

INFO:tensorflow:global_step/sec: 121.425

INFO:tensorflow:global_step/sec: 121.425

INFO:tensorflow:loss = 40.920486, step = 1600 (0.824 sec)

INFO:tensorflow:loss = 40.920486, step = 1600 (0.824 sec)

INFO:tensorflow:global_step/sec: 119.283

INFO:tensorflow:global_step/sec: 119.283

INFO:tensorflow:loss = 41.593758, step = 1700 (0.838 sec)

INFO:tensorflow:loss = 41.593758, step = 1700 (0.838 sec)

INFO:tensorflow:global_step/sec: 120.527

INFO:tensorflow:global_step/sec: 120.527

INFO:tensorflow:loss = 52.07851, step = 1800 (0.830 sec)

INFO:tensorflow:loss = 52.07851, step = 1800 (0.830 sec)

INFO:tensorflow:global_step/sec: 121.508

INFO:tensorflow:global_step/sec: 121.508

INFO:tensorflow:loss = 47.40565, step = 1900 (0.823 sec)

INFO:tensorflow:loss = 47.40565, step = 1900 (0.823 sec)

INFO:tensorflow:global_step/sec: 120.495

INFO:tensorflow:global_step/sec: 120.495

INFO:tensorflow:loss = 37.15222, step = 2000 (0.830 sec)

INFO:tensorflow:loss = 37.15222, step = 2000 (0.830 sec)

INFO:tensorflow:global_step/sec: 121.44

INFO:tensorflow:global_step/sec: 121.44

INFO:tensorflow:loss = 40.43554, step = 2100 (0.824 sec)

INFO:tensorflow:loss = 40.43554, step = 2100 (0.824 sec)

INFO:tensorflow:global_step/sec: 121.327

INFO:tensorflow:global_step/sec: 121.327

INFO:tensorflow:loss = 34.854286, step = 2200 (0.824 sec)

INFO:tensorflow:loss = 34.854286, step = 2200 (0.824 sec)

INFO:tensorflow:global_step/sec: 121.412

INFO:tensorflow:global_step/sec: 121.412

INFO:tensorflow:loss = 39.517056, step = 2300 (0.824 sec)

INFO:tensorflow:loss = 39.517056, step = 2300 (0.824 sec)

INFO:tensorflow:global_step/sec: 121.074

INFO:tensorflow:global_step/sec: 121.074

INFO:tensorflow:loss = 42.165592, step = 2400 (0.826 sec)

INFO:tensorflow:loss = 42.165592, step = 2400 (0.826 sec)

INFO:tensorflow:global_step/sec: 120.49

INFO:tensorflow:global_step/sec: 120.49

INFO:tensorflow:loss = 41.839676, step = 2500 (0.830 sec)

INFO:tensorflow:loss = 41.839676, step = 2500 (0.830 sec)

INFO:tensorflow:global_step/sec: 120.712

INFO:tensorflow:global_step/sec: 120.712

INFO:tensorflow:loss = 39.65165, step = 2600 (0.829 sec)

INFO:tensorflow:loss = 39.65165, step = 2600 (0.829 sec)

INFO:tensorflow:global_step/sec: 119.787

INFO:tensorflow:global_step/sec: 119.787

INFO:tensorflow:loss = 42.94733, step = 2700 (0.835 sec)

INFO:tensorflow:loss = 42.94733, step = 2700 (0.835 sec)

INFO:tensorflow:global_step/sec: 121.19

INFO:tensorflow:global_step/sec: 121.19

INFO:tensorflow:loss = 41.764587, step = 2800 (0.825 sec)

INFO:tensorflow:loss = 41.764587, step = 2800 (0.825 sec)

INFO:tensorflow:global_step/sec: 120.891

INFO:tensorflow:global_step/sec: 120.891

INFO:tensorflow:loss = 51.07626, step = 2900 (0.827 sec)

INFO:tensorflow:loss = 51.07626, step = 2900 (0.827 sec)

INFO:tensorflow:global_step/sec: 120.676

INFO:tensorflow:global_step/sec: 120.676

INFO:tensorflow:loss = 44.027252, step = 3000 (0.829 sec)

INFO:tensorflow:loss = 44.027252, step = 3000 (0.829 sec)

INFO:tensorflow:global_step/sec: 120.657

INFO:tensorflow:global_step/sec: 120.657

INFO:tensorflow:loss = 39.74543, step = 3100 (0.829 sec)

INFO:tensorflow:loss = 39.74543, step = 3100 (0.829 sec)

INFO:tensorflow:global_step/sec: 119.515

INFO:tensorflow:global_step/sec: 119.515

INFO:tensorflow:loss = 37.307327, step = 3200 (0.837 sec)

INFO:tensorflow:loss = 37.307327, step = 3200 (0.837 sec)

INFO:tensorflow:global_step/sec: 119.363

INFO:tensorflow:global_step/sec: 119.363

INFO:tensorflow:loss = 38.945374, step = 3300 (0.838 sec)

INFO:tensorflow:loss = 38.945374, step = 3300 (0.838 sec)

INFO:tensorflow:global_step/sec: 120.96

INFO:tensorflow:global_step/sec: 120.96

INFO:tensorflow:loss = 35.931507, step = 3400 (0.827 sec)

INFO:tensorflow:loss = 35.931507, step = 3400 (0.827 sec)

INFO:tensorflow:global_step/sec: 119.252

INFO:tensorflow:global_step/sec: 119.252

INFO:tensorflow:loss = 40.045303, step = 3500 (0.838 sec)

INFO:tensorflow:loss = 40.045303, step = 3500 (0.838 sec)

INFO:tensorflow:global_step/sec: 121.723

INFO:tensorflow:global_step/sec: 121.723

INFO:tensorflow:loss = 38.345215, step = 3600 (0.822 sec)

INFO:tensorflow:loss = 38.345215, step = 3600 (0.822 sec)

INFO:tensorflow:global_step/sec: 122.203

INFO:tensorflow:global_step/sec: 122.203

INFO:tensorflow:loss = 47.337906, step = 3700 (0.818 sec)

INFO:tensorflow:loss = 47.337906, step = 3700 (0.818 sec)

INFO:tensorflow:global_step/sec: 120.313

INFO:tensorflow:global_step/sec: 120.313

INFO:tensorflow:loss = 51.29358, step = 3800 (0.831 sec)

INFO:tensorflow:loss = 51.29358, step = 3800 (0.831 sec)

INFO:tensorflow:global_step/sec: 120.656

INFO:tensorflow:global_step/sec: 120.656

INFO:tensorflow:loss = 43.02392, step = 3900 (0.829 sec)

INFO:tensorflow:loss = 43.02392, step = 3900 (0.829 sec)

INFO:tensorflow:global_step/sec: 121.049

INFO:tensorflow:global_step/sec: 121.049

INFO:tensorflow:loss = 34.73887, step = 4000 (0.826 sec)

INFO:tensorflow:loss = 34.73887, step = 4000 (0.826 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...

INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmputnc4riv/model.ckpt.

INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmputnc4riv/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...

INFO:tensorflow:Loss for final step: 38.014317.

INFO:tensorflow:Loss for final step: 38.014317.

Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_3:0\022\tworkclass"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_5:0\022\teducation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_7:0\022\016marital-status"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\013\n\tConst_9:0\022\noccupation"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_11:0\022\014relationship"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_13:0\022\004race"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_15:0\022\003sex"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


Warning:tensorflow:Expected binary or unicode string, got type_url: "type.googleapis.com/tensorflow.AssetFileDef"
value: "\n\014\n\nConst_17:0\022\016native-country"


INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']

INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']

INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']

INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Restoring parameters from /tmp/tmputnc4riv/model.ckpt-4071

INFO:tensorflow:Restoring parameters from /tmp/tmputnc4riv/model.ckpt-4071

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/exported_model_dir/temp-1597742714/assets

INFO:tensorflow:Assets written to: /tmp/exported_model_dir/temp-1597742714/assets

INFO:tensorflow:SavedModel written to: /tmp/exported_model_dir/temp-1597742714/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/exported_model_dir/temp-1597742714/saved_model.pb

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-08-18T09:25:17Z

INFO:tensorflow:Starting evaluation at 2020-08-18T09:25:17Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmp/tmputnc4riv/model.ckpt-4071

INFO:tensorflow:Restoring parameters from /tmp/tmputnc4riv/model.ckpt-4071

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Evaluation [1628/16281]

INFO:tensorflow:Evaluation [1628/16281]

INFO:tensorflow:Evaluation [3256/16281]

INFO:tensorflow:Evaluation [3256/16281]

INFO:tensorflow:Evaluation [4884/16281]

INFO:tensorflow:Evaluation [4884/16281]

INFO:tensorflow:Evaluation [6512/16281]

INFO:tensorflow:Evaluation [6512/16281]

INFO:tensorflow:Evaluation [8140/16281]

INFO:tensorflow:Evaluation [8140/16281]

INFO:tensorflow:Evaluation [9768/16281]

INFO:tensorflow:Evaluation [9768/16281]

INFO:tensorflow:Evaluation [11396/16281]

INFO:tensorflow:Evaluation [11396/16281]

INFO:tensorflow:Evaluation [13024/16281]

INFO:tensorflow:Evaluation [13024/16281]

INFO:tensorflow:Evaluation [14652/16281]

INFO:tensorflow:Evaluation [14652/16281]

INFO:tensorflow:Evaluation [16280/16281]

INFO:tensorflow:Evaluation [16280/16281]

INFO:tensorflow:Evaluation [16281/16281]

INFO:tensorflow:Evaluation [16281/16281]

INFO:tensorflow:Inference Time : 129.68261s

INFO:tensorflow:Inference Time : 129.68261s

INFO:tensorflow:Finished evaluation at 2020-08-18-09:27:26

INFO:tensorflow:Finished evaluation at 2020-08-18-09:27:26

INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.85037774, accuracy_baseline = 0.76377374, auc = 0.901911, auc_precision_recall = 0.96723056, average_loss = 0.3240134, global_step = 4071, label/mean = 0.76377374, loss = 0.3240134, precision = 0.8783774, prediction/mean = 0.7628007, recall = 0.93333334

INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.85037774, accuracy_baseline = 0.76377374, auc = 0.901911, auc_precision_recall = 0.96723056, average_loss = 0.3240134, global_step = 4071, label/mean = 0.76377374, loss = 0.3240134, precision = 0.8783774, prediction/mean = 0.7628007, recall = 0.93333334

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmputnc4riv/model.ckpt-4071

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmputnc4riv/model.ckpt-4071

{'accuracy': 0.85037774,
 'accuracy_baseline': 0.76377374,
 'auc': 0.901911,
 'auc_precision_recall': 0.96723056,
 'average_loss': 0.3240134,
 'global_step': 4071,
 'label/mean': 0.76377374,
 'loss': 0.3240134,
 'precision': 0.8783774,
 'prediction/mean': 0.7628007,
 'recall': 0.93333334}

Co zrobiliśmy

W tym przykładzie użyliśmy tf.Transform aby wstępnie przetworzyć zbiór danych spisu powszechnego i wytrenować model z oczyszczonymi i przekształconymi danymi. Stworzyliśmy również funkcję wejściową, której możemy użyć, gdy wdrażamy nasz wyszkolony model w środowisku produkcyjnym w celu wykonywania wnioskowania. Używając tego samego kodu zarówno do uczenia, jak i wnioskowania, unikamy wszelkich problemów z wypaczeniem danych. Po drodze dowiedzieliśmy się o tworzeniu transformacji Apache Beam w celu wykonania transformacji potrzebnej do oczyszczenia danych i opakowaliśmy nasze dane w TensorFlow FeatureColumns . To tylko niewielka część tego, co może zrobić TensorFlow Transform! Zachęcamy do zanurzenia się w tf.Transform i tf.Transform co może dla Ciebie zrobić.