ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more

Preprocessing data with TensorFlow Transform

The Feature Engineering Component of TensorFlow Extended (TFX)

This example colab notebook provides a somewhat more advanced example of how TensorFlow Transform (tf.Transform) can be used to preprocess data using exactly the same code for both training a model and serving inferences in production.

TensorFlow Transform is a library for preprocessing input data for TensorFlow, including creating features that require a full pass over the training dataset. For example, using TensorFlow Transform you could:

  • Normalize an input value by using the mean and standard deviation
  • Convert strings to integers by generating a vocabulary over all of the input values
  • Convert floats to integers by assigning them to buckets, based on the observed data distribution

TensorFlow has built-in support for manipulations on a single example or a batch of examples. tf.Transform extends these capabilities to support full passes over the entire training dataset.

The output of tf.Transform is exported as a TensorFlow graph which you can use for both training and serving. Using the same graph for both training and serving can prevent skew, since the same transformations are applied in both stages.

What we're doing in this example

In this example we'll be processing a widely used dataset containing census data, and training a model to do classification. Along the way we'll be transforming the data using tf.Transform.

Upgrade Pip

To avoid upgrading Pip in a system when running locally, check to make sure that we're running in Colab. Local systems can of course be upgraded separately.

try:
  import colab
  !pip install --upgrade pip
except:
  pass

Install TensorFlow Transform

pip install tensorflow-transform

Python check, imports, and globals

First we'll make sure that we're using Python 3, and then go ahead and install and import the stuff we need.

import sys

# Confirm that we're using Python 3
assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime > Change runtime type'
import math
import os
import pprint

import tensorflow as tf
print('TF: {}'.format(tf.__version__))

import apache_beam as beam
print('Beam: {}'.format(beam.__version__))

import tensorflow_transform as tft
import tensorflow_transform.beam as tft_beam
print('Transform: {}'.format(tft.__version__))

from tfx_bsl.public import tfxio
from tfx_bsl.coders.example_coder import RecordBatchToExamples

!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test

train = './adult.data'
test = './adult.test'
TF: 2.5.1
Beam: 2.32.0
Transform: 1.2.0
--2021-09-30 02:30:53--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data
Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.97.128, 64.233.189.128, 64.233.188.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.97.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3974305 (3.8M) [application/octet-stream]
Saving to: ‘adult.data’

adult.data          100%[===================>]   3.79M  --.-KB/s    in 0.03s   

2021-09-30 02:30:54 (148 MB/s) - ‘adult.data’ saved [3974305/3974305]

--2021-09-30 02:30:54--  https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test
Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.97.128, 64.233.189.128, 64.233.188.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.97.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2003153 (1.9M) [application/octet-stream]
Saving to: ‘adult.test’

adult.test          100%[===================>]   1.91M  --.-KB/s    in 0.01s   

2021-09-30 02:30:54 (129 MB/s) - ‘adult.test’ saved [2003153/2003153]

Name our columns

We'll create some handy lists for referencing the columns in our dataset.

CATEGORICAL_FEATURE_KEYS = [
    'workclass',
    'education',
    'marital-status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'native-country',
]
NUMERIC_FEATURE_KEYS = [
    'age',
    'capital-gain',
    'capital-loss',
    'hours-per-week',
]
OPTIONAL_NUMERIC_FEATURE_KEYS = [
    'education-num',
]
ORDERED_CSV_COLUMNS = [
    'age', 'workclass', 'fnlwgt', 'education', 'education-num',
    'marital-status', 'occupation', 'relationship', 'race', 'sex',
    'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'label'
]
LABEL_KEY = 'label'

Define our features and schema

Let's define a schema based on what types the columns are in our input. Among other things this will help with importing them correctly.

RAW_DATA_FEATURE_SPEC = dict(
    [(name, tf.io.FixedLenFeature([], tf.string))
     for name in CATEGORICAL_FEATURE_KEYS] +
    [(name, tf.io.FixedLenFeature([], tf.float32))
     for name in NUMERIC_FEATURE_KEYS] +
    [(name, tf.io.VarLenFeature(tf.float32))
     for name in OPTIONAL_NUMERIC_FEATURE_KEYS] +
    [(LABEL_KEY, tf.io.FixedLenFeature([], tf.string))]
)

SCHEMA = tft.tf_metadata.dataset_metadata.DatasetMetadata(
    tft.tf_metadata.schema_utils.schema_from_feature_spec(RAW_DATA_FEATURE_SPEC)).schema

Setting hyperparameters and basic housekeeping

Constants and hyperparameters used for training. The bucket size includes all listed categories in the dataset description as well as one extra for "?" which represents unknown.

testing = os.getenv("WEB_TEST_BROWSER", False)
NUM_OOV_BUCKETS = 1
if testing:
  TRAIN_NUM_EPOCHS = 1
  NUM_TRAIN_INSTANCES = 1
  TRAIN_BATCH_SIZE = 1
  NUM_TEST_INSTANCES = 1
else:
  TRAIN_NUM_EPOCHS = 16
  NUM_TRAIN_INSTANCES = 32561
  TRAIN_BATCH_SIZE = 128
  NUM_TEST_INSTANCES = 16281

# Names of temp files
TRANSFORMED_TRAIN_DATA_FILEBASE = 'train_transformed'
TRANSFORMED_TEST_DATA_FILEBASE = 'test_transformed'
EXPORTED_MODEL_DIR = 'exported_model_dir'

Preprocessing with tf.Transform

Create a tf.Transform preprocessing_fn

The preprocessing function is the most important concept of tf.Transform. A preprocessing function is where the transformation of the dataset really happens. It accepts and returns a dictionary of tensors, where a tensor means a Tensor or SparseTensor. There are two main groups of API calls that typically form the heart of a preprocessing function:

  1. TensorFlow Ops: Any function that accepts and returns tensors, which usually means TensorFlow ops. These add TensorFlow operations to the graph that transforms raw data into transformed data one feature vector at a time. These will run for every example, during both training and serving.
  2. TensorFlow Transform Analyzers: Any of the analyzers provided by tf.Transform. Analyzers also accept and return tensors, but unlike TensorFlow ops they only run once, during training, and typically make a full pass over the entire training dataset. They create tensor constants, which are added to your graph. For example, tft.min computes the minimum of a tensor over the training dataset. tf.Transform provides a fixed set of analyzers, but this will be extended in future versions.
def preprocessing_fn(inputs):
  """Preprocess input columns into transformed columns."""
  # Since we are modifying some features and leaving others unchanged, we
  # start by setting `outputs` to a copy of `inputs.
  outputs = inputs.copy()

  # Scale numeric columns to have range [0, 1].
  for key in NUMERIC_FEATURE_KEYS:
    outputs[key] = tft.scale_to_0_1(inputs[key])

  for key in OPTIONAL_NUMERIC_FEATURE_KEYS:
    # This is a SparseTensor because it is optional. Here we fill in a default
    # value when it is missing.
    sparse = tf.sparse.SparseTensor(inputs[key].indices, inputs[key].values,
                                    [inputs[key].dense_shape[0], 1])
    dense = tf.sparse.to_dense(sp_input=sparse, default_value=0.)
    # Reshaping from a batch of vectors of size 1 to a batch to scalars.
    dense = tf.squeeze(dense, axis=1)
    outputs[key] = tft.scale_to_0_1(dense)

  # For all categorical columns except the label column, we generate a
  # vocabulary but do not modify the feature.  This vocabulary is instead
  # used in the trainer, by means of a feature column, to convert the feature
  # from a string to an integer id.
  for key in CATEGORICAL_FEATURE_KEYS:
    outputs[key] = tft.compute_and_apply_vocabulary(
        tf.strings.strip(inputs[key]),
        num_oov_buckets=NUM_OOV_BUCKETS,
        vocab_filename=key)

  # For the label column we provide the mapping from string to index.
  table_keys = ['>50K', '<=50K']
  with tf.init_scope():
    initializer = tf.lookup.KeyValueTensorInitializer(
        keys=table_keys,
        values=tf.cast(tf.range(len(table_keys)), tf.int64),
        key_dtype=tf.string,
        value_dtype=tf.int64)
    table = tf.lookup.StaticHashTable(initializer, default_value=-1)
  # Remove trailing periods for test data when the data is read with tf.data.
  label_str = tf.strings.regex_replace(inputs[LABEL_KEY], r'\.', '')
  label_str = tf.strings.strip(label_str)
  data_labels = table.lookup(label_str)
  transformed_label = tf.one_hot(
      indices=data_labels, depth=len(table_keys), on_value=1.0, off_value=0.0)
  outputs[LABEL_KEY] = tf.reshape(transformed_label, [-1, len(table_keys)])

  return outputs

Transform the data

Now we're ready to start transforming our data in an Apache Beam pipeline.

  1. Read in the data using the CSV reader
  2. Transform it using a preprocessing pipeline that scales numeric data and converts categorical data from strings to int64 values indices, by creating a vocabulary for each category
  3. Write out the result as a TFRecord of Example protos, which we will use for training a model later
def transform_data(train_data_file, test_data_file, working_dir):
  """Transform the data and write out as a TFRecord of Example protos.

  Read in the data using the CSV reader, and transform it using a
  preprocessing pipeline that scales numeric data and converts categorical data
  from strings to int64 values indices, by creating a vocabulary for each
  category.

  Args:
    train_data_file: File containing training data
    test_data_file: File containing test data
    working_dir: Directory to write transformed data and metadata to
  """

  # The "with" block will create a pipeline, and run that pipeline at the exit
  # of the block.
  with beam.Pipeline() as pipeline:
    with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
      # Create a TFXIO to read the census data with the schema. To do this we
      # need to list all columns in order since the schema doesn't specify the
      # order of columns in the csv.
      # We first read CSV files and use BeamRecordCsvTFXIO whose .BeamSource()
      # accepts a PCollection[bytes] because we need to patch the records first
      # (see "FixCommasTrainData" below). Otherwise, tfxio.CsvTFXIO can be used
      # to both read the CSV files and parse them to TFT inputs:
      # csv_tfxio = tfxio.CsvTFXIO(...)
      # raw_data = (pipeline | 'ToRecordBatches' >> csv_tfxio.BeamSource())
      csv_tfxio = tfxio.BeamRecordCsvTFXIO(
          physical_format='text',
          column_names=ORDERED_CSV_COLUMNS,
          schema=SCHEMA)

      # Read in raw data and convert using CSV TFXIO.  Note that we apply
      # some Beam transformations here, which will not be encoded in the TF
      # graph since we don't do the from within tf.Transform's methods
      # (AnalyzeDataset, TransformDataset etc.).  These transformations are just
      # to get data into a format that the CSV TFXIO can read, in particular
      # removing spaces after commas.
      raw_data = (
          pipeline
          | 'ReadTrainData' >> beam.io.ReadFromText(
              train_data_file, coder=beam.coders.BytesCoder())
          | 'FixCommasTrainData' >> beam.Map(
              lambda line: line.replace(b', ', b','))
          | 'DecodeTrainData' >> csv_tfxio.BeamSource())

      # Combine data and schema into a dataset tuple.  Note that we already used
      # the schema to read the CSV data, but we also need it to interpret
      # raw_data.
      raw_dataset = (raw_data, csv_tfxio.TensorAdapterConfig())

      # The TFXIO output format is chosen for improved performance.
      transformed_dataset, transform_fn = (
          raw_dataset | tft_beam.AnalyzeAndTransformDataset(
              preprocessing_fn, output_record_batches=True))

      # Transformed metadata is not necessary for encoding.
      transformed_data, _ = transformed_dataset

      # Extract transformed RecordBatches, encode and write them to the given
      # directory.
      _ = (
          transformed_data
          | 'EncodeTrainData' >>
          beam.FlatMapTuple(lambda batch, _: RecordBatchToExamples(batch))
          | 'WriteTrainData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE)))

      # Now apply transform function to test data.  In this case we remove the
      # trailing period at the end of each line, and also ignore the header line
      # that is present in the test data file.
      raw_test_data = (
          pipeline
          | 'ReadTestData' >> beam.io.ReadFromText(
              test_data_file, skip_header_lines=1,
              coder=beam.coders.BytesCoder())
          | 'FixCommasTestData' >> beam.Map(
              lambda line: line.replace(b', ', b','))
          | 'RemoveTrailingPeriodsTestData' >> beam.Map(lambda line: line[:-1])
          | 'DecodeTestData' >> csv_tfxio.BeamSource())

      raw_test_dataset = (raw_test_data, csv_tfxio.TensorAdapterConfig())

      # The TFXIO output format is chosen for improved performance.
      transformed_test_dataset = (
          (raw_test_dataset, transform_fn)
          | tft_beam.TransformDataset(output_record_batches=True))

      # Transformed metadata is not necessary for encoding.
      transformed_test_data, _ = transformed_test_dataset

      # Extract transformed RecordBatches, encode and write them to the given
      # directory.
      _ = (
          transformed_test_data
          | 'EncodeTestData' >>
          beam.FlatMapTuple(lambda batch, _: RecordBatchToExamples(batch))
          | 'WriteTestData' >> beam.io.WriteToTFRecord(
              os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE)))

      # Will write a SavedModel and metadata to working_dir, which can then
      # be read by the tft.TFTransformOutput class.
      _ = (
          transform_fn
          | 'WriteTransformFn' >> tft_beam.WriteTransformFn(working_dir))

Using our preprocessed data to train a model using tf.keras

To show how tf.Transform enables us to use the same code for both training and serving, and thus prevent skew, we're going to train a model. To train our model and prepare our trained model for production we need to create input functions. The main difference between our training input function and our serving input function is that training data contains the labels, and production data does not. The arguments and returns are also somewhat different.

Create an input function for training

def _make_training_input_fn(tf_transform_output, transformed_examples,
                            batch_size):
  """An input function reading from transformed data, converting to model input.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    transformed_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input data for training or eval, in the form of k.
  """
  def input_fn():
    return tf.data.experimental.make_batched_features_dataset(
        file_pattern=transformed_examples,
        batch_size=batch_size,
        features=tf_transform_output.transformed_feature_spec(),
        reader=tf.data.TFRecordDataset,
        label_key=LABEL_KEY,
        shuffle=True).prefetch(tf.data.experimental.AUTOTUNE)

  return input_fn

Create an input function for serving

Let's create an input function that we could use in production, and prepare our trained model for serving.

def _make_serving_input_fn(tf_transform_output, raw_examples, batch_size):
  """An input function reading from raw data, converting to model input.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    raw_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input data for training or eval, in the form of k.
  """

  def get_ordered_raw_data_dtypes():
    result = []
    for col in ORDERED_CSV_COLUMNS:
      if col not in RAW_DATA_FEATURE_SPEC:
        result.append(0.0)
        continue
      spec = RAW_DATA_FEATURE_SPEC[col]
      if isinstance(spec, tf.io.FixedLenFeature):
        result.append(spec.dtype)
      else:
        result.append(0.0)
    return result

  def input_fn():
    dataset = tf.data.experimental.make_csv_dataset(
        file_pattern=raw_examples,
        batch_size=batch_size,
        column_names=ORDERED_CSV_COLUMNS,
        column_defaults=get_ordered_raw_data_dtypes(),
        prefetch_buffer_size=0,
        ignore_errors=True)

    tft_layer = tf_transform_output.transform_features_layer()

    def transform_dataset(data):
      raw_features = {}
      for key, val in data.items():
        if key not in RAW_DATA_FEATURE_SPEC:
          continue
        if isinstance(RAW_DATA_FEATURE_SPEC[key], tf.io.VarLenFeature):
          raw_features[key] = tf.RaggedTensor.from_tensor(
              tf.expand_dims(val, -1)).to_sparse()
          continue
        raw_features[key] = val
      transformed_features = tft_layer(raw_features)
      data_labels = transformed_features.pop(LABEL_KEY)
      return (transformed_features, data_labels)

    return dataset.map(
        transform_dataset,
        num_parallel_calls=tf.data.experimental.AUTOTUNE).prefetch(
            tf.data.experimental.AUTOTUNE)

  return input_fn

Train, Evaluate, and Export our model

def export_serving_model(tf_transform_output, model, output_dir):
  """Exports a keras model for serving.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    model: A keras model to export for serving.
    output_dir: A directory where the model will be exported to.
  """
  # The layer has to be saved to the model for keras tracking purpases.
  model.tft_layer = tf_transform_output.transform_features_layer()

  @tf.function
  def serve_tf_examples_fn(serialized_tf_examples):
    """Serving tf.function model wrapper."""
    feature_spec = RAW_DATA_FEATURE_SPEC.copy()
    feature_spec.pop(LABEL_KEY)
    parsed_features = tf.io.parse_example(serialized_tf_examples, feature_spec)
    transformed_features = model.tft_layer(parsed_features)
    outputs = model(transformed_features)
    classes_names = tf.constant([['0', '1']])
    classes = tf.tile(classes_names, [tf.shape(outputs)[0], 1])
    return {'classes': classes, 'scores': outputs}

  concrete_serving_fn = serve_tf_examples_fn.get_concrete_function(
      tf.TensorSpec(shape=[None], dtype=tf.string, name='inputs'))
  signatures = {'serving_default': concrete_serving_fn}

  # This is required in order to make this model servable with model_server.
  versioned_output_dir = os.path.join(output_dir, '1')
  model.save(versioned_output_dir, save_format='tf', signatures=signatures)
def train_and_evaluate(working_dir,
                       num_train_instances=NUM_TRAIN_INSTANCES,
                       num_test_instances=NUM_TEST_INSTANCES):
  """Train the model on training data and evaluate on test data.

  Args:
    working_dir: The location of the Transform output.
    num_train_instances: Number of instances in train set
    num_test_instances: Number of instances in test set

  Returns:
    The results from the estimator's 'evaluate' method
  """
  train_data_path_pattern = os.path.join(working_dir,
                                 TRANSFORMED_TRAIN_DATA_FILEBASE + '*')
  eval_data_path_pattern = os.path.join(working_dir,
                            TRANSFORMED_TEST_DATA_FILEBASE + '*')
  tf_transform_output = tft.TFTransformOutput(working_dir)

  train_input_fn = _make_training_input_fn(
      tf_transform_output, train_data_path_pattern, batch_size=TRAIN_BATCH_SIZE)
  train_dataset = train_input_fn()

  # Evaluate model on test dataset.
  eval_input_fn = _make_training_input_fn(
      tf_transform_output, eval_data_path_pattern, batch_size=TRAIN_BATCH_SIZE)
  validation_dataset = eval_input_fn()

  feature_spec = tf_transform_output.transformed_feature_spec().copy()
  feature_spec.pop(LABEL_KEY)

  inputs = {}
  for key, spec in feature_spec.items():
    if isinstance(spec, tf.io.VarLenFeature):
      inputs[key] = tf.keras.layers.Input(
          shape=[None], name=key, dtype=spec.dtype, sparse=True)
    elif isinstance(spec, tf.io.FixedLenFeature):
      inputs[key] = tf.keras.layers.Input(
          shape=spec.shape, name=key, dtype=spec.dtype)
    else:
      raise ValueError('Spec type is not supported: ', key, spec)

  encoded_inputs = {}
  for key in inputs:
    feature = tf.expand_dims(inputs[key], -1)
    if key in CATEGORICAL_FEATURE_KEYS:
      num_buckets = tf_transform_output.num_buckets_for_transformed_feature(key)
      encoding_layer = (
          tf.keras.layers.experimental.preprocessing.CategoryEncoding(
              max_tokens=num_buckets, output_mode='binary', sparse=False))
      encoded_inputs[key] = encoding_layer(feature)
    else:
      encoded_inputs[key] = feature

  stacked_inputs = tf.concat(tf.nest.flatten(encoded_inputs), axis=1)
  output = tf.keras.layers.Dense(100, activation='relu')(stacked_inputs)
  output = tf.keras.layers.Dense(70, activation='relu')(output)
  output = tf.keras.layers.Dense(50, activation='relu')(output)
  output = tf.keras.layers.Dense(20, activation='relu')(output)
  output = tf.keras.layers.Dense(2, activation='sigmoid')(output)
  model = tf.keras.Model(inputs=inputs, outputs=output)

  model.compile(optimizer='adam',
                loss='binary_crossentropy',
                metrics=['accuracy'])
  pprint.pprint(model.summary())

  model.fit(train_dataset, validation_data=validation_dataset,
            epochs=TRAIN_NUM_EPOCHS,
            steps_per_epoch=math.ceil(num_train_instances / TRAIN_BATCH_SIZE),
            validation_steps=math.ceil(num_test_instances / TRAIN_BATCH_SIZE))

  # Export the model.
  exported_model_dir = os.path.join(working_dir, EXPORTED_MODEL_DIR)
  export_serving_model(tf_transform_output, model, exported_model_dir)

  metrics_values = model.evaluate(validation_dataset, steps=num_test_instances)
  metrics_labels = model.metrics_names
  return {l: v for l, v in zip(metrics_labels, metrics_values)}

Put it all together

We've created all the stuff we need to preprocess our census data, train a model, and prepare it for serving. So far we've just been getting things ready. It's time to start running!

import tempfile
temp = os.path.join(tempfile.gettempdir(), 'keras')

transform_data(train, test, temp)
results = train_and_evaluate(temp)
pprint.pprint(results)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow_transform/tf_utils.py:261: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow_transform/tf_utils.py:261: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
2021-09-30 02:31:01.850979: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
INFO:tensorflow:Assets written to: /tmp/tmpd1cex_au/tftransform_tmp/21d20776ffa64c869670b7553b785f36/assets
INFO:tensorflow:Assets written to: /tmp/tmpd1cex_au/tftransform_tmp/21d20776ffa64c869670b7553b785f36/assets
INFO:tensorflow:Assets written to: /tmp/tmpd1cex_au/tftransform_tmp/f01aba569c414eb78512ef41e9e6b35e/assets
INFO:tensorflow:Assets written to: /tmp/tmpd1cex_au/tftransform_tmp/f01aba569c414eb78512ef41e9e6b35e/assets
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
WARNING:tensorflow:max_tokens is deprecated, please use num_tokens instead.
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
education (InputLayer)          [(None,)]            0                                            
__________________________________________________________________________________________________
marital-status (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
native-country (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
occupation (InputLayer)         [(None,)]            0                                            
__________________________________________________________________________________________________
race (InputLayer)               [(None,)]            0                                            
__________________________________________________________________________________________________
relationship (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
sex (InputLayer)                [(None,)]            0                                            
__________________________________________________________________________________________________
workclass (InputLayer)          [(None,)]            0                                            
__________________________________________________________________________________________________
age (InputLayer)                [(None,)]            0                                            
__________________________________________________________________________________________________
capital-gain (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
capital-loss (InputLayer)       [(None,)]            0                                            
__________________________________________________________________________________________________
tf.expand_dims_3 (TFOpLambda)   (None, 1)            0           education[0][0]                  
__________________________________________________________________________________________________
education-num (InputLayer)      [(None,)]            0                                            
__________________________________________________________________________________________________
hours-per-week (InputLayer)     [(None,)]            0                                            
__________________________________________________________________________________________________
tf.expand_dims_6 (TFOpLambda)   (None, 1)            0           marital-status[0][0]             
__________________________________________________________________________________________________
tf.expand_dims_7 (TFOpLambda)   (None, 1)            0           native-country[0][0]             
__________________________________________________________________________________________________
tf.expand_dims_8 (TFOpLambda)   (None, 1)            0           occupation[0][0]                 
__________________________________________________________________________________________________
tf.expand_dims_9 (TFOpLambda)   (None, 1)            0           race[0][0]                       
__________________________________________________________________________________________________
tf.expand_dims_10 (TFOpLambda)  (None, 1)            0           relationship[0][0]               
__________________________________________________________________________________________________
tf.expand_dims_11 (TFOpLambda)  (None, 1)            0           sex[0][0]                        
__________________________________________________________________________________________________
tf.expand_dims_12 (TFOpLambda)  (None, 1)            0           workclass[0][0]                  
__________________________________________________________________________________________________
tf.expand_dims (TFOpLambda)     (None, 1)            0           age[0][0]                        
__________________________________________________________________________________________________
tf.expand_dims_1 (TFOpLambda)   (None, 1)            0           capital-gain[0][0]               
__________________________________________________________________________________________________
tf.expand_dims_2 (TFOpLambda)   (None, 1)            0           capital-loss[0][0]               
__________________________________________________________________________________________________
category_encoding (CategoryEnco (None, 17)           0           tf.expand_dims_3[0][0]           
__________________________________________________________________________________________________
tf.expand_dims_4 (TFOpLambda)   (None, 1)            0           education-num[0][0]              
__________________________________________________________________________________________________
tf.expand_dims_5 (TFOpLambda)   (None, 1)            0           hours-per-week[0][0]             
__________________________________________________________________________________________________
category_encoding_1 (CategoryEn (None, 8)            0           tf.expand_dims_6[0][0]           
__________________________________________________________________________________________________
category_encoding_2 (CategoryEn (None, 43)           0           tf.expand_dims_7[0][0]           
__________________________________________________________________________________________________
category_encoding_3 (CategoryEn (None, 16)           0           tf.expand_dims_8[0][0]           
__________________________________________________________________________________________________
category_encoding_4 (CategoryEn (None, 6)            0           tf.expand_dims_9[0][0]           
__________________________________________________________________________________________________
category_encoding_5 (CategoryEn (None, 7)            0           tf.expand_dims_10[0][0]          
__________________________________________________________________________________________________
category_encoding_6 (CategoryEn (None, 3)            0           tf.expand_dims_11[0][0]          
__________________________________________________________________________________________________
category_encoding_7 (CategoryEn (None, 10)           0           tf.expand_dims_12[0][0]          
__________________________________________________________________________________________________
tf.concat (TFOpLambda)          (None, 115)          0           tf.expand_dims[0][0]             
                                                                 tf.expand_dims_1[0][0]           
                                                                 tf.expand_dims_2[0][0]           
                                                                 category_encoding[0][0]          
                                                                 tf.expand_dims_4[0][0]           
                                                                 tf.expand_dims_5[0][0]           
                                                                 category_encoding_1[0][0]        
                                                                 category_encoding_2[0][0]        
                                                                 category_encoding_3[0][0]        
                                                                 category_encoding_4[0][0]        
                                                                 category_encoding_5[0][0]        
                                                                 category_encoding_6[0][0]        
                                                                 category_encoding_7[0][0]        
__________________________________________________________________________________________________
dense (Dense)                   (None, 100)          11600       tf.concat[0][0]                  
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 70)           7070        dense[0][0]                      
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 50)           3550        dense_1[0][0]                    
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 20)           1020        dense_2[0][0]                    
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 2)            42          dense_3[0][0]                    
==================================================================================================
Total params: 23,282
Trainable params: 23,282
Non-trainable params: 0
__________________________________________________________________________________________________
None
Epoch 1/16
255/255 [==============================] - 3s 7ms/step - loss: 0.3795 - accuracy: 0.8211 - val_loss: 0.3425 - val_accuracy: 0.8447
Epoch 2/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3341 - accuracy: 0.8449 - val_loss: 0.3290 - val_accuracy: 0.8486
Epoch 3/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3234 - accuracy: 0.8491 - val_loss: 0.3244 - val_accuracy: 0.8482
Epoch 4/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3150 - accuracy: 0.8520 - val_loss: 0.3207 - val_accuracy: 0.8486
Epoch 5/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3087 - accuracy: 0.8549 - val_loss: 0.3206 - val_accuracy: 0.8504
Epoch 6/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3045 - accuracy: 0.8569 - val_loss: 0.3267 - val_accuracy: 0.8470
Epoch 7/16
255/255 [==============================] - 2s 6ms/step - loss: 0.3013 - accuracy: 0.8578 - val_loss: 0.3223 - val_accuracy: 0.8531
Epoch 8/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2987 - accuracy: 0.8592 - val_loss: 0.3230 - val_accuracy: 0.8488
Epoch 9/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2943 - accuracy: 0.8626 - val_loss: 0.3308 - val_accuracy: 0.8516
Epoch 10/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2907 - accuracy: 0.8633 - val_loss: 0.3291 - val_accuracy: 0.8488
Epoch 11/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2870 - accuracy: 0.8669 - val_loss: 0.3279 - val_accuracy: 0.8514
Epoch 12/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2818 - accuracy: 0.8679 - val_loss: 0.3414 - val_accuracy: 0.8469
Epoch 13/16
255/255 [==============================] - 2s 7ms/step - loss: 0.2795 - accuracy: 0.8666 - val_loss: 0.3416 - val_accuracy: 0.8482
Epoch 14/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2779 - accuracy: 0.8698 - val_loss: 0.3355 - val_accuracy: 0.8497
Epoch 15/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2735 - accuracy: 0.8701 - val_loss: 0.3512 - val_accuracy: 0.8476
Epoch 16/16
255/255 [==============================] - 2s 6ms/step - loss: 0.2712 - accuracy: 0.8728 - val_loss: 0.3571 - val_accuracy: 0.8472
INFO:tensorflow:Assets written to: /tmp/keras/exported_model_dir/1/assets
INFO:tensorflow:Assets written to: /tmp/keras/exported_model_dir/1/assets
16281/16281 [==============================] - 60s 4ms/step - loss: 0.3571 - accuracy: 0.8474
{'accuracy': 0.8473681211471558, 'loss': 0.357058048248291}

(Optional) Using our preprocessed data to train a model using tf.estimator

If you would rather use an Estimator model instead of a Keras model, the code in this section shows how to do that.

Create an input function for training

def _make_training_input_fn(tf_transform_output, transformed_examples,
                            batch_size):
  """Creates an input function reading from transformed data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.
    transformed_examples: Base filename of examples.
    batch_size: Batch size.

  Returns:
    The input function for training or eval.
  """
  def input_fn():
    """Input function for training and eval."""
    dataset = tf.data.experimental.make_batched_features_dataset(
        file_pattern=transformed_examples,
        batch_size=batch_size,
        features=tf_transform_output.transformed_feature_spec(),
        reader=tf.data.TFRecordDataset,
        shuffle=True)

    transformed_features = tf.compat.v1.data.make_one_shot_iterator(
        dataset).get_next()

    # Extract features and label from the transformed tensors.
    transformed_labels = tf.where(
        tf.equal(transformed_features.pop(LABEL_KEY), 1))

    return transformed_features, transformed_labels[:,1]

  return input_fn

Create an input function for serving

Let's create an input function that we could use in production, and prepare our trained model for serving.

def _make_serving_input_fn(tf_transform_output):
  """Creates an input function reading from raw data.

  Args:
    tf_transform_output: Wrapper around output of tf.Transform.

  Returns:
    The serving input function.
  """
  raw_feature_spec = RAW_DATA_FEATURE_SPEC.copy()
  # Remove label since it is not available during serving.
  raw_feature_spec.pop(LABEL_KEY)

  def serving_input_fn():
    """Input function for serving."""
    # Get raw features by generating the basic serving input_fn and calling it.
    # Here we generate an input_fn that expects a parsed Example proto to be fed
    # to the model at serving time.  See also
    # tf.estimator.export.build_raw_serving_input_receiver_fn.
    raw_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(
        raw_feature_spec, default_batch_size=None)
    serving_input_receiver = raw_input_fn()

    # Apply the transform function that was used to generate the materialized
    # data.
    raw_features = serving_input_receiver.features
    transformed_features = tf_transform_output.transform_raw_features(
        raw_features)

    return tf.estimator.export.ServingInputReceiver(
        transformed_features, serving_input_receiver.receiver_tensors)

  return serving_input_fn

Wrap our input data in FeatureColumns

Our model will expect our data in TensorFlow FeatureColumns.

def get_feature_columns(tf_transform_output):
  """Returns the FeatureColumns for the model.

  Args:
    tf_transform_output: A `TFTransformOutput` object.

  Returns:
    A list of FeatureColumns.
  """
  # Wrap scalars as real valued columns.
  real_valued_columns = [tf.feature_column.numeric_column(key, shape=())
                         for key in NUMERIC_FEATURE_KEYS]

  # Wrap categorical columns.
  one_hot_columns = [
      tf.feature_column.indicator_column(
          tf.feature_column.categorical_column_with_identity(
              key=key,
              num_buckets=(NUM_OOV_BUCKETS +
                  tf_transform_output.vocabulary_size_by_name(
                      vocab_filename=key))))
      for key in CATEGORICAL_FEATURE_KEYS]

  return real_valued_columns + one_hot_columns

Train, Evaluate, and Export our model

def train_and_evaluate(working_dir, num_train_instances=NUM_TRAIN_INSTANCES,
                       num_test_instances=NUM_TEST_INSTANCES):
  """Train the model on training data and evaluate on test data.

  Args:
    working_dir: Directory to read transformed data and metadata from and to
        write exported model to.
    num_train_instances: Number of instances in train set
    num_test_instances: Number of instances in test set

  Returns:
    The results from the estimator's 'evaluate' method
  """
  tf_transform_output = tft.TFTransformOutput(working_dir)

  run_config = tf.estimator.RunConfig()

  estimator = tf.estimator.LinearClassifier(
      feature_columns=get_feature_columns(tf_transform_output),
      config=run_config,
      loss_reduction=tf.losses.Reduction.SUM)

  # Fit the model using the default optimizer.
  train_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE + '*'),
      batch_size=TRAIN_BATCH_SIZE)
  estimator.train(
      input_fn=train_input_fn,
      max_steps=TRAIN_NUM_EPOCHS * num_train_instances / TRAIN_BATCH_SIZE)

  # Evaluate model on test dataset.
  eval_input_fn = _make_training_input_fn(
      tf_transform_output,
      os.path.join(working_dir, TRANSFORMED_TEST_DATA_FILEBASE + '*'),
      batch_size=1)

  # Export the model.
  serving_input_fn = _make_serving_input_fn(tf_transform_output)
  exported_model_dir = os.path.join(working_dir, EXPORTED_MODEL_DIR)
  estimator.export_saved_model(exported_model_dir, serving_input_fn)

  return estimator.evaluate(input_fn=eval_input_fn, steps=num_test_instances)

Put it all together

We've created all the stuff we need to preprocess our census data, train a model, and prepare it for serving. So far we've just been getting things ready. It's time to start running!

import tempfile
temp = os.path.join(tempfile.gettempdir(), 'estimator')

transform_data(train, test, temp)
results = train_and_evaluate(temp)
pprint.pprint(results)
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
Exception ignored in: <function CapturableResource.__del__ at 0x7fcea4afc290>
Traceback (most recent call last):
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 277, in __del__
    self._destroy_resource()
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3022, in __call__
    filtered_flat_args) = self._maybe_define_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
AttributeError: 'NoneType' object has no attribute '__wrapped__'
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:tensorflow:Assets written to: /tmp/tmp72gs5nsi/tftransform_tmp/09e8562d89874e559c930ddf078c1286/assets
INFO:tensorflow:Assets written to: /tmp/tmp72gs5nsi/tftransform_tmp/09e8562d89874e559c930ddf078c1286/assets
INFO:tensorflow:Assets written to: /tmp/tmp72gs5nsi/tftransform_tmp/fb207a9c6d2e4357abb07318f22b5085/assets
INFO:tensorflow:Assets written to: /tmp/tmp72gs5nsi/tftransform_tmp/fb207a9c6d2e4357abb07318f22b5085/assets
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp8qyug3l8
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp8qyug3l8
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp8qyug3l8', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp8qyug3l8', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
/home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1700: UserWarning: `layer.add_variable` is deprecated and will be removed in a future version. Please use `layer.add_weight` method instead.
  warnings.warn('`layer.add_variable` is deprecated and '
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:149: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:149: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp8qyug3l8/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp8qyug3l8/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 88.72284, step = 0
INFO:tensorflow:loss = 88.72284, step = 0
INFO:tensorflow:global_step/sec: 185.378
INFO:tensorflow:global_step/sec: 185.378
INFO:tensorflow:loss = 38.7204, step = 100 (0.541 sec)
INFO:tensorflow:loss = 38.7204, step = 100 (0.541 sec)
INFO:tensorflow:global_step/sec: 245.551
INFO:tensorflow:global_step/sec: 245.551
INFO:tensorflow:loss = 46.631004, step = 200 (0.407 sec)
INFO:tensorflow:loss = 46.631004, step = 200 (0.407 sec)
INFO:tensorflow:global_step/sec: 245.097
INFO:tensorflow:global_step/sec: 245.097
INFO:tensorflow:loss = 42.11092, step = 300 (0.408 sec)
INFO:tensorflow:loss = 42.11092, step = 300 (0.408 sec)
INFO:tensorflow:global_step/sec: 249.879
INFO:tensorflow:global_step/sec: 249.879
INFO:tensorflow:loss = 45.452286, step = 400 (0.400 sec)
INFO:tensorflow:loss = 45.452286, step = 400 (0.400 sec)
INFO:tensorflow:global_step/sec: 250.479
INFO:tensorflow:global_step/sec: 250.479
INFO:tensorflow:loss = 47.122536, step = 500 (0.399 sec)
INFO:tensorflow:loss = 47.122536, step = 500 (0.399 sec)
INFO:tensorflow:global_step/sec: 251.807
INFO:tensorflow:global_step/sec: 251.807
INFO:tensorflow:loss = 47.914326, step = 600 (0.397 sec)
INFO:tensorflow:loss = 47.914326, step = 600 (0.397 sec)
INFO:tensorflow:global_step/sec: 249.916
INFO:tensorflow:global_step/sec: 249.916
INFO:tensorflow:loss = 33.26215, step = 700 (0.400 sec)
INFO:tensorflow:loss = 33.26215, step = 700 (0.400 sec)
INFO:tensorflow:global_step/sec: 253.629
INFO:tensorflow:global_step/sec: 253.629
INFO:tensorflow:loss = 40.877037, step = 800 (0.394 sec)
INFO:tensorflow:loss = 40.877037, step = 800 (0.394 sec)
INFO:tensorflow:global_step/sec: 251.941
INFO:tensorflow:global_step/sec: 251.941
INFO:tensorflow:loss = 47.78914, step = 900 (0.397 sec)
INFO:tensorflow:loss = 47.78914, step = 900 (0.397 sec)
INFO:tensorflow:global_step/sec: 250.837
INFO:tensorflow:global_step/sec: 250.837
INFO:tensorflow:loss = 46.0111, step = 1000 (0.399 sec)
INFO:tensorflow:loss = 46.0111, step = 1000 (0.399 sec)
INFO:tensorflow:global_step/sec: 245.293
INFO:tensorflow:global_step/sec: 245.293
INFO:tensorflow:loss = 43.548885, step = 1100 (0.408 sec)
INFO:tensorflow:loss = 43.548885, step = 1100 (0.408 sec)
INFO:tensorflow:global_step/sec: 246.393
INFO:tensorflow:global_step/sec: 246.393
INFO:tensorflow:loss = 39.584225, step = 1200 (0.406 sec)
INFO:tensorflow:loss = 39.584225, step = 1200 (0.406 sec)
INFO:tensorflow:global_step/sec: 244.852
INFO:tensorflow:global_step/sec: 244.852
INFO:tensorflow:loss = 38.676952, step = 1300 (0.408 sec)
INFO:tensorflow:loss = 38.676952, step = 1300 (0.408 sec)
INFO:tensorflow:global_step/sec: 249.666
INFO:tensorflow:global_step/sec: 249.666
INFO:tensorflow:loss = 30.786205, step = 1400 (0.401 sec)
INFO:tensorflow:loss = 30.786205, step = 1400 (0.401 sec)
INFO:tensorflow:global_step/sec: 251.171
INFO:tensorflow:global_step/sec: 251.171
INFO:tensorflow:loss = 46.162445, step = 1500 (0.398 sec)
INFO:tensorflow:loss = 46.162445, step = 1500 (0.398 sec)
INFO:tensorflow:global_step/sec: 246.803
INFO:tensorflow:global_step/sec: 246.803
INFO:tensorflow:loss = 38.054405, step = 1600 (0.405 sec)
INFO:tensorflow:loss = 38.054405, step = 1600 (0.405 sec)
INFO:tensorflow:global_step/sec: 250.874
INFO:tensorflow:global_step/sec: 250.874
INFO:tensorflow:loss = 39.51439, step = 1700 (0.399 sec)
INFO:tensorflow:loss = 39.51439, step = 1700 (0.399 sec)
INFO:tensorflow:global_step/sec: 249.77
INFO:tensorflow:global_step/sec: 249.77
INFO:tensorflow:loss = 52.209587, step = 1800 (0.400 sec)
INFO:tensorflow:loss = 52.209587, step = 1800 (0.400 sec)
INFO:tensorflow:global_step/sec: 250.912
INFO:tensorflow:global_step/sec: 250.912
INFO:tensorflow:loss = 45.748035, step = 1900 (0.399 sec)
INFO:tensorflow:loss = 45.748035, step = 1900 (0.399 sec)
INFO:tensorflow:global_step/sec: 248.3
INFO:tensorflow:global_step/sec: 248.3
INFO:tensorflow:loss = 33.066315, step = 2000 (0.403 sec)
INFO:tensorflow:loss = 33.066315, step = 2000 (0.403 sec)
INFO:tensorflow:global_step/sec: 249.278
INFO:tensorflow:global_step/sec: 249.278
INFO:tensorflow:loss = 42.234577, step = 2100 (0.401 sec)
INFO:tensorflow:loss = 42.234577, step = 2100 (0.401 sec)
INFO:tensorflow:global_step/sec: 249.358
INFO:tensorflow:global_step/sec: 249.358
INFO:tensorflow:loss = 40.97232, step = 2200 (0.401 sec)
INFO:tensorflow:loss = 40.97232, step = 2200 (0.401 sec)
INFO:tensorflow:global_step/sec: 247.775
INFO:tensorflow:global_step/sec: 247.775
INFO:tensorflow:loss = 42.90811, step = 2300 (0.404 sec)
INFO:tensorflow:loss = 42.90811, step = 2300 (0.404 sec)
INFO:tensorflow:global_step/sec: 251.49
INFO:tensorflow:global_step/sec: 251.49
INFO:tensorflow:loss = 34.21136, step = 2400 (0.398 sec)
INFO:tensorflow:loss = 34.21136, step = 2400 (0.398 sec)
INFO:tensorflow:global_step/sec: 251.484
INFO:tensorflow:global_step/sec: 251.484
INFO:tensorflow:loss = 47.288605, step = 2500 (0.398 sec)
INFO:tensorflow:loss = 47.288605, step = 2500 (0.398 sec)
INFO:tensorflow:global_step/sec: 251.154
INFO:tensorflow:global_step/sec: 251.154
INFO:tensorflow:loss = 53.636517, step = 2600 (0.398 sec)
INFO:tensorflow:loss = 53.636517, step = 2600 (0.398 sec)
INFO:tensorflow:global_step/sec: 251.953
INFO:tensorflow:global_step/sec: 251.953
INFO:tensorflow:loss = 36.662205, step = 2700 (0.397 sec)
INFO:tensorflow:loss = 36.662205, step = 2700 (0.397 sec)
INFO:tensorflow:global_step/sec: 250.658
INFO:tensorflow:global_step/sec: 250.658
INFO:tensorflow:loss = 41.134422, step = 2800 (0.399 sec)
INFO:tensorflow:loss = 41.134422, step = 2800 (0.399 sec)
INFO:tensorflow:global_step/sec: 249.268
INFO:tensorflow:global_step/sec: 249.268
INFO:tensorflow:loss = 35.20195, step = 2900 (0.401 sec)
INFO:tensorflow:loss = 35.20195, step = 2900 (0.401 sec)
INFO:tensorflow:global_step/sec: 250.445
INFO:tensorflow:global_step/sec: 250.445
INFO:tensorflow:loss = 43.653954, step = 3000 (0.399 sec)
INFO:tensorflow:loss = 43.653954, step = 3000 (0.399 sec)
INFO:tensorflow:global_step/sec: 247.714
INFO:tensorflow:global_step/sec: 247.714
INFO:tensorflow:loss = 35.921513, step = 3100 (0.404 sec)
INFO:tensorflow:loss = 35.921513, step = 3100 (0.404 sec)
INFO:tensorflow:global_step/sec: 246.162
INFO:tensorflow:global_step/sec: 246.162
INFO:tensorflow:loss = 41.18482, step = 3200 (0.406 sec)
INFO:tensorflow:loss = 41.18482, step = 3200 (0.406 sec)
INFO:tensorflow:global_step/sec: 252.253
INFO:tensorflow:global_step/sec: 252.253
INFO:tensorflow:loss = 43.12972, step = 3300 (0.396 sec)
INFO:tensorflow:loss = 43.12972, step = 3300 (0.396 sec)
INFO:tensorflow:global_step/sec: 251.81
INFO:tensorflow:global_step/sec: 251.81
INFO:tensorflow:loss = 49.742702, step = 3400 (0.397 sec)
INFO:tensorflow:loss = 49.742702, step = 3400 (0.397 sec)
INFO:tensorflow:global_step/sec: 248.348
INFO:tensorflow:global_step/sec: 248.348
INFO:tensorflow:loss = 35.813183, step = 3500 (0.402 sec)
INFO:tensorflow:loss = 35.813183, step = 3500 (0.402 sec)
INFO:tensorflow:global_step/sec: 251.73
INFO:tensorflow:global_step/sec: 251.73
INFO:tensorflow:loss = 45.16124, step = 3600 (0.397 sec)
INFO:tensorflow:loss = 45.16124, step = 3600 (0.397 sec)
INFO:tensorflow:global_step/sec: 253.182
INFO:tensorflow:global_step/sec: 253.182
INFO:tensorflow:loss = 34.189198, step = 3700 (0.395 sec)
INFO:tensorflow:loss = 34.189198, step = 3700 (0.395 sec)
INFO:tensorflow:global_step/sec: 251.82
INFO:tensorflow:global_step/sec: 251.82
INFO:tensorflow:loss = 49.45624, step = 3800 (0.397 sec)
INFO:tensorflow:loss = 49.45624, step = 3800 (0.397 sec)
INFO:tensorflow:global_step/sec: 251.734
INFO:tensorflow:global_step/sec: 251.734
INFO:tensorflow:loss = 50.020947, step = 3900 (0.397 sec)
INFO:tensorflow:loss = 50.020947, step = 3900 (0.397 sec)
INFO:tensorflow:global_step/sec: 246.923
INFO:tensorflow:global_step/sec: 246.923
INFO:tensorflow:loss = 40.07228, step = 4000 (0.405 sec)
INFO:tensorflow:loss = 40.07228, step = 4000 (0.405 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 4071...
INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmp8qyug3l8/model.ckpt.
INFO:tensorflow:Saving checkpoints for 4071 into /tmp/tmp8qyug3l8/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 4071...
INFO:tensorflow:Loss for final step: 48.046616.
INFO:tensorflow:Loss for final step: 48.046616.
WARNING:tensorflow:Loading a TF2 SavedModel but eager mode seems disabled.
WARNING:tensorflow:Loading a TF2 SavedModel but eager mode seems disabled.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:145: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /home/kbuilder/.local/lib/python3.7/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:145: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']
INFO:tensorflow:Signatures INCLUDED in export for Classify: ['serving_default', 'classification']
INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']
INFO:tensorflow:Signatures INCLUDED in export for Regress: ['regression']
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Restoring parameters from /tmp/tmp8qyug3l8/model.ckpt-4071
INFO:tensorflow:Restoring parameters from /tmp/tmp8qyug3l8/model.ckpt-4071
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/estimator/exported_model_dir/temp-1632969202/assets
INFO:tensorflow:Assets written to: /tmp/estimator/exported_model_dir/temp-1632969202/assets
INFO:tensorflow:SavedModel written to: /tmp/estimator/exported_model_dir/temp-1632969202/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/estimator/exported_model_dir/temp-1632969202/saved_model.pb
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-09-30T02:33:24
INFO:tensorflow:Starting evaluation at 2021-09-30T02:33:24
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmp8qyug3l8/model.ckpt-4071
INFO:tensorflow:Restoring parameters from /tmp/tmp8qyug3l8/model.ckpt-4071
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1628/16281]
INFO:tensorflow:Evaluation [1628/16281]
INFO:tensorflow:Evaluation [3256/16281]
INFO:tensorflow:Evaluation [3256/16281]
INFO:tensorflow:Evaluation [4884/16281]
INFO:tensorflow:Evaluation [4884/16281]
INFO:tensorflow:Evaluation [6512/16281]
INFO:tensorflow:Evaluation [6512/16281]
INFO:tensorflow:Evaluation [8140/16281]
INFO:tensorflow:Evaluation [8140/16281]
INFO:tensorflow:Evaluation [9768/16281]
INFO:tensorflow:Evaluation [9768/16281]
INFO:tensorflow:Evaluation [11396/16281]
INFO:tensorflow:Evaluation [11396/16281]
INFO:tensorflow:Evaluation [13024/16281]
INFO:tensorflow:Evaluation [13024/16281]
INFO:tensorflow:Evaluation [14652/16281]
INFO:tensorflow:Evaluation [14652/16281]
INFO:tensorflow:Evaluation [16280/16281]
INFO:tensorflow:Evaluation [16280/16281]
INFO:tensorflow:Evaluation [16281/16281]
INFO:tensorflow:Evaluation [16281/16281]
INFO:tensorflow:Inference Time : 63.96106s
INFO:tensorflow:Inference Time : 63.96106s
INFO:tensorflow:Finished evaluation at 2021-09-30-02:34:28
INFO:tensorflow:Finished evaluation at 2021-09-30-02:34:28
INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.850562, accuracy_baseline = 0.76377374, auc = 0.90161204, auc_precision_recall = 0.967104, average_loss = 0.32443467, global_step = 4071, label/mean = 0.76377374, loss = 0.32443467, precision = 0.88210577, prediction/mean = 0.75769806, recall = 0.9284278
INFO:tensorflow:Saving dict for global step 4071: accuracy = 0.850562, accuracy_baseline = 0.76377374, auc = 0.90161204, auc_precision_recall = 0.967104, average_loss = 0.32443467, global_step = 4071, label/mean = 0.76377374, loss = 0.32443467, precision = 0.88210577, prediction/mean = 0.75769806, recall = 0.9284278
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmp8qyug3l8/model.ckpt-4071
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4071: /tmp/tmp8qyug3l8/model.ckpt-4071
{'accuracy': 0.850562,
 'accuracy_baseline': 0.76377374,
 'auc': 0.90161204,
 'auc_precision_recall': 0.967104,
 'average_loss': 0.32443467,
 'global_step': 4071,
 'label/mean': 0.76377374,
 'loss': 0.32443467,
 'precision': 0.88210577,
 'prediction/mean': 0.75769806,
 'recall': 0.9284278}

What we did

In this example we used tf.Transform to preprocess a dataset of census data, and train a model with the cleaned and transformed data. We also created an input function that we could use when we deploy our trained model in a production environment to perform inference. By using the same code for both training and inference we avoid any issues with data skew. Along the way we learned about creating an Apache Beam transform to perform the transformation that we needed for cleaning the data. We also saw how to use this transformed data to train a model using either tf.keras or tf.estimator. This is just a small piece of what TensorFlow Transform can do! We encourage you to dive into tf.Transform and discover what it can do for you.