Build, train and evaluate models with TensorFlow Decision Forests

View on TensorFlow.org Run in Google Colab View on GitHub Download notebook

Introduction

Decision Forests (DF) are a large family of Machine Learning algorithms for supervised classification, regression and ranking. As the name suggests, DFs use decision trees as a building block. Today, the two most popular DF training algorithms are Random Forests and Gradient Boosted Decision Trees. Both algorithms are ensemble techniques that use multiple decision trees, but differ on how they do it.

TensorFlow Decision Forests (TF-DF) is a library for the training, evaluation, interpretation and inference of Decision Forest models.

In this tutorial, you will learn how to:

  1. Train a binary classification Random Forest on a dataset containing numerical, categorical and missing features.
  2. Evaluate the model on a test dataset.
  3. Prepare the model for TensorFlow Serving.
  4. Examine the overall structure of the model and the importance of each feature.
  5. Re-train the model with a different learning algorithm (Gradient Boosted Decision Trees).
  6. Use a different set of input features.
  7. Change the hyperparameters of the model.
  8. Preprocess the features.
  9. Train a model for regression.
  10. Train a model for ranking.

Detailed documentation is available in the user manual. The example directory contains other end-to-end examples.

Installing TensorFlow Decision Forests

Install TF-DF by running the following cell.

pip install tensorflow_decision_forests

Wurlitzer is needed to display the detailed training logs in Colabs (when using verbose=2 in the model constructor).

pip install wurlitzer

Importing libraries

import tensorflow_decision_forests as tfdf

import os
import numpy as np
import pandas as pd
import tensorflow as tf
import math

The hidden code cell limits the output height in colab.

# Check the version of TensorFlow Decision Forests
print("Found TensorFlow Decision Forests v" + tfdf.__version__)
Found TensorFlow Decision Forests v0.2.7

Training a Random Forest model

In this section, we train, evaluate, analyse and export a binary classification Random Forest trained on the Palmer's Penguins dataset.

Load the dataset and convert it in a tf.Dataset

This dataset is very small (300 examples) and stored as a .csv-like file. Therefore, use Pandas to load it.

Let's assemble the dataset into a csv file (i.e. add the header), and load it:

# Download the dataset
!wget -q https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins.csv -O /tmp/penguins.csv

# Load a dataset into a Pandas Dataframe.
dataset_df = pd.read_csv("/tmp/penguins.csv")

# Display the first 3 examples.
dataset_df.head(3)

The dataset contains a mix of numerical (e.g. bill_depth_mm), categorical (e.g. island) and missing features. TF-DF supports all these feature types natively (differently than NN based models), therefore there is no need for preprocessing in the form of one-hot encoding, normalization or extra is_present feature.

Labels are a bit different: Keras metrics expect integers. The label (species) is stored as a string, so let's convert it into an integer.

# Encode the categorical labels as integers.
#
# Details:
# This stage is necessary if your classification label is represented as a
# string since Keras expects integer classification labels.
# When using `pd_dataframe_to_tf_dataset` (see below), this step can be skipped.

# Name of the label column.
label = "species"

classes = dataset_df[label].unique().tolist()
print(f"Label classes: {classes}")

dataset_df[label] = dataset_df[label].map(classes.index)
Label classes: ['Adelie', 'Gentoo', 'Chinstrap']

Next split the dataset into training and testing:

# Split the dataset into a training and a testing dataset.

def split_dataset(dataset, test_ratio=0.30):
  """Splits a panda dataframe in two."""
  test_indices = np.random.rand(len(dataset)) < test_ratio
  return dataset[~test_indices], dataset[test_indices]


train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))
242 examples in training, 102 examples for testing.

And finally, convert the pandas dataframe (pd.Dataframe) into tensorflow datasets (tf.data.Dataset):

train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label)
test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_ds_pd, label=label)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)

Notes: Recall that pd_dataframe_to_tf_dataset converts string labels to integers if necessary.

If you want to create the tf.data.Dataset yourself, there are a couple of things to remember:

  • The learning algorithms work with a one-epoch dataset and without shuffling.
  • The batch size does not impact the training algorithm, but a small value might slow down reading the dataset.

Train the model

%set_cell_height 300

# Specify the model.
model_1 = tfdf.keras.RandomForestModel(verbose=2)

# Train the model.
model_1.fit(x=train_ds)
<IPython.core.display.Javascript object>
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmp4ebv7hkj as temporary training directory
Reading training dataset...
Training tensor examples:
Features: {'island': <tf.Tensor 'data_4:0' shape=(None,) dtype=string>, 'bill_length_mm': <tf.Tensor 'data_1:0' shape=(None,) dtype=float64>, 'bill_depth_mm': <tf.Tensor 'data:0' shape=(None,) dtype=float64>, 'flipper_length_mm': <tf.Tensor 'data_3:0' shape=(None,) dtype=float64>, 'body_mass_g': <tf.Tensor 'data_2:0' shape=(None,) dtype=float64>, 'sex': <tf.Tensor 'data_5:0' shape=(None,) dtype=string>, 'year': <tf.Tensor 'data_6:0' shape=(None,) dtype=int64>}
Label: Tensor("data_7:0", shape=(None,), dtype=int64)
Weights: None
Normalized tensor features:
 {'island': SemanticTensor(semantic=<Semantic.CATEGORICAL: 2>, tensor=<tf.Tensor 'data_4:0' shape=(None,) dtype=string>), 'bill_length_mm': SemanticTensor(semantic=<Semantic.NUMERICAL: 1>, tensor=<tf.Tensor 'Cast:0' shape=(None,) dtype=float32>), 'bill_depth_mm': SemanticTensor(semantic=<Semantic.NUMERICAL: 1>, tensor=<tf.Tensor 'Cast_1:0' shape=(None,) dtype=float32>), 'flipper_length_mm': SemanticTensor(semantic=<Semantic.NUMERICAL: 1>, tensor=<tf.Tensor 'Cast_2:0' shape=(None,) dtype=float32>), 'body_mass_g': SemanticTensor(semantic=<Semantic.NUMERICAL: 1>, tensor=<tf.Tensor 'Cast_3:0' shape=(None,) dtype=float32>), 'sex': SemanticTensor(semantic=<Semantic.CATEGORICAL: 2>, tensor=<tf.Tensor 'data_5:0' shape=(None,) dtype=string>), 'year': SemanticTensor(semantic=<Semantic.NUMERICAL: 1>, tensor=<tf.Tensor 'Cast_4:0' shape=(None,) dtype=float32>)}
Training dataset read in 0:00:03.021331. Found 242 examples.
Training model...
Standard output detected as not visible to the user e.g. running in a notebook. Creating a training log redirection. If training get stuck, try calling tfdf.keras.set_training_logs_redirection(False).
[INFO kernel.cc:813] Start Yggdrasil model training
[INFO kernel.cc:814] Collect training examples
[INFO kernel.cc:422] Number of batches: 1
[INFO kernel.cc:423] Number of examples: 242
[INFO kernel.cc:836] Training dataset:
Number of records: 242
Number of columns: 8

Number of columns by type:
    NUMERICAL: 5 (62.5%)
    CATEGORICAL: 3 (37.5%)

Columns:

NUMERICAL: 5 (62.5%)
    0: "bill_depth_mm" NUMERICAL num-nas:2 (0.826446%) mean:17.2567 min:13.2 max:21.5 sd:1.98672
    1: "bill_length_mm" NUMERICAL num-nas:2 (0.826446%) mean:43.8558 min:32.1 max:59.6 sd:5.56876
    2: "body_mass_g" NUMERICAL num-nas:2 (0.826446%) mean:4170.21 min:2700 max:6300 sd:780.423
    3: "flipper_length_mm" NUMERICAL num-nas:2 (0.826446%) mean:200.575 min:172 max:230 sd:13.8514
    6: "year" NUMERICAL mean:2008.07 min:2007 max:2009 sd:0.807521

CATEGORICAL: 3 (37.5%)
    4: "island" CATEGORICAL has-dict vocab-size:4 zero-ood-items most-frequent:"Biscoe" 115 (47.5207%)
    5: "sex" CATEGORICAL num-nas:11 (4.54545%) has-dict vocab-size:3 zero-ood-items most-frequent:"male" 120 (51.9481%)
    7: "__LABEL" CATEGORICAL integerized vocab-size:4 no-ood-item

Terminology:
    nas: Number of non-available (i.e. missing) values.
    ood: Out of dictionary.
    manually-defined: Attribute which type is manually defined by the user i.e. the type was not automatically inferred.
    tokenized: The attribute value is obtained through tokenization.
    has-dict: The attribute is attached to a string dictionary e.g. a categorical attribute stored as a string.
    vocab-size: Number of unique values.

[INFO kernel.cc:882] Configure learner
[INFO kernel.cc:912] Training config:
learner: "RANDOM_FOREST"
features: "bill_depth_mm"
features: "bill_length_mm"
features: "body_mass_g"
features: "flipper_length_mm"
features: "island"
features: "sex"
features: "year"
label: "__LABEL"
task: CLASSIFICATION
random_seed: 123456
metadata {
  framework: "TF Keras"
}
pure_serving_model: false
[yggdrasil_decision_forests.model.random_forest.proto.random_forest_config] {
  num_trees: 300
  decision_tree {
    max_depth: 16
    min_examples: 5
    in_split_min_examples_check: true
    keep_non_leaf_label_distribution: true
    num_candidate_attributes: 0
    missing_value_policy: GLOBAL_IMPUTATION
    allow_na_conditions: false
    categorical_set_greedy_forward {
      sampling: 0.1
      max_num_items: -1
      min_item_frequency: 1
    }
    growing_strategy_local {
    }
    categorical {
      cart {
      }
    }
    axis_aligned_split {
    }
    internal {
      sorting_strategy: PRESORTED
    }
    uplift {
      min_examples_in_treatment: 5
      split_score: KULLBACK_LEIBLER
    }
  }
  winner_take_all_inference: true
  compute_oob_performances: true
  compute_oob_variable_importances: false
  num_oob_variable_importances_permutations: 1
  bootstrap_training_dataset: true
  bootstrap_size_ratio: 1
  adapt_bootstrap_size_ratio_for_maximum_training_duration: false
  sampling_with_replacement: true
}

[INFO kernel.cc:915] Deployment config:
cache_path: "/tmpfs/tmp/tmp4ebv7hkj/working_cache"
num_threads: 32
try_resume_training: true

[INFO kernel.cc:944] Train model
[INFO random_forest.cc:407] Training random forest on 242 example(s) and 7 feature(s).
[INFO random_forest.cc:796] Training of tree  1/300 (tree index:3) done accuracy:0.956989 logloss:1.55026
[INFO random_forest.cc:796] Training of tree  11/300 (tree index:34) done accuracy:0.9625 logloss:0.343657
[INFO random_forest.cc:796] Training of tree  21/300 (tree index:9) done accuracy:0.966942 logloss:0.350169
[INFO random_forest.cc:796] Training of tree  40/300 (tree index:25) done accuracy:0.971074 logloss:0.348873
[INFO random_forest.cc:796] Training of tree  50/300 (tree index:49) done accuracy:0.966942 logloss:0.213431
[INFO random_forest.cc:796] Training of tree  61/300 (tree index:61) done accuracy:0.975207 logloss:0.217023
[INFO random_forest.cc:796] Training of tree  71/300 (tree index:70) done accuracy:0.983471 logloss:0.216868
[INFO random_forest.cc:796] Training of tree  82/300 (tree index:82) done accuracy:0.979339 logloss:0.220044
[INFO random_forest.cc:796] Training of tree  92/300 (tree index:90) done accuracy:0.975207 logloss:0.221188
[INFO random_forest.cc:796] Training of tree  102/300 (tree index:101) done accuracy:0.979339 logloss:0.0813704
[INFO random_forest.cc:796] Training of tree  112/300 (tree index:111) done accuracy:0.975207 logloss:0.0810937
[INFO random_forest.cc:796] Training of tree  127/300 (tree index:127) done accuracy:0.979339 logloss:0.0816566
[INFO random_forest.cc:796] Training of tree  139/300 (tree index:134) done accuracy:0.975207 logloss:0.078605
[INFO random_forest.cc:796] Training of tree  149/300 (tree index:149) done accuracy:0.979339 logloss:0.0794525
[INFO random_forest.cc:796] Training of tree  159/300 (tree index:158) done accuracy:0.983471 logloss:0.0790214
[INFO random_forest.cc:796] Training of tree  169/300 (tree index:169) done accuracy:0.983471 logloss:0.0795807
[INFO random_forest.cc:796] Training of tree  180/300 (tree index:177) done accuracy:0.979339 logloss:0.0796508
[INFO random_forest.cc:796] Training of tree  190/300 (tree index:190) done accuracy:0.983471 logloss:0.0774147
[INFO random_forest.cc:796] Training of tree  202/300 (tree index:202) done accuracy:0.979339 logloss:0.0782929
[INFO random_forest.cc:796] Training of tree  213/300 (tree index:214) done accuracy:0.979339 logloss:0.0779711
[INFO random_forest.cc:796] Training of tree  223/300 (tree index:220) done accuracy:0.979339 logloss:0.0776364
[INFO random_forest.cc:796] Training of tree  233/300 (tree index:232) done accuracy:0.983471 logloss:0.0774042
[INFO random_forest.cc:796] Training of tree  244/300 (tree index:243) done accuracy:0.983471 logloss:0.0775225
[INFO random_forest.cc:796] Training of tree  254/300 (tree index:254) done accuracy:0.983471 logloss:0.0777649
[INFO random_forest.cc:796] Training of tree  264/300 (tree index:263) done accuracy:0.983471 logloss:0.0774339
[INFO random_forest.cc:796] Training of tree  274/300 (tree index:271) done accuracy:0.983471 logloss:0.076491
[INFO random_forest.cc:796] Training of tree  284/300 (tree index:284) done accuracy:0.983471 logloss:0.076347
[INFO random_forest.cc:796] Training of tree  294/300 (tree index:292) done accuracy:0.983471 logloss:0.075455
[INFO random_forest.cc:796] Training of tree  300/300 (tree index:299) done accuracy:0.979339 logloss:0.0751547
[INFO random_forest.cc:876] Final OOB metrics: accuracy:0.979339 logloss:0.0751547
[INFO kernel.cc:961] Export model in log directory: /tmpfs/tmp/tmp4ebv7hkj with prefix 7861ae1e9c0e4ac9
[INFO kernel.cc:978] Save model in resources
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmp4ebv7hkj/model/ with prefix 7861ae1e9c0e4ac9
[INFO decision_forest.cc:639] Model loaded with 300 root(s), 4014 node(s), and 7 input feature(s).
[INFO abstract_model.cc:1248] Engine "RandomForestGeneric" built
[INFO kernel.cc:1022] Use fast generic engine
Model trained in 0:00:00.042860
Compiling model...
WARNING:tensorflow:AutoGraph could not transform <function simple_ml_inference_op_with_handle at 0x7fc32ed03280> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: could not get source code
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <function simple_ml_inference_op_with_handle at 0x7fc32ed03280> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: could not get source code
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <function simple_ml_inference_op_with_handle at 0x7fc32ed03280> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: could not get source code
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
Model compiled.
<keras.callbacks.History at 0x7fc32ed78b80>

Remarks

  • No input features are specified. Therefore, all the columns will be used as input features except for the label. The feature used by the model are shown in the training logs and in the model.summary().
  • DFs consume natively numerical, categorical, categorical-set features and missing-values. Numerical features do not need to be normalized. Categorical string values do not need to be encoded in a dictionary.
  • No training hyper-parameters are specified. Therefore the default hyper-parameters will be used. Default hyper-parameters provide reasonable results in most situations.
  • Calling compile on the model before the fit is optional. Compile can be used to provide extra evaluation metrics.
  • Training algorithms do not need validation datasets. If a validation dataset is provided, it will only be used to show metrics.
  • Tweak the verbose argument to RandomForestModel to control the amount of displayed training logs. Set verbose=0 to hide most of the logs. Set verbose=2 to show all the logs.

Evaluate the model

Let's evaluate our model on the test dataset.

model_1.compile(metrics=["accuracy"])
evaluation = model_1.evaluate(test_ds, return_dict=True)
print()

for name, value in evaluation.items():
  print(f"{name}: {value:.4f}")
1/1 [==============================] - 0s 380ms/step - loss: 0.0000e+00 - accuracy: 0.9510

loss: 0.0000
accuracy: 0.9510

Remark: The test accuracy is close to the Out-of-bag accuracy shown in the training logs.

See the Model Self Evaluation section below for more evaluation methods.

Prepare this model for TensorFlow Serving.

Export the model to the SavedModel format for later re-use e.g. TensorFlow Serving.

model_1.save("/tmp/my_saved_model")
WARNING:absl:Found untraced functions such as call_get_leaves while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /tmp/my_saved_model/assets
INFO:tensorflow:Assets written to: /tmp/my_saved_model/assets

Plot the model

Plotting a decision tree and following the first branches helps learning about decision forests. In some cases, plotting a model can even be used for debugging.

Because of the difference in the way they are trained, some models are more interesting to plan than others. Because of the noise injected during training and the depth of the trees, plotting Random Forest is less informative than plotting a CART or the first tree of a Gradient Boosted Tree.

Never the less, let's plot the first tree of our Random Forest model:

tfdf.model_plotter.plot_model_in_colab(model_1, tree_idx=0, max_depth=3)

The root node on the left contains the first condition (bill_depth_mm >= 16.55), number of examples (240) and label distribution (the red-blue-green bar).

Examples that evaluates true to bill_depth_mm >= 16.55 are branched to the green path. The other ones are branched to the red path.

The deeper the node, the more pure they become i.e. the label distribution is biased toward a subset of classes.

Model structure and feature importance

The overall structure of the model is show with .summary(). You will see:

  • Type: The learning algorithm used to train the model (Random Forest in our case).
  • Task: The problem solved by the model (Classification in our case).
  • Input Features: The input features of the model.
  • Variable Importance: Different measures of the importance of each feature for the model.
  • Out-of-bag evaluation: The out-of-bag evaluation of the model. This is a cheap and efficient alternative to cross-validation.
  • Number of {trees, nodes} and other metrics: Statistics about the structure of the decisions forests.

Remark: The summary's content depends on the learning algorithm (e.g. Out-of-bag is only available for Random Forest) and the hyper-parameters (e.g. the mean-decrease-in-accuracy variable importance can be disabled in the hyper-parameters).

%set_cell_height 300
model_1.summary()
<IPython.core.display.Javascript object>
Model: "random_forest_model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
=================================================================
Total params: 1
Trainable params: 0
Non-trainable params: 1
_________________________________________________________________
Type: "RANDOM_FOREST"
Task: CLASSIFICATION
Label: "__LABEL"

Input Features (7):
    bill_depth_mm
    bill_length_mm
    body_mass_g
    flipper_length_mm
    island
    sex
    year

No weights

Variable Importance: MEAN_MIN_DEPTH:

    1.           "__LABEL"  3.057408 ################
    2.              "year"  3.040498 ###############
    3.               "sex"  3.027410 ###############
    4.       "body_mass_g"  2.617562 ############
    5.     "bill_depth_mm"  2.354820 ##########
    6.            "island"  1.976905 #######
    7. "flipper_length_mm"  1.271189 #
    8.    "bill_length_mm"  1.133561 

Variable Importance: NUM_AS_ROOT:

    1. "flipper_length_mm" 135.000000 ################
    2.    "bill_length_mm" 123.000000 ##############
    3.     "bill_depth_mm" 23.000000 
    4.            "island" 19.000000 

Variable Importance: NUM_NODES:

    1.    "bill_length_mm" 579.000000 ################
    2. "flipper_length_mm" 373.000000 ##########
    3.     "bill_depth_mm" 340.000000 #########
    4.            "island" 294.000000 #######
    5.       "body_mass_g" 236.000000 ######
    6.               "sex" 21.000000 
    7.              "year" 14.000000 

Variable Importance: SUM_SCORE:

    1.    "bill_length_mm" 29164.326866 ################
    2. "flipper_length_mm" 21660.315810 ###########
    3.            "island" 13479.335545 #######
    4.     "bill_depth_mm" 6678.495304 ###
    5.       "body_mass_g" 2129.157265 #
    6.               "sex" 189.244600 
    7.              "year" 33.780576 



Winner take all: true
Out-of-bag evaluation: accuracy:0.979339 logloss:0.0751547
Number of trees: 300
Total number of nodes: 4014

Number of nodes by tree:
Count: 300 Average: 13.38 StdDev: 3.17106
Min: 7 Max: 25 Ignored: 0
----------------------------------------------
[  7,  8)  6   2.00%   2.00% #
[  8,  9)  0   0.00%   2.00%
[  9, 10) 29   9.67%  11.67% ####
[ 10, 11)  0   0.00%  11.67%
[ 11, 12) 71  23.67%  35.33% #########
[ 12, 13)  0   0.00%  35.33%
[ 13, 14) 81  27.00%  62.33% ##########
[ 14, 15)  0   0.00%  62.33%
[ 15, 16) 64  21.33%  83.67% ########
[ 16, 17)  0   0.00%  83.67%
[ 17, 18) 24   8.00%  91.67% ###
[ 18, 19)  0   0.00%  91.67%
[ 19, 20) 16   5.33%  97.00% ##
[ 20, 21)  0   0.00%  97.00%
[ 21, 22)  3   1.00%  98.00%
[ 22, 23)  0   0.00%  98.00%
[ 23, 24)  4   1.33%  99.33%
[ 24, 25)  0   0.00%  99.33%
[ 25, 25]  2   0.67% 100.00%

Depth by leafs:
Count: 2157 Average: 3.14604 StdDev: 0.993488
Min: 1 Max: 7 Ignored: 0
----------------------------------------------
[ 1, 2)  17   0.79%   0.79%
[ 2, 3) 609  28.23%  29.02% ########
[ 3, 4) 802  37.18%  66.20% ##########
[ 4, 5) 535  24.80%  91.01% #######
[ 5, 6) 165   7.65%  98.66% ##
[ 6, 7)  23   1.07%  99.72%
[ 7, 7]   6   0.28% 100.00%

Number of training obs by leaf:
Count: 2157 Average: 33.6579 StdDev: 33.4413
Min: 5 Max: 122 Ignored: 0
----------------------------------------------
[   5,  10) 1004  46.55%  46.55% ##########
[  10,  16)  102   4.73%  51.27% #
[  16,  22)   69   3.20%  54.47% #
[  22,  28)   62   2.87%  57.35% #
[  28,  34)   58   2.69%  60.04% #
[  34,  40)   78   3.62%  63.65% #
[  40,  46)   90   4.17%  67.83% #
[  46,  52)   66   3.06%  70.89% #
[  52,  58)   48   2.23%  73.11%
[  58,  64)   32   1.48%  74.59%
[  64,  69)   58   2.69%  77.28% #
[  69,  75)   72   3.34%  80.62% #
[  75,  81)  117   5.42%  86.05% #
[  81,  87)   82   3.80%  89.85% #
[  87,  93)   69   3.20%  93.05% #
[  93,  99)   46   2.13%  95.18%
[  99, 105)   61   2.83%  98.01% #
[ 105, 111)   28   1.30%  99.30%
[ 111, 117)   11   0.51%  99.81%
[ 117, 122]    4   0.19% 100.00%

Attribute in nodes:
    579 : bill_length_mm [NUMERICAL]
    373 : flipper_length_mm [NUMERICAL]
    340 : bill_depth_mm [NUMERICAL]
    294 : island [CATEGORICAL]
    236 : body_mass_g [NUMERICAL]
    21 : sex [CATEGORICAL]
    14 : year [NUMERICAL]

Attribute in nodes with depth <= 0:
    135 : flipper_length_mm [NUMERICAL]
    123 : bill_length_mm [NUMERICAL]
    23 : bill_depth_mm [NUMERICAL]
    19 : island [CATEGORICAL]

Attribute in nodes with depth <= 1:
    252 : bill_length_mm [NUMERICAL]
    229 : flipper_length_mm [NUMERICAL]
    189 : island [CATEGORICAL]
    161 : bill_depth_mm [NUMERICAL]
    52 : body_mass_g [NUMERICAL]

Attribute in nodes with depth <= 2:
    420 : bill_length_mm [NUMERICAL]
    330 : flipper_length_mm [NUMERICAL]
    268 : bill_depth_mm [NUMERICAL]
    265 : island [CATEGORICAL]
    150 : body_mass_g [NUMERICAL]
    6 : sex [CATEGORICAL]
    1 : year [NUMERICAL]

Attribute in nodes with depth <= 3:
    532 : bill_length_mm [NUMERICAL]
    361 : flipper_length_mm [NUMERICAL]
    324 : bill_depth_mm [NUMERICAL]
    289 : island [CATEGORICAL]
    217 : body_mass_g [NUMERICAL]
    20 : sex [CATEGORICAL]
    9 : year [NUMERICAL]

Attribute in nodes with depth <= 5:
    577 : bill_length_mm [NUMERICAL]
    373 : flipper_length_mm [NUMERICAL]
    339 : bill_depth_mm [NUMERICAL]
    294 : island [CATEGORICAL]
    236 : body_mass_g [NUMERICAL]
    21 : sex [CATEGORICAL]
    14 : year [NUMERICAL]

Condition type in nodes:
    1542 : HigherCondition
    315 : ContainsBitmapCondition
Condition type in nodes with depth <= 0:
    281 : HigherCondition
    19 : ContainsBitmapCondition
Condition type in nodes with depth <= 1:
    694 : HigherCondition
    189 : ContainsBitmapCondition
Condition type in nodes with depth <= 2:
    1169 : HigherCondition
    271 : ContainsBitmapCondition
Condition type in nodes with depth <= 3:
    1443 : HigherCondition
    309 : ContainsBitmapCondition
Condition type in nodes with depth <= 5:
    1539 : HigherCondition
    315 : ContainsBitmapCondition
Node format: NOT_SET

Training OOB:
    trees: 1, Out-of-bag evaluation: accuracy:0.956989 logloss:1.55026
    trees: 11, Out-of-bag evaluation: accuracy:0.9625 logloss:0.343657
    trees: 21, Out-of-bag evaluation: accuracy:0.966942 logloss:0.350169
    trees: 40, Out-of-bag evaluation: accuracy:0.971074 logloss:0.348873
    trees: 50, Out-of-bag evaluation: accuracy:0.966942 logloss:0.213431
    trees: 61, Out-of-bag evaluation: accuracy:0.975207 logloss:0.217023
    trees: 71, Out-of-bag evaluation: accuracy:0.983471 logloss:0.216868
    trees: 82, Out-of-bag evaluation: accuracy:0.979339 logloss:0.220044
    trees: 92, Out-of-bag evaluation: accuracy:0.975207 logloss:0.221188
    trees: 102, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0813704
    trees: 112, Out-of-bag evaluation: accuracy:0.975207 logloss:0.0810937
    trees: 127, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0816566
    trees: 139, Out-of-bag evaluation: accuracy:0.975207 logloss:0.078605
    trees: 149, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0794525
    trees: 159, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0790214
    trees: 169, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0795807
    trees: 180, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0796508
    trees: 190, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0774147
    trees: 202, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0782929
    trees: 213, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0779711
    trees: 223, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0776364
    trees: 233, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0774042
    trees: 244, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0775225
    trees: 254, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0777649
    trees: 264, Out-of-bag evaluation: accuracy:0.983471 logloss:0.0774339
    trees: 274, Out-of-bag evaluation: accuracy:0.983471 logloss:0.076491
    trees: 284, Out-of-bag evaluation: accuracy:0.983471 logloss:0.076347
    trees: 294, Out-of-bag evaluation: accuracy:0.983471 logloss:0.075455
    trees: 300, Out-of-bag evaluation: accuracy:0.979339 logloss:0.0751547

The information in summary are all available programatically using the model inspector:

# The input features
model_1.make_inspector().features()
["bill_depth_mm" (1; #0),
 "bill_length_mm" (1; #1),
 "body_mass_g" (1; #2),
 "flipper_length_mm" (1; #3),
 "island" (4; #4),
 "sex" (4; #5),
 "year" (1; #6)]
# The feature importances
model_1.make_inspector().variable_importances()
{'SUM_SCORE': [("bill_length_mm" (1; #1), 29164.326866285875),
  ("flipper_length_mm" (1; #3), 21660.31581015885),
  ("island" (4; #4), 13479.335544526577),
  ("bill_depth_mm" (1; #0), 6678.4953044727445),
  ("body_mass_g" (1; #2), 2129.157264975831),
  ("sex" (4; #5), 189.24459952116013),
  ("year" (1; #6), 33.780576441437006)],
 'NUM_AS_ROOT': [("flipper_length_mm" (1; #3), 135.0),
  ("bill_length_mm" (1; #1), 123.0),
  ("bill_depth_mm" (1; #0), 23.0),
  ("island" (4; #4), 19.0)],
 'NUM_NODES': [("bill_length_mm" (1; #1), 579.0),
  ("flipper_length_mm" (1; #3), 373.0),
  ("bill_depth_mm" (1; #0), 340.0),
  ("island" (4; #4), 294.0),
  ("body_mass_g" (1; #2), 236.0),
  ("sex" (4; #5), 21.0),
  ("year" (1; #6), 14.0)],
 'MEAN_MIN_DEPTH': [("__LABEL" (4; #7), 3.057407573907569),
  ("year" (1; #6), 3.0404981869981826),
  ("sex" (4; #5), 3.0274096181596137),
  ("body_mass_g" (1; #2), 2.6175624098124106),
  ("bill_depth_mm" (1; #0), 2.3548201243201254),
  ("island" (4; #4), 1.976905344655344),
  ("flipper_length_mm" (1; #3), 1.2711885521885529),
  ("bill_length_mm" (1; #1), 1.1335610593110603)]}

The content of the summary and the inspector depends on the learning algorithm (tfdf.keras.RandomForestModel in this case) and its hyper-parameters (e.g. compute_oob_variable_importances=True will trigger the computation of Out-of-bag variable importances for the Random Forest learner).

Model Self Evaluation

During training TFDF models can self evaluate even if no validation dataset is provided to the fit() method. The exact logic depends on the model. For example, Random Forest will use Out-of-bag evaluation while Gradient Boosted Trees will use internal train-validation.

The model self evaluation is available with the inspector's evaluation():

model_1.make_inspector().evaluation()
Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07515470768047944, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)

Plotting the training logs

The training logs show the quality of the model (e.g. accuracy evaluated on the out-of-bag or validation dataset) according to the number of trees in the model. These logs are helpful to study the balance between model size and model quality.

The logs are available in multiple ways:

  1. Displayed in during training if fit() is wrapped in with sys_pipes(): (see example above).
  2. At the end of the model summary i.e. model.summary() (see example above).
  3. Programmatically, using the model inspector i.e. model.make_inspector().training_logs().
  4. Using TensorBoard

Let's try the options 2 and 3:

%set_cell_height 150
model_1.make_inspector().training_logs()
<IPython.core.display.Javascript object>
[TrainLog(num_trees=1, evaluation=Evaluation(num_examples=93, accuracy=0.956989247311828, loss=1.5502645841208837, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=11, evaluation=Evaluation(num_examples=240, accuracy=0.9625, loss=0.3436569623028239, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=21, evaluation=Evaluation(num_examples=242, accuracy=0.9669421487603306, loss=0.35016938982423673, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=40, evaluation=Evaluation(num_examples=242, accuracy=0.9710743801652892, loss=0.3488732372556836, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=50, evaluation=Evaluation(num_examples=242, accuracy=0.9669421487603306, loss=0.21343091718298346, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=61, evaluation=Evaluation(num_examples=242, accuracy=0.9752066115702479, loss=0.2170233132148331, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=71, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.21686807902883892, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=82, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.22004372069288877, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=92, evaluation=Evaluation(num_examples=242, accuracy=0.9752066115702479, loss=0.22118825428495723, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=102, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.08137040281252673, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=112, evaluation=Evaluation(num_examples=242, accuracy=0.9752066115702479, loss=0.08109374480583698, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=127, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.0816565806446366, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=139, evaluation=Evaluation(num_examples=242, accuracy=0.9752066115702479, loss=0.07860502579888283, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=149, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07945252900312016, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=159, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07902135570679815, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=169, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.079580716257684, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=180, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.0796508065401769, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=190, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07741469333394746, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=202, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07829285810955545, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=213, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07797106273456915, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=223, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07763638938892602, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=233, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07740418245046099, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=244, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07752250102618016, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=254, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07776488542310463, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=264, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07743393323068654, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=274, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07649095226771944, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=284, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07634701544018693, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=294, evaluation=Evaluation(num_examples=242, accuracy=0.9834710743801653, loss=0.07545497547835112, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)),
 TrainLog(num_trees=300, evaluation=Evaluation(num_examples=242, accuracy=0.9793388429752066, loss=0.07515470768047944, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None))]

Let's plot it:

import matplotlib.pyplot as plt

logs = model_1.make_inspector().training_logs()

plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot([log.num_trees for log in logs], [log.evaluation.accuracy for log in logs])
plt.xlabel("Number of trees")
plt.ylabel("Accuracy (out-of-bag)")

plt.subplot(1, 2, 2)
plt.plot([log.num_trees for log in logs], [log.evaluation.loss for log in logs])
plt.xlabel("Number of trees")
plt.ylabel("Logloss (out-of-bag)")

plt.show()

png

This dataset is small. You can see the model converging almost immediately.

Let's use TensorBoard:

# This cell start TensorBoard that can be slow.
# Load the TensorBoard notebook extension
%load_ext tensorboard
# Google internal version
# %load_ext google3.learning.brain.tensorboard.notebook.extension
# Clear existing results (if any)
rm -fr "/tmp/tensorboard_logs"
# Export the meta-data to tensorboard.
model_1.make_inspector().export_to_tensorboard("/tmp/tensorboard_logs")
# docs_infra: no_execute
# Start a tensorboard instance.
%tensorboard --logdir "/tmp/tensorboard_logs"

Re-train the model with a different learning algorithm

The learning algorithm is defined by the model class. For example, tfdf.keras.RandomForestModel() trains a Random Forest, while tfdf.keras.GradientBoostedTreesModel() trains a Gradient Boosted Decision Trees.

The learning algorithms are listed by calling tfdf.keras.get_all_models() or in the learner list.

tfdf.keras.get_all_models()
[tensorflow_decision_forests.keras.RandomForestModel,
 tensorflow_decision_forests.keras.GradientBoostedTreesModel,
 tensorflow_decision_forests.keras.CartModel,
 tensorflow_decision_forests.keras.DistributedGradientBoostedTreesModel]

The description of the learning algorithms and their hyper-parameters are also available in the API reference and builtin help:

# help works anywhere.
help(tfdf.keras.RandomForestModel)

# ? only works in ipython or notebooks, it usually opens on a separate panel.
tfdf.keras.RandomForestModel?
Help on class RandomForestModel in module tensorflow_decision_forests.keras:

class RandomForestModel(tensorflow_decision_forests.keras.wrappers.RandomForestModel)
 |  RandomForestModel(*args, **kwargs)
 |  
 |  Method resolution order:
 |      RandomForestModel
 |      tensorflow_decision_forests.keras.wrappers.RandomForestModel
 |      tensorflow_decision_forests.keras.core.CoreModel
 |      keras.engine.training.Model
 |      keras.engine.base_layer.Layer
 |      tensorflow.python.module.module.Module
 |      tensorflow.python.training.tracking.autotrackable.AutoTrackable
 |      tensorflow.python.training.tracking.base.Trackable
 |      keras.utils.version_utils.LayerVersionSelector
 |      keras.utils.version_utils.ModelVersionSelector
 |      builtins.object
 |  
 |  Methods inherited from tensorflow_decision_forests.keras.wrappers.RandomForestModel:
 |  
 |  __init__ = wrapper(*args, **kargs)
 |  
 |  ----------------------------------------------------------------------
 |  Static methods inherited from tensorflow_decision_forests.keras.wrappers.RandomForestModel:
 |  
 |  capabilities() -> yggdrasil_decision_forests.learner.abstract_learner_pb2.LearnerCapabilities
 |      Lists the capabilities of the learning algorithm.
 |  
 |  predefined_hyperparameters() -> List[tensorflow_decision_forests.keras.core.HyperParameterTemplate]
 |      Returns a better than default set of hyper-parameters.
 |      
 |      They can be used directly with the `hyperparameter_template` argument of the
 |      model constructor.
 |      
 |      These hyper-parameters outperform the default hyper-parameters (either
 |      generally or in specific scenarios). Like default hyper-parameters, existing
 |      pre-defined hyper-parameters cannot change.
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from tensorflow_decision_forests.keras.core.CoreModel:
 |  
 |  call(self, inputs, training=False)
 |      Inference of the model.
 |      
 |      This method is used for prediction and evaluation of a trained model.
 |      
 |      Args:
 |        inputs: Input tensors.
 |        training: Is the model being trained. Always False.
 |      
 |      Returns:
 |        Model predictions.
 |  
 |  call_get_leaves(self, inputs)
 |      Computes the index of the active leaf in each tree.
 |      
 |      The active leaf is the leave that that receive the example during inference.
 |      
 |      The returned value "leaves[i,j]" is the index of the active leave for the
 |      i-th example and the j-th tree. Leaves are indexed by depth first
 |      exploration with the negative child visited before the positive one
 |      (similarly as "iterate_on_nodes()" iteration). Leaf indices are also
 |      available with LeafNode.leaf_idx.
 |      
 |      Args:
 |        inputs: Input tensors. Same signature as the model's "call(inputs)".
 |      
 |      Returns:
 |        Index of the active leaf for each tree in the model.
 |  
 |  collect_data_step(self, data, is_training_example)
 |      Collect examples e.g. training or validation.
 |  
 |  compile(self, metrics=None, weighted_metrics=None)
 |      Configure the model for training.
 |      
 |      Unlike for most Keras model, calling "compile" is optional before calling
 |      "fit".
 |      
 |      Args:
 |        metrics: List of metrics to be evaluated by the model during training and
 |          testing.
 |        weighted_metrics: List of metrics to be evaluated and weighted by
 |          `sample_weight` or `class_weight` during training and testing.
 |      
 |      Raises:
 |        ValueError: Invalid arguments.
 |  
 |  fit(self, x=None, y=None, callbacks=None, verbose: Optional[Any] = None, validation_steps: Optional[int] = None, validation_data: Optional[Any] = None, sample_weight: Optional[Any] = None, steps_per_epoch: Optional[Any] = None, class_weight: Optional[Any] = None, **kwargs) -> keras.callbacks.History
 |      Trains the model.
 |      
 |      Local training
 |      ==============
 |      
 |      It is recommended to use a Pandas Dataframe dataset and to convert it to
 |      a TensorFlow dataset with "pd_dataframe_to_tf_dataset()":
 |      
 |        pd_dataset = pandas.Dataframe(...)
 |        tf_dataset = pd_dataframe_to_tf_dataset(dataset, label="my_label")
 |        model.fit(pd_dataset)
 |      
 |      The following dataset formats are supported:
 |      
 |        1. "x" is a tf.data.Dataset containing a tuple "(features, labels)".
 |           "features" can be a dictionary a tensor, a list of tensors or a
 |           dictionary of tensors (recommended). "labels" is a tensor.
 |      
 |        2. "x" is a tensor, list of tensors or dictionary of tensors containing
 |           the input features. "y" is a tensor.
 |      
 |        3. "x" is a numpy-array, list of numpy-arrays or dictionary of
 |           numpy-arrays containing the input features. "y" is a numpy-array.
 |      
 |      IMPORTANT: This model trains on the entire dataset at once. This has the
 |      following consequences:
 |      
 |        1. The dataset need to be read exactly once. If you use a TensorFlow
 |           dataset, make sure NOT to add a "repeat" operation.
 |        2. The algorithm does not benefit from shuffling the dataset. If you use a
 |           TensorFlow dataset, make sure NOT to add a "shuffle" operation.
 |        3. The dataset needs to be batched (i.e. with a "batch" operation).
 |           However, the number of elements per batch has not impact on the model.
 |           Generally, it is recommended to use batches as large as possible as its
 |           speeds-up reading the dataset in TensorFlow.
 |      
 |      Input features do not need to be normalized (e.g. dividing numerical values
 |      by the variance) or indexed (e.g. replacing categorical string values by
 |      an integer). Additionnaly, missing values can be consumed natigely.
 |      
 |      Distributed training
 |      ====================
 |      
 |      Some of the learning algorithms will support distributed training with the
 |      ParameterServerStrategy.
 |      
 |      In this case, the dataset is read asynchronously in between the workers. The
 |      distribution of the training depends on the learning algorithm.
 |      
 |      Like for non-distributed training, the dataset should be read eactly once.
 |      The simplest solution is to divide the dataset into different files (i.e.
 |      shards) and have each of the worker read a non overlapping subset of shards.
 |      
 |      IMPORTANT: The training dataset should not be infinite i.e. the training
 |      dataset should not contain any repeat operation.
 |      
 |      Currently (to be changed), the validation dataset (if provided) is simply
 |      feed to the "model.evaluate()" method. Therefore, it should satify Keras'
 |      evaluate API. Notably, for distributed training, the validation dataset
 |      should be infinite (i.e. have a repeat operation).
 |      
 |      See https://www.tensorflow.org/decision_forests/distributed_training for
 |      more details and examples.
 |      
 |      Here is a single example of distributed training using PSS for both dataset
 |      reading and training distribution.
 |      
 |        def dataset_fn(context, paths, training=True):
 |          ds_path = tf.data.Dataset.from_tensor_slices(paths)
 |      
 |      
 |          if context is not None:
 |            # Train on at least 2 workers.
 |            current_worker = tfdf.keras.get_worker_idx_and_num_workers(context)
 |            assert current_worker.num_workers > 2
 |      
 |            # Split the dataset's examples among the workers.
 |            ds_path = ds_path.shard(
 |                num_shards=current_worker.num_workers,
 |                index=current_worker.worker_idx)
 |      
 |          def read_csv_file(path):
 |            numerical = tf.constant([math.nan], dtype=tf.float32)
 |            categorical_string = tf.constant([""], dtype=tf.string)
 |            csv_columns = [
 |                numerical,  # age
 |                categorical_string,  # workclass
 |                numerical,  # fnlwgt
 |                ...
 |            ]
 |            column_names = [
 |              "age", "workclass", "fnlwgt", ...
 |            ]
 |            label_name = "label"
 |            return tf.data.experimental.CsvDataset(path, csv_columns, header=True)
 |      
 |          ds_columns = ds_path.interleave(read_csv_file)
 |      
 |          def map_features(*columns):
 |            assert len(column_names) == len(columns)
 |            features = {column_names[i]: col for i, col in enumerate(columns)}
 |            label = label_table.lookup(features.pop(label_name))
 |            return features, label
 |      
 |          ds_dataset = ds_columns.map(map_features)
 |          if not training:
 |            dataset = dataset.repeat(None)
 |          ds_dataset = ds_dataset.batch(batch_size)
 |          return ds_dataset
 |      
 |        strategy = tf.distribute.experimental.ParameterServerStrategy(...)
 |        sharded_train_paths = [list of dataset files]
 |        with strategy.scope():
 |          model = DistributedGradientBoostedTreesModel()
 |          train_dataset = strategy.distribute_datasets_from_function(
 |            lambda context: dataset_fn(context, sharded_train_paths))
 |      
 |          test_dataset = strategy.distribute_datasets_from_function(
 |            lambda context: dataset_fn(context, sharded_test_paths))
 |      
 |        model.fit(sharded_train_paths)
 |        evaluation = model.evaluate(test_dataset, steps=num_test_examples //
 |          batch_size)
 |      
 |      Args:
 |        x: Training dataset (See details above for the supported formats).
 |        y: Label of the training dataset. Only used if "x" does not contains the
 |          labels.
 |        callbacks: Callbacks triggered during the training. The training runs in a
 |          single epoch, itself run in a single step. Therefore, callback logic can
 |          be called equivalently before/after the fit function.
 |        verbose: Verbosity mode. 0 = silent, 1 = small details, 2 = full details.
 |        validation_steps: Number of steps in the evaluation dataset when
 |          evaluating the trained model with model.evaluate(). If not specified,
 |          evaluates the model on the entire dataset (generally recommended; not
 |          yet supported for distributed datasets).
 |        validation_data: Validation dataset. If specified, the learner might use
 |          this dataset to help training e.g. early stopping.
 |        sample_weight: Training weights. Note: training weights can also be
 |          provided as the third output in a tf.data.Dataset e.g. (features, label,
 |          weights).
 |        steps_per_epoch: [Parameter will be removed] Number of training batch to
 |          load before training the model. Currently, only supported for
 |          distributed training.
 |        class_weight: For binary classification only. Mapping class indices
 |          (integers) to a weight (float) value. Only available for non-Distributed
 |          training. For maximum compatibility, feed example weights through the
 |          tf.data.Dataset or using the `weight` argument of
 |          pd_dataframe_to_tf_dataset.
 |        **kwargs: Extra arguments passed to the core keras model's fit. Note that
 |          not all keras' model fit arguments are supported.
 |      
 |      Returns:
 |        A `History` object. Its `History.history` attribute is not yet
 |        implemented for decision forests algorithms, and will return empty.
 |        All other fields are filled as usual for `Keras.Mode.fit()`.
 |  
 |  fit_on_dataset_path(self, train_path: str, label_key: str, weight_key: Optional[str] = None, ranking_key: Optional[str] = None, valid_path: Optional[str] = None, dataset_format: Optional[str] = 'csv', max_num_scanned_rows_to_accumulate_statistics: Optional[int] = 100000, try_resume_training: Optional[bool] = True, input_model_signature_fn: Optional[Callable[[tensorflow_decision_forests.component.inspector.inspector.AbstractInspector], Any]] = <function build_default_input_model_signature at 0x7fc32fdd5160>)
 |      Trains the model on a dataset stored on disk.
 |      
 |      This solution is generally more efficient and easier that loading the
 |      dataset with a tf.Dataset both for local and distributed training.
 |      
 |      Usage example:
 |      
 |        # Local training
 |        model = model = keras.GradientBoostedTreesModel()
 |        model.fit_on_dataset_path(
 |          train_path="/path/to/dataset.csv",
 |          label_key="label",
 |          dataset_format="csv")
 |        model.save("/model/path")
 |      
 |        # Distributed training
 |        with tf.distribute.experimental.ParameterServerStrategy(...).scope():
 |          model = model = keras.DistributedGradientBoostedTreesModel()
 |        model.fit_on_dataset_path(
 |          train_path="/path/to/dataset@10",
 |          label_key="label",
 |          dataset_format="tfrecord+tfe")
 |        model.save("/model/path")
 |      
 |      Args:
 |         train_path: Path to the training dataset. Support comma separated files,
 |           shard and glob notation.
 |         label_key: Name of the label column.
 |         weight_key: Name of the weighing column.
 |         ranking_key: Name of the ranking column.
 |         valid_path: Path to the validation dataset. If not provided, or if the
 |           learning algorithm does not support/need a validation dataset,
 |           `valid_path` is ignored.
 |         dataset_format: Format of the dataset. Should be one of the registered
 |           dataset format (see
 |           https://github.com/google/yggdrasil-decision-forests/blob/main/documentation/user_manual#dataset-path-and-format
 |             for more details). The format "csv" always available but it is
 |             generally only suited for small datasets.
 |        max_num_scanned_rows_to_accumulate_statistics: Maximum number of examples
 |          to scan to determine the statistics of the features (i.e. the dataspec,
 |          e.g. mean value, dictionaries). (Currently) the "first" examples of the
 |          dataset are scanned (e.g. the first examples of the dataset is a single
 |          file). Therefore, it is important that the sampled dataset is relatively
 |          uniformly sampled, notably the scanned examples should contains all the
 |          possible categorical values (otherwise the not seen value will be
 |          treated as out-of-vocabulary). If set to None, the entire dataset is
 |          scanned. This parameter has no effect if the dataset is stored in a
 |          format that already contains those values.
 |        try_resume_training: If true, tries to resume training from the model
 |          checkpoint stored in the `temp_directory` directory. If `temp_directory`
 |          does not contain any model checkpoint, start the training from the
 |          start. Works in the following three situations: (1) The training was
 |          interrupted by the user (e.g. ctrl+c). (2) the training job was
 |          interrupted (e.g. rescheduling), ond (3) the hyper-parameter of the
 |          model were changed such that an initially completed training is now
 |          incomplete (e.g. increasing the number of trees).
 |        input_model_signature_fn: A lambda that returns the
 |          (Dense,Sparse,Ragged)TensorSpec (or structure of TensorSpec e.g.
 |          dictionary, list) corresponding to input signature of the model. If not
 |          specified, the input model signature is created by
 |          "build_default_input_model_signature". For example, specify
 |          "input_model_signature_fn" if an numerical input feature (which is
 |          consumed as DenseTensorSpec(float32) by default) will be feed
 |          differently (e.g. RaggedTensor(int64)).
 |      
 |      Returns:
 |        A `History` object. Its `History.history` attribute is not yet
 |        implemented for decision forests algorithms, and will return empty.
 |        All other fields are filled as usual for `Keras.Mode.fit()`.
 |  
 |  load_weights(self, *args, **kwargs)
 |      No-op for TensorFlow Decision Forests models.
 |      
 |      `load_weights` is not supported by TensorFlow Decision Forests models.
 |      To save and restore a model, use the SavedModel API i.e.
 |      `model.save(...)` and `tf.keras.models.load_model(...)`. To resume the
 |      training of an existing model, create the model with
 |      `try_resume_training=True` (default value) and with a similar
 |      `temp_directory` argument. See documentation of `try_resume_training`
 |      for more details.
 |      
 |      Args:
 |        *args: Passed through to base `keras.Model` implemenation.
 |        **kwargs: Passed through to base `keras.Model` implemenation.
 |  
 |  make_inspector(self) -> tensorflow_decision_forests.component.inspector.inspector.AbstractInspector
 |      Creates an inspector to access the internal model structure.
 |      
 |      Usage example:
 |      
 |      ```python
 |      inspector = model.make_inspector()
 |      print(inspector.num_trees())
 |      print(inspector.variable_importances())
 |      ```
 |      
 |      Returns:
 |        A model inspector.
 |  
 |  make_predict_function(self)
 |      Prediction of the model (!= evaluation).
 |  
 |  make_test_function(self)
 |      Predictions for evaluation.
 |  
 |  predict_get_leaves(self, x)
 |      Gets the index of the active leaf of each tree.
 |      
 |      The active leaf is the leave that that receive the example during inference.
 |      
 |      The returned value "leaves[i,j]" is the index of the active leave for the
 |      i-th example and the j-th tree. Leaves are indexed by depth first
 |      exploration with the negative child visited before the positive one
 |      (similarly as "iterate_on_nodes()" iteration). Leaf indices are also
 |      available with LeafNode.leaf_idx.
 |      
 |      Args:
 |        x: Input samples as a tf.data.Dataset.
 |      
 |      Returns:
 |        Index of the active leaf for each tree in the model.
 |  
 |  save(self, filepath: str, overwrite: Optional[bool] = True, **kwargs)
 |      Saves the model as a TensorFlow SavedModel.
 |      
 |      The exported SavedModel contains a standalone Yggdrasil Decision Forests
 |      model in the "assets" sub-directory. The Yggdrasil model can be used
 |      directly using the Yggdrasil API. However, this model does not contain the
 |      "preprocessing" layer (if any).
 |      
 |      Args:
 |        filepath: Path to the output model.
 |        overwrite: If true, override an already existing model. If false, raise an
 |          error if a model already exist.
 |        **kwargs: Arguments passed to the core keras model's save.
 |  
 |  summary(self, line_length=None, positions=None, print_fn=None)
 |      Shows information about the model.
 |  
 |  train_step(self, data)
 |      Collects training examples.
 |  
 |  valid_step(self, data)
 |      Collects validation examples.
 |  
 |  yggdrasil_model_path_tensor(self) -> Optional[tensorflow.python.framework.ops.Tensor]
 |      Gets the path to yggdrasil model, if available.
 |      
 |      The effective path can be obtained with:
 |      
 |      ```python
 |      yggdrasil_model_path_tensor().numpy().decode("utf-8")
 |      ```
 |      
 |      Returns:
 |        Path to the Yggdrasil model.
 |  
 |  yggdrasil_model_prefix(self) -> str
 |      Gets the prefix of the internal yggdrasil model.
 |  
 |  ----------------------------------------------------------------------
 |  Readonly properties inherited from tensorflow_decision_forests.keras.core.CoreModel:
 |  
 |  exclude_non_specified_features
 |      If true, only use the features specified in "features".
 |  
 |  learner
 |      Name of the learning algorithm used to train the model.
 |  
 |  learner_params
 |      Gets the dictionary of hyper-parameters passed in the model constructor.
 |      
 |      Changing this dictionary will impact the training.
 |  
 |  num_threads
 |      Number of threads used to train the model.
 |  
 |  num_training_examples
 |      Number of training examples.
 |  
 |  num_validation_examples
 |      Number of validation examples.
 |  
 |  task
 |      Task to solve (e.g. CLASSIFICATION, REGRESSION, RANKING).
 |  
 |  training_model_id
 |      Identifier of the model.
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from keras.engine.training.Model:
 |  
 |  __call__(self, *args, **kwargs)
 |  
 |  __copy__(self)
 |  
 |  __deepcopy__(self, memo)
 |  
 |  __reduce__(self)
 |      Helper for pickle.
 |  
 |  __setattr__(self, name, value)
 |      Support self.foo = trackable syntax.
 |  
 |  build(self, input_shape)
 |      Builds the model based on input shapes received.
 |      
 |      This is to be used for subclassed models, which do not know at instantiation
 |      time what their inputs look like.
 |      
 |      This method only exists for users who want to call `model.build()` in a
 |      standalone way (as a substitute for calling the model on real data to
 |      build it). It will never be called by the framework (and thus it will
 |      never throw unexpected errors in an unrelated workflow).
 |      
 |      Args:
 |       input_shape: Single tuple, `TensorShape` instance, or list/dict of shapes,
 |         where shapes are tuples, integers, or `TensorShape` instances.
 |      
 |      Raises:
 |        ValueError:
 |          1. In case of invalid user-provided data (not of type tuple,
 |             list, `TensorShape`, or dict).
 |          2. If the model requires call arguments that are agnostic
 |             to the input shapes (positional or keyword arg in call signature).
 |          3. If not all layers were properly built.
 |          4. If float type inputs are not supported within the layers.
 |      
 |        In each of these cases, the user should build their model by calling it
 |        on real tensor data.
 |  
 |  compute_loss(self, x=None, y=None, y_pred=None, sample_weight=None)
 |      Compute the total loss, validate it, and return it.
 |      
 |      Subclasses can optionally override this method to provide custom loss
 |      computation logic.
 |      
 |      Example:
 |      ```python
 |      class MyModel(tf.keras.Model):
 |      
 |        def __init__(self, *args, **kwargs):
 |          super(MyModel, self).__init__(*args, **kwargs)
 |          self.loss_tracker = tf.keras.metrics.Mean(name='loss')
 |      
 |        def compute_loss(self, x, y, y_pred, sample_weight):
 |          loss = tf.reduce_mean(tf.math.squared_difference(y_pred, y))
 |          loss += tf.add_n(self.losses)
 |          self.loss_tracker.update_state(loss)
 |          return loss
 |      
 |        def reset_metrics(self):
 |          self.loss_tracker.reset_states()
 |      
 |        @property
 |        def metrics(self):
 |          return [self.loss_tracker]
 |      
 |      tensors = tf.random.uniform((10, 10)), tf.random.uniform((10,))
 |      dataset = tf.data.Dataset.from_tensor_slices(tensors).repeat().batch(1)
 |      
 |      inputs = tf.keras.layers.Input(shape=(10,), name='my_input')
 |      outputs = tf.keras.layers.Dense(10)(inputs)
 |      model = MyModel(inputs, outputs)
 |      model.add_loss(tf.reduce_sum(outputs))
 |      
 |      optimizer = tf.keras.optimizers.SGD()
 |      model.compile(optimizer, loss='mse', steps_per_execution=10)
 |      model.fit(dataset, epochs=2, steps_per_epoch=10)
 |      print('My custom loss: ', model.loss_tracker.result().numpy())
 |      ```
 |      
 |      Args:
 |        x: Input data.
 |        y: Target data.
 |        y_pred: Predictions returned by the model (output of `model(x)`)
 |        sample_weight: Sample weights for weighting the loss function.
 |      
 |      Returns:
 |        The total loss as a `tf.Tensor`, or `None` if no loss results (which is
 |        the case when called by `Model.test_step`).
 |  
 |  compute_metrics(self, x, y, y_pred, sample_weight)
 |      Update metric states and collect all metrics to be returned.
 |      
 |      Subclasses can optionally override this method to provide custom metric
 |      updating and collection logic.
 |      
 |      Example:
 |      ```python
 |      class MyModel(tf.keras.Sequential):
 |      
 |        def compute_metrics(self, x, y, y_pred, sample_weight):
 |      
 |          # This super call updates `self.compiled_metrics` and returns results
 |          # for all metrics listed in `self.metrics`.
 |          metric_results = super(MyModel, self).compute_metrics(
 |              x, y, y_pred, sample_weight)
 |      
 |          # Note that `self.custom_metric` is not listed in `self.metrics`.
 |          self.custom_metric.update_state(x, y, y_pred, sample_weight)
 |          metric_results['custom_metric_name'] = self.custom_metric.result()
 |          return metric_results
 |      ```
 |      
 |      Args:
 |        x: Input data.
 |        y: Target data.
 |        y_pred: Predictions returned by the model (output of `model.call(x)`)
 |        sample_weight: Sample weights for weighting the loss function.
 |      
 |      Returns:
 |        A `dict` containing values that will be passed to
 |        `tf.keras.callbacks.CallbackList.on_train_batch_end()`. Typically, the
 |        values of the metrics listed in `self.metrics` are returned. Example:
 |        `{'loss': 0.2, 'accuracy': 0.7}`.
 |  
 |  evaluate(self, x=None, y=None, batch_size=None, verbose='auto', sample_weight=None, steps=None, callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False, return_dict=False, **kwargs)
 |      Returns the loss value & metrics values for the model in test mode.
 |      
 |      Computation is done in batches (see the `batch_size` arg.)
 |      
 |      Args:
 |          x: Input data. It could be:
 |            - A Numpy array (or array-like), or a list of arrays
 |              (in case the model has multiple inputs).
 |            - A TensorFlow tensor, or a list of tensors
 |              (in case the model has multiple inputs).
 |            - A dict mapping input names to the corresponding array/tensors,
 |              if the model has named inputs.
 |            - A `tf.data` dataset. Should return a tuple
 |              of either `(inputs, targets)` or
 |              `(inputs, targets, sample_weights)`.
 |            - A generator or `keras.utils.Sequence` returning `(inputs, targets)`
 |              or `(inputs, targets, sample_weights)`.
 |            A more detailed description of unpacking behavior for iterator types
 |            (Dataset, generator, Sequence) is given in the `Unpacking behavior
 |            for iterator-like inputs` section of `Model.fit`.
 |          y: Target data. Like the input data `x`, it could be either Numpy
 |            array(s) or TensorFlow tensor(s). It should be consistent with `x`
 |            (you cannot have Numpy inputs and tensor targets, or inversely). If
 |            `x` is a dataset, generator or `keras.utils.Sequence` instance, `y`
 |            should not be specified (since targets will be obtained from the
 |            iterator/dataset).
 |          batch_size: Integer or `None`. Number of samples per batch of
 |            computation. If unspecified, `batch_size` will default to 32. Do not
 |            specify the `batch_size` if your data is in the form of a dataset,
 |            generators, or `keras.utils.Sequence` instances (since they generate
 |            batches).
 |          verbose: `"auto"`, 0, 1, or 2. Verbosity mode.
 |              0 = silent, 1 = progress bar, 2 = single line.
 |              `"auto"` defaults to 1 for most cases, and to 2 when used with
 |              `ParameterServerStrategy`. Note that the progress bar is not
 |              particularly useful when logged to a file, so `verbose=2` is
 |              recommended when not running interactively (e.g. in a production
 |              environment).
 |          sample_weight: Optional Numpy array of weights for the test samples,
 |            used for weighting the loss function. You can either pass a flat (1D)
 |            Numpy array with the same length as the input samples
 |              (1:1 mapping between weights and samples), or in the case of
 |                temporal data, you can pass a 2D array with shape `(samples,
 |                sequence_length)`, to apply a different weight to every timestep
 |                of every sample. This argument is not supported when `x` is a
 |                dataset, instead pass sample weights as the third element of `x`.
 |          steps: Integer or `None`. Total number of steps (batches of samples)
 |            before declaring the evaluation round finished. Ignored with the
 |            default value of `None`. If x is a `tf.data` dataset and `steps` is
 |            None, 'evaluate' will run until the dataset is exhausted. This
 |            argument is not supported with array inputs.
 |          callbacks: List of `keras.callbacks.Callback` instances. List of
 |            callbacks to apply during evaluation. See
 |            [callbacks](/api_docs/python/tf/keras/callbacks).
 |          max_queue_size: Integer. Used for generator or `keras.utils.Sequence`
 |            input only. Maximum size for the generator queue. If unspecified,
 |            `max_queue_size` will default to 10.
 |          workers: Integer. Used for generator or `keras.utils.Sequence` input
 |            only. Maximum number of processes to spin up when using process-based
 |            threading. If unspecified, `workers` will default to 1.
 |          use_multiprocessing: Boolean. Used for generator or
 |            `keras.utils.Sequence` input only. If `True`, use process-based
 |            threading. If unspecified, `use_multiprocessing` will default to
 |            `False`. Note that because this implementation relies on
 |            multiprocessing, you should not pass non-picklable arguments to the
 |            generator as they can't be passed easily to children processes.
 |          return_dict: If `True`, loss and metric results are returned as a dict,
 |            with each key being the name of the metric. If `False`, they are
 |            returned as a list.
 |          **kwargs: Unused at this time.
 |      
 |      See the discussion of `Unpacking behavior for iterator-like inputs` for
 |      `Model.fit`.
 |      
 |      Returns:
 |          Scalar test loss (if the model has a single output and no metrics)
 |          or list of scalars (if the model has multiple outputs
 |          and/or metrics). The attribute `model.metrics_names` will give you
 |          the display labels for the scalar outputs.
 |      
 |      Raises:
 |          RuntimeError: If `model.evaluate` is wrapped in a `tf.function`.
 |  
 |  evaluate_generator(self, generator, steps=None, callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False, verbose=0)
 |      Evaluates the model on a data generator.
 |      
 |      DEPRECATED:
 |        `Model.evaluate` now supports generators, so there is no longer any need
 |        to use this endpoint.
 |  
 |  fit_generator(self, generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, validation_data=None, validation_steps=None, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)
 |      Fits the model on data yielded batch-by-batch by a Python generator.
 |      
 |      DEPRECATED:
 |        `Model.fit` now supports generators, so there is no longer any need to use
 |        this endpoint.
 |  
 |  get_config(self)
 |      Returns the config of the `Model`.
 |      
 |      Config is a Python dictionary (serializable) containing the configuration of
 |      an object, which in this case is a `Model`. This allows the `Model` to be
 |      be reinstantiated later (without its trained weights) from this
 |      configuration.
 |      
 |      Note that `get_config()` does not guarantee to return a fresh copy of dict
 |      every time it is called. The callers should make a copy of the returned dict
 |      if they want to modify it.
 |      
 |      Developers of subclassed `Model` are advised to override this method, and
 |      continue to update the dict from `super(MyModel, self).get_config()`
 |      to provide the proper configuration of this `Model`. The default config
 |      is an empty dict. Optionally, raise `NotImplementedError` to allow Keras to
 |      attempt a default serialization.
 |      
 |      Returns:
 |          Python dictionary containing the configuration of this `Model`.
 |  
 |  get_layer(self, name=None, index=None)
 |      Retrieves a layer based on either its name (unique) or index.
 |      
 |      If `name` and `index` are both provided, `index` will take precedence.
 |      Indices are based on order of horizontal graph traversal (bottom-up).
 |      
 |      Args:
 |          name: String, name of layer.
 |          index: Integer, index of layer.
 |      
 |      Returns:
 |          A layer instance.
 |  
 |  get_weights(self)
 |      Retrieves the weights of the model.
 |      
 |      Returns:
 |          A flat list of Numpy arrays.
 |  
 |  make_train_function(self, force=False)
 |      Creates a function that executes one step of training.
 |      
 |      This method can be overridden to support custom training logic.
 |      This method is called by `Model.fit` and `Model.train_on_batch`.
 |      
 |      Typically, this method directly controls `tf.function` and
 |      `tf.distribute.Strategy` settings, and delegates the actual training
 |      logic to `Model.train_step`.
 |      
 |      This function is cached the first time `Model.fit` or
 |      `Model.train_on_batch` is called. The cache is cleared whenever
 |      `Model.compile` is called. You can skip the cache and generate again the
 |      function with `force=True`.
 |      
 |      Args:
 |        force: Whether to regenerate the train function and skip the cached
 |          function if available.
 |      
 |      Returns:
 |        Function. The function created by this method should accept a
 |        `tf.data.Iterator`, and return a `dict` containing values that will
 |        be passed to `tf.keras.Callbacks.on_train_batch_end`, such as
 |        `{'loss': 0.2, 'accuracy': 0.7}`.
 |  
 |  predict(self, x, batch_size=None, verbose='auto', steps=None, callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False)
 |      Generates output predictions for the input samples.
 |      
 |      Computation is done in batches. This method is designed for batch processing
 |      of large numbers of inputs. It is not intended for use inside of loops
 |      that iterate over your data and process small numbers of inputs at a time.
 |      
 |      For small numbers of inputs that fit in one batch,
 |      directly use `__call__()` for faster execution, e.g.,
 |      `model(x)`, or `model(x, training=False)` if you have layers such as
 |      `tf.keras.layers.BatchNormalization` that behave differently during
 |      inference. You may pair the individual model call with a `tf.function`
 |      for additional performance inside your inner loop.
 |      If you need access to numpy array values instead of tensors after your
 |      model call, you can use `tensor.numpy()` to get the numpy array value of
 |      an eager tensor.
 |      
 |      Also, note the fact that test loss is not affected by
 |      regularization layers like noise and dropout.
 |      
 |      Note: See [this FAQ entry](
 |      https://keras.io/getting_started/faq/#whats-the-difference-between-model-methods-predict-and-call)
 |      for more details about the difference between `Model` methods `predict()`
 |      and `__call__()`.
 |      
 |      Args:
 |          x: Input samples. It could be:
 |            - A Numpy array (or array-like), or a list of arrays
 |              (in case the model has multiple inputs).
 |            - A TensorFlow tensor, or a list of tensors
 |              (in case the model has multiple inputs).
 |            - A `tf.data` dataset.
 |            - A generator or `keras.utils.Sequence` instance.
 |            A more detailed description of unpacking behavior for iterator types
 |            (Dataset, generator, Sequence) is given in the `Unpacking behavior
 |            for iterator-like inputs` section of `Model.fit`.
 |          batch_size: Integer or `None`.
 |              Number of samples per batch.
 |              If unspecified, `batch_size` will default to 32.
 |              Do not specify the `batch_size` if your data is in the
 |              form of dataset, generators, or `keras.utils.Sequence` instances
 |              (since they generate batches).
 |          verbose: `"auto"`, 0, 1, or 2. Verbosity mode.
 |              0 = silent, 1 = progress bar, 2 = single line.
 |              `"auto"` defaults to 1 for most cases, and to 2 when used with
 |              `ParameterServerStrategy`. Note that the progress bar is not
 |              particularly useful when logged to a file, so `verbose=2` is
 |              recommended when not running interactively (e.g. in a production
 |              environment).
 |          steps: Total number of steps (batches of samples)
 |              before declaring the prediction round finished.
 |              Ignored with the default value of `None`. If x is a `tf.data`
 |              dataset and `steps` is None, `predict()` will
 |              run until the input dataset is exhausted.
 |          callbacks: List of `keras.callbacks.Callback` instances.
 |              List of callbacks to apply during prediction.
 |              See [callbacks](/api_docs/python/tf/keras/callbacks).
 |          max_queue_size: Integer. Used for generator or `keras.utils.Sequence`
 |              input only. Maximum size for the generator queue.
 |              If unspecified, `max_queue_size` will default to 10.
 |          workers: Integer. Used for generator or `keras.utils.Sequence` input
 |              only. Maximum number of processes to spin up when using
 |              process-based threading. If unspecified, `workers` will default
 |              to 1.
 |          use_multiprocessing: Boolean. Used for generator or
 |              `keras.utils.Sequence` input only. If `True`, use process-based
 |              threading. If unspecified, `use_multiprocessing` will default to
 |              `False`. Note that because this implementation relies on
 |              multiprocessing, you should not pass non-picklable arguments to
 |              the generator as they can't be passed easily to children processes.
 |      
 |      See the discussion of `Unpacking behavior for iterator-like inputs` for
 |      `Model.fit`. Note that Model.predict uses the same interpretation rules as
 |      `Model.fit` and `Model.evaluate`, so inputs must be unambiguous for all
 |      three methods.
 |      
 |      Returns:
 |          Numpy array(s) of predictions.
 |      
 |      Raises:
 |          RuntimeError: If `model.predict` is wrapped in a `tf.function`.
 |          ValueError: In case of mismatch between the provided
 |              input data and the model's expectations,
 |              or in case a stateful model receives a number of samples
 |              that is not a multiple of the batch size.
 |  
 |  predict_generator(self, generator, steps=None, callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False, verbose=0)
 |      Generates predictions for the input samples from a data generator.
 |      
 |      DEPRECATED:
 |        `Model.predict` now supports generators, so there is no longer any need
 |        to use this endpoint.
 |  
 |  predict_on_batch(self, x)
 |      Returns predictions for a single batch of samples.
 |      
 |      Args:
 |          x: Input data. It could be:
 |            - A Numpy array (or array-like), or a list of arrays (in case the
 |                model has multiple inputs).
 |            - A TensorFlow tensor, or a list of tensors (in case the model has
 |                multiple inputs).
 |      
 |      Returns:
 |          Numpy array(s) of predictions.
 |      
 |      Raises:
 |          RuntimeError: If `model.predict_on_batch` is wrapped in a `tf.function`.
 |  
 |  predict_step(self, data)
 |      The logic for one inference step.
 |      
 |      This method can be overridden to support custom inference logic.
 |      This method is called by `Model.make_predict_function`.
 |      
 |      This method should contain the mathematical logic for one step of inference.
 |      This typically includes the forward pass.
 |      
 |      Configuration details for *how* this logic is run (e.g. `tf.function` and
 |      `tf.distribute.Strategy` settings), should be left to
 |      `Model.make_predict_function`, which can also be overridden.
 |      
 |      Args:
 |        data: A nested structure of `Tensor`s.
 |      
 |      Returns:
 |        The result of one inference step, typically the output of calling the
 |        `Model` on data.
 |  
 |  reset_metrics(self)
 |      Resets the state of all the metrics in the model.
 |      
 |      Examples:
 |      
 |      >>> inputs = tf.keras.layers.Input(shape=(3,))
 |      >>> outputs = tf.keras.layers.Dense(2)(inputs)
 |      >>> model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
 |      >>> model.compile(optimizer="Adam", loss="mse", metrics=["mae"])
 |      
 |      >>> x = np.random.random((2, 3))
 |      >>> y = np.random.randint(0, 2, (2, 2))
 |      >>> _ = model.fit(x, y, verbose=0)
 |      >>> assert all(float(m.result()) for m in model.metrics)
 |      
 |      >>> model.reset_metrics()
 |      >>> assert all(float(m.result()) == 0 for m in model.metrics)
 |  
 |  reset_states(self)
 |  
 |  save_spec(self, dynamic_batch=True)
 |      Returns the `tf.TensorSpec` of call inputs as a tuple `(args, kwargs)`.
 |      
 |      This value is automatically defined after calling the model for the first
 |      time. Afterwards, you can use it when exporting the model for serving:
 |      
 |      ```python
 |      model = tf.keras.Model(...)
 |      
 |      @tf.function
 |      def serve(*args, **kwargs):
 |        outputs = model(*args, **kwargs)
 |        # Apply postprocessing steps, or add additional outputs.
 |        ...
 |        return outputs
 |      
 |      # arg_specs is `[tf.TensorSpec(...), ...]`. kwarg_specs, in this example, is
 |      # an empty dict since functional models do not use keyword arguments.
 |      arg_specs, kwarg_specs = model.save_spec()
 |      
 |      model.save(path, signatures={
 |        'serving_default': serve.get_concrete_function(*arg_specs, **kwarg_specs)
 |      })
 |      ```
 |      
 |      Args:
 |        dynamic_batch: Whether to set the batch sizes of all the returned
 |          `tf.TensorSpec` to `None`. (Note that when defining functional or
 |          Sequential models with `tf.keras.Input([...], batch_size=X)`, the
 |          batch size will always be preserved). Defaults to `True`.
 |      Returns:
 |        If the model inputs are defined, returns a tuple `(args, kwargs)`. All
 |        elements in `args` and `kwargs` are `tf.TensorSpec`.
 |        If the model inputs are not defined, returns `None`.
 |        The model inputs are automatically set when calling the model,
 |        `model.fit`, `model.evaluate` or `model.predict`.
 |  
 |  save_weights(self, filepath, overwrite=True, save_format=None, options=None)
 |      Saves all layer weights.
 |      
 |      Either saves in HDF5 or in TensorFlow format based on the `save_format`
 |      argument.
 |      
 |      When saving in HDF5 format, the weight file has:
 |        - `layer_names` (attribute), a list of strings
 |            (ordered names of model layers).
 |        - For every layer, a `group` named `layer.name`
 |            - For every such layer group, a group attribute `weight_names`,
 |                a list of strings
 |                (ordered names of weights tensor of the layer).
 |            - For every weight in the layer, a dataset
 |                storing the weight value, named after the weight tensor.
 |      
 |      When saving in TensorFlow format, all objects referenced by the network are
 |      saved in the same format as `tf.train.Checkpoint`, including any `Layer`
 |      instances or `Optimizer` instances assigned to object attributes. For
 |      networks constructed from inputs and outputs using `tf.keras.Model(inputs,
 |      outputs)`, `Layer` instances used by the network are tracked/saved
 |      automatically. For user-defined classes which inherit from `tf.keras.Model`,
 |      `Layer` instances must be assigned to object attributes, typically in the
 |      constructor. See the documentation of `tf.train.Checkpoint` and
 |      `tf.keras.Model` for details.
 |      
 |      While the formats are the same, do not mix `save_weights` and
 |      `tf.train.Checkpoint`. Checkpoints saved by `Model.save_weights` should be
 |      loaded using `Model.load_weights`. Checkpoints saved using
 |      `tf.train.Checkpoint.save` should be restored using the corresponding
 |      `tf.train.Checkpoint.restore`. Prefer `tf.train.Checkpoint` over
 |      `save_weights` for training checkpoints.
 |      
 |      The TensorFlow format matches objects and variables by starting at a root
 |      object, `self` for `save_weights`, and greedily matching attribute
 |      names. For `Model.save` this is the `Model`, and for `Checkpoint.save` this
 |      is the `Checkpoint` even if the `Checkpoint` has a model attached. This
 |      means saving a `tf.keras.Model` using `save_weights` and loading into a
 |      `tf.train.Checkpoint` with a `Model` attached (or vice versa) will not match
 |      the `Model`'s variables. See the
 |      [guide to training checkpoints](https://www.tensorflow.org/guide/checkpoint)
 |      for details on the TensorFlow format.
 |      
 |      Args:
 |          filepath: String or PathLike, path to the file to save the weights to.
 |              When saving in TensorFlow format, this is the prefix used for
 |              checkpoint files (multiple files are generated). Note that the '.h5'
 |              suffix causes weights to be saved in HDF5 format.
 |          overwrite: Whether to silently overwrite any existing file at the
 |              target location, or provide the user with a manual prompt.
 |          save_format: Either 'tf' or 'h5'. A `filepath` ending in '.h5' or
 |              '.keras' will default to HDF5 if `save_format` is `None`. Otherwise
 |              `None` defaults to 'tf'.
 |          options: Optional `tf.train.CheckpointOptions` object that specifies
 |              options for saving weights.
 |      
 |      Raises:
 |          ImportError: If `h5py` is not available when attempting to save in HDF5
 |              format.
 |  
 |  test_on_batch(self, x, y=None, sample_weight=None, reset_metrics=True, return_dict=False)
 |      Test the model on a single batch of samples.
 |      
 |      Args:
 |          x: Input data. It could be:
 |            - A Numpy array (or array-like), or a list of arrays (in case the
 |                model has multiple inputs).
 |            - A TensorFlow tensor, or a list of tensors (in case the model has
 |                multiple inputs).
 |            - A dict mapping input names to the corresponding array/tensors, if
 |                the model has named inputs.
 |          y: Target data. Like the input data `x`, it could be either Numpy
 |            array(s) or TensorFlow tensor(s). It should be consistent with `x`
 |            (you cannot have Numpy inputs and tensor targets, or inversely).
 |          sample_weight: Optional array of the same length as x, containing
 |            weights to apply to the model's loss for each sample. In the case of
 |            temporal data, you can pass a 2D array with shape (samples,
 |            sequence_length), to apply a different weight to every timestep of
 |            every sample.
 |          reset_metrics: If `True`, the metrics returned will be only for this
 |            batch. If `False`, the metrics will be statefully accumulated across
 |            batches.
 |          return_dict: If `True`, loss and metric results are returned as a dict,
 |            with each key being the name of the metric. If `False`, they are
 |            returned as a list.
 |      
 |      Returns:
 |          Scalar test loss (if the model has a single output and no metrics)
 |          or list of scalars (if the model has multiple outputs
 |          and/or metrics). The attribute `model.metrics_names` will give you
 |          the display labels for the scalar outputs.
 |      
 |      Raises:
 |          RuntimeError: If `model.test_on_batch` is wrapped in a `tf.function`.
 |  
 |  test_step(self, data)
 |      The logic for one evaluation step.
 |      
 |      This method can be overridden to support custom evaluation logic.
 |      This method is called by `Model.make_test_function`.
 |      
 |      This function should contain the mathematical logic for one step of
 |      evaluation.
 |      This typically includes the forward pass, loss calculation, and metrics
 |      updates.
 |      
 |      Configuration details for *how* this logic is run (e.g. `tf.function` and
 |      `tf.distribute.Strategy` settings), should be left to
 |      `Model.make_test_function`, which can also be overridden.
 |      
 |      Args:
 |        data: A nested structure of `Tensor`s.
 |      
 |      Returns:
 |        A `dict` containing values that will be passed to
 |        `tf.keras.callbacks.CallbackList.on_train_batch_end`. Typically, the
 |        values of the `Model`'s metrics are returned.
 |  
 |  to_json(self, **kwargs)
 |      Returns a JSON string containing the network configuration.
 |      
 |      To load a network from a JSON save file, use
 |      `keras.models.model_from_json(json_string, custom_objects={})`.
 |      
 |      Args:
 |          **kwargs: Additional keyword arguments
 |              to be passed to `json.dumps()`.
 |      
 |      Returns:
 |          A JSON string.
 |  
 |  to_yaml(self, **kwargs)
 |      Returns a yaml string containing the network configuration.
 |      
 |      Note: Since TF 2.6, this method is no longer supported and will raise a
 |      RuntimeError.
 |      
 |      To load a network from a yaml save file, use
 |      `keras.models.model_from_yaml(yaml_string, custom_objects={})`.
 |      
 |      `custom_objects` should be a dictionary mapping
 |      the names of custom losses / layers / etc to the corresponding
 |      functions / classes.
 |      
 |      Args:
 |          **kwargs: Additional keyword arguments
 |              to be passed to `yaml.dump()`.
 |      
 |      Returns:
 |          A YAML string.
 |      
 |      Raises:
 |          RuntimeError: announces that the method poses a security risk
 |  
 |  train_on_batch(self, x, y=None, sample_weight=None, class_weight=None, reset_metrics=True, return_dict=False)
 |      Runs a single gradient update on a single batch of data.
 |      
 |      Args:
 |          x: Input data. It could be:
 |            - A Numpy array (or array-like), or a list of arrays
 |                (in case the model has multiple inputs).
 |            - A TensorFlow tensor, or a list of tensors
 |                (in case the model has multiple inputs).
 |            - A dict mapping input names to the corresponding array/tensors,
 |                if the model has named inputs.
 |          y: Target data. Like the input data `x`, it could be either Numpy
 |            array(s) or TensorFlow tensor(s).
 |          sample_weight: Optional array of the same length as x, containing
 |            weights to apply to the model's loss for each sample. In the case of
 |            temporal data, you can pass a 2D array with shape (samples,
 |            sequence_length), to apply a different weight to every timestep of
 |            every sample.
 |          class_weight: Optional dictionary mapping class indices (integers) to a
 |            weight (float) to apply to the model's loss for the samples from this
 |            class during training. This can be useful to tell the model to "pay
 |            more attention" to samples from an under-represented class.
 |          reset_metrics: If `True`, the metrics returned will be only for this
 |            batch. If `False`, the metrics will be statefully accumulated across
 |            batches.
 |          return_dict: If `True`, loss and metric results are returned as a dict,
 |            with each key being the name of the metric. If `False`, they are
 |            returned as a list.
 |      
 |      Returns:
 |          Scalar training loss
 |          (if the model has a single output and no metrics)
 |          or list of scalars (if the model has multiple outputs
 |          and/or metrics). The attribute `model.metrics_names` will give you
 |          the display labels for the scalar outputs.
 |      
 |      Raises:
 |        RuntimeError: If `model.train_on_batch` is wrapped in a `tf.function`.
 |  
 |  ----------------------------------------------------------------------
 |  Class methods inherited from keras.engine.training.Model:
 |  
 |  from_config(config, custom_objects=None) from builtins.type
 |      Creates a layer from its config.
 |      
 |      This method is the reverse of `get_config`,
 |      capable of instantiating the same layer from the config
 |      dictionary. It does not handle layer connectivity
 |      (handled by Network), nor weights (handled by `set_weights`).
 |      
 |      Args:
 |          config: A Python dictionary, typically the
 |              output of get_config.
 |      
 |      Returns:
 |          A layer instance.
 |  
 |  ----------------------------------------------------------------------
 |  Static methods inherited from keras.engine.training.Model:
 |  
 |  __new__(cls, *args, **kwargs)
 |      Create and return a new object.  See help(type) for accurate signature.
 |  
 |  ----------------------------------------------------------------------
 |  Readonly properties inherited from keras.engine.training.Model:
 |  
 |  distribute_strategy
 |      The `tf.distribute.Strategy` this model was created under.
 |  
 |  metrics
 |      Returns the model's metrics added using `compile()`, `add_metric()` APIs.
 |      
 |      Note: Metrics passed to `compile()` are available only after a `keras.Model`
 |      has been trained/evaluated on actual data.
 |      
 |      Examples:
 |      
 |      >>> inputs = tf.keras.layers.Input(shape=(3,))
 |      >>> outputs = tf.keras.layers.Dense(2)(inputs)
 |      >>> model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
 |      >>> model.compile(optimizer="Adam", loss="mse", metrics=["mae"])
 |      >>> [m.name for m in model.metrics]
 |      []
 |      
 |      >>> x = np.random.random((2, 3))
 |      >>> y = np.random.randint(0, 2, (2, 2))
 |      >>> model.fit(x, y)
 |      >>> [m.name for m in model.metrics]
 |      ['loss', 'mae']
 |      
 |      >>> inputs = tf.keras.layers.Input(shape=(3,))
 |      >>> d = tf.keras.layers.Dense(2, name='out')
 |      >>> output_1 = d(inputs)
 |      >>> output_2 = d(inputs)
 |      >>> model = tf.keras.models.Model(
 |      ...    inputs=inputs, outputs=[output_1, output_2])
 |      >>> model.add_metric(
 |      ...    tf.reduce_sum(output_2), name='mean', aggregation='mean')
 |      >>> model.compile(optimizer="Adam", loss="mse", metrics=["mae", "acc"])
 |      >>> model.fit(x, (y, y))
 |      >>> [m.name for m in model.metrics]
 |      ['loss', 'out_loss', 'out_1_loss', 'out_mae', 'out_acc', 'out_1_mae',
 |      'out_1_acc', 'mean']
 |  
 |  metrics_names
 |      Returns the model's display labels for all outputs.
 |      
 |      Note: `metrics_names` are available only after a `keras.Model` has been
 |      trained/evaluated on actual data.
 |      
 |      Examples:
 |      
 |      >>> inputs = tf.keras.layers.Input(shape=(3,))
 |      >>> outputs = tf.keras.layers.Dense(2)(inputs)
 |      >>> model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
 |      >>> model.compile(optimizer="Adam", loss="mse", metrics=["mae"])
 |      >>> model.metrics_names
 |      []
 |      
 |      >>> x = np.random.random((2, 3))
 |      >>> y = np.random.randint(0, 2, (2, 2))
 |      >>> model.fit(x, y)
 |      >>> model.metrics_names
 |      ['loss', 'mae']
 |      
 |      >>> inputs = tf.keras.layers.Input(shape=(3,))
 |      >>> d = tf.keras.layers.Dense(2, name='out')
 |      >>> output_1 = d(inputs)
 |      >>> output_2 = d(inputs)
 |      >>> model = tf.keras.models.Model(
 |      ...    inputs=inputs, outputs=[output_1, output_2])
 |      >>> model.compile(optimizer="Adam", loss="mse", metrics=["mae", "acc"])
 |      >>> model.fit(x, (y, y))
 |      >>> model.metrics_names
 |      ['loss', 'out_loss', 'out_1_loss', 'out_mae', 'out_acc', 'out_1_mae',
 |      'out_1_acc']
 |  
 |  non_trainable_weights
 |      List of all non-trainable weights tracked by this layer.
 |      
 |      Non-trainable weights are *not* updated during training. They are expected
 |      to be updated manually in `call()`.
 |      
 |      Returns:
 |        A list of non-trainable variables.
 |  
 |  state_updates
 |      Deprecated, do NOT use!
 |      
 |      Returns the `updates` from all layers that are stateful.
 |      
 |      This is useful for separating training updates and
 |      state updates, e.g. when we need to update a layer's internal state
 |      during prediction.
 |      
 |      Returns:
 |          A list of update ops.
 |  
 |  trainable_weights
 |      List of all trainable weights tracked by this layer.
 |      
 |      Trainable weights are updated via gradient descent during training.
 |      
 |      Returns:
 |        A list of trainable variables.
 |  
 |  weights
 |      Returns the list of all layer variables/weights.
 |      
 |      Note: This will not track the weights of nested `tf.Modules` that are not
 |      themselves Keras layers.
 |      
 |      Returns:
 |        A list of variables.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from keras.engine.training.Model:
 |  
 |  layers
 |  
 |  run_eagerly
 |      Settable attribute indicating whether the model should run eagerly.
 |      
 |      Running eagerly means that your model will be run step by step,
 |      like Python code. Your model might run slower, but it should become easier
 |      for you to debug it by stepping into individual layer calls.
 |      
 |      By default, we will attempt to compile your model to a static graph to
 |      deliver the best execution performance.
 |      
 |      Returns:
 |        Boolean, whether the model should run eagerly.
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from keras.engine.base_layer.Layer:
 |  
 |  __delattr__(self, name)
 |      Implement delattr(self, name).
 |  
 |  __getstate__(self)
 |  
 |  __setstate__(self, state)
 |  
 |  add_loss(self, losses, **kwargs)
 |      Add loss tensor(s), potentially dependent on layer inputs.
 |      
 |      Some losses (for instance, activity regularization losses) may be dependent
 |      on the inputs passed when calling a layer. Hence, when reusing the same
 |      layer on different inputs `a` and `b`, some entries in `layer.losses` may
 |      be dependent on `a` and some on `b`. This method automatically keeps track
 |      of dependencies.
 |      
 |      This method can be used inside a subclassed layer or model's `call`
 |      function, in which case `losses` should be a Tensor or list of Tensors.
 |      
 |      Example:
 |      
 |      ```python
 |      class MyLayer(tf.keras.layers.Layer):
 |        def call(self, inputs):
 |          self.add_loss(tf.abs(tf.reduce_mean(inputs)))
 |          return inputs
 |      ```
 |      
 |      This method can also be called directly on a Functional Model during
 |      construction. In this case, any loss Tensors passed to this Model must
 |      be symbolic and be able to be traced back to the model's `Input`s. These
 |      losses become part of the model's topology and are tracked in `get_config`.
 |      
 |      Example:
 |      
 |      ```python
 |      inputs = tf.keras.Input(shape=(10,))
 |      x = tf.keras.layers.Dense(10)(inputs)
 |      outputs = tf.keras.layers.Dense(1)(x)
 |      model = tf.keras.Model(inputs, outputs)
 |      # Activity regularization.
 |      model.add_loss(tf.abs(tf.reduce_mean(x)))
 |      ```
 |      
 |      If this is not the case for your loss (if, for example, your loss references
 |      a `Variable` of one of the model's layers), you can wrap your loss in a
 |      zero-argument lambda. These losses are not tracked as part of the model's
 |      topology since they can't be serialized.
 |      
 |      Example:
 |      
 |      ```python
 |      inputs = tf.keras.Input(shape=(10,))
 |      d = tf.keras.layers.Dense(10)
 |      x = d(inputs)
 |      outputs = tf.keras.layers.Dense(1)(x)
 |      model = tf.keras.Model(inputs, outputs)
 |      # Weight regularization.
 |      model.add_loss(lambda: tf.reduce_mean(d.kernel))
 |      ```
 |      
 |      Args:
 |        losses: Loss tensor, or list/tuple of tensors. Rather than tensors, losses
 |          may also be zero-argument callables which create a loss tensor.
 |        **kwargs: Used for backwards compatibility only.
 |  
 |  add_metric(self, value, name=None, **kwargs)
 |      Adds metric tensor to the layer.
 |      
 |      This method can be used inside the `call()` method of a subclassed layer
 |      or model.
 |      
 |      ```python
 |      class MyMetricLayer(tf.keras.layers.Layer):
 |        def __init__(self):
 |          super(MyMetricLayer, self).__init__(name='my_metric_layer')
 |          self.mean = tf.keras.metrics.Mean(name='metric_1')
 |      
 |        def call(self, inputs):
 |          self.add_metric(self.mean(inputs))
 |          self.add_metric(tf.reduce_sum(inputs), name='metric_2')
 |          return inputs
 |      ```
 |      
 |      This method can also be called directly on a Functional Model during
 |      construction. In this case, any tensor passed to this Model must
 |      be symbolic and be able to be traced back to the model's `Input`s. These
 |      metrics become part of the model's topology and are tracked when you
 |      save the model via `save()`.
 |      
 |      ```python
 |      inputs = tf.keras.Input(shape=(10,))
 |      x = tf.keras.layers.Dense(10)(inputs)
 |      outputs = tf.keras.layers.Dense(1)(x)
 |      model = tf.keras.Model(inputs, outputs)
 |      model.add_metric(math_ops.reduce_sum(x), name='metric_1')
 |      ```
 |      
 |      Note: Calling `add_metric()` with the result of a metric object on a
 |      Functional Model, as shown in the example below, is not supported. This is
 |      because we cannot trace the metric result tensor back to the model's inputs.
 |      
 |      ```python
 |      inputs = tf.keras.Input(shape=(10,))
 |      x = tf.keras.layers.Dense(10)(inputs)
 |      outputs = tf.keras.layers.Dense(1)(x)
 |      model = tf.keras.Model(inputs, outputs)
 |      model.add_metric(tf.keras.metrics.Mean()(x), name='metric_1')
 |      ```
 |      
 |      Args:
 |        value: Metric tensor.
 |        name: String metric name.
 |        **kwargs: Additional keyword arguments for backward compatibility.
 |          Accepted values:
 |          `aggregation` - When the `value` tensor provided is not the result of
 |          calling a `keras.Metric` instance, it will be aggregated by default
 |          using a `keras.Metric.Mean`.
 |  
 |  add_update(self, updates)
 |      Add update op(s), potentially dependent on layer inputs.
 |      
 |      Weight updates (for instance, the updates of the moving mean and variance
 |      in a BatchNormalization layer) may be dependent on the inputs passed
 |      when calling a layer. Hence, when reusing the same layer on
 |      different inputs `a` and `b`, some entries in `layer.updates` may be
 |      dependent on `a` and some on `b`. This method automatically keeps track
 |      of dependencies.
 |      
 |      This call is ignored when eager execution is enabled (in that case, variable
 |      updates are run on the fly and thus do not need to be tracked for later
 |      execution).
 |      
 |      Args:
 |        updates: Update op, or list/tuple of update ops, or zero-arg callable
 |          that returns an update op. A zero-arg callable should be passed in
 |          order to disable running the updates by setting `trainable=False`
 |          on this Layer, when executing in Eager mode.
 |  
 |  add_variable(self, *args, **kwargs)
 |      Deprecated, do NOT use! Alias for `add_weight`.
 |  
 |  add_weight(self, name=None, shape=None, dtype=None, initializer=None, regularizer=None, trainable=None, constraint=None, use_resource=None, synchronization=<VariableSynchronization.AUTO: 0>, aggregation=<VariableAggregationV2.NONE: 0>, **kwargs)
 |      Adds a new variable to the layer.
 |      
 |      Args:
 |        name: Variable name.
 |        shape: Variable shape. Defaults to scalar if unspecified.
 |        dtype: The type of the variable. Defaults to `self.dtype`.
 |        initializer: Initializer instance (callable).
 |        regularizer: Regularizer instance (callable).
 |        trainable: Boolean, whether the variable should be part of the layer's
 |          "trainable_variables" (e.g. variables, biases)
 |          or "non_trainable_variables" (e.g. BatchNorm mean and variance).
 |          Note that `trainable` cannot be `True` if `synchronization`
 |          is set to `ON_READ`.
 |        constraint: Constraint instance (callable).
 |        use_resource: Whether to use `ResourceVariable`.
 |        synchronization: Indicates when a distributed a variable will be
 |          aggregated. Accepted values are constants defined in the class
 |          `tf.VariableSynchronization`. By default the synchronization is set to
 |          `AUTO` and the current `DistributionStrategy` chooses
 |          when to synchronize. If `synchronization` is set to `ON_READ`,
 |          `trainable` must not be set to `True`.
 |        aggregation: Indicates how a distributed variable will be aggregated.
 |          Accepted values are constants defined in the class
 |          `tf.VariableAggregation`.
 |        **kwargs: Additional keyword arguments. Accepted values are `getter`,
 |          `collections`, `experimental_autocast` and `caching_device`.
 |      
 |      Returns:
 |        The variable created.
 |      
 |      Raises:
 |        ValueError: When giving unsupported dtype and no initializer or when
 |          trainable has been set to True with synchronization set as `ON_READ`.
 |  
 |  compute_mask(self, inputs, mask=None)
 |      Computes an output mask tensor.
 |      
 |      Args:
 |          inputs: Tensor or list of tensors.
 |          mask: Tensor or list of tensors.
 |      
 |      Returns:
 |          None or a tensor (or list of tensors,
 |              one per output tensor of the layer).
 |  
 |  compute_output_shape(self, input_shape)
 |      Computes the output shape of the layer.
 |      
 |      This method will cause the layer's state to be built, if that has not
 |      happened before. This requires that the layer will later be used with
 |      inputs that match the input shape provided here.
 |      
 |      Args:
 |          input_shape: Shape tuple (tuple of integers)
 |              or list of shape tuples (one per output tensor of the layer).
 |              Shape tuples can include None for free dimensions,
 |              instead of an integer.
 |      
 |      Returns:
 |          An input shape tuple.
 |  
 |  compute_output_signature(self, input_signature)
 |      Compute the output tensor signature of the layer based on the inputs.
 |      
 |      Unlike a TensorShape object, a TensorSpec object contains both shape
 |      and dtype information for a tensor. This method allows layers to provide
 |      output dtype information if it is different from the input dtype.
 |      For any layer that doesn't implement this function,
 |      the framework will fall back to use `compute_output_shape`, and will
 |      assume that the output dtype matches the input dtype.
 |      
 |      Args:
 |        input_signature: Single TensorSpec or nested structure of TensorSpec
 |          objects, describing a candidate input for the layer.
 |      
 |      Returns:
 |        Single TensorSpec or nested structure of TensorSpec objects, describing
 |          how the layer would transform the provided input.
 |      
 |      Raises:
 |        TypeError: If input_signature contains a non-TensorSpec object.
 |  
 |  count_params(self)
 |      Count the total number of scalars composing the weights.
 |      
 |      Returns:
 |          An integer count.
 |      
 |      Raises:
 |          ValueError: if the layer isn't yet built
 |            (in which case its weights aren't yet defined).
 |  
 |  finalize_state(self)
 |      Finalizes the layers state after updating layer weights.
 |      
 |      This function can be subclassed in a layer and will be called after updating
 |      a layer weights. It can be overridden to finalize any additional layer state
 |      after a weight update.
 |      
 |      This function will be called after weights of a layer have been restored
 |      from a loaded model.
 |  
 |  get_input_at(self, node_index)
 |      Retrieves the input tensor(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first input node of the layer.
 |      
 |      Returns:
 |          A tensor (or list of tensors if the layer has multiple inputs).
 |      
 |      Raises:
 |        RuntimeError: If called in Eager mode.
 |  
 |  get_input_mask_at(self, node_index)
 |      Retrieves the input mask tensor(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first time the layer was called.
 |      
 |      Returns:
 |          A mask tensor
 |          (or list of tensors if the layer has multiple inputs).
 |  
 |  get_input_shape_at(self, node_index)
 |      Retrieves the input shape(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first time the layer was called.
 |      
 |      Returns:
 |          A shape tuple
 |          (or list of shape tuples if the layer has multiple inputs).
 |      
 |      Raises:
 |        RuntimeError: If called in Eager mode.
 |  
 |  get_output_at(self, node_index)
 |      Retrieves the output tensor(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first output node of the layer.
 |      
 |      Returns:
 |          A tensor (or list of tensors if the layer has multiple outputs).
 |      
 |      Raises:
 |        RuntimeError: If called in Eager mode.
 |  
 |  get_output_mask_at(self, node_index)
 |      Retrieves the output mask tensor(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first time the layer was called.
 |      
 |      Returns:
 |          A mask tensor
 |          (or list of tensors if the layer has multiple outputs).
 |  
 |  get_output_shape_at(self, node_index)
 |      Retrieves the output shape(s) of a layer at a given node.
 |      
 |      Args:
 |          node_index: Integer, index of the node
 |              from which to retrieve the attribute.
 |              E.g. `node_index=0` will correspond to the
 |              first time the layer was called.
 |      
 |      Returns:
 |          A shape tuple
 |          (or list of shape tuples if the layer has multiple outputs).
 |      
 |      Raises:
 |        RuntimeError: If called in Eager mode.
 |  
 |  set_weights(self, weights)
 |      Sets the weights of the layer, from NumPy arrays.
 |      
 |      The weights of a layer represent the state of the layer. This function
 |      sets the weight values from numpy arrays. The weight values should be
 |      passed in the order they are created by the layer. Note that the layer's
 |      weights must be instantiated before calling this function, by calling
 |      the layer.
 |      
 |      For example, a `Dense` layer returns a list of two values: the kernel matrix
 |      and the bias vector. These can be used to set the weights of another
 |      `Dense` layer:
 |      
 |      >>> layer_a = tf.keras.layers.Dense(1,
 |      ...   kernel_initializer=tf.constant_initializer(1.))
 |      >>> a_out = layer_a(tf.convert_to_tensor([[1., 2., 3.]]))
 |      >>> layer_a.get_weights()
 |      [array([[1.],
 |             [1.],
 |             [1.]], dtype=float32), array([0.], dtype=float32)]
 |      >>> layer_b = tf.keras.layers.Dense(1,
 |      ...   kernel_initializer=tf.constant_initializer(2.))
 |      >>> b_out = layer_b(tf.convert_to_tensor([[10., 20., 30.]]))
 |      >>> layer_b.get_weights()
 |      [array([[2.],
 |             [2.],
 |             [2.]], dtype=float32), array([0.], dtype=float32)]
 |      >>> layer_b.set_weights(layer_a.get_weights())
 |      >>> layer_b.get_weights()
 |      [array([[1.],
 |             [1.],
 |             [1.]], dtype=float32), array([0.], dtype=float32)]
 |      
 |      Args:
 |        weights: a list of NumPy arrays. The number
 |          of arrays and their shape must match
 |          number of the dimensions of the weights
 |          of the layer (i.e. it should match the
 |          output of `get_weights`).
 |      
 |      Raises:
 |        ValueError: If the provided weights list does not match the
 |          layer's specifications.
 |  
 |  ----------------------------------------------------------------------
 |  Readonly properties inherited from keras.engine.base_layer.Layer:
 |  
 |  compute_dtype
 |      The dtype of the layer's computations.
 |      
 |      This is equivalent to `Layer.dtype_policy.compute_dtype`. Unless
 |      mixed precision is used, this is the same as `Layer.dtype`, the dtype of
 |      the weights.
 |      
 |      Layers automatically cast their inputs to the compute dtype, which causes
 |      computations and the output to be in the compute dtype as well. This is done
 |      by the base Layer class in `Layer.__call__`, so you do not have to insert
 |      these casts if implementing your own layer.
 |      
 |      Layers often perform certain internal computations in higher precision when
 |      `compute_dtype` is float16 or bfloat16 for numeric stability. The output
 |      will still typically be float16 or bfloat16 in such cases.
 |      
 |      Returns:
 |        The layer's compute dtype.
 |  
 |  dtype
 |      The dtype of the layer weights.
 |      
 |      This is equivalent to `Layer.dtype_policy.variable_dtype`. Unless
 |      mixed precision is used, this is the same as `Layer.compute_dtype`, the
 |      dtype of the layer's computations.
 |  
 |  dtype_policy
 |      The dtype policy associated with this layer.
 |      
 |      This is an instance of a `tf.keras.mixed_precision.Policy`.
 |  
 |  dynamic
 |      Whether the layer is dynamic (eager-only); set in the constructor.
 |  
 |  inbound_nodes
 |      Return Functional API nodes upstream of this layer.
 |  
 |  input
 |      Retrieves the input tensor(s) of a layer.
 |      
 |      Only applicable if the layer has exactly one input,
 |      i.e. if it is connected to one incoming layer.
 |      
 |      Returns:
 |          Input tensor or list of input tensors.
 |      
 |      Raises:
 |        RuntimeError: If called in Eager mode.
 |        AttributeError: If no inbound nodes are found.
 |  
 |  input_mask
 |      Retrieves the input mask tensor(s) of a layer.
 |      
 |      Only applicable if the layer has exactly one inbound node,
 |      i.e. if it is connected to one incoming layer.
 |      
 |      Returns:
 |          Input mask tensor (potentially None) or list of input
 |          mask tensors.
 |      
 |      Raises:
 |          AttributeError: if the layer is connected to
 |          more than one incoming layers.
 |  
 |  input_shape
 |      Retrieves the input shape(s) of a layer.
 |      
 |      Only applicable if the layer has exactly one input,
 |      i.e. if it is connected to one incoming layer, or if all inputs
 |      have the same shape.
 |      
 |      Returns:
 |          Input shape, as an integer shape tuple
 |          (or list of shape tuples, one tuple per input tensor).
 |      
 |      Raises:
 |          AttributeError: if the layer has no defined input_shape.
 |          RuntimeError: if called in Eager mode.
 |  
 |  losses
 |      List of losses added using the `add_loss()` API.
 |      
 |      Variable regularization tensors are created when this property is accessed,
 |      so it is eager safe: accessing `losses` under a `tf.GradientTape` will
 |      propagate gradients back to the corresponding variables.
 |      
 |      Examples:
 |      
 |      >>> class MyLayer(tf.keras.layers.Layer):
 |      ...   def call(self, inputs):
 |      ...     self.add_loss(tf.abs(tf.reduce_mean(inputs)))
 |      ...     return inputs
 |      >>> l = MyLayer()
 |      >>> l(np.ones((10, 1)))
 |      >>> l.losses
 |      [1.0]
 |      
 |      >>> inputs = tf.keras.Input(shape=(10,))
 |      >>> x = tf.keras.layers.Dense(10)(inputs)
 |      >>> outputs = tf.keras.layers.Dense(1)(x)
 |      >>> model = tf.keras.Model(inputs, outputs)
 |      >>> # Activity regularization.
 |      >>> len(model.losses)
 |      0
 |      >>> model.add_loss(tf.abs(tf.reduce_mean(x)))
 |      >>> len(model.losses)
 |      1
 |      
 |      >>> inputs = tf.keras.Input(shape=(10,))
 |      >>> d = tf.keras.layers.Dense(10, kernel_initializer='ones')
 |      >>> x = d(inputs)
 |      >>> outputs = tf.keras.layers.Dense(1)(x)
 |      >>> model = tf.keras.Model(inputs, outputs)
 |      >>> # Weight regularization.
 |      >>> model.add_loss(lambda: tf.reduce_mean(d.kernel))
 |      >>> model.losses
 |      [<tf.Tensor: shape=(), dtype=float32, numpy=1.0>]
 |      
 |      Returns:
 |        A list of tensors.
 |  
 |  name
 |      Name of the layer (string), set in the constructor.
 |  
 |  non_trainable_variables
 |      Sequence of non-trainable variables owned by this module and its submodules.
 |      
 |      Note: this method uses reflection to find variables on the current instance
 |      and submodules. For performance reasons you may wish to cache the result
 |      of calling this method if you don't expect the return value to change.
 |      
 |      Returns:
 |        A sequence of variables for the current module (sorted by attribute
 |        name) followed by variables from all submodules recursively (breadth
 |        first).
 |  
 |  outbound_nodes
 |      Return Functional API nodes downstream of this layer.
 |  
 |  output
 |      Retrieves the output tensor(s) of a layer.
 |      
 |      Only applicable if the layer has exactly one output,
 |      i.e. if it is connected to one incoming layer.
 |      
 |      Returns:
 |        Output tensor or list of output tensors.
 |      
 |      Raises:
 |        AttributeError: if the layer is connected to more than one incoming
 |          layers.
 |        RuntimeError: if called in Eager mode.
 |  
 |  output_mask
 |      Retrieves the output mask tensor(s) of a layer.
 |      
 |      Only applicable if the layer has exactly one inbound node,
 |      i.e. if it is connected to one incoming layer.
 |      
 |      Returns:
 |          Output mask tensor (potentially None) or list of output
 |          mask tensors.
 |      
 |      Raises:
 |          AttributeError: if the layer is connected to
 |          more than one incoming layers.
 |  
 |  output_shape
 |      Retrieves the output shape(s) of a layer.
 |      
 |      Only applicable if the layer has one output,
 |      or if all outputs have the same shape.
 |      
 |      Returns:
 |          Output shape, as an integer shape tuple
 |          (or list of shape tuples, one tuple per output tensor).
 |      
 |      Raises:
 |          AttributeError: if the layer has no defined output shape.
 |          RuntimeError: if called in Eager mode.
 |  
 |  trainable_variables
 |      Sequence of trainable variables owned by this module and its submodules.
 |      
 |      Note: this method uses reflection to find variables on the current instance
 |      and submodules. For performance reasons you may wish to cache the result
 |      of calling this method if you don't expect the return value to change.
 |      
 |      Returns:
 |        A sequence of variables for the current module (sorted by attribute
 |        name) followed by variables from all submodules recursively (breadth
 |        first).
 |  
 |  updates
 |  
 |  variable_dtype
 |      Alias of `Layer.dtype`, the dtype of the weights.
 |  
 |  variables
 |      Returns the list of all layer variables/weights.
 |      
 |      Alias of `self.weights`.
 |      
 |      Note: This will not track the weights of nested `tf.Modules` that are not
 |      themselves Keras layers.
 |      
 |      Returns:
 |        A list of variables.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from keras.engine.base_layer.Layer:
 |  
 |  activity_regularizer
 |      Optional regularizer function for the output of this layer.
 |  
 |  input_spec
 |      `InputSpec` instance(s) describing the input format for this layer.
 |      
 |      When you create a layer subclass, you can set `self.input_spec` to enable
 |      the layer to run input compatibility checks when it is called.
 |      Consider a `Conv2D` layer: it can only be called on a single input tensor
 |      of rank 4. As such, you can set, in `__init__()`:
 |      
 |      ```python
 |      self.input_spec = tf.keras.layers.InputSpec(ndim=4)
 |      ```
 |      
 |      Now, if you try to call the layer on an input that isn't rank 4
 |      (for instance, an input of shape `(2,)`, it will raise a nicely-formatted
 |      error:
 |      
 |      ```
 |      ValueError: Input 0 of layer conv2d is incompatible with the layer:
 |      expected ndim=4, found ndim=1. Full shape received: [2]
 |      ```
 |      
 |      Input checks that can be specified via `input_spec` include:
 |      - Structure (e.g. a single input, a list of 2 inputs, etc)
 |      - Shape
 |      - Rank (ndim)
 |      - Dtype
 |      
 |      For more information, see `tf.keras.layers.InputSpec`.
 |      
 |      Returns:
 |        A `tf.keras.layers.InputSpec` instance, or nested structure thereof.
 |  
 |  stateful
 |  
 |  supports_masking
 |      Whether this layer supports computing a mask using `compute_mask`.
 |  
 |  trainable
 |  
 |  ----------------------------------------------------------------------
 |  Class methods inherited from tensorflow.python.module.module.Module:
 |  
 |  with_name_scope(method) from builtins.type
 |      Decorator to automatically enter the module name scope.
 |      
 |      >>> class MyModule(tf.Module):
 |      ...   @tf.Module.with_name_scope
 |      ...   def __call__(self, x):
 |      ...     if not hasattr(self, 'w'):
 |      ...       self.w = tf.Variable(tf.random.normal([x.shape[1], 3]))
 |      ...     return tf.matmul(x, self.w)
 |      
 |      Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose
 |      names included the module name:
 |      
 |      >>> mod = MyModule()
 |      >>> mod(tf.ones([1, 2]))
 |      <tf.Tensor: shape=(1, 3), dtype=float32, numpy=..., dtype=float32)>
 |      >>> mod.w
 |      <tf.Variable 'my_module/Variable:0' shape=(2, 3) dtype=float32,
 |      numpy=..., dtype=float32)>
 |      
 |      Args:
 |        method: The method to wrap.
 |      
 |      Returns:
 |        The original method wrapped such that it enters the module's name scope.
 |  
 |  ----------------------------------------------------------------------
 |  Readonly properties inherited from tensorflow.python.module.module.Module:
 |  
 |  name_scope
 |      Returns a `tf.name_scope` instance for this class.
 |  
 |  submodules
 |      Sequence of all sub-modules.
 |      
 |      Submodules are modules which are properties of this module, or found as
 |      properties of modules which are properties of this module (and so on).
 |      
 |      >>> a = tf.Module()
 |      >>> b = tf.Module()
 |      >>> c = tf.Module()
 |      >>> a.b = b
 |      >>> b.c = c
 |      >>> list(a.submodules) == [b, c]
 |      True
 |      >>> list(b.submodules) == [c]
 |      True
 |      >>> list(c.submodules) == []
 |      True
 |      
 |      Returns:
 |        A sequence of all submodules.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from tensorflow.python.training.tracking.base.Trackable:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)

Using a subset of features

The previous example did not specify the features, so all the columns were used as input feature (except for the label). The following example shows how to specify input features.

feature_1 = tfdf.keras.FeatureUsage(name="bill_length_mm")
feature_2 = tfdf.keras.FeatureUsage(name="island")

all_features = [feature_1, feature_2]

# Note: This model is only trained with two features. It will not be as good as
# the one trained on all features.

model_2 = tfdf.keras.GradientBoostedTreesModel(
    features=all_features, exclude_non_specified_features=True)

model_2.compile(metrics=["accuracy"])
model_2.fit(x=train_ds, validation_data=test_ds)

print(model_2.evaluate(test_ds, return_dict=True))
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpr3aho19l as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.129188. Found 242 examples.
Reading validation dataset...
Num validation examples: tf.Tensor(102, shape=(), dtype=int32)
Validation dataset read in 0:00:00.181865. Found 102 examples.
Training model...
Model trained in 0:00:00.417688
Compiling model...
Model compiled.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpr3aho19l/model/ with prefix 3cc2423f6d294eac
[INFO kernel.cc:1022] Use fast generic engine
1/1 [==============================] - 0s 73ms/step - loss: 0.0000e+00 - accuracy: 0.9412
{'loss': 0.0, 'accuracy': 0.9411764740943909}

TF-DF attaches a semantics to each feature. This semantics controls how the feature is used by the model. The following semantics are currently supported:

  • Numerical: Generally for quantities or counts with full ordering. For example, the age of a person, or the number of items in a bag. Can be a float or an integer. Missing values are represented with float(Nan) or with an empty sparse tensor.
  • Categorical: Generally for a type/class in finite set of possible values without ordering. For example, the color RED in the set {RED, BLUE, GREEN}. Can be a string or an integer. Missing values are represented as "" (empty sting), value -2 or with an empty sparse tensor.
  • Categorical-Set: A set of categorical values. Great to represent tokenized text. Can be a string or an integer in a sparse tensor or a ragged tensor (recommended). The order/index of each item doesn't matter.

If not specified, the semantics is inferred from the representation type and shown in the training logs:

  • int, float (dense or sparse) → Numerical semantics.
  • str (dense or sparse) → Categorical semantics
  • int, str (ragged) → Categorical-Set semantics

In some cases, the inferred semantics is incorrect. For example: An Enum stored as an integer is semantically categorical, but it will be detected as numerical. In this case, you should specify the semantic argument in the input. The education_num field of the Adult dataset is classical example.

This dataset doesn't contain such a feature. However, for the demonstration, we will make the model treat the year as a categorical feature:

%set_cell_height 300

feature_1 = tfdf.keras.FeatureUsage(name="year", semantic=tfdf.keras.FeatureSemantic.CATEGORICAL)
feature_2 = tfdf.keras.FeatureUsage(name="bill_length_mm")
feature_3 = tfdf.keras.FeatureUsage(name="sex")
all_features = [feature_1, feature_2, feature_3]

model_3 = tfdf.keras.GradientBoostedTreesModel(features=all_features, exclude_non_specified_features=True)
model_3.compile( metrics=["accuracy"])

model_3.fit(x=train_ds, validation_data=test_ds)
<IPython.core.display.Javascript object>
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpx_dqjdop as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.132409. Found 242 examples.
Reading validation dataset...
Num validation examples: tf.Tensor(102, shape=(), dtype=int32)
Validation dataset read in 0:00:00.134877. Found 102 examples.
Training model...
Model trained in 0:00:00.220214
Compiling model...
Model compiled.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpx_dqjdop/model/ with prefix f2810cc4eebc448c
[INFO kernel.cc:1022] Use fast generic engine
<keras.callbacks.History at 0x7fc22864dd00>

Note that year is in the list of CATEGORICAL features (unlike the first run).

Hyper-parameters

Hyper-parameters are parameters of the training algorithm that impact the quality of the final model. They are specified in the model class constructor. The list of hyper-parameters is visible with the question mark colab command (e.g. ?tfdf.keras.GradientBoostedTreesModel).

Alternatively, you can find them on the TensorFlow Decision Forest Github or the Yggdrasil Decision Forest documentation.

The default hyper-parameters of each algorithm matches approximatively the initial publication paper. To ensure consistancy, new features and their matching hyper-parameters are always disable by default. That's why it is a good idea to tune your hyper-parameters.

# A classical but slighly more complex model.
model_6 = tfdf.keras.GradientBoostedTreesModel(
    num_trees=500, growing_strategy="BEST_FIRST_GLOBAL", max_depth=8)
model_6.fit(x=train_ds)
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpuy09za7o as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.303553. Found 242 examples.
Training model...
Model trained in 0:00:00.450126
Compiling model...
Model compiled.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpuy09za7o/model/ with prefix b33dfe74e5494ba0
[INFO kernel.cc:1022] Use fast generic engine
<keras.callbacks.History at 0x7fc228568cd0>
# A more complex, but possibly, more accurate model.
model_7 = tfdf.keras.GradientBoostedTreesModel(
    num_trees=500,
    growing_strategy="BEST_FIRST_GLOBAL",
    max_depth=8,
    split_axis="SPARSE_OBLIQUE",
    categorical_algorithm="RANDOM",
    )
model_7.fit(x=train_ds)
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpcebt04oj as temporary training directory
Reading training dataset...
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel._consumes_training_examples_until_eof at 0x7fc32ed37b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel._consumes_training_examples_until_eof at 0x7fc32ed37b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Training dataset read in 0:00:00.156549. Found 242 examples.
Training model...
Model trained in 0:00:00.542584
Compiling model...
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7fc24007b4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpcebt04oj/model/ with prefix cc51ae07237a4335
[INFO kernel.cc:1022] Use fast generic engine
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7fc24007b4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Model compiled.
<keras.callbacks.History at 0x7fc2c4047130>

As new training methods are published and implemented, combinaisons of hyper-parameters can emerge as good or almost-always-better than the default parameters. To avoid changing the default hyper-parameter values these good combinaisons are indexed and available as hyper-parameter templates.

For example, the benchmark_rank1 template is the best combinaison on our internal benchmarks. Those templates are versioned to allow training configuration stability e.g. benchmark_rank1@v1.

# A good template of hyper-parameters.
model_8 = tfdf.keras.GradientBoostedTreesModel(hyperparameter_template="benchmark_rank1")
model_8.fit(x=train_ds)
Resolve hyper-parameter template "benchmark_rank1" to "benchmark_rank1@v1" -> {'growing_strategy': 'BEST_FIRST_GLOBAL', 'categorical_algorithm': 'RANDOM', 'split_axis': 'SPARSE_OBLIQUE', 'sparse_oblique_normalization': 'MIN_MAX', 'sparse_oblique_num_projections_exponent': 1.0}.
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpqywlo4xq as temporary training directory
Reading training dataset...
WARNING:tensorflow:6 out of the last 6 calls to <function CoreModel._consumes_training_examples_until_eof at 0x7fc32ed37b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function CoreModel._consumes_training_examples_until_eof at 0x7fc32ed37b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Training dataset read in 0:00:00.156228. Found 242 examples.
Training model...
Model trained in 0:00:00.409198
Compiling model...
WARNING:tensorflow:6 out of the last 6 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7fc2c4031af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpqywlo4xq/model/ with prefix 0930440b96e14ffa
[INFO kernel.cc:1022] Use fast generic engine
WARNING:tensorflow:6 out of the last 6 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7fc2c4031af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Model compiled.
<keras.callbacks.History at 0x7fc367617850>

The available tempaltes are available with predefined_hyperparameters. Note that different learning algorithms have different templates, even if the name is similar.

# The hyper-parameter templates of the Gradient Boosted Tree model.
print(tfdf.keras.GradientBoostedTreesModel.predefined_hyperparameters())
[HyperParameterTemplate(name='better_default', version=1, parameters={'growing_strategy': 'BEST_FIRST_GLOBAL'}, description='A configuration that is generally better than the default parameters without being more expensive.'), HyperParameterTemplate(name='benchmark_rank1', version=1, parameters={'growing_strategy': 'BEST_FIRST_GLOBAL', 'categorical_algorithm': 'RANDOM', 'split_axis': 'SPARSE_OBLIQUE', 'sparse_oblique_normalization': 'MIN_MAX', 'sparse_oblique_num_projections_exponent': 1.0}, description='Top ranking hyper-parameters on our benchmark slightly modified to run in reasonable time.')]

Feature Preprocessing

Pre-processing features is sometimes necessary to consume signals with complex structures, to regularize the model or to apply transfer learning. Pre-processing can be done in one of three ways:

  1. Preprocessing on the Pandas dataframe. This solution is easy to implement and generally suitable for experimentation. However, the pre-processing logic will not be exported in the model by model.save().

  2. Keras Preprocessing: While more complex than the previous solution, Keras Preprocessing is packaged in the model.

  3. TensorFlow Feature Columns: This API is part of the TF Estimator library (!= Keras) and planned for deprecation. This solution is interesting when using existing preprocessing code.

In the next example, pre-process the body_mass_g feature into body_mass_kg = body_mass_g / 1000. The bill_length_mm is consumed without pre-processing. Note that such monotonic transformations have generally no impact on decision forest models.

%set_cell_height 300

body_mass_g = tf.keras.layers.Input(shape=(1,), name="body_mass_g")
body_mass_kg = body_mass_g / 1000.0

bill_length_mm = tf.keras.layers.Input(shape=(1,), name="bill_length_mm")

raw_inputs = {"body_mass_g": body_mass_g, "bill_length_mm": bill_length_mm}
processed_inputs = {"body_mass_kg": body_mass_kg, "bill_length_mm": bill_length_mm}

# "preprocessor" contains the preprocessing logic.
preprocessor = tf.keras.Model(inputs=raw_inputs, outputs=processed_inputs)

# "model_4" contains both the pre-processing logic and the decision forest.
model_4 = tfdf.keras.RandomForestModel(preprocessing=preprocessor)
model_4.fit(x=train_ds)

model_4.summary()
<IPython.core.display.Javascript object>
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpk07ankuj as temporary training directory
Reading training dataset...
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/keras/engine/functional.py:566: UserWarning: Input dict contained keys ['island', 'bill_depth_mm', 'flipper_length_mm', 'sex', 'year'] which did not match any model input. They will be ignored by the model.
  inputs = self._flatten_to_reference_inputs(inputs)
Training dataset read in 0:00:00.214896. Found 242 examples.
Training model...
Model trained in 0:00:00.035897
Compiling model...
Model compiled.
WARNING:tensorflow:5 out of the last 12 calls to <function CoreModel.yggdrasil_model_path_tensor at 0x7fc2283bd160> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpk07ankuj/model/ with prefix 368c5e6025ca41f0
[INFO kernel.cc:1022] Use fast generic engine
WARNING:tensorflow:5 out of the last 12 calls to <function CoreModel.yggdrasil_model_path_tensor at 0x7fc2283bd160> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Model: "random_forest_model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 model (Functional)          {'body_mass_kg': (None,   0         
                             1),                                 
                              'bill_length_mm': (None            
                             , 1)}                               
                                                                 
=================================================================
Total params: 1
Trainable params: 0
Non-trainable params: 1
_________________________________________________________________
Type: "RANDOM_FOREST"
Task: CLASSIFICATION
Label: "__LABEL"

Input Features (2):
    bill_length_mm
    body_mass_kg

No weights

Variable Importance: MEAN_MIN_DEPTH:

    1.        "__LABEL"  3.845138 ################
    2.   "body_mass_kg"  1.265615 #####
    3. "bill_length_mm"  0.000000 

Variable Importance: NUM_AS_ROOT:

    1. "bill_length_mm" 300.000000 

Variable Importance: NUM_NODES:

    1. "bill_length_mm" 1483.000000 ################
    2.   "body_mass_kg" 1334.000000 

Variable Importance: SUM_SCORE:

    1. "bill_length_mm" 46749.496410 ################
    2.   "body_mass_kg" 23736.953397 



Winner take all: true
Out-of-bag evaluation: accuracy:0.92562 logloss:0.741038
Number of trees: 300
Total number of nodes: 5934

Number of nodes by tree:
Count: 300 Average: 19.78 StdDev: 3.19452
Min: 11 Max: 31 Ignored: 0
----------------------------------------------
[ 11, 12)  2   0.67%   0.67%
[ 12, 13)  0   0.00%   0.67%
[ 13, 14) 10   3.33%   4.00% #
[ 14, 15)  0   0.00%   4.00%
[ 15, 16) 23   7.67%  11.67% ###
[ 16, 17)  0   0.00%  11.67%
[ 17, 18) 45  15.00%  26.67% ######
[ 18, 19)  0   0.00%  26.67%
[ 19, 20) 76  25.33%  52.00% ##########
[ 20, 21)  0   0.00%  52.00%
[ 21, 22) 77  25.67%  77.67% ##########
[ 22, 23)  0   0.00%  77.67%
[ 23, 24) 43  14.33%  92.00% ######
[ 24, 25)  0   0.00%  92.00%
[ 25, 26) 17   5.67%  97.67% ##
[ 26, 27)  0   0.00%  97.67%
[ 27, 28)  4   1.33%  99.00% #
[ 28, 29)  0   0.00%  99.00%
[ 29, 30)  2   0.67%  99.67%
[ 30, 31]  1   0.33% 100.00%

Depth by leafs:
Count: 3117 Average: 3.88611 StdDev: 1.28546
Min: 1 Max: 8 Ignored: 0
----------------------------------------------
[ 1, 2)  25   0.80%   0.80%
[ 2, 3) 419  13.44%  14.24% ####
[ 3, 4) 802  25.73%  39.97% ########
[ 4, 5) 980  31.44%  71.41% ##########
[ 5, 6) 515  16.52%  87.94% #####
[ 6, 7) 285   9.14%  97.08% ###
[ 7, 8)  89   2.86%  99.94% #
[ 8, 8]   2   0.06% 100.00%

Number of training obs by leaf:
Count: 3117 Average: 23.2916 StdDev: 28.3772
Min: 5 Max: 121 Ignored: 0
----------------------------------------------
[   5,  10) 1934  62.05%  62.05% ##########
[  10,  16)  247   7.92%  69.97% #
[  16,  22)   25   0.80%  70.77%
[  22,  28)   22   0.71%  71.48%
[  28,  34)   37   1.19%  72.67%
[  34,  40)  115   3.69%  76.36% #
[  40,  45)   91   2.92%  79.27%
[  45,  51)   95   3.05%  82.32%
[  51,  57)   72   2.31%  84.63%
[  57,  63)   69   2.21%  86.85%
[  63,  69)   63   2.02%  88.87%
[  69,  75)   47   1.51%  90.38%
[  75,  81)   37   1.19%  91.56%
[  81,  86)   49   1.57%  93.13%
[  86,  92)   71   2.28%  95.41%
[  92,  98)   56   1.80%  97.21%
[  98, 104)   54   1.73%  98.94%
[ 104, 110)   22   0.71%  99.65%
[ 110, 116)    8   0.26%  99.90%
[ 116, 121]    3   0.10% 100.00%

Attribute in nodes:
    1483 : bill_length_mm [NUMERICAL]
    1334 : body_mass_kg [NUMERICAL]

Attribute in nodes with depth <= 0:
    300 : bill_length_mm [NUMERICAL]

Attribute in nodes with depth <= 1:
    514 : bill_length_mm [NUMERICAL]
    361 : body_mass_kg [NUMERICAL]

Attribute in nodes with depth <= 2:
    854 : bill_length_mm [NUMERICAL]
    752 : body_mass_kg [NUMERICAL]

Attribute in nodes with depth <= 3:
    1179 : bill_length_mm [NUMERICAL]
    1087 : body_mass_kg [NUMERICAL]

Attribute in nodes with depth <= 5:
    1465 : bill_length_mm [NUMERICAL]
    1306 : body_mass_kg [NUMERICAL]

Condition type in nodes:
    2817 : HigherCondition
Condition type in nodes with depth <= 0:
    300 : HigherCondition
Condition type in nodes with depth <= 1:
    875 : HigherCondition
Condition type in nodes with depth <= 2:
    1606 : HigherCondition
Condition type in nodes with depth <= 3:
    2266 : HigherCondition
Condition type in nodes with depth <= 5:
    2771 : HigherCondition
Node format: NOT_SET

Training OOB:
    trees: 1, Out-of-bag evaluation: accuracy:0.933333 logloss:2.40291
    trees: 12, Out-of-bag evaluation: accuracy:0.908696 logloss:1.93969
    trees: 23, Out-of-bag evaluation: accuracy:0.925311 logloss:1.86117
    trees: 34, Out-of-bag evaluation: accuracy:0.92562 logloss:1.28498
    trees: 44, Out-of-bag evaluation: accuracy:0.929752 logloss:1.1449
    trees: 58, Out-of-bag evaluation: accuracy:0.929752 logloss:1.15294
    trees: 68, Out-of-bag evaluation: accuracy:0.92562 logloss:1.01023
    trees: 81, Out-of-bag evaluation: accuracy:0.92562 logloss:1.0134
    trees: 91, Out-of-bag evaluation: accuracy:0.92562 logloss:1.0138
    trees: 102, Out-of-bag evaluation: accuracy:0.92562 logloss:0.878691
    trees: 112, Out-of-bag evaluation: accuracy:0.92562 logloss:0.875412
    trees: 122, Out-of-bag evaluation: accuracy:0.92562 logloss:0.873907
    trees: 132, Out-of-bag evaluation: accuracy:0.921488 logloss:0.87135
    trees: 142, Out-of-bag evaluation: accuracy:0.921488 logloss:0.737539
    trees: 152, Out-of-bag evaluation: accuracy:0.921488 logloss:0.739482
    trees: 163, Out-of-bag evaluation: accuracy:0.921488 logloss:0.739229
    trees: 174, Out-of-bag evaluation: accuracy:0.92562 logloss:0.74163
    trees: 184, Out-of-bag evaluation: accuracy:0.92562 logloss:0.741984
    trees: 194, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742052
    trees: 204, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742382
    trees: 216, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742003
    trees: 226, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742988
    trees: 236, Out-of-bag evaluation: accuracy:0.92562 logloss:0.741734
    trees: 247, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742444
    trees: 257, Out-of-bag evaluation: accuracy:0.92562 logloss:0.743115
    trees: 271, Out-of-bag evaluation: accuracy:0.92562 logloss:0.742596
    trees: 282, Out-of-bag evaluation: accuracy:0.92562 logloss:0.743563
    trees: 293, Out-of-bag evaluation: accuracy:0.92562 logloss:0.741565
    trees: 300, Out-of-bag evaluation: accuracy:0.92562 logloss:0.741038

The following example re-implements the same logic using TensorFlow Feature Columns.

def g_to_kg(x):
  return x / 1000

feature_columns = [
    tf.feature_column.numeric_column("body_mass_g", normalizer_fn=g_to_kg),
    tf.feature_column.numeric_column("bill_length_mm"),
]

preprocessing = tf.keras.layers.DenseFeatures(feature_columns)

model_5 = tfdf.keras.RandomForestModel(preprocessing=preprocessing)
model_5.fit(x=train_ds)
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmp9hfi6k8e as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.150437. Found 242 examples.
Training model...
Model trained in 0:00:00.036151
Compiling model...
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmp9hfi6k8e/model/ with prefix fbc45516d44f47f9
[INFO kernel.cc:1022] Use fast generic engine
Model compiled.
WARNING:tensorflow:6 out of the last 13 calls to <function CoreModel.yggdrasil_model_path_tensor at 0x7fc228343700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 13 calls to <function CoreModel.yggdrasil_model_path_tensor at 0x7fc228343700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
<keras.callbacks.History at 0x7fc2401124f0>

Training a regression model

The previous example trains a classification model (TF-DF does not differentiate between binary classification and multi-class classification). In the next example, train a regression model on the Abalone dataset. The objective of this dataset is to predict the number of shell's rings of an abalone.

# Download the dataset.
!wget -q https://storage.googleapis.com/download.tensorflow.org/data/abalone_raw.csv -O /tmp/abalone.csv

dataset_df = pd.read_csv("/tmp/abalone.csv")
print(dataset_df.head(3))
Type  LongestShell  Diameter  Height  WholeWeight  ShuckedWeight  \
0    M         0.455     0.365   0.095       0.5140         0.2245   
1    M         0.350     0.265   0.090       0.2255         0.0995   
2    F         0.530     0.420   0.135       0.6770         0.2565   

   VisceraWeight  ShellWeight  Rings  
0         0.1010         0.15     15  
1         0.0485         0.07      7  
2         0.1415         0.21      9
# Split the dataset into a training and testing dataset.
train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))

# Name of the label column.
label = "Rings"

train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)
test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)
2892 examples in training, 1285 examples for testing.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)
%set_cell_height 300

# Configure the model.
model_7 = tfdf.keras.RandomForestModel(task = tfdf.keras.Task.REGRESSION)

# Train the model.
model_7.fit(x=train_ds)
<IPython.core.display.Javascript object>
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmplx2zl6g8 as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.187042. Found 2892 examples.
Training model...
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmplx2zl6g8/model/ with prefix 85f479400f0c4f33
Model trained in 0:00:01.218478
Compiling model...
[INFO abstract_model.cc:1248] Engine "RandomForestOptPred" built
[INFO kernel.cc:1022] Use fast generic engine
Model compiled.
<keras.callbacks.History at 0x7fc22829b1f0>
# Evaluate the model on the test dataset.
model_7.compile(metrics=["mse"])
evaluation = model_7.evaluate(test_ds, return_dict=True)

print(evaluation)
print()
print(f"MSE: {evaluation['mse']}")
print(f"RMSE: {math.sqrt(evaluation['mse'])}")
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel.make_test_function.<locals>.test_function at 0x7fc2281cc940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 5 calls to <function CoreModel.make_test_function.<locals>.test_function at 0x7fc2281cc940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
2/2 [==============================] - 0s 14ms/step - loss: 0.0000e+00 - mse: 4.6607
{'loss': 0.0, 'mse': 4.660676956176758}

MSE: 4.660676956176758
RMSE: 2.1588601057448713

Training a ranking model

Finaly, after having trained a classification and a regression models, train a ranking model.

The goal of a ranking is to order items by importance. The "value" of relevance does not matter directly. Ranking a set of documents with regard to a user query is an example of ranking problem: It is only important to get the right order, where the top documents matter more.

TF-DF expects for ranking datasets to be presented in a "flat" format. A document+query dataset might look like that:

query document_id feature_1 feature_2 relevance/label
cat 1 0.1 blue 4
cat 2 0.5 green 1
cat 3 0.2 red 2
dog 4 NA red 0
dog 5 0.2 red 1
dog 6 0.6 green 1

The relevance/label is a floating point numerical value between 0 and 5 (generally between 0 and 4) where 0 means "completely unrelated", 4 means "very relevant" and 5 means "the same as the query".

Interestingly, decision forests are often good rankers, and many state-of-the-art ranking models are decision forests.

In this example, use a sample of the LETOR3 dataset. More precisely, we want to download the OHSUMED.zip from the LETOR3 repo. This dataset is stored in the libsvm format, so we will need to convert it to csv.

%set_cell_height 200

archive_path = tf.keras.utils.get_file("letor.zip",
  "https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip",
  extract=True)

# Path to the train and test dataset using libsvm format.
raw_dataset_path = os.path.join(os.path.dirname(archive_path),"OHSUMED/Data/All/OHSUMED.txt")
<IPython.core.display.Javascript object>
Downloading data from https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip
61824018/61824018 [==============================] - 11s 0us/step

The dataset is stored as a .txt file in a specific format, so first convert it into a csv file.

def convert_libsvm_to_csv(src_path, dst_path):
  """Converts a libsvm ranking dataset into a flat csv file.

  Note: This code is specific to the LETOR3 dataset.
  """
  dst_handle = open(dst_path, "w")
  first_line = True
  for src_line in open(src_path,"r"):
    # Note: The last 3 items are comments.
    items = src_line.split(" ")[:-3]
    relevance = items[0]
    group = items[1].split(":")[1]
    features = [ item.split(":") for item in items[2:]]

    if first_line:
      # Csv header
      dst_handle.write("relevance,group," + ",".join(["f_" + feature[0] for feature in features]) + "\n")
      first_line = False
    dst_handle.write(relevance + ",g_" + group + "," + (",".join([feature[1] for feature in features])) + "\n")
  dst_handle.close()

# Convert the dataset.
csv_dataset_path="/tmp/ohsumed.csv"
convert_libsvm_to_csv(raw_dataset_path, csv_dataset_path)

# Load a dataset into a Pandas Dataframe.
dataset_df = pd.read_csv(csv_dataset_path)

# Display the first 3 examples.
dataset_df.head(3)
train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))

# Display the first 3 examples of the training dataset.
train_ds_pd.head(3)
11328 examples in training, 4812 examples for testing.

In this dataset, the relevance defines the ground-truth rank among rows of the same group.

# Name of the relevance and grouping columns.
relevance = "relevance"

ranking_train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)
ranking_test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_decision_forests/keras/core.py:2574: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
  features_dataframe = dataframe.drop(label, 1)
%set_cell_height 400

model_8 = tfdf.keras.GradientBoostedTreesModel(
    task=tfdf.keras.Task.RANKING,
    ranking_group="group",
    num_trees=50)

model_8.fit(x=ranking_train_ds)
<IPython.core.display.Javascript object>
Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.
Use /tmpfs/tmp/tmpwfuzm2k9 as temporary training directory
Reading training dataset...
Training dataset read in 0:00:00.456315. Found 11328 examples.
Training model...
Model trained in 0:00:00.718546
Compiling model...
Model compiled.
[INFO kernel.cc:1176] Loading model from path /tmpfs/tmp/tmpwfuzm2k9/model/ with prefix ffc2a342af0b4045
[INFO abstract_model.cc:1248] Engine "GradientBoostedTreesQuickScorerExtended" built
[INFO kernel.cc:1022] Use fast generic engine
<keras.callbacks.History at 0x7fc2282e4ee0>

At this point, keras does not propose any ranking metrics. Instead, the training and validation (a GBDT uses a validation dataset) are shown in the training logs. In this case the loss is LAMBDA_MART_NDCG5, and the final (i.e. at the end of the training) NDCG (normalized discounted cumulative gain) is 0.510136 (see line Final model valid-loss: -0.510136).

Note that the NDCG is a value between 0 and 1. The larget the NDCG, the better the model. For this reason, the loss to be -NDCG.

As before, the model can be analysed:

%set_cell_height 400

model_8.summary()
<IPython.core.display.Javascript object>
Model: "gradient_boosted_trees_model_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
=================================================================
Total params: 1
Trainable params: 0
Non-trainable params: 1
_________________________________________________________________
Type: "GRADIENT_BOOSTED_TREES"
Task: RANKING
Label: "__LABEL"
Rank group: "__RANK_GROUP"

Input Features (25):
    f_1
    f_10
    f_11
    f_12
    f_13
    f_14
    f_15
    f_16
    f_17
    f_18
    f_19
    f_2
    f_20
    f_21
    f_22
    f_23
    f_24
    f_25
    f_3
    f_4
    f_5
    f_6
    f_7
    f_8
    f_9

No weights

Variable Importance: MEAN_MIN_DEPTH:

    1. "__RANK_GROUP"  4.515667 ################
    2.      "__LABEL"  4.515667 ################
    3.          "f_1"  4.507402 ###############
    4.         "f_11"  4.506097 ###############
    5.          "f_5"  4.506097 ###############
    6.         "f_17"  4.505566 ###############
    7.         "f_13"  4.499138 ###############
    8.          "f_4"  4.485180 ###############
    9.         "f_18"  4.470461 ###############
   10.         "f_14"  4.469504 ###############
   11.          "f_2"  4.444389 ###############
   12.          "f_7"  4.428890 ###############
   13.         "f_10"  4.380584 ###############
   14.         "f_20"  4.371072 ###############
   15.          "f_9"  4.341373 ###############
   16.          "f_3"  4.313097 ###############
   17.          "f_6"  4.312361 ###############
   18.         "f_24"  4.224279 ##############
   19.         "f_15"  4.219596 ##############
   20.         "f_21"  4.070796 ##############
   21.         "f_22"  4.069954 ##############
   22.         "f_23"  3.957579 #############
   23.         "f_12"  3.465309 ###########
   24.         "f_16"  3.464898 ###########
   25.         "f_19"  3.273077 ##########
   26.         "f_25"  2.710960 ########
   27.          "f_8"  0.580808 

Variable Importance: NUM_AS_ROOT:

    1.  "f_8"  8.000000 ################
    2. "f_25"  2.000000 ##
    3. "f_19"  1.000000 

Variable Importance: NUM_NODES:

    1. "f_25" 23.000000 ################
    2.  "f_8" 23.000000 ################
    3. "f_19" 18.000000 ############
    4. "f_23" 17.000000 ###########
    5. "f_12" 16.000000 ##########
    6. "f_22" 16.000000 ##########
    7. "f_21" 11.000000 #######
    8. "f_24" 10.000000 ######
    9. "f_10"  9.000000 #####
   10. "f_20"  8.000000 #####
   11. "f_16"  7.000000 ####
   12.  "f_3"  6.000000 ###
   13.  "f_9"  6.000000 ###
   14. "f_14"  5.000000 ##
   15. "f_18"  5.000000 ##
   16. "f_15"  3.000000 #
   17.  "f_2"  3.000000 #
   18.  "f_4"  3.000000 #
   19.  "f_6"  3.000000 #
   20. "f_13"  2.000000 
   21.  "f_7"  2.000000 
   22.  "f_1"  1.000000 
   23. "f_11"  1.000000 
   24. "f_17"  1.000000 
   25.  "f_5"  1.000000 

Variable Importance: SUM_SCORE:

    1.  "f_8" 5576.371454 ################
    2. "f_25" 3851.723576 ###########
    3. "f_23" 2329.174742 ######
    4. "f_19" 1850.520707 #####
    5. "f_22" 1678.418310 ####
    6. "f_16" 1427.985957 ####
    7. "f_12" 1265.388689 ###
    8. "f_24" 878.971907 ##
    9.  "f_3" 816.531551 ##
   10. "f_20" 748.182166 ##
   11. "f_21" 734.607612 ##
   12.  "f_9" 597.981601 #
   13.  "f_6" 582.720355 #
   14. "f_15" 543.677861 #
   15. "f_14" 442.961323 #
   16. "f_10" 351.735157 
   17.  "f_4" 300.268641 
   18.  "f_2" 287.530042 
   19. "f_17" 272.195190 
   20. "f_18" 238.749069 
   21.  "f_7" 205.329028 
   22. "f_11" 117.736936 
   23.  "f_1" 64.810715 
   24. "f_13" 44.639788 
   25.  "f_5" 33.620414 



Loss: LAMBDA_MART_NDCG5
Validation loss value: -0.507803
Number of trees per iteration: 1
Node format: NOT_SET
Number of trees: 11
Total number of nodes: 411

Number of nodes by tree:
Count: 11 Average: 37.3636 StdDev: 4.88547
Min: 29 Max: 43 Ignored: 0
----------------------------------------------
[ 29, 30) 2  18.18%  18.18% #######
[ 30, 31) 0   0.00%  18.18%
[ 31, 32) 0   0.00%  18.18%
[ 32, 33) 0   0.00%  18.18%
[ 33, 34) 0   0.00%  18.18%
[ 34, 35) 0   0.00%  18.18%
[ 35, 36) 2  18.18%  36.36% #######
[ 36, 37) 0   0.00%  36.36%
[ 37, 38) 2  18.18%  54.55% #######
[ 38, 39) 0   0.00%  54.55%
[ 39, 40) 1   9.09%  63.64% ###
[ 40, 41) 0   0.00%  63.64%
[ 41, 42) 1   9.09%  72.73% ###
[ 42, 43) 0   0.00%  72.73%
[ 43, 43] 3  27.27% 100.00% ##########

Depth by leafs:
Count: 211 Average: 4.52607 StdDev: 0.756039
Min: 1 Max: 5 Ignored: 0
----------------------------------------------
[ 1, 2)   1   0.47%   0.47%
[ 2, 3)   2   0.95%   1.42%
[ 3, 4)  22  10.43%  11.85% ##
[ 4, 5)  46  21.80%  33.65% ###
[ 5, 5] 140  66.35% 100.00% ##########

Number of training obs by leaf:
Count: 211 Average: 536.289 StdDev: 1922.15
Min: 5 Max: 10071 Ignored: 0
----------------------------------------------
[     5,   508) 193  91.47%  91.47% ##########
[   508,  1011)   0   0.00%  91.47%
[  1011,  1515)   1   0.47%  91.94%
[  1515,  2018)   1   0.47%  92.42%
[  2018,  2521)   0   0.00%  92.42%
[  2521,  3025)   1   0.47%  92.89%
[  3025,  3528)   4   1.90%  94.79%
[  3528,  4031)   0   0.00%  94.79%
[  4031,  4535)   0   0.00%  94.79%
[  4535,  5038)   1   0.47%  95.26%
[  5038,  5541)   0   0.00%  95.26%
[  5541,  6045)   0   0.00%  95.26%
[  6045,  6548)   0   0.00%  95.26%
[  6548,  7051)   3   1.42%  96.68%
[  7051,  7555)   1   0.47%  97.16%
[  7555,  8058)   0   0.00%  97.16%
[  8058,  8561)   0   0.00%  97.16%
[  8561,  9065)   1   0.47%  97.63%
[  9065,  9568)   0   0.00%  97.63%
[  9568, 10071]   5   2.37% 100.00%

Attribute in nodes:
    23 : f_8 [NUMERICAL]
    23 : f_25 [NUMERICAL]
    18 : f_19 [NUMERICAL]
    17 : f_23 [NUMERICAL]
    16 : f_22 [NUMERICAL]
    16 : f_12 [NUMERICAL]
    11 : f_21 [NUMERICAL]
    10 : f_24 [NUMERICAL]
    9 : f_10 [NUMERICAL]
    8 : f_20 [NUMERICAL]
    7 : f_16 [NUMERICAL]
    6 : f_9 [NUMERICAL]
    6 : f_3 [NUMERICAL]
    5 : f_18 [NUMERICAL]
    5 : f_14 [NUMERICAL]
    3 : f_6 [NUMERICAL]
    3 : f_4 [NUMERICAL]
    3 : f_2 [NUMERICAL]
    3 : f_15 [NUMERICAL]
    2 : f_7 [NUMERICAL]
    2 : f_13 [NUMERICAL]
    1 : f_5 [NUMERICAL]
    1 : f_17 [NUMERICAL]
    1 : f_11 [NUMERICAL]
    1 : f_1 [NUMERICAL]

Attribute in nodes with depth <= 0:
    8 : f_8 [NUMERICAL]
    2 : f_25 [NUMERICAL]
    1 : f_19 [NUMERICAL]

Attribute in nodes with depth <= 1:
    10 : f_8 [NUMERICAL]
    7 : f_25 [NUMERICAL]
    5 : f_16 [NUMERICAL]
    3 : f_12 [NUMERICAL]
    2 : f_19 [NUMERICAL]
    2 : f_15 [NUMERICAL]
    1 : f_9 [NUMERICAL]
    1 : f_6 [NUMERICAL]
    1 : f_24 [NUMERICAL]

Attribute in nodes with depth <= 2:
    14 : f_25 [NUMERICAL]
    11 : f_19 [NUMERICAL]
    10 : f_8 [NUMERICAL]
    7 : f_23 [NUMERICAL]
    7 : f_12 [NUMERICAL]
    6 : f_16 [NUMERICAL]
    3 : f_21 [NUMERICAL]
    2 : f_22 [NUMERICAL]
    2 : f_20 [NUMERICAL]
    2 : f_15 [NUMERICAL]
    1 : f_9 [NUMERICAL]
    1 : f_7 [NUMERICAL]
    1 : f_6 [NUMERICAL]
    1 : f_3 [NUMERICAL]
    1 : f_24 [NUMERICAL]
    1 : f_2 [NUMERICAL]
    1 : f_18 [NUMERICAL]
    1 : f_14 [NUMERICAL]

Attribute in nodes with depth <= 3:
    20 : f_25 [NUMERICAL]
    16 : f_8 [NUMERICAL]
    16 : f_19 [NUMERICAL]
    13 : f_12 [NUMERICAL]
    11 : f_23 [NUMERICAL]
    10 : f_22 [NUMERICAL]
    8 : f_21 [NUMERICAL]
    6 : f_24 [NUMERICAL]
    6 : f_16 [NUMERICAL]
    5 : f_20 [NUMERICAL]
    3 : f_3 [NUMERICAL]
    3 : f_15 [NUMERICAL]
    3 : f_10 [NUMERICAL]
    2 : f_9 [NUMERICAL]
    2 : f_7 [NUMERICAL]
    2 : f_6 [NUMERICAL]
    1 : f_2 [NUMERICAL]
    1 : f_18 [NUMERICAL]
    1 : f_14 [NUMERICAL]
    1 : f_13 [NUMERICAL]

Attribute in nodes with depth <= 5:
    23 : f_8 [NUMERICAL]
    23 : f_25 [NUMERICAL]
    18 : f_19 [NUMERICAL]
    17 : f_23 [NUMERICAL]
    16 : f_22 [NUMERICAL]
    16 : f_12 [NUMERICAL]
    11 : f_21 [NUMERICAL]
    10 : f_24 [NUMERICAL]
    9 : f_10 [NUMERICAL]
    8 : f_20 [NUMERICAL]
    7 : f_16 [NUMERICAL]
    6 : f_9 [NUMERICAL]
    6 : f_3 [NUMERICAL]
    5 : f_18 [NUMERICAL]
    5 : f_14 [NUMERICAL]
    3 : f_6 [NUMERICAL]
    3 : f_4 [NUMERICAL]
    3 : f_2 [NUMERICAL]
    3 : f_15 [NUMERICAL]
    2 : f_7 [NUMERICAL]
    2 : f_13 [NUMERICAL]
    1 : f_5 [NUMERICAL]
    1 : f_17 [NUMERICAL]
    1 : f_11 [NUMERICAL]
    1 : f_1 [NUMERICAL]

Condition type in nodes:
    200 : HigherCondition
Condition type in nodes with depth <= 0:
    11 : HigherCondition
Condition type in nodes with depth <= 1:
    32 : HigherCondition
Condition type in nodes with depth <= 2:
    72 : HigherCondition
Condition type in nodes with depth <= 3:
    130 : HigherCondition
Condition type in nodes with depth <= 5:
    200 : HigherCondition