![]() |
![]() |
![]() |
![]() |
This tutorial demonstrates how to classify a highly imbalanced dataset in which the number of examples in one class greatly outnumbers the examples in another. You will work with the Credit Card Fraud Detection dataset hosted on Kaggle. The aim is to detect a mere 492 fraudulent transactions from 284,807 transactions in total. You will use Keras to define the model and class weights to help the model learn from the imbalanced data. .
This tutorial contains complete code to:
- Load a CSV file using Pandas.
- Create train, validation, and test sets.
- Define and train a model using Keras (including setting class weights).
- Evaluate the model using various metrics (including precision and recall).
- Try common techniques for dealing with imbalanced data like:
- Class weighting
- Oversampling
Setup
import tensorflow as tf
from tensorflow import keras
import os
import tempfile
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import sklearn
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
mpl.rcParams['figure.figsize'] = (12, 10)
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
Data processing and exploration
Download the Kaggle Credit Card Fraud data set
Pandas is a Python library with many helpful utilities for loading and working with structured data. It can be used to download CSVs into a Pandas DataFrame.
file = tf.keras.utils
raw_df = pd.read_csv('https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv')
raw_df.head()
raw_df[['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V26', 'V27', 'V28', 'Amount', 'Class']].describe()
Examine the class label imbalance
Let's look at the dataset imbalance:
neg, pos = np.bincount(raw_df['Class'])
total = neg + pos
print('Examples:\n Total: {}\n Positive: {} ({:.2f}% of total)\n'.format(
total, pos, 100 * pos / total))
Examples: Total: 284807 Positive: 492 (0.17% of total)
This shows the small fraction of positive samples.
Clean, split and normalize the data
The raw data has a few issues. First the Time
and Amount
columns are too variable to use directly. Drop the Time
column (since it's not clear what it means) and take the log of the Amount
column to reduce its range.
cleaned_df = raw_df.copy()
# You don't want the `Time` column.
cleaned_df.pop('Time')
# The `Amount` column covers a huge range. Convert to log-space.
eps = 0.001 # 0 => 0.1¢
cleaned_df['Log Amount'] = np.log(cleaned_df.pop('Amount')+eps)
Split the dataset into train, validation, and test sets. The validation set is used during the model fitting to evaluate the loss and any metrics, however the model is not fit with this data. The test set is completely unused during the training phase and is only used at the end to evaluate how well the model generalizes to new data. This is especially important with imbalanced datasets where overfitting is a significant concern from the lack of training data.
# Use a utility from sklearn to split and shuffle your dataset.
train_df, test_df = train_test_split(cleaned_df, test_size=0.2)
train_df, val_df = train_test_split(train_df, test_size=0.2)
# Form np arrays of labels and features.
train_labels = np.array(train_df.pop('Class'))
bool_train_labels = train_labels != 0
val_labels = np.array(val_df.pop('Class'))
test_labels = np.array(test_df.pop('Class'))
train_features = np.array(train_df)
val_features = np.array(val_df)
test_features = np.array(test_df)
Normalize the input features using the sklearn StandardScaler. This will set the mean to 0 and standard deviation to 1.
scaler = StandardScaler()
train_features = scaler.fit_transform(train_features)
val_features = scaler.transform(val_features)
test_features = scaler.transform(test_features)
train_features = np.clip(train_features, -5, 5)
val_features = np.clip(val_features, -5, 5)
test_features = np.clip(test_features, -5, 5)
print('Training labels shape:', train_labels.shape)
print('Validation labels shape:', val_labels.shape)
print('Test labels shape:', test_labels.shape)
print('Training features shape:', train_features.shape)
print('Validation features shape:', val_features.shape)
print('Test features shape:', test_features.shape)
Training labels shape: (182276,) Validation labels shape: (45569,) Test labels shape: (56962,) Training features shape: (182276, 29) Validation features shape: (45569, 29) Test features shape: (56962, 29)
Look at the data distribution
Next compare the distributions of the positive and negative examples over a few features. Good questions to ask yourself at this point are:
- Do these distributions make sense?
- Yes. You've normalized the input and these are mostly concentrated in the
+/- 2
range.
- Yes. You've normalized the input and these are mostly concentrated in the
- Can you see the difference between the distributions?
- Yes the positive examples contain a much higher rate of extreme values.
pos_df = pd.DataFrame(train_features[ bool_train_labels], columns=train_df.columns)
neg_df = pd.DataFrame(train_features[~bool_train_labels], columns=train_df.columns)
sns.jointplot(x=pos_df['V5'], y=pos_df['V6'],
kind='hex', xlim=(-5,5), ylim=(-5,5))
plt.suptitle("Positive distribution")
sns.jointplot(x=neg_df['V5'], y=neg_df['V6'],
kind='hex', xlim=(-5,5), ylim=(-5,5))
_ = plt.suptitle("Negative distribution")
Define the model and metrics
Define a function that creates a simple neural network with a densly connected hidden layer, a dropout layer to reduce overfitting, and an output sigmoid layer that returns the probability of a transaction being fraudulent:
METRICS = [
keras.metrics.TruePositives(name='tp'),
keras.metrics.FalsePositives(name='fp'),
keras.metrics.TrueNegatives(name='tn'),
keras.metrics.FalseNegatives(name='fn'),
keras.metrics.BinaryAccuracy(name='accuracy'),
keras.metrics.Precision(name='precision'),
keras.metrics.Recall(name='recall'),
keras.metrics.AUC(name='auc'),
keras.metrics.AUC(name='prc', curve='PR'), # precision-recall curve
]
def make_model(metrics=METRICS, output_bias=None):
if output_bias is not None:
output_bias = tf.keras.initializers.Constant(output_bias)
model = keras.Sequential([
keras.layers.Dense(
16, activation='relu',
input_shape=(train_features.shape[-1],)),
keras.layers.Dropout(0.5),
keras.layers.Dense(1, activation='sigmoid',
bias_initializer=output_bias),
])
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=1e-3),
loss=keras.losses.BinaryCrossentropy(),
metrics=metrics)
return model
Understanding useful metrics
Notice that there are a few metrics defined above that can be computed by the model that will be helpful when evaluating the performance.
- False negatives and false positives are samples that were incorrectly classified
- True negatives and true positives are samples that were correctly classified
- Accuracy is the percentage of examples correctly classified > \(\frac{\text{true samples} }{\text{total samples} }\)
- Precision is the percentage of predicted positives that were correctly classified > \(\frac{\text{true positives} }{\text{true positives + false positives} }\)
- Recall is the percentage of actual positives that were correctly classified > \(\frac{\text{true positives} }{\text{true positives + false negatives} }\)
- AUC refers to the Area Under the Curve of a Receiver Operating Characteristic curve (ROC-AUC). This metric is equal to the probability that a classifier will rank a random positive sample higher than a random negative sample.
- AUPRC refers to Area Under the Curve of the Precision-Recall Curve. This metric computes precision-recall pairs for different probability thresholds.
Read more:
- True vs. False and Positive vs. Negative
- Accuracy
- Precision and Recall
- ROC-AUC
- Relationship between Precision-Recall and ROC Curves
Baseline model
Build the model
Now create and train your model using the function that was defined earlier. Notice that the model is fit using a larger than default batch size of 2048, this is important to ensure that each batch has a decent chance of containing a few positive samples. If the batch size was too small, they would likely have no fraudulent transactions to learn from.
EPOCHS = 100
BATCH_SIZE = 2048
early_stopping = tf.keras.callbacks.EarlyStopping(
monitor='val_prc',
verbose=1,
patience=10,
mode='max',
restore_best_weights=True)
model = make_model()
model.summary()
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 16) 480 dropout (Dropout) (None, 16) 0 dense_1 (Dense) (None, 1) 17 ================================================================= Total params: 497 Trainable params: 497 Non-trainable params: 0 _________________________________________________________________
Test run the model:
model.predict(train_features[:10])
1/1 [==============================] - 0s 344ms/step array([[0.6088169 ], [0.4941456 ], [0.6518074 ], [0.6477403 ], [0.3832439 ], [0.4007728 ], [0.32497266], [0.3713796 ], [0.42784396], [0.2801294 ]], dtype=float32)
Optional: Set the correct initial bias.
These initial guesses are not great. You know the dataset is imbalanced. Set the output layer's bias to reflect that (See: A Recipe for Training Neural Networks: "init well"). This can help with initial convergence.
With the default bias initialization the loss should be about math.log(2) = 0.69314
results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0)
print("Loss: {:0.4f}".format(results[0]))
Loss: 0.6775
The correct bias to set can be derived from:
\[ p_0 = pos/(pos + neg) = 1/(1+e^{-b_0}) \]
\[ b_0 = -log_e(1/p_0 - 1) \]
\[ b_0 = log_e(pos/neg)\]
initial_bias = np.log([pos/neg])
initial_bias
array([-6.35935934])
Set that as the initial bias, and the model will give much more reasonable initial guesses.
It should be near: pos/total = 0.0018
model = make_model(output_bias=initial_bias)
model.predict(train_features[:10])
1/1 [==============================] - 0s 40ms/step array([[0.00102003], [0.00334534], [0.00060373], [0.00167802], [0.0011223 ], [0.0013187 ], [0.00594859], [0.00176013], [0.0008566 ], [0.5312835 ]], dtype=float32)
With this initialization the initial loss should be approximately:
\[-p_0log(p_0)-(1-p_0)log(1-p_0) = 0.01317\]
results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0)
print("Loss: {:0.4f}".format(results[0]))
Loss: 0.0209
This initial loss is about 50 times less than if would have been with naive initialization.
This way the model doesn't need to spend the first few epochs just learning that positive examples are unlikely. This also makes it easier to read plots of the loss during training.
Checkpoint the initial weights
To make the various training runs more comparable, keep this initial model's weights in a checkpoint file, and load them into each model before training:
initial_weights = os.path.join(tempfile.mkdtemp(), 'initial_weights')
model.save_weights(initial_weights)
Confirm that the bias fix helps
Before moving on, confirm quick that the careful bias initialization actually helped.
Train the model for 20 epochs, with and without this careful initialization, and compare the losses:
model = make_model()
model.load_weights(initial_weights)
model.layers[-1].bias.assign([0.0])
zero_bias_history = model.fit(
train_features,
train_labels,
batch_size=BATCH_SIZE,
epochs=20,
validation_data=(val_features, val_labels),
verbose=0)
model = make_model()
model.load_weights(initial_weights)
careful_bias_history = model.fit(
train_features,
train_labels,
batch_size=BATCH_SIZE,
epochs=20,
validation_data=(val_features, val_labels),
verbose=0)
def plot_loss(history, label, n):
# Use a log scale on y-axis to show the wide range of values.
plt.semilogy(history.epoch, history.history['loss'],
color=colors[n], label='Train ' + label)
plt.semilogy(history.epoch, history.history['val_loss'],
color=colors[n], label='Val ' + label,
linestyle="--")
plt.xlabel('Epoch')
plt.ylabel('Loss')
plot_loss(zero_bias_history, "Zero Bias", 0)
plot_loss(careful_bias_history, "Careful Bias", 1)
The above figure makes it clear: In terms of validation loss, on this problem, this careful initialization gives a clear advantage.
Train the model
model = make_model()
model.load_weights(initial_weights)
baseline_history = model.fit(
train_features,
train_labels,
batch_size=BATCH_SIZE,
epochs=EPOCHS,
callbacks=[early_stopping],
validation_data=(val_features, val_labels))
Epoch 1/100 90/90 [==============================] - 3s 14ms/step - loss: 0.0179 - tp: 62.0000 - fp: 150.0000 - tn: 227299.0000 - fn: 334.0000 - accuracy: 0.9979 - precision: 0.2925 - recall: 0.1566 - auc: 0.6586 - prc: 0.1123 - val_loss: 0.0092 - val_tp: 0.0000e+00 - val_fp: 0.0000e+00 - val_tn: 45496.0000 - val_fn: 73.0000 - val_accuracy: 0.9984 - val_precision: 0.0000e+00 - val_recall: 0.0000e+00 - val_auc: 0.7654 - val_prc: 0.3283 Epoch 2/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0095 - tp: 77.0000 - fp: 51.0000 - tn: 181902.0000 - fn: 246.0000 - accuracy: 0.9984 - precision: 0.6016 - recall: 0.2384 - auc: 0.8065 - prc: 0.3097 - val_loss: 0.0056 - val_tp: 26.0000 - val_fp: 8.0000 - val_tn: 45488.0000 - val_fn: 47.0000 - val_accuracy: 0.9988 - val_precision: 0.7647 - val_recall: 0.3562 - val_auc: 0.9107 - val_prc: 0.6655 Epoch 3/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0070 - tp: 136.0000 - fp: 33.0000 - tn: 181920.0000 - fn: 187.0000 - accuracy: 0.9988 - precision: 0.8047 - recall: 0.4211 - auc: 0.8727 - prc: 0.5329 - val_loss: 0.0047 - val_tp: 37.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 36.0000 - val_accuracy: 0.9990 - val_precision: 0.8043 - val_recall: 0.5068 - val_auc: 0.9244 - val_prc: 0.6576 Epoch 4/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0061 - tp: 157.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 166.0000 - accuracy: 0.9990 - precision: 0.8626 - recall: 0.4861 - auc: 0.8871 - prc: 0.6036 - val_loss: 0.0043 - val_tp: 42.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 31.0000 - val_accuracy: 0.9991 - val_precision: 0.8077 - val_recall: 0.5753 - val_auc: 0.9244 - val_prc: 0.6553 Epoch 5/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0058 - tp: 172.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 151.0000 - accuracy: 0.9990 - precision: 0.8731 - recall: 0.5325 - auc: 0.8942 - prc: 0.6344 - val_loss: 0.0041 - val_tp: 45.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 28.0000 - val_accuracy: 0.9992 - val_precision: 0.8182 - val_recall: 0.6164 - val_auc: 0.9312 - val_prc: 0.6639 Epoch 6/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0055 - tp: 172.0000 - fp: 22.0000 - tn: 181931.0000 - fn: 151.0000 - accuracy: 0.9991 - precision: 0.8866 - recall: 0.5325 - auc: 0.8976 - prc: 0.6494 - val_loss: 0.0039 - val_tp: 52.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 21.0000 - val_accuracy: 0.9993 - val_precision: 0.8387 - val_recall: 0.7123 - val_auc: 0.9380 - val_prc: 0.6824 Epoch 7/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0057 - tp: 180.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 143.0000 - accuracy: 0.9991 - precision: 0.8654 - recall: 0.5573 - auc: 0.8946 - prc: 0.6345 - val_loss: 0.0037 - val_tp: 52.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 21.0000 - val_accuracy: 0.9993 - val_precision: 0.8387 - val_recall: 0.7123 - val_auc: 0.9448 - val_prc: 0.6873 Epoch 8/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0053 - tp: 181.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 142.0000 - accuracy: 0.9991 - precision: 0.8744 - recall: 0.5604 - auc: 0.9027 - prc: 0.6565 - val_loss: 0.0036 - val_tp: 52.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 21.0000 - val_accuracy: 0.9993 - val_precision: 0.8525 - val_recall: 0.7123 - val_auc: 0.9448 - val_prc: 0.6884 Epoch 9/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0053 - tp: 180.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 143.0000 - accuracy: 0.9990 - precision: 0.8531 - recall: 0.5573 - auc: 0.8980 - prc: 0.6595 - val_loss: 0.0035 - val_tp: 53.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 20.0000 - val_accuracy: 0.9993 - val_precision: 0.8413 - val_recall: 0.7260 - val_auc: 0.9449 - val_prc: 0.7112 Epoch 10/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0049 - tp: 189.0000 - fp: 32.0000 - tn: 181921.0000 - fn: 134.0000 - accuracy: 0.9991 - precision: 0.8552 - recall: 0.5851 - auc: 0.9123 - prc: 0.6817 - val_loss: 0.0034 - val_tp: 53.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 20.0000 - val_accuracy: 0.9994 - val_precision: 0.8548 - val_recall: 0.7260 - val_auc: 0.9448 - val_prc: 0.7094 Epoch 11/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0048 - tp: 189.0000 - fp: 32.0000 - tn: 181921.0000 - fn: 134.0000 - accuracy: 0.9991 - precision: 0.8552 - recall: 0.5851 - auc: 0.9215 - prc: 0.6940 - val_loss: 0.0034 - val_tp: 54.0000 - val_fp: 10.0000 - val_tn: 45486.0000 - val_fn: 19.0000 - val_accuracy: 0.9994 - val_precision: 0.8438 - val_recall: 0.7397 - val_auc: 0.9448 - val_prc: 0.7151 Epoch 12/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0050 - tp: 183.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 140.0000 - accuracy: 0.9991 - precision: 0.8551 - recall: 0.5666 - auc: 0.9091 - prc: 0.6663 - val_loss: 0.0032 - val_tp: 52.0000 - val_fp: 8.0000 - val_tn: 45488.0000 - val_fn: 21.0000 - val_accuracy: 0.9994 - val_precision: 0.8667 - val_recall: 0.7123 - val_auc: 0.9448 - val_prc: 0.7642 Epoch 13/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0047 - tp: 187.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 136.0000 - accuracy: 0.9991 - precision: 0.8578 - recall: 0.5789 - auc: 0.9278 - prc: 0.6926 - val_loss: 0.0032 - val_tp: 54.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 19.0000 - val_accuracy: 0.9994 - val_precision: 0.8571 - val_recall: 0.7397 - val_auc: 0.9449 - val_prc: 0.7562 Epoch 14/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 198.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 125.0000 - accuracy: 0.9991 - precision: 0.8684 - recall: 0.6130 - auc: 0.9170 - prc: 0.7135 - val_loss: 0.0031 - val_tp: 52.0000 - val_fp: 8.0000 - val_tn: 45488.0000 - val_fn: 21.0000 - val_accuracy: 0.9994 - val_precision: 0.8667 - val_recall: 0.7123 - val_auc: 0.9449 - val_prc: 0.7759 Epoch 15/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 192.0000 - fp: 27.0000 - tn: 181926.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8767 - recall: 0.5944 - auc: 0.9216 - prc: 0.7232 - val_loss: 0.0031 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9448 - val_prc: 0.7478 Epoch 16/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0046 - tp: 192.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8727 - recall: 0.5944 - auc: 0.9200 - prc: 0.6946 - val_loss: 0.0030 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9448 - val_prc: 0.7806 Epoch 17/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 196.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 127.0000 - accuracy: 0.9992 - precision: 0.8869 - recall: 0.6068 - auc: 0.9231 - prc: 0.7186 - val_loss: 0.0030 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9449 - val_prc: 0.7846 Epoch 18/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0046 - tp: 189.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 134.0000 - accuracy: 0.9991 - precision: 0.8832 - recall: 0.5851 - auc: 0.9107 - prc: 0.6852 - val_loss: 0.0030 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9448 - val_prc: 0.7923 Epoch 19/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 195.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8628 - recall: 0.6037 - auc: 0.9247 - prc: 0.6924 - val_loss: 0.0029 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9448 - val_prc: 0.7919 Epoch 20/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 202.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 121.0000 - accuracy: 0.9992 - precision: 0.8745 - recall: 0.6254 - auc: 0.9262 - prc: 0.7067 - val_loss: 0.0028 - val_tp: 56.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9994 - val_precision: 0.8615 - val_recall: 0.7671 - val_auc: 0.9449 - val_prc: 0.7965 Epoch 21/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 184.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 139.0000 - accuracy: 0.9991 - precision: 0.8638 - recall: 0.5697 - auc: 0.9294 - prc: 0.6954 - val_loss: 0.0029 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.7957 Epoch 22/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 205.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8798 - recall: 0.6347 - auc: 0.9170 - prc: 0.7232 - val_loss: 0.0029 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.7826 Epoch 23/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 201.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8664 - recall: 0.6223 - auc: 0.9185 - prc: 0.6926 - val_loss: 0.0028 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.7982 Epoch 24/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 196.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 127.0000 - accuracy: 0.9991 - precision: 0.8711 - recall: 0.6068 - auc: 0.9217 - prc: 0.7141 - val_loss: 0.0028 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.7979 Epoch 25/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 199.0000 - fp: 32.0000 - tn: 181921.0000 - fn: 124.0000 - accuracy: 0.9991 - precision: 0.8615 - recall: 0.6161 - auc: 0.9278 - prc: 0.7046 - val_loss: 0.0027 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8015 Epoch 26/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 191.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 132.0000 - accuracy: 0.9991 - precision: 0.8643 - recall: 0.5913 - auc: 0.9232 - prc: 0.6977 - val_loss: 0.0027 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8049 Epoch 27/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 196.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 127.0000 - accuracy: 0.9991 - precision: 0.8634 - recall: 0.6068 - auc: 0.9170 - prc: 0.6967 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8081 Epoch 28/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 198.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 125.0000 - accuracy: 0.9991 - precision: 0.8684 - recall: 0.6130 - auc: 0.9247 - prc: 0.7149 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9448 - val_prc: 0.8068 Epoch 29/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 194.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8661 - recall: 0.6006 - auc: 0.9123 - prc: 0.7028 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8058 Epoch 30/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 195.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8744 - recall: 0.6037 - auc: 0.9217 - prc: 0.7177 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8049 Epoch 31/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 193.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 130.0000 - accuracy: 0.9991 - precision: 0.8694 - recall: 0.5975 - auc: 0.9200 - prc: 0.7029 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8092 Epoch 32/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 195.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 128.0000 - accuracy: 0.9992 - precision: 0.8864 - recall: 0.6037 - auc: 0.9310 - prc: 0.7208 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9448 - val_prc: 0.8128 Epoch 33/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 199.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 124.0000 - accuracy: 0.9992 - precision: 0.8884 - recall: 0.6161 - auc: 0.9217 - prc: 0.7202 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8134 Epoch 34/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 210.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 113.0000 - accuracy: 0.9992 - precision: 0.8824 - recall: 0.6502 - auc: 0.9341 - prc: 0.7453 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9448 - val_prc: 0.8131 Epoch 35/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 200.0000 - fp: 34.0000 - tn: 181919.0000 - fn: 123.0000 - accuracy: 0.9991 - precision: 0.8547 - recall: 0.6192 - auc: 0.9310 - prc: 0.7165 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8199 Epoch 36/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 184.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 139.0000 - accuracy: 0.9991 - precision: 0.8598 - recall: 0.5697 - auc: 0.9278 - prc: 0.7054 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8162 Epoch 37/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 208.0000 - fp: 27.0000 - tn: 181926.0000 - fn: 115.0000 - accuracy: 0.9992 - precision: 0.8851 - recall: 0.6440 - auc: 0.9264 - prc: 0.7460 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8162 Epoch 38/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 201.0000 - fp: 33.0000 - tn: 181920.0000 - fn: 122.0000 - accuracy: 0.9991 - precision: 0.8590 - recall: 0.6223 - auc: 0.9263 - prc: 0.7016 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8199 Epoch 39/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 204.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8793 - recall: 0.6316 - auc: 0.9311 - prc: 0.7300 - val_loss: 0.0026 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8207 Epoch 40/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 202.0000 - fp: 24.0000 - tn: 181929.0000 - fn: 121.0000 - accuracy: 0.9992 - precision: 0.8938 - recall: 0.6254 - auc: 0.9296 - prc: 0.7514 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8199 Epoch 41/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 202.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 121.0000 - accuracy: 0.9992 - precision: 0.8783 - recall: 0.6254 - auc: 0.9372 - prc: 0.7448 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8200 Epoch 42/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 197.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 126.0000 - accuracy: 0.9991 - precision: 0.8678 - recall: 0.6099 - auc: 0.9325 - prc: 0.7317 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8260 Epoch 43/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 204.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8870 - recall: 0.6316 - auc: 0.9280 - prc: 0.7542 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8231 Epoch 44/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 189.0000 - fp: 36.0000 - tn: 181917.0000 - fn: 134.0000 - accuracy: 0.9991 - precision: 0.8400 - recall: 0.5851 - auc: 0.9231 - prc: 0.6901 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8268 Epoch 45/100 90/90 [==============================] - 0s 6ms/step - loss: 0.0040 - tp: 204.0000 - fp: 22.0000 - tn: 181931.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.9027 - recall: 0.6316 - auc: 0.9325 - prc: 0.7257 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8315 Epoch 46/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 198.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 125.0000 - accuracy: 0.9991 - precision: 0.8646 - recall: 0.6130 - auc: 0.9357 - prc: 0.7480 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8322 Epoch 47/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 205.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8723 - recall: 0.6347 - auc: 0.9263 - prc: 0.7404 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8269 Epoch 48/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0036 - tp: 217.0000 - fp: 21.0000 - tn: 181932.0000 - fn: 106.0000 - accuracy: 0.9993 - precision: 0.9118 - recall: 0.6718 - auc: 0.9357 - prc: 0.7610 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8306 Epoch 49/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0036 - tp: 211.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 112.0000 - accuracy: 0.9992 - precision: 0.8941 - recall: 0.6533 - auc: 0.9356 - prc: 0.7632 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8247 Epoch 50/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 205.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8761 - recall: 0.6347 - auc: 0.9340 - prc: 0.7286 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8288 Epoch 51/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 203.0000 - fp: 27.0000 - tn: 181926.0000 - fn: 120.0000 - accuracy: 0.9992 - precision: 0.8826 - recall: 0.6285 - auc: 0.9279 - prc: 0.7410 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8276 Epoch 52/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 201.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8777 - recall: 0.6223 - auc: 0.9342 - prc: 0.7278 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8306 Epoch 53/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 201.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8739 - recall: 0.6223 - auc: 0.9325 - prc: 0.7294 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8342 Epoch 54/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 201.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8739 - recall: 0.6223 - auc: 0.9372 - prc: 0.7202 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8329 Epoch 55/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 196.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 127.0000 - accuracy: 0.9991 - precision: 0.8711 - recall: 0.6068 - auc: 0.9372 - prc: 0.7500 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8337 Epoch 56/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 205.0000 - fp: 35.0000 - tn: 181918.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8542 - recall: 0.6347 - auc: 0.9231 - prc: 0.7172 - val_loss: 0.0024 - val_tp: 55.0000 - val_fp: 7.0000 - val_tn: 45489.0000 - val_fn: 18.0000 - val_accuracy: 0.9995 - val_precision: 0.8871 - val_recall: 0.7534 - val_auc: 0.9449 - val_prc: 0.8335 Epoch 57/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 195.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8705 - recall: 0.6037 - auc: 0.9341 - prc: 0.7320 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8340 Epoch 58/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 197.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 126.0000 - accuracy: 0.9992 - precision: 0.8756 - recall: 0.6099 - auc: 0.9294 - prc: 0.7280 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8317 Epoch 59/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 206.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 117.0000 - accuracy: 0.9992 - precision: 0.8879 - recall: 0.6378 - auc: 0.9340 - prc: 0.7334 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8415 Epoch 60/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 194.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8661 - recall: 0.6006 - auc: 0.9325 - prc: 0.7462 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8330 Epoch 61/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 204.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8681 - recall: 0.6316 - auc: 0.9341 - prc: 0.7380 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8344 Epoch 62/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 204.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8908 - recall: 0.6316 - auc: 0.9403 - prc: 0.7393 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8384 Epoch 63/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 204.0000 - fp: 27.0000 - tn: 181926.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8831 - recall: 0.6316 - auc: 0.9372 - prc: 0.7546 - val_loss: 0.0025 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8325 Epoch 64/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 208.0000 - fp: 37.0000 - tn: 181916.0000 - fn: 115.0000 - accuracy: 0.9992 - precision: 0.8490 - recall: 0.6440 - auc: 0.9248 - prc: 0.7373 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8350 Epoch 65/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 199.0000 - fp: 24.0000 - tn: 181929.0000 - fn: 124.0000 - accuracy: 0.9992 - precision: 0.8924 - recall: 0.6161 - auc: 0.9388 - prc: 0.7484 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9449 - val_prc: 0.8350 Epoch 66/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 214.0000 - fp: 23.0000 - tn: 181930.0000 - fn: 109.0000 - accuracy: 0.9993 - precision: 0.9030 - recall: 0.6625 - auc: 0.9403 - prc: 0.7531 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8392 Epoch 67/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 205.0000 - fp: 33.0000 - tn: 181920.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8613 - recall: 0.6347 - auc: 0.9372 - prc: 0.7505 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8314 Epoch 68/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 213.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 110.0000 - accuracy: 0.9992 - precision: 0.8838 - recall: 0.6594 - auc: 0.9465 - prc: 0.7519 - val_loss: 0.0024 - val_tp: 55.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 18.0000 - val_accuracy: 0.9995 - val_precision: 0.9016 - val_recall: 0.7534 - val_auc: 0.9517 - val_prc: 0.8420 Epoch 69/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 198.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 125.0000 - accuracy: 0.9991 - precision: 0.8684 - recall: 0.6130 - auc: 0.9386 - prc: 0.7377 - val_loss: 0.0023 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8423 Epoch 70/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0036 - tp: 205.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8761 - recall: 0.6347 - auc: 0.9449 - prc: 0.7606 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9517 - val_prc: 0.8433 Epoch 71/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 196.0000 - fp: 24.0000 - tn: 181929.0000 - fn: 127.0000 - accuracy: 0.9992 - precision: 0.8909 - recall: 0.6068 - auc: 0.9340 - prc: 0.7519 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9518 - val_prc: 0.8424 Epoch 72/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0035 - tp: 219.0000 - fp: 28.0000 - tn: 181925.0000 - fn: 104.0000 - accuracy: 0.9993 - precision: 0.8866 - recall: 0.6780 - auc: 0.9373 - prc: 0.7570 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9517 - val_prc: 0.8435 Epoch 73/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 212.0000 - fp: 31.0000 - tn: 181922.0000 - fn: 111.0000 - accuracy: 0.9992 - precision: 0.8724 - recall: 0.6563 - auc: 0.9418 - prc: 0.7575 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9517 - val_prc: 0.8434 Epoch 74/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 201.0000 - fp: 24.0000 - tn: 181929.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8933 - recall: 0.6223 - auc: 0.9402 - prc: 0.7614 - val_loss: 0.0024 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9518 - val_prc: 0.8406 Epoch 75/100 90/90 [==============================] - 0s 6ms/step - loss: 0.0036 - tp: 210.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 113.0000 - accuracy: 0.9992 - precision: 0.8750 - recall: 0.6502 - auc: 0.9356 - prc: 0.7504 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9517 - val_prc: 0.8416 Epoch 76/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0035 - tp: 206.0000 - fp: 30.0000 - tn: 181923.0000 - fn: 117.0000 - accuracy: 0.9992 - precision: 0.8729 - recall: 0.6378 - auc: 0.9573 - prc: 0.7699 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8415 Epoch 77/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0035 - tp: 216.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 107.0000 - accuracy: 0.9993 - precision: 0.8926 - recall: 0.6687 - auc: 0.9388 - prc: 0.7629 - val_loss: 0.0023 - val_tp: 55.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 18.0000 - val_accuracy: 0.9995 - val_precision: 0.9016 - val_recall: 0.7534 - val_auc: 0.9517 - val_prc: 0.8398 Epoch 78/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 202.0000 - fp: 29.0000 - tn: 181924.0000 - fn: 121.0000 - accuracy: 0.9992 - precision: 0.8745 - recall: 0.6254 - auc: 0.9403 - prc: 0.7551 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9518 - val_prc: 0.8426 Epoch 79/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0034 - tp: 213.0000 - fp: 25.0000 - tn: 181928.0000 - fn: 110.0000 - accuracy: 0.9993 - precision: 0.8950 - recall: 0.6594 - auc: 0.9450 - prc: 0.7702 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9517 - val_prc: 0.8427 Epoch 80/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 204.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 119.0000 - accuracy: 0.9992 - precision: 0.8870 - recall: 0.6316 - auc: 0.9417 - prc: 0.7328 - val_loss: 0.0023 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7808 - val_auc: 0.9518 - val_prc: 0.8401 Epoch 81/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0035 - tp: 205.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8874 - recall: 0.6347 - auc: 0.9402 - prc: 0.7628 - val_loss: 0.0023 - val_tp: 56.0000 - val_fp: 6.0000 - val_tn: 45490.0000 - val_fn: 17.0000 - val_accuracy: 0.9995 - val_precision: 0.9032 - val_recall: 0.7671 - val_auc: 0.9517 - val_prc: 0.8406 Epoch 82/100 82/90 [==========================>...] - ETA: 0s - loss: 0.0037 - tp: 186.0000 - fp: 21.0000 - tn: 167618.0000 - fn: 111.0000 - accuracy: 0.9992 - precision: 0.8986 - recall: 0.6263 - auc: 0.9367 - prc: 0.7555Restoring model weights from the end of the best epoch: 72. 90/90 [==============================] - 1s 6ms/step - loss: 0.0037 - tp: 205.0000 - fp: 26.0000 - tn: 181927.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8874 - recall: 0.6347 - auc: 0.9386 - prc: 0.7444 - val_loss: 0.0023 - val_tp: 58.0000 - val_fp: 9.0000 - val_tn: 45487.0000 - val_fn: 15.0000 - val_accuracy: 0.9995 - val_precision: 0.8657 - val_recall: 0.7945 - val_auc: 0.9517 - val_prc: 0.8395 Epoch 82: early stopping
Check training history
In this section, you will produce plots of your model's accuracy and loss on the training and validation set. These are useful to check for overfitting, which you can learn more about in the Overfit and underfit tutorial.
Additionally, you can produce these plots for any of the metrics you created above. False negatives are included as an example.
def plot_metrics(history):
metrics = ['loss', 'prc', 'precision', 'recall']
for n, metric in enumerate(metrics):
name = metric.replace("_"," ").capitalize()
plt.subplot(2,2,n+1)
plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train')
plt.plot(history.epoch, history.history['val_'+metric],
color=colors[0], linestyle="--", label='Val')
plt.xlabel('Epoch')
plt.ylabel(name)
if metric == 'loss':
plt.ylim([0, plt.ylim()[1]])
elif metric == 'auc':
plt.ylim([0.8,1])
else:
plt.ylim([0,1])
plt.legend();
plot_metrics(baseline_history)
Evaluate metrics
You can use a confusion matrix to summarize the actual vs. predicted labels, where the X axis is the predicted label and the Y axis is the actual label:
train_predictions_baseline = model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_baseline = model.predict(test_features, batch_size=BATCH_SIZE)
90/90 [==============================] - 0s 1ms/step 28/28 [==============================] - 0s 1ms/step
def plot_cm(labels, predictions, p=0.5):
cm = confusion_matrix(labels, predictions > p)
plt.figure(figsize=(5,5))
sns.heatmap(cm, annot=True, fmt="d")
plt.title('Confusion matrix @{:.2f}'.format(p))
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
print('Legitimate Transactions Detected (True Negatives): ', cm[0][0])
print('Legitimate Transactions Incorrectly Detected (False Positives): ', cm[0][1])
print('Fraudulent Transactions Missed (False Negatives): ', cm[1][0])
print('Fraudulent Transactions Detected (True Positives): ', cm[1][1])
print('Total Fraudulent Transactions: ', np.sum(cm[1]))
Evaluate your model on the test dataset and display the results for the metrics you created above:
baseline_results = model.evaluate(test_features, test_labels,
batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(model.metrics_names, baseline_results):
print(name, ': ', value)
print()
plot_cm(test_labels, test_predictions_baseline)
loss : 0.002767325611785054 tp : 78.0 fp : 9.0 tn : 56857.0 fn : 18.0 accuracy : 0.9995260238647461 precision : 0.8965517282485962 recall : 0.8125 auc : 0.9475902915000916 prc : 0.847183108329773 Legitimate Transactions Detected (True Negatives): 56857 Legitimate Transactions Incorrectly Detected (False Positives): 9 Fraudulent Transactions Missed (False Negatives): 18 Fraudulent Transactions Detected (True Positives): 78 Total Fraudulent Transactions: 96
If the model had predicted everything perfectly, this would be a diagonal matrix where values off the main diagonal, indicating incorrect predictions, would be zero. In this case the matrix shows that you have relatively few false positives, meaning that there were relatively few legitimate transactions that were incorrectly flagged. However, you would likely want to have even fewer false negatives despite the cost of increasing the number of false positives. This trade off may be preferable because false negatives would allow fraudulent transactions to go through, whereas false positives may cause an email to be sent to a customer to ask them to verify their card activity.
Plot the ROC
Now plot the ROC. This plot is useful because it shows, at a glance, the range of performance the model can reach just by tuning the output threshold.
def plot_roc(name, labels, predictions, **kwargs):
fp, tp, _ = sklearn.metrics.roc_curve(labels, predictions)
plt.plot(100*fp, 100*tp, label=name, linewidth=2, **kwargs)
plt.xlabel('False positives [%]')
plt.ylabel('True positives [%]')
plt.xlim([-0.5,20])
plt.ylim([80,100.5])
plt.grid(True)
ax = plt.gca()
ax.set_aspect('equal')
plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plt.legend(loc='lower right');
Plot the AUPRC
Now plot the AUPRC. Area under the interpolated precision-recall curve, obtained by plotting (recall, precision) points for different values of the classification threshold. Depending on how it's calculated, PR AUC may be equivalent to the average precision of the model.
def plot_prc(name, labels, predictions, **kwargs):
precision, recall, _ = sklearn.metrics.precision_recall_curve(labels, predictions)
plt.plot(precision, recall, label=name, linewidth=2, **kwargs)
plt.xlabel('Precision')
plt.ylabel('Recall')
plt.grid(True)
ax = plt.gca()
ax.set_aspect('equal')
plot_prc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_prc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plt.legend(loc='lower right');
It looks like the precision is relatively high, but the recall and the area under the ROC curve (AUC) aren't as high as you might like. Classifiers often face challenges when trying to maximize both precision and recall, which is especially true when working with imbalanced datasets. It is important to consider the costs of different types of errors in the context of the problem you care about. In this example, a false negative (a fraudulent transaction is missed) may have a financial cost, while a false positive (a transaction is incorrectly flagged as fraudulent) may decrease user happiness.
Class weights
Calculate class weights
The goal is to identify fraudulent transactions, but you don't have very many of those positive samples to work with, so you would want to have the classifier heavily weight the few examples that are available. You can do this by passing Keras weights for each class through a parameter. These will cause the model to "pay more attention" to examples from an under-represented class.
# Scaling by total/2 helps keep the loss to a similar magnitude.
# The sum of the weights of all examples stays the same.
weight_for_0 = (1 / neg) * (total / 2.0)
weight_for_1 = (1 / pos) * (total / 2.0)
class_weight = {0: weight_for_0, 1: weight_for_1}
print('Weight for class 0: {:.2f}'.format(weight_for_0))
print('Weight for class 1: {:.2f}'.format(weight_for_1))
Weight for class 0: 0.50 Weight for class 1: 289.44
Train a model with class weights
Now try re-training and evaluating the model with class weights to see how that affects the predictions.
weighted_model = make_model()
weighted_model.load_weights(initial_weights)
weighted_history = weighted_model.fit(
train_features,
train_labels,
batch_size=BATCH_SIZE,
epochs=EPOCHS,
callbacks=[early_stopping],
validation_data=(val_features, val_labels),
# The class weights go here
class_weight=class_weight)
Epoch 1/100 90/90 [==============================] - 3s 14ms/step - loss: 3.3363 - tp: 112.0000 - fp: 540.0000 - tn: 238279.0000 - fn: 307.0000 - accuracy: 0.9965 - precision: 0.1718 - recall: 0.2673 - auc: 0.7141 - prc: 0.0899 - val_loss: 0.0121 - val_tp: 23.0000 - val_fp: 71.0000 - val_tn: 45425.0000 - val_fn: 50.0000 - val_accuracy: 0.9973 - val_precision: 0.2447 - val_recall: 0.3151 - val_auc: 0.8876 - val_prc: 0.2189 Epoch 2/100 90/90 [==============================] - 1s 6ms/step - loss: 1.2405 - tp: 141.0000 - fp: 1034.0000 - tn: 180919.0000 - fn: 182.0000 - accuracy: 0.9933 - precision: 0.1200 - recall: 0.4365 - auc: 0.8562 - prc: 0.1437 - val_loss: 0.0157 - val_tp: 48.0000 - val_fp: 124.0000 - val_tn: 45372.0000 - val_fn: 25.0000 - val_accuracy: 0.9967 - val_precision: 0.2791 - val_recall: 0.6575 - val_auc: 0.9392 - val_prc: 0.4586 Epoch 3/100 90/90 [==============================] - 1s 6ms/step - loss: 0.8540 - tp: 198.0000 - fp: 1492.0000 - tn: 180461.0000 - fn: 125.0000 - accuracy: 0.9911 - precision: 0.1172 - recall: 0.6130 - auc: 0.8861 - prc: 0.1848 - val_loss: 0.0216 - val_tp: 59.0000 - val_fp: 184.0000 - val_tn: 45312.0000 - val_fn: 14.0000 - val_accuracy: 0.9957 - val_precision: 0.2428 - val_recall: 0.8082 - val_auc: 0.9582 - val_prc: 0.5453 Epoch 4/100 90/90 [==============================] - 1s 6ms/step - loss: 0.6551 - tp: 225.0000 - fp: 2166.0000 - tn: 179787.0000 - fn: 98.0000 - accuracy: 0.9876 - precision: 0.0941 - recall: 0.6966 - auc: 0.9086 - prc: 0.2311 - val_loss: 0.0281 - val_tp: 62.0000 - val_fp: 235.0000 - val_tn: 45261.0000 - val_fn: 11.0000 - val_accuracy: 0.9946 - val_precision: 0.2088 - val_recall: 0.8493 - val_auc: 0.9627 - val_prc: 0.5504 Epoch 5/100 90/90 [==============================] - 1s 6ms/step - loss: 0.5006 - tp: 247.0000 - fp: 2986.0000 - tn: 178967.0000 - fn: 76.0000 - accuracy: 0.9832 - precision: 0.0764 - recall: 0.7647 - auc: 0.9274 - prc: 0.2182 - val_loss: 0.0367 - val_tp: 64.0000 - val_fp: 356.0000 - val_tn: 45140.0000 - val_fn: 9.0000 - val_accuracy: 0.9920 - val_precision: 0.1524 - val_recall: 0.8767 - val_auc: 0.9676 - val_prc: 0.5536 Epoch 6/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4861 - tp: 250.0000 - fp: 3848.0000 - tn: 178105.0000 - fn: 73.0000 - accuracy: 0.9785 - precision: 0.0610 - recall: 0.7740 - auc: 0.9276 - prc: 0.1976 - val_loss: 0.0454 - val_tp: 64.0000 - val_fp: 507.0000 - val_tn: 44989.0000 - val_fn: 9.0000 - val_accuracy: 0.9887 - val_precision: 0.1121 - val_recall: 0.8767 - val_auc: 0.9736 - val_prc: 0.5522 Epoch 7/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4173 - tp: 260.0000 - fp: 4601.0000 - tn: 177352.0000 - fn: 63.0000 - accuracy: 0.9744 - precision: 0.0535 - recall: 0.8050 - auc: 0.9349 - prc: 0.1939 - val_loss: 0.0533 - val_tp: 65.0000 - val_fp: 622.0000 - val_tn: 44874.0000 - val_fn: 8.0000 - val_accuracy: 0.9862 - val_precision: 0.0946 - val_recall: 0.8904 - val_auc: 0.9757 - val_prc: 0.5307 Epoch 8/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4603 - tp: 258.0000 - fp: 5399.0000 - tn: 176554.0000 - fn: 65.0000 - accuracy: 0.9700 - precision: 0.0456 - recall: 0.7988 - auc: 0.9225 - prc: 0.1723 - val_loss: 0.0617 - val_tp: 65.0000 - val_fp: 728.0000 - val_tn: 44768.0000 - val_fn: 8.0000 - val_accuracy: 0.9838 - val_precision: 0.0820 - val_recall: 0.8904 - val_auc: 0.9811 - val_prc: 0.5223 Epoch 9/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3903 - tp: 260.0000 - fp: 6230.0000 - tn: 175723.0000 - fn: 63.0000 - accuracy: 0.9655 - precision: 0.0401 - recall: 0.8050 - auc: 0.9424 - prc: 0.1609 - val_loss: 0.0720 - val_tp: 65.0000 - val_fp: 852.0000 - val_tn: 44644.0000 - val_fn: 8.0000 - val_accuracy: 0.9811 - val_precision: 0.0709 - val_recall: 0.8904 - val_auc: 0.9822 - val_prc: 0.5203 Epoch 10/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3662 - tp: 270.0000 - fp: 6567.0000 - tn: 175386.0000 - fn: 53.0000 - accuracy: 0.9637 - precision: 0.0395 - recall: 0.8359 - auc: 0.9400 - prc: 0.1673 - val_loss: 0.0744 - val_tp: 65.0000 - val_fp: 870.0000 - val_tn: 44626.0000 - val_fn: 8.0000 - val_accuracy: 0.9807 - val_precision: 0.0695 - val_recall: 0.8904 - val_auc: 0.9828 - val_prc: 0.5165 Epoch 11/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3496 - tp: 274.0000 - fp: 7039.0000 - tn: 174914.0000 - fn: 49.0000 - accuracy: 0.9611 - precision: 0.0375 - recall: 0.8483 - auc: 0.9415 - prc: 0.1715 - val_loss: 0.0760 - val_tp: 65.0000 - val_fp: 881.0000 - val_tn: 44615.0000 - val_fn: 8.0000 - val_accuracy: 0.9805 - val_precision: 0.0687 - val_recall: 0.8904 - val_auc: 0.9830 - val_prc: 0.5239 Epoch 12/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4208 - tp: 264.0000 - fp: 6923.0000 - tn: 175030.0000 - fn: 59.0000 - accuracy: 0.9617 - precision: 0.0367 - recall: 0.8173 - auc: 0.9244 - prc: 0.1632 - val_loss: 0.0788 - val_tp: 65.0000 - val_fp: 905.0000 - val_tn: 44591.0000 - val_fn: 8.0000 - val_accuracy: 0.9800 - val_precision: 0.0670 - val_recall: 0.8904 - val_auc: 0.9837 - val_prc: 0.5425 Epoch 13/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3524 - tp: 274.0000 - fp: 7253.0000 - tn: 174700.0000 - fn: 49.0000 - accuracy: 0.9599 - precision: 0.0364 - recall: 0.8483 - auc: 0.9415 - prc: 0.1615 - val_loss: 0.0842 - val_tp: 65.0000 - val_fp: 956.0000 - val_tn: 44540.0000 - val_fn: 8.0000 - val_accuracy: 0.9788 - val_precision: 0.0637 - val_recall: 0.8904 - val_auc: 0.9837 - val_prc: 0.5044 Epoch 14/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3047 - tp: 278.0000 - fp: 7509.0000 - tn: 174444.0000 - fn: 45.0000 - accuracy: 0.9586 - precision: 0.0357 - recall: 0.8607 - auc: 0.9541 - prc: 0.1673 - val_loss: 0.0879 - val_tp: 65.0000 - val_fp: 990.0000 - val_tn: 44506.0000 - val_fn: 8.0000 - val_accuracy: 0.9781 - val_precision: 0.0616 - val_recall: 0.8904 - val_auc: 0.9840 - val_prc: 0.4995 Epoch 15/100 89/90 [============================>.] - ETA: 0s - loss: 0.2886 - tp: 283.0000 - fp: 7751.0000 - tn: 174198.0000 - fn: 40.0000 - accuracy: 0.9573 - precision: 0.0352 - recall: 0.8762 - auc: 0.9542 - prc: 0.1691Restoring model weights from the end of the best epoch: 5. 90/90 [==============================] - 1s 6ms/step - loss: 0.2886 - tp: 283.0000 - fp: 7752.0000 - tn: 174201.0000 - fn: 40.0000 - accuracy: 0.9573 - precision: 0.0352 - recall: 0.8762 - auc: 0.9542 - prc: 0.1691 - val_loss: 0.0896 - val_tp: 66.0000 - val_fp: 1005.0000 - val_tn: 44491.0000 - val_fn: 7.0000 - val_accuracy: 0.9778 - val_precision: 0.0616 - val_recall: 0.9041 - val_auc: 0.9843 - val_prc: 0.4827 Epoch 15: early stopping
Check training history
plot_metrics(weighted_history)
Evaluate metrics
train_predictions_weighted = weighted_model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_weighted = weighted_model.predict(test_features, batch_size=BATCH_SIZE)
90/90 [==============================] - 0s 1ms/step 28/28 [==============================] - 0s 1ms/step
weighted_results = weighted_model.evaluate(test_features, test_labels,
batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(weighted_model.metrics_names, weighted_results):
print(name, ': ', value)
print()
plot_cm(test_labels, test_predictions_weighted)
loss : 0.0376315712928772 tp : 84.0 fp : 454.0 tn : 56412.0 fn : 12.0 accuracy : 0.9918190836906433 precision : 0.15613383054733276 recall : 0.875 auc : 0.9684351086616516 prc : 0.5385129451751709 Legitimate Transactions Detected (True Negatives): 56412 Legitimate Transactions Incorrectly Detected (False Positives): 454 Fraudulent Transactions Missed (False Negatives): 12 Fraudulent Transactions Detected (True Positives): 84 Total Fraudulent Transactions: 96
Here you can see that with class weights the accuracy and precision are lower because there are more false positives, but conversely the recall and AUC are higher because the model also found more true positives. Despite having lower accuracy, this model has higher recall (and identifies more fraudulent transactions). Of course, there is a cost to both types of error (you wouldn't want to bug users by flagging too many legitimate transactions as fraudulent, either). Carefully consider the trade-offs between these different types of errors for your application.
Plot the ROC
plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')
plt.legend(loc='lower right');
Plot the AUPRC
plot_prc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_prc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plot_prc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_prc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')
plt.legend(loc='lower right');
Oversampling
Oversample the minority class
A related approach would be to resample the dataset by oversampling the minority class.
pos_features = train_features[bool_train_labels]
neg_features = train_features[~bool_train_labels]
pos_labels = train_labels[bool_train_labels]
neg_labels = train_labels[~bool_train_labels]
Using NumPy
You can balance the dataset manually by choosing the right number of random indices from the positive examples:
ids = np.arange(len(pos_features))
choices = np.random.choice(ids, len(neg_features))
res_pos_features = pos_features[choices]
res_pos_labels = pos_labels[choices]
res_pos_features.shape
(181953, 29)
resampled_features = np.concatenate([res_pos_features, neg_features], axis=0)
resampled_labels = np.concatenate([res_pos_labels, neg_labels], axis=0)
order = np.arange(len(resampled_labels))
np.random.shuffle(order)
resampled_features = resampled_features[order]
resampled_labels = resampled_labels[order]
resampled_features.shape
(363906, 29)
Using tf.data
If you're using tf.data
the easiest way to produce balanced examples is to start with a positive
and a negative
dataset, and merge them. See the tf.data guide for more examples.
BUFFER_SIZE = 100000
def make_ds(features, labels):
ds = tf.data.Dataset.from_tensor_slices((features, labels))#.cache()
ds = ds.shuffle(BUFFER_SIZE).repeat()
return ds
pos_ds = make_ds(pos_features, pos_labels)
neg_ds = make_ds(neg_features, neg_labels)
Each dataset provides (feature, label)
pairs:
for features, label in pos_ds.take(1):
print("Features:\n", features.numpy())
print()
print("Label: ", label.numpy())
Features: [-2.25920707 -0.0072081 -3.40638234 4.89298529 3.87783585 -3.25107162 -2.14285219 0.28594133 -3.52636688 -3.65233852 5. -5. -1.1397116 -5. 0.28999929 -1.26264317 0.74955735 1.1855264 -0.84786706 -0.13985126 0.33724841 -0.06842706 -0.3580829 -0.65930289 1.63937591 1.36920315 1.22563042 2.86007288 -1.4516158 ] Label: 1
Merge the two together using tf.data.Dataset.sample_from_datasets
:
resampled_ds = tf.data.Dataset.sample_from_datasets([pos_ds, neg_ds], weights=[0.5, 0.5])
resampled_ds = resampled_ds.batch(BATCH_SIZE).prefetch(2)
for features, label in resampled_ds.take(1):
print(label.numpy().mean())
0.498046875
To use this dataset, you'll need the number of steps per epoch.
The definition of "epoch" in this case is less clear. Say it's the number of batches required to see each negative example once:
resampled_steps_per_epoch = np.ceil(2.0*neg/BATCH_SIZE)
resampled_steps_per_epoch
278.0
Train on the oversampled data
Now try training the model with the resampled data set instead of using class weights to see how these methods compare.
resampled_model = make_model()
resampled_model.load_weights(initial_weights)
# Reset the bias to zero, since this dataset is balanced.
output_layer = resampled_model.layers[-1]
output_layer.bias.assign([0])
val_ds = tf.data.Dataset.from_tensor_slices((val_features, val_labels)).cache()
val_ds = val_ds.batch(BATCH_SIZE).prefetch(2)
resampled_history = resampled_model.fit(
resampled_ds,
epochs=EPOCHS,
steps_per_epoch=resampled_steps_per_epoch,
callbacks=[early_stopping],
validation_data=val_ds)
Epoch 1/100 278/278 [==============================] - 8s 22ms/step - loss: 0.6368 - tp: 214568.0000 - fp: 82297.0000 - tn: 259040.0000 - fn: 70401.0000 - accuracy: 0.7562 - precision: 0.7228 - recall: 0.7530 - auc: 0.8294 - prc: 0.8637 - val_loss: 0.2618 - val_tp: 68.0000 - val_fp: 1550.0000 - val_tn: 43946.0000 - val_fn: 5.0000 - val_accuracy: 0.9659 - val_precision: 0.0420 - val_recall: 0.9315 - val_auc: 0.9874 - val_prc: 0.7153 Epoch 2/100 278/278 [==============================] - 5s 19ms/step - loss: 0.2339 - tp: 253726.0000 - fp: 20423.0000 - tn: 263865.0000 - fn: 31330.0000 - accuracy: 0.9091 - precision: 0.9255 - recall: 0.8901 - auc: 0.9628 - prc: 0.9711 - val_loss: 0.1346 - val_tp: 67.0000 - val_fp: 1014.0000 - val_tn: 44482.0000 - val_fn: 6.0000 - val_accuracy: 0.9776 - val_precision: 0.0620 - val_recall: 0.9178 - val_auc: 0.9945 - val_prc: 0.7481 Epoch 3/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1753 - tp: 259428.0000 - fp: 12400.0000 - tn: 271726.0000 - fn: 25790.0000 - accuracy: 0.9329 - precision: 0.9544 - recall: 0.9096 - auc: 0.9792 - prc: 0.9830 - val_loss: 0.0976 - val_tp: 67.0000 - val_fp: 867.0000 - val_tn: 44629.0000 - val_fn: 6.0000 - val_accuracy: 0.9808 - val_precision: 0.0717 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7559 Epoch 4/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1520 - tp: 261697.0000 - fp: 10110.0000 - tn: 274249.0000 - fn: 23288.0000 - accuracy: 0.9413 - precision: 0.9628 - recall: 0.9183 - auc: 0.9849 - prc: 0.9871 - val_loss: 0.0841 - val_tp: 67.0000 - val_fp: 876.0000 - val_tn: 44620.0000 - val_fn: 6.0000 - val_accuracy: 0.9806 - val_precision: 0.0710 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7494 Epoch 5/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1376 - tp: 262584.0000 - fp: 9120.0000 - tn: 275971.0000 - fn: 21669.0000 - accuracy: 0.9459 - precision: 0.9664 - recall: 0.9238 - auc: 0.9882 - prc: 0.9894 - val_loss: 0.0762 - val_tp: 67.0000 - val_fp: 853.0000 - val_tn: 44643.0000 - val_fn: 6.0000 - val_accuracy: 0.9811 - val_precision: 0.0728 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7422 Epoch 6/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1267 - tp: 264133.0000 - fp: 8618.0000 - tn: 276116.0000 - fn: 20477.0000 - accuracy: 0.9489 - precision: 0.9684 - recall: 0.9281 - auc: 0.9904 - prc: 0.9911 - val_loss: 0.0696 - val_tp: 67.0000 - val_fp: 822.0000 - val_tn: 44674.0000 - val_fn: 6.0000 - val_accuracy: 0.9818 - val_precision: 0.0754 - val_recall: 0.9178 - val_auc: 0.9946 - val_prc: 0.7418 Epoch 7/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1189 - tp: 265211.0000 - fp: 8283.0000 - tn: 276289.0000 - fn: 19561.0000 - accuracy: 0.9511 - precision: 0.9697 - recall: 0.9313 - auc: 0.9918 - prc: 0.9922 - val_loss: 0.0640 - val_tp: 67.0000 - val_fp: 771.0000 - val_tn: 44725.0000 - val_fn: 6.0000 - val_accuracy: 0.9829 - val_precision: 0.0800 - val_recall: 0.9178 - val_auc: 0.9942 - val_prc: 0.7416 Epoch 8/100 278/278 [==============================] - 5s 20ms/step - loss: 0.1113 - tp: 265521.0000 - fp: 7871.0000 - tn: 277346.0000 - fn: 18606.0000 - accuracy: 0.9535 - precision: 0.9712 - recall: 0.9345 - auc: 0.9931 - prc: 0.9932 - val_loss: 0.0580 - val_tp: 67.0000 - val_fp: 702.0000 - val_tn: 44794.0000 - val_fn: 6.0000 - val_accuracy: 0.9845 - val_precision: 0.0871 - val_recall: 0.9178 - val_auc: 0.9938 - val_prc: 0.7420 Epoch 9/100 278/278 [==============================] - 5s 20ms/step - loss: 0.1055 - tp: 266759.0000 - fp: 7648.0000 - tn: 277146.0000 - fn: 17791.0000 - accuracy: 0.9553 - precision: 0.9721 - recall: 0.9375 - auc: 0.9938 - prc: 0.9939 - val_loss: 0.0528 - val_tp: 67.0000 - val_fp: 675.0000 - val_tn: 44821.0000 - val_fn: 6.0000 - val_accuracy: 0.9851 - val_precision: 0.0903 - val_recall: 0.9178 - val_auc: 0.9934 - val_prc: 0.7426 Epoch 10/100 278/278 [==============================] - 5s 19ms/step - loss: 0.1016 - tp: 267885.0000 - fp: 7444.0000 - tn: 276857.0000 - fn: 17158.0000 - accuracy: 0.9568 - precision: 0.9730 - recall: 0.9398 - auc: 0.9943 - prc: 0.9942 - val_loss: 0.0496 - val_tp: 67.0000 - val_fp: 662.0000 - val_tn: 44834.0000 - val_fn: 6.0000 - val_accuracy: 0.9853 - val_precision: 0.0919 - val_recall: 0.9178 - val_auc: 0.9930 - val_prc: 0.7243 Epoch 11/100 278/278 [==============================] - 5s 19ms/step - loss: 0.0974 - tp: 268749.0000 - fp: 7419.0000 - tn: 277074.0000 - fn: 16102.0000 - accuracy: 0.9587 - precision: 0.9731 - recall: 0.9435 - auc: 0.9947 - prc: 0.9946 - val_loss: 0.0465 - val_tp: 67.0000 - val_fp: 632.0000 - val_tn: 44864.0000 - val_fn: 6.0000 - val_accuracy: 0.9860 - val_precision: 0.0959 - val_recall: 0.9178 - val_auc: 0.9928 - val_prc: 0.7248 Epoch 12/100 278/278 [==============================] - 5s 19ms/step - loss: 0.0944 - tp: 269498.0000 - fp: 7384.0000 - tn: 277085.0000 - fn: 15377.0000 - accuracy: 0.9600 - precision: 0.9733 - recall: 0.9460 - auc: 0.9950 - prc: 0.9949 - val_loss: 0.0455 - val_tp: 67.0000 - val_fp: 641.0000 - val_tn: 44855.0000 - val_fn: 6.0000 - val_accuracy: 0.9858 - val_precision: 0.0946 - val_recall: 0.9178 - val_auc: 0.9927 - val_prc: 0.7070 Epoch 13/100 278/278 [==============================] - ETA: 0s - loss: 0.0912 - tp: 269571.0000 - fp: 7399.0000 - tn: 277708.0000 - fn: 14666.0000 - accuracy: 0.9612 - precision: 0.9733 - recall: 0.9484 - auc: 0.9953 - prc: 0.9951Restoring model weights from the end of the best epoch: 3. 278/278 [==============================] - 5s 19ms/step - loss: 0.0912 - tp: 269571.0000 - fp: 7399.0000 - tn: 277708.0000 - fn: 14666.0000 - accuracy: 0.9612 - precision: 0.9733 - recall: 0.9484 - auc: 0.9953 - prc: 0.9951 - val_loss: 0.0422 - val_tp: 67.0000 - val_fp: 591.0000 - val_tn: 44905.0000 - val_fn: 6.0000 - val_accuracy: 0.9869 - val_precision: 0.1018 - val_recall: 0.9178 - val_auc: 0.9926 - val_prc: 0.7169 Epoch 13: early stopping
If the training process were considering the whole dataset on each gradient update, this oversampling would be basically identical to the class weighting.
But when training the model batch-wise, as you did here, the oversampled data provides a smoother gradient signal: Instead of each positive example being shown in one batch with a large weight, they're shown in many different batches each time with a small weight.
This smoother gradient signal makes it easier to train the model.
Check training history
Note that the distributions of metrics will be different here, because the training data has a totally different distribution from the validation and test data.
plot_metrics(resampled_history)
Re-train
Because training is easier on the balanced data, the above training procedure may overfit quickly.
So break up the epochs to give the tf.keras.callbacks.EarlyStopping
finer control over when to stop training.
resampled_model = make_model()
resampled_model.load_weights(initial_weights)
# Reset the bias to zero, since this dataset is balanced.
output_layer = resampled_model.layers[-1]
output_layer.bias.assign([0])
resampled_history = resampled_model.fit(
resampled_ds,
# These are not real epochs
steps_per_epoch=20,
epochs=10*EPOCHS,
callbacks=[early_stopping],
validation_data=(val_ds))
Epoch 1/1000 20/20 [==============================] - 3s 76ms/step - loss: 2.0054 - tp: 6299.0000 - fp: 10480.0000 - tn: 55419.0000 - fn: 14331.0000 - accuracy: 0.7133 - precision: 0.3754 - recall: 0.3053 - auc: 0.6631 - prc: 0.4161 - val_loss: 0.7601 - val_tp: 21.0000 - val_fp: 21680.0000 - val_tn: 23816.0000 - val_fn: 52.0000 - val_accuracy: 0.5231 - val_precision: 9.6770e-04 - val_recall: 0.2877 - val_auc: 0.3172 - val_prc: 0.0014 Epoch 2/1000 20/20 [==============================] - 0s 24ms/step - loss: 1.2359 - tp: 10247.0000 - fp: 10132.0000 - tn: 10397.0000 - fn: 10184.0000 - accuracy: 0.5040 - precision: 0.5028 - recall: 0.5015 - auc: 0.4999 - prc: 0.6413 - val_loss: 0.7564 - val_tp: 48.0000 - val_fp: 21320.0000 - val_tn: 24176.0000 - val_fn: 25.0000 - val_accuracy: 0.5316 - val_precision: 0.0022 - val_recall: 0.6575 - val_auc: 0.6533 - val_prc: 0.0485 Epoch 3/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.8576 - tp: 13612.0000 - fp: 9694.0000 - tn: 10667.0000 - fn: 6987.0000 - accuracy: 0.5927 - precision: 0.5841 - recall: 0.6608 - auc: 0.6607 - prc: 0.7634 - val_loss: 0.7143 - val_tp: 63.0000 - val_fp: 19194.0000 - val_tn: 26302.0000 - val_fn: 10.0000 - val_accuracy: 0.5786 - val_precision: 0.0033 - val_recall: 0.8630 - val_auc: 0.8577 - val_prc: 0.1507 Epoch 4/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.6847 - tp: 15195.0000 - fp: 8650.0000 - tn: 11869.0000 - fn: 5246.0000 - accuracy: 0.6607 - precision: 0.6372 - recall: 0.7434 - auc: 0.7550 - prc: 0.8294 - val_loss: 0.6522 - val_tp: 65.0000 - val_fp: 16012.0000 - val_tn: 29484.0000 - val_fn: 8.0000 - val_accuracy: 0.6484 - val_precision: 0.0040 - val_recall: 0.8904 - val_auc: 0.9065 - val_prc: 0.2609 Epoch 5/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.5836 - tp: 16133.0000 - fp: 7573.0000 - tn: 12751.0000 - fn: 4503.0000 - accuracy: 0.7052 - precision: 0.6805 - recall: 0.7818 - auc: 0.8069 - prc: 0.8688 - val_loss: 0.5896 - val_tp: 66.0000 - val_fp: 12822.0000 - val_tn: 32674.0000 - val_fn: 7.0000 - val_accuracy: 0.7185 - val_precision: 0.0051 - val_recall: 0.9041 - val_auc: 0.9259 - val_prc: 0.3880 Epoch 6/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.5186 - tp: 16451.0000 - fp: 6555.0000 - tn: 13918.0000 - fn: 4036.0000 - accuracy: 0.7414 - precision: 0.7151 - recall: 0.8030 - auc: 0.8409 - prc: 0.8907 - val_loss: 0.5300 - val_tp: 67.0000 - val_fp: 9680.0000 - val_tn: 35816.0000 - val_fn: 6.0000 - val_accuracy: 0.7874 - val_precision: 0.0069 - val_recall: 0.9178 - val_auc: 0.9434 - val_prc: 0.5008 Epoch 7/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.4782 - tp: 16712.0000 - fp: 5743.0000 - tn: 14709.0000 - fn: 3796.0000 - accuracy: 0.7671 - precision: 0.7442 - recall: 0.8149 - auc: 0.8600 - prc: 0.9032 - val_loss: 0.4763 - val_tp: 68.0000 - val_fp: 7080.0000 - val_tn: 38416.0000 - val_fn: 5.0000 - val_accuracy: 0.8445 - val_precision: 0.0095 - val_recall: 0.9315 - val_auc: 0.9555 - val_prc: 0.5643 Epoch 8/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.4357 - tp: 16815.0000 - fp: 4970.0000 - tn: 15687.0000 - fn: 3488.0000 - accuracy: 0.7935 - precision: 0.7719 - recall: 0.8282 - auc: 0.8799 - prc: 0.9157 - val_loss: 0.4287 - val_tp: 68.0000 - val_fp: 4974.0000 - val_tn: 40522.0000 - val_fn: 5.0000 - val_accuracy: 0.8907 - val_precision: 0.0135 - val_recall: 0.9315 - val_auc: 0.9642 - val_prc: 0.6281 Epoch 9/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.4052 - tp: 16931.0000 - fp: 4132.0000 - tn: 16459.0000 - fn: 3438.0000 - accuracy: 0.8152 - precision: 0.8038 - recall: 0.8312 - auc: 0.8922 - prc: 0.9238 - val_loss: 0.3893 - val_tp: 68.0000 - val_fp: 3646.0000 - val_tn: 41850.0000 - val_fn: 5.0000 - val_accuracy: 0.9199 - val_precision: 0.0183 - val_recall: 0.9315 - val_auc: 0.9709 - val_prc: 0.6681 Epoch 10/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.3701 - tp: 17362.0000 - fp: 3603.0000 - tn: 16820.0000 - fn: 3175.0000 - accuracy: 0.8345 - precision: 0.8281 - recall: 0.8454 - auc: 0.9093 - prc: 0.9364 - val_loss: 0.3556 - val_tp: 69.0000 - val_fp: 2818.0000 - val_tn: 42678.0000 - val_fn: 4.0000 - val_accuracy: 0.9381 - val_precision: 0.0239 - val_recall: 0.9452 - val_auc: 0.9771 - val_prc: 0.6858 Epoch 11/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.3507 - tp: 17566.0000 - fp: 3172.0000 - tn: 17183.0000 - fn: 3039.0000 - accuracy: 0.8484 - precision: 0.8470 - recall: 0.8525 - auc: 0.9176 - prc: 0.9416 - val_loss: 0.3269 - val_tp: 69.0000 - val_fp: 2256.0000 - val_tn: 43240.0000 - val_fn: 4.0000 - val_accuracy: 0.9504 - val_precision: 0.0297 - val_recall: 0.9452 - val_auc: 0.9811 - val_prc: 0.6952 Epoch 12/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.3325 - tp: 17372.0000 - fp: 2859.0000 - tn: 17890.0000 - fn: 2839.0000 - accuracy: 0.8609 - precision: 0.8587 - recall: 0.8595 - auc: 0.9268 - prc: 0.9460 - val_loss: 0.2991 - val_tp: 68.0000 - val_fp: 1867.0000 - val_tn: 43629.0000 - val_fn: 5.0000 - val_accuracy: 0.9589 - val_precision: 0.0351 - val_recall: 0.9315 - val_auc: 0.9837 - val_prc: 0.7017 Epoch 13/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.3169 - tp: 17690.0000 - fp: 2524.0000 - tn: 17896.0000 - fn: 2850.0000 - accuracy: 0.8688 - precision: 0.8751 - recall: 0.8612 - auc: 0.9308 - prc: 0.9503 - val_loss: 0.2772 - val_tp: 68.0000 - val_fp: 1629.0000 - val_tn: 43867.0000 - val_fn: 5.0000 - val_accuracy: 0.9641 - val_precision: 0.0401 - val_recall: 0.9315 - val_auc: 0.9858 - val_prc: 0.7106 Epoch 14/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2969 - tp: 17677.0000 - fp: 2190.0000 - tn: 18385.0000 - fn: 2708.0000 - accuracy: 0.8804 - precision: 0.8898 - recall: 0.8672 - auc: 0.9400 - prc: 0.9555 - val_loss: 0.2586 - val_tp: 68.0000 - val_fp: 1477.0000 - val_tn: 44019.0000 - val_fn: 5.0000 - val_accuracy: 0.9675 - val_precision: 0.0440 - val_recall: 0.9315 - val_auc: 0.9874 - val_prc: 0.7132 Epoch 15/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2867 - tp: 17761.0000 - fp: 2037.0000 - tn: 18478.0000 - fn: 2684.0000 - accuracy: 0.8847 - precision: 0.8971 - recall: 0.8687 - auc: 0.9435 - prc: 0.9583 - val_loss: 0.2422 - val_tp: 68.0000 - val_fp: 1392.0000 - val_tn: 44104.0000 - val_fn: 5.0000 - val_accuracy: 0.9693 - val_precision: 0.0466 - val_recall: 0.9315 - val_auc: 0.9886 - val_prc: 0.7195 Epoch 16/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.2744 - tp: 18001.0000 - fp: 1857.0000 - tn: 18569.0000 - fn: 2533.0000 - accuracy: 0.8928 - precision: 0.9065 - recall: 0.8766 - auc: 0.9482 - prc: 0.9613 - val_loss: 0.2278 - val_tp: 67.0000 - val_fp: 1315.0000 - val_tn: 44181.0000 - val_fn: 6.0000 - val_accuracy: 0.9710 - val_precision: 0.0485 - val_recall: 0.9178 - val_auc: 0.9898 - val_prc: 0.7274 Epoch 17/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2630 - tp: 17977.0000 - fp: 1754.0000 - tn: 18791.0000 - fn: 2438.0000 - accuracy: 0.8977 - precision: 0.9111 - recall: 0.8806 - auc: 0.9528 - prc: 0.9643 - val_loss: 0.2148 - val_tp: 67.0000 - val_fp: 1254.0000 - val_tn: 44242.0000 - val_fn: 6.0000 - val_accuracy: 0.9723 - val_precision: 0.0507 - val_recall: 0.9178 - val_auc: 0.9909 - val_prc: 0.7300 Epoch 18/1000 20/20 [==============================] - 1s 26ms/step - loss: 0.2541 - tp: 18176.0000 - fp: 1670.0000 - tn: 18783.0000 - fn: 2331.0000 - accuracy: 0.9023 - precision: 0.9159 - recall: 0.8863 - auc: 0.9562 - prc: 0.9667 - val_loss: 0.2034 - val_tp: 66.0000 - val_fp: 1235.0000 - val_tn: 44261.0000 - val_fn: 7.0000 - val_accuracy: 0.9727 - val_precision: 0.0507 - val_recall: 0.9041 - val_auc: 0.9915 - val_prc: 0.7287 Epoch 19/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2463 - tp: 18088.0000 - fp: 1630.0000 - tn: 18963.0000 - fn: 2279.0000 - accuracy: 0.9046 - precision: 0.9173 - recall: 0.8881 - auc: 0.9588 - prc: 0.9683 - val_loss: 0.1922 - val_tp: 66.0000 - val_fp: 1178.0000 - val_tn: 44318.0000 - val_fn: 7.0000 - val_accuracy: 0.9740 - val_precision: 0.0531 - val_recall: 0.9041 - val_auc: 0.9921 - val_prc: 0.7335 Epoch 20/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.2341 - tp: 18073.0000 - fp: 1470.0000 - tn: 19196.0000 - fn: 2221.0000 - accuracy: 0.9099 - precision: 0.9248 - recall: 0.8906 - auc: 0.9623 - prc: 0.9709 - val_loss: 0.1821 - val_tp: 66.0000 - val_fp: 1144.0000 - val_tn: 44352.0000 - val_fn: 7.0000 - val_accuracy: 0.9747 - val_precision: 0.0545 - val_recall: 0.9041 - val_auc: 0.9927 - val_prc: 0.7345 Epoch 21/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2331 - tp: 18126.0000 - fp: 1436.0000 - tn: 19189.0000 - fn: 2209.0000 - accuracy: 0.9110 - precision: 0.9266 - recall: 0.8914 - auc: 0.9632 - prc: 0.9712 - val_loss: 0.1746 - val_tp: 66.0000 - val_fp: 1125.0000 - val_tn: 44371.0000 - val_fn: 7.0000 - val_accuracy: 0.9752 - val_precision: 0.0554 - val_recall: 0.9041 - val_auc: 0.9930 - val_prc: 0.7352 Epoch 22/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2236 - tp: 18526.0000 - fp: 1343.0000 - tn: 18861.0000 - fn: 2230.0000 - accuracy: 0.9128 - precision: 0.9324 - recall: 0.8926 - auc: 0.9655 - prc: 0.9737 - val_loss: 0.1672 - val_tp: 66.0000 - val_fp: 1097.0000 - val_tn: 44399.0000 - val_fn: 7.0000 - val_accuracy: 0.9758 - val_precision: 0.0567 - val_recall: 0.9041 - val_auc: 0.9935 - val_prc: 0.7383 Epoch 23/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2197 - tp: 18364.0000 - fp: 1309.0000 - tn: 19065.0000 - fn: 2222.0000 - accuracy: 0.9138 - precision: 0.9335 - recall: 0.8921 - auc: 0.9673 - prc: 0.9744 - val_loss: 0.1606 - val_tp: 66.0000 - val_fp: 1095.0000 - val_tn: 44401.0000 - val_fn: 7.0000 - val_accuracy: 0.9758 - val_precision: 0.0568 - val_recall: 0.9041 - val_auc: 0.9938 - val_prc: 0.7416 Epoch 24/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2136 - tp: 18337.0000 - fp: 1215.0000 - tn: 19260.0000 - fn: 2148.0000 - accuracy: 0.9179 - precision: 0.9379 - recall: 0.8951 - auc: 0.9689 - prc: 0.9754 - val_loss: 0.1540 - val_tp: 66.0000 - val_fp: 1069.0000 - val_tn: 44427.0000 - val_fn: 7.0000 - val_accuracy: 0.9764 - val_precision: 0.0581 - val_recall: 0.9041 - val_auc: 0.9941 - val_prc: 0.7417 Epoch 25/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2137 - tp: 18513.0000 - fp: 1200.0000 - tn: 19114.0000 - fn: 2133.0000 - accuracy: 0.9186 - precision: 0.9391 - recall: 0.8967 - auc: 0.9692 - prc: 0.9759 - val_loss: 0.1487 - val_tp: 66.0000 - val_fp: 1055.0000 - val_tn: 44441.0000 - val_fn: 7.0000 - val_accuracy: 0.9767 - val_precision: 0.0589 - val_recall: 0.9041 - val_auc: 0.9941 - val_prc: 0.7407 Epoch 26/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.2051 - tp: 18203.0000 - fp: 1144.0000 - tn: 19546.0000 - fn: 2067.0000 - accuracy: 0.9216 - precision: 0.9409 - recall: 0.8980 - auc: 0.9716 - prc: 0.9767 - val_loss: 0.1432 - val_tp: 67.0000 - val_fp: 1023.0000 - val_tn: 44473.0000 - val_fn: 6.0000 - val_accuracy: 0.9774 - val_precision: 0.0615 - val_recall: 0.9178 - val_auc: 0.9942 - val_prc: 0.7437 Epoch 27/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.2017 - tp: 18422.0000 - fp: 1109.0000 - tn: 19356.0000 - fn: 2073.0000 - accuracy: 0.9223 - precision: 0.9432 - recall: 0.8989 - auc: 0.9727 - prc: 0.9778 - val_loss: 0.1376 - val_tp: 67.0000 - val_fp: 995.0000 - val_tn: 44501.0000 - val_fn: 6.0000 - val_accuracy: 0.9780 - val_precision: 0.0631 - val_recall: 0.9178 - val_auc: 0.9944 - val_prc: 0.7452 Epoch 28/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.1993 - tp: 18544.0000 - fp: 1137.0000 - tn: 19232.0000 - fn: 2047.0000 - accuracy: 0.9223 - precision: 0.9422 - recall: 0.9006 - auc: 0.9736 - prc: 0.9787 - val_loss: 0.1327 - val_tp: 67.0000 - val_fp: 985.0000 - val_tn: 44511.0000 - val_fn: 6.0000 - val_accuracy: 0.9783 - val_precision: 0.0637 - val_recall: 0.9178 - val_auc: 0.9946 - val_prc: 0.7468 Epoch 29/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1967 - tp: 18420.0000 - fp: 1099.0000 - tn: 19427.0000 - fn: 2014.0000 - accuracy: 0.9240 - precision: 0.9437 - recall: 0.9014 - auc: 0.9741 - prc: 0.9790 - val_loss: 0.1287 - val_tp: 67.0000 - val_fp: 971.0000 - val_tn: 44525.0000 - val_fn: 6.0000 - val_accuracy: 0.9786 - val_precision: 0.0645 - val_recall: 0.9178 - val_auc: 0.9947 - val_prc: 0.7482 Epoch 30/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.1900 - tp: 18615.0000 - fp: 974.0000 - tn: 19378.0000 - fn: 1993.0000 - accuracy: 0.9276 - precision: 0.9503 - recall: 0.9033 - auc: 0.9756 - prc: 0.9805 - val_loss: 0.1254 - val_tp: 67.0000 - val_fp: 975.0000 - val_tn: 44521.0000 - val_fn: 6.0000 - val_accuracy: 0.9785 - val_precision: 0.0643 - val_recall: 0.9178 - val_auc: 0.9948 - val_prc: 0.7485 Epoch 31/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1842 - tp: 18482.0000 - fp: 958.0000 - tn: 19592.0000 - fn: 1928.0000 - accuracy: 0.9295 - precision: 0.9507 - recall: 0.9055 - auc: 0.9770 - prc: 0.9811 - val_loss: 0.1222 - val_tp: 67.0000 - val_fp: 967.0000 - val_tn: 44529.0000 - val_fn: 6.0000 - val_accuracy: 0.9786 - val_precision: 0.0648 - val_recall: 0.9178 - val_auc: 0.9949 - val_prc: 0.7494 Epoch 32/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1817 - tp: 18693.0000 - fp: 926.0000 - tn: 19423.0000 - fn: 1918.0000 - accuracy: 0.9306 - precision: 0.9528 - recall: 0.9069 - auc: 0.9776 - prc: 0.9817 - val_loss: 0.1195 - val_tp: 67.0000 - val_fp: 963.0000 - val_tn: 44533.0000 - val_fn: 6.0000 - val_accuracy: 0.9787 - val_precision: 0.0650 - val_recall: 0.9178 - val_auc: 0.9949 - val_prc: 0.7502 Epoch 33/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1804 - tp: 18668.0000 - fp: 971.0000 - tn: 19405.0000 - fn: 1916.0000 - accuracy: 0.9295 - precision: 0.9506 - recall: 0.9069 - auc: 0.9780 - prc: 0.9821 - val_loss: 0.1164 - val_tp: 67.0000 - val_fp: 957.0000 - val_tn: 44539.0000 - val_fn: 6.0000 - val_accuracy: 0.9789 - val_precision: 0.0654 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7516 Epoch 34/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1772 - tp: 18630.0000 - fp: 878.0000 - tn: 19607.0000 - fn: 1845.0000 - accuracy: 0.9335 - precision: 0.9550 - recall: 0.9099 - auc: 0.9789 - prc: 0.9825 - val_loss: 0.1132 - val_tp: 67.0000 - val_fp: 938.0000 - val_tn: 44558.0000 - val_fn: 6.0000 - val_accuracy: 0.9793 - val_precision: 0.0667 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7518 Epoch 35/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1747 - tp: 18664.0000 - fp: 893.0000 - tn: 19523.0000 - fn: 1880.0000 - accuracy: 0.9323 - precision: 0.9543 - recall: 0.9085 - auc: 0.9793 - prc: 0.9829 - val_loss: 0.1110 - val_tp: 67.0000 - val_fp: 936.0000 - val_tn: 44560.0000 - val_fn: 6.0000 - val_accuracy: 0.9793 - val_precision: 0.0668 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7523 Epoch 36/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1753 - tp: 18668.0000 - fp: 877.0000 - tn: 19531.0000 - fn: 1884.0000 - accuracy: 0.9326 - precision: 0.9551 - recall: 0.9083 - auc: 0.9790 - prc: 0.9830 - val_loss: 0.1089 - val_tp: 67.0000 - val_fp: 936.0000 - val_tn: 44560.0000 - val_fn: 6.0000 - val_accuracy: 0.9793 - val_precision: 0.0668 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7560 Epoch 37/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1691 - tp: 18684.0000 - fp: 866.0000 - tn: 19598.0000 - fn: 1812.0000 - accuracy: 0.9346 - precision: 0.9557 - recall: 0.9116 - auc: 0.9810 - prc: 0.9842 - val_loss: 0.1062 - val_tp: 67.0000 - val_fp: 909.0000 - val_tn: 44587.0000 - val_fn: 6.0000 - val_accuracy: 0.9799 - val_precision: 0.0686 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7560 Epoch 38/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1691 - tp: 18741.0000 - fp: 830.0000 - tn: 19553.0000 - fn: 1836.0000 - accuracy: 0.9349 - precision: 0.9576 - recall: 0.9108 - auc: 0.9810 - prc: 0.9842 - val_loss: 0.1041 - val_tp: 67.0000 - val_fp: 903.0000 - val_tn: 44593.0000 - val_fn: 6.0000 - val_accuracy: 0.9801 - val_precision: 0.0691 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7564 Epoch 39/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.1697 - tp: 18739.0000 - fp: 841.0000 - tn: 19556.0000 - fn: 1824.0000 - accuracy: 0.9349 - precision: 0.9570 - recall: 0.9113 - auc: 0.9811 - prc: 0.9845 - val_loss: 0.1018 - val_tp: 67.0000 - val_fp: 888.0000 - val_tn: 44608.0000 - val_fn: 6.0000 - val_accuracy: 0.9804 - val_precision: 0.0702 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7563 Epoch 40/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1637 - tp: 18653.0000 - fp: 808.0000 - tn: 19662.0000 - fn: 1837.0000 - accuracy: 0.9354 - precision: 0.9585 - recall: 0.9103 - auc: 0.9820 - prc: 0.9850 - val_loss: 0.1005 - val_tp: 67.0000 - val_fp: 893.0000 - val_tn: 44603.0000 - val_fn: 6.0000 - val_accuracy: 0.9803 - val_precision: 0.0698 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7564 Epoch 41/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1624 - tp: 18820.0000 - fp: 790.0000 - tn: 19638.0000 - fn: 1712.0000 - accuracy: 0.9389 - precision: 0.9597 - recall: 0.9166 - auc: 0.9827 - prc: 0.9855 - val_loss: 0.0986 - val_tp: 67.0000 - val_fp: 888.0000 - val_tn: 44608.0000 - val_fn: 6.0000 - val_accuracy: 0.9804 - val_precision: 0.0702 - val_recall: 0.9178 - val_auc: 0.9950 - val_prc: 0.7568 Epoch 42/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1605 - tp: 18760.0000 - fp: 761.0000 - tn: 19682.0000 - fn: 1757.0000 - accuracy: 0.9385 - precision: 0.9610 - recall: 0.9144 - auc: 0.9828 - prc: 0.9857 - val_loss: 0.0976 - val_tp: 67.0000 - val_fp: 899.0000 - val_tn: 44597.0000 - val_fn: 6.0000 - val_accuracy: 0.9801 - val_precision: 0.0694 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7566 Epoch 43/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1600 - tp: 18767.0000 - fp: 766.0000 - tn: 19694.0000 - fn: 1733.0000 - accuracy: 0.9390 - precision: 0.9608 - recall: 0.9155 - auc: 0.9832 - prc: 0.9859 - val_loss: 0.0958 - val_tp: 67.0000 - val_fp: 881.0000 - val_tn: 44615.0000 - val_fn: 6.0000 - val_accuracy: 0.9805 - val_precision: 0.0707 - val_recall: 0.9178 - val_auc: 0.9951 - val_prc: 0.7568 Epoch 44/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1598 - tp: 18753.0000 - fp: 778.0000 - tn: 19681.0000 - fn: 1748.0000 - accuracy: 0.9383 - precision: 0.9602 - recall: 0.9147 - auc: 0.9831 - prc: 0.9856 - val_loss: 0.0945 - val_tp: 67.0000 - val_fp: 876.0000 - val_tn: 44620.0000 - val_fn: 6.0000 - val_accuracy: 0.9806 - val_precision: 0.0710 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7568 Epoch 45/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1579 - tp: 18598.0000 - fp: 790.0000 - tn: 19840.0000 - fn: 1732.0000 - accuracy: 0.9384 - precision: 0.9593 - recall: 0.9148 - auc: 0.9834 - prc: 0.9859 - val_loss: 0.0932 - val_tp: 67.0000 - val_fp: 869.0000 - val_tn: 44627.0000 - val_fn: 6.0000 - val_accuracy: 0.9808 - val_precision: 0.0716 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7566 Epoch 46/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1561 - tp: 18780.0000 - fp: 760.0000 - tn: 19692.0000 - fn: 1728.0000 - accuracy: 0.9393 - precision: 0.9611 - recall: 0.9157 - auc: 0.9843 - prc: 0.9864 - val_loss: 0.0917 - val_tp: 67.0000 - val_fp: 858.0000 - val_tn: 44638.0000 - val_fn: 6.0000 - val_accuracy: 0.9810 - val_precision: 0.0724 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7566 Epoch 47/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1583 - tp: 18786.0000 - fp: 769.0000 - tn: 19661.0000 - fn: 1744.0000 - accuracy: 0.9386 - precision: 0.9607 - recall: 0.9151 - auc: 0.9835 - prc: 0.9861 - val_loss: 0.0905 - val_tp: 67.0000 - val_fp: 851.0000 - val_tn: 44645.0000 - val_fn: 6.0000 - val_accuracy: 0.9812 - val_precision: 0.0730 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7568 Epoch 48/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1526 - tp: 18597.0000 - fp: 679.0000 - tn: 19964.0000 - fn: 1720.0000 - accuracy: 0.9414 - precision: 0.9648 - recall: 0.9153 - auc: 0.9848 - prc: 0.9868 - val_loss: 0.0891 - val_tp: 67.0000 - val_fp: 842.0000 - val_tn: 44654.0000 - val_fn: 6.0000 - val_accuracy: 0.9814 - val_precision: 0.0737 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7489 Epoch 49/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1523 - tp: 18795.0000 - fp: 701.0000 - tn: 19731.0000 - fn: 1733.0000 - accuracy: 0.9406 - precision: 0.9640 - recall: 0.9156 - auc: 0.9845 - prc: 0.9870 - val_loss: 0.0886 - val_tp: 67.0000 - val_fp: 851.0000 - val_tn: 44645.0000 - val_fn: 6.0000 - val_accuracy: 0.9812 - val_precision: 0.0730 - val_recall: 0.9178 - val_auc: 0.9954 - val_prc: 0.7494 Epoch 50/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1479 - tp: 18782.0000 - fp: 682.0000 - tn: 19808.0000 - fn: 1688.0000 - accuracy: 0.9421 - precision: 0.9650 - recall: 0.9175 - auc: 0.9858 - prc: 0.9877 - val_loss: 0.0883 - val_tp: 67.0000 - val_fp: 867.0000 - val_tn: 44629.0000 - val_fn: 6.0000 - val_accuracy: 0.9808 - val_precision: 0.0717 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7491 Epoch 51/1000 20/20 [==============================] - 0s 25ms/step - loss: 0.1490 - tp: 18867.0000 - fp: 712.0000 - tn: 19683.0000 - fn: 1698.0000 - accuracy: 0.9412 - precision: 0.9636 - recall: 0.9174 - auc: 0.9856 - prc: 0.9877 - val_loss: 0.0883 - val_tp: 67.0000 - val_fp: 884.0000 - val_tn: 44612.0000 - val_fn: 6.0000 - val_accuracy: 0.9805 - val_precision: 0.0705 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7487 Epoch 52/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1485 - tp: 18946.0000 - fp: 716.0000 - tn: 19653.0000 - fn: 1645.0000 - accuracy: 0.9424 - precision: 0.9636 - recall: 0.9201 - auc: 0.9858 - prc: 0.9878 - val_loss: 0.0876 - val_tp: 67.0000 - val_fp: 891.0000 - val_tn: 44605.0000 - val_fn: 6.0000 - val_accuracy: 0.9803 - val_precision: 0.0699 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7488 Epoch 53/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1466 - tp: 18795.0000 - fp: 684.0000 - tn: 19882.0000 - fn: 1599.0000 - accuracy: 0.9443 - precision: 0.9649 - recall: 0.9216 - auc: 0.9860 - prc: 0.9877 - val_loss: 0.0870 - val_tp: 67.0000 - val_fp: 897.0000 - val_tn: 44599.0000 - val_fn: 6.0000 - val_accuracy: 0.9802 - val_precision: 0.0695 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7488 Epoch 54/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1482 - tp: 19074.0000 - fp: 738.0000 - tn: 19480.0000 - fn: 1668.0000 - accuracy: 0.9413 - precision: 0.9627 - recall: 0.9196 - auc: 0.9860 - prc: 0.9880 - val_loss: 0.0856 - val_tp: 67.0000 - val_fp: 885.0000 - val_tn: 44611.0000 - val_fn: 6.0000 - val_accuracy: 0.9804 - val_precision: 0.0704 - val_recall: 0.9178 - val_auc: 0.9953 - val_prc: 0.7490 Epoch 55/1000 20/20 [==============================] - 0s 23ms/step - loss: 0.1456 - tp: 18935.0000 - fp: 713.0000 - tn: 19657.0000 - fn: 1655.0000 - accuracy: 0.9422 - precision: 0.9637 - recall: 0.9196 - auc: 0.9864 - prc: 0.9883 - val_loss: 0.0850 - val_tp: 67.0000 - val_fp: 894.0000 - val_tn: 44602.0000 - val_fn: 6.0000 - val_accuracy: 0.9802 - val_precision: 0.0697 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7491 Epoch 56/1000 20/20 [==============================] - 0s 24ms/step - loss: 0.1457 - tp: 18855.0000 - fp: 730.0000 - tn: 19772.0000 - fn: 1603.0000 - accuracy: 0.9430 - precision: 0.9627 - recall: 0.9216 - auc: 0.9865 - prc: 0.9879 - val_loss: 0.0834 - val_tp: 67.0000 - val_fp: 881.0000 - val_tn: 44615.0000 - val_fn: 6.0000 - val_accuracy: 0.9805 - val_precision: 0.0707 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7494 Epoch 57/1000 20/20 [==============================] - ETA: 0s - loss: 0.1436 - tp: 18714.0000 - fp: 662.0000 - tn: 19927.0000 - fn: 1657.0000 - accuracy: 0.9434 - precision: 0.9658 - recall: 0.9187 - auc: 0.9868 - prc: 0.9884Restoring model weights from the end of the best epoch: 47. 20/20 [==============================] - 0s 24ms/step - loss: 0.1436 - tp: 18714.0000 - fp: 662.0000 - tn: 19927.0000 - fn: 1657.0000 - accuracy: 0.9434 - precision: 0.9658 - recall: 0.9187 - auc: 0.9868 - prc: 0.9884 - val_loss: 0.0822 - val_tp: 67.0000 - val_fp: 865.0000 - val_tn: 44631.0000 - val_fn: 6.0000 - val_accuracy: 0.9809 - val_precision: 0.0719 - val_recall: 0.9178 - val_auc: 0.9952 - val_prc: 0.7494 Epoch 57: early stopping
Re-check training history
plot_metrics(resampled_history)
Evaluate metrics
train_predictions_resampled = resampled_model.predict(train_features, batch_size=BATCH_SIZE)
test_predictions_resampled = resampled_model.predict(test_features, batch_size=BATCH_SIZE)
90/90 [==============================] - 0s 1ms/step 28/28 [==============================] - 0s 1ms/step
resampled_results = resampled_model.evaluate(test_features, test_labels,
batch_size=BATCH_SIZE, verbose=0)
for name, value in zip(resampled_model.metrics_names, resampled_results):
print(name, ': ', value)
print()
plot_cm(test_labels, test_predictions_resampled)
loss : 0.0926642045378685 tp : 87.0 fp : 1080.0 tn : 55786.0 fn : 9.0 accuracy : 0.9808819890022278 precision : 0.07455012947320938 recall : 0.90625 auc : 0.9739617109298706 prc : 0.6837594509124756 Legitimate Transactions Detected (True Negatives): 55786 Legitimate Transactions Incorrectly Detected (False Positives): 1080 Fraudulent Transactions Missed (False Negatives): 9 Fraudulent Transactions Detected (True Positives): 87 Total Fraudulent Transactions: 96
Plot the ROC
plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')
plot_roc("Train Resampled", train_labels, train_predictions_resampled, color=colors[2])
plot_roc("Test Resampled", test_labels, test_predictions_resampled, color=colors[2], linestyle='--')
plt.legend(loc='lower right');
Plot the AUPRC
plot_prc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0])
plot_prc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')
plot_prc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1])
plot_prc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')
plot_prc("Train Resampled", train_labels, train_predictions_resampled, color=colors[2])
plot_prc("Test Resampled", test_labels, test_predictions_resampled, color=colors[2], linestyle='--')
plt.legend(loc='lower right');
Applying this tutorial to your problem
Imbalanced data classification is an inherently difficult task since there are so few samples to learn from. You should always start with the data first and do your best to collect as many samples as possible and give substantial thought to what features may be relevant so the model can get the most out of your minority class. At some point your model may struggle to improve and yield the results you want, so it is important to keep in mind the context of your problem and the trade offs between different types of errors.