Implement Differential Privacy with TensorFlow Privacy

View on TensorFlow.org

Run in Google Colab

View source on GitHub

Download notebook

Overview

Differential privacy (DP) is a framework for measuring the privacy guarantees provided by an algorithm. Through the lens of differential privacy, you can design machine learning algorithms that responsibly train models on private data. Learning with differential privacy provides measurable guarantees of privacy, helping to mitigate the risk of exposing sensitive training data in machine learning. Intuitively, a model trained with differential privacy should not be affected by any single training example, or small set of training examples, in its data set. This helps mitigate the risk of exposing sensitive training data in ML.

The basic idea of this approach, called differentially private stochastic gradient descent (DP-SGD), is to modify the gradients used in stochastic gradient descent (SGD), which lies at the core of almost all deep learning algorithms. Models trained with DP-SGD provide provable differential privacy guarantees for their input data. There are two modifications made to the vanilla SGD algorithm:

First, the sensitivity of each gradient needs to be bounded. In other words, you need to limit how much each individual training point sampled in a minibatch can influence gradient computations and the resulting updates applied to model parameters. This can be done by clipping each gradient computed on each training point.
Random noise is sampled and added to the clipped gradients to make it statistically impossible to know whether or not a particular data point was included in the training dataset by comparing the updates SGD applies when it operates with or without this particular data point in the training dataset.

This tutorial uses tf.keras to train a convolutional neural network (CNN) to recognize handwritten digits with the DP-SGD optimizer provided by the TensorFlow Privacy library. TensorFlow Privacy provides code that wraps an existing TensorFlow optimizer to create a variant that implements DP-SGD.

Setup

Begin by importing the necessary libraries:

import tensorflow as tf
tf.compat.v1.disable_v2_behavior()

import numpy as np

tf.get_logger().setLevel('ERROR')

2022-12-12 10:43:10.411464: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2022-12-12 10:43:10.411665: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2022-12-12 10:43:10.411681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term

Install TensorFlow Privacy.

pip install tensorflow-privacy

import tensorflow_privacy

from tensorflow_privacy.privacy.analysis import compute_dp_sgd_privacy

Load and pre-process the dataset

Load the MNIST dataset and prepare the data for training.

train, test = tf.keras.datasets.mnist.load_data()
train_data, train_labels = train
test_data, test_labels = test

train_data = np.array(train_data, dtype=np.float32) / 255
test_data = np.array(test_data, dtype=np.float32) / 255

train_data = train_data.reshape(train_data.shape[0], 28, 28, 1)
test_data = test_data.reshape(test_data.shape[0], 28, 28, 1)

train_labels = np.array(train_labels, dtype=np.int32)
test_labels = np.array(test_labels, dtype=np.int32)

train_labels = tf.keras.utils.to_categorical(train_labels, num_classes=10)
test_labels = tf.keras.utils.to_categorical(test_labels, num_classes=10)

assert train_data.min() == 0.
assert train_data.max() == 1.
assert test_data.min() == 0.
assert test_data.max() == 1.

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
6766592/11490434 [================>.............] - ETA: 0s
11490434/11490434 [==============================] - 0s 0us/step

Define the hyperparameters

Set learning model hyperparamter values.

epochs = 3
batch_size = 250

DP-SGD has three privacy-specific hyperparameters and one existing hyperamater that you must tune:

l2_norm_clip (float) - The maximum Euclidean (L2) norm of each gradient that is applied to update model parameters. This hyperparameter is used to bound the optimizer's sensitivity to individual training points.
noise_multiplier (float) - The amount of noise sampled and added to gradients during training. Generally, more noise results in better privacy (often, but not necessarily, at the expense of lower utility).
microbatches (int) - Each batch of data is split in smaller units called microbatches. By default, each microbatch should contain a single training example. This allows us to clip gradients on a per-example basis rather than after they have been averaged across the minibatch. This in turn decreases the (negative) effect of clipping on signal found in the gradient and typically maximizes utility. However, computational overhead can be reduced by increasing the size of microbatches to include more than one training examples. The average gradient across these multiple training examples is then clipped. The total number of examples consumed in a batch, i.e., one step of gradient descent, remains the same. The number of microbatches should evenly divide the batch size.
learning_rate (float) - This hyperparameter already exists in vanilla SGD. The higher the learning rate, the more each update matters. If the updates are noisy (such as when the additive noise is large compared to the clipping threshold), a low learning rate may help the training procedure converge.

Use the hyperparameter values below to obtain a reasonably accurate model (95% test accuracy):

l2_norm_clip = 1.5
noise_multiplier = 1.3
num_microbatches = 250
learning_rate = 0.25

if batch_size % num_microbatches != 0:
  raise ValueError('Batch size should be an integer multiple of the number of microbatches')

Build the model

Define a convolutional neural network as the learning model.

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(16, 8,
                           strides=2,
                           padding='same',
                           activation='relu',
                           input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPool2D(2, 1),
    tf.keras.layers.Conv2D(32, 4,
                           strides=2,
                           padding='valid',
                           activation='relu'),
    tf.keras.layers.MaxPool2D(2, 1),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(10)
])

Define the optimizer and loss function for the learning model. Compute the loss as a vector of losses per-example rather than as the mean over a minibatch to support gradient manipulation over each training point.

optimizer = tensorflow_privacy.DPKerasSGDOptimizer(
    l2_norm_clip=l2_norm_clip,
    noise_multiplier=noise_multiplier,
    num_microbatches=num_microbatches,
    learning_rate=learning_rate)

loss = tf.keras.losses.CategoricalCrossentropy(
    from_logits=True, reduction=tf.losses.Reduction.NONE)

Train the model

model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

model.fit(train_data, train_labels,
          epochs=epochs,
          validation_data=(test_data, test_labels),
          batch_size=batch_size)

Train on 60000 samples, validate on 10000 samples
Epoch 1/3
2022-12-12 10:43:20.126016: W tensorflow/c/c_api.cc:291] Operation '{name:'training/SGD/momentum/Assign' id:500 op device:{requested: '', assigned: ''} def:{ { {node training/SGD/momentum/Assign} } = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](training/SGD/momentum, training/SGD/momentum/Initializer/initial_value)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
250/60000 [..............................] - ETA: 9:24 - loss: 2.3168 - acc: 0.0840
500/60000 [..............................] - ETA: 5:17 - loss: 2.3069 - acc: 0.0960
750/60000 [..............................] - ETA: 3:55 - loss: 2.3018 - acc: 0.0987
1000/60000 [..............................] - ETA: 3:14 - loss: 2.2993 - acc: 0.0950
1250/60000 [..............................] - ETA: 2:49 - loss: 2.2964 - acc: 0.0960
1500/60000 [..............................] - ETA: 2:33 - loss: 2.2895 - acc: 0.1040
1750/60000 [..............................] - ETA: 2:21 - loss: 2.2847 - acc: 0.1057
2000/60000 [>.............................] - ETA: 2:11 - loss: 2.2816 - acc: 0.1170
2250/60000 [>.............................] - ETA: 2:04 - loss: 2.2776 - acc: 0.1284
2500/60000 [>.............................] - ETA: 1:59 - loss: 2.2721 - acc: 0.1352
2750/60000 [>.............................] - ETA: 1:54 - loss: 2.2694 - acc: 0.1422
3000/60000 [>.............................] - ETA: 1:50 - loss: 2.2647 - acc: 0.1533
3250/60000 [>.............................] - ETA: 1:47 - loss: 2.2595 - acc: 0.1631
3500/60000 [>.............................] - ETA: 1:44 - loss: 2.2543 - acc: 0.1729
3750/60000 [>.............................] - ETA: 1:41 - loss: 2.2479 - acc: 0.1872
4000/60000 [=>............................] - ETA: 1:38 - loss: 2.2433 - acc: 0.1920
4250/60000 [=>............................] - ETA: 1:36 - loss: 2.2389 - acc: 0.1967
4500/60000 [=>............................] - ETA: 1:35 - loss: 2.2318 - acc: 0.2058
4750/60000 [=>............................] - ETA: 1:33 - loss: 2.2261 - acc: 0.2131
5000/60000 [=>............................] - ETA: 1:31 - loss: 2.2197 - acc: 0.2202
5250/60000 [=>............................] - ETA: 1:30 - loss: 2.2153 - acc: 0.2223
5500/60000 [=>............................] - ETA: 1:28 - loss: 2.2092 - acc: 0.2318
5750/60000 [=>............................] - ETA: 1:27 - loss: 2.2017 - acc: 0.2423
6000/60000 [==>...........................] - ETA: 1:26 - loss: 2.1956 - acc: 0.2483
6250/60000 [==>...........................] - ETA: 1:25 - loss: 2.1869 - acc: 0.2584
6500/60000 [==>...........................] - ETA: 1:23 - loss: 2.1807 - acc: 0.2672
6750/60000 [==>...........................] - ETA: 1:22 - loss: 2.1729 - acc: 0.2733
7000/60000 [==>...........................] - ETA: 1:21 - loss: 2.1655 - acc: 0.2786
7250/60000 [==>...........................] - ETA: 1:21 - loss: 2.1600 - acc: 0.2822
7500/60000 [==>...........................] - ETA: 1:20 - loss: 2.1525 - acc: 0.2869
7750/60000 [==>...........................] - ETA: 1:19 - loss: 2.1449 - acc: 0.2920
8000/60000 [===>..........................] - ETA: 1:18 - loss: 2.1364 - acc: 0.2979
8250/60000 [===>..........................] - ETA: 1:17 - loss: 2.1262 - acc: 0.3055
8500/60000 [===>..........................] - ETA: 1:16 - loss: 2.1186 - acc: 0.3104
8750/60000 [===>..........................] - ETA: 1:16 - loss: 2.1090 - acc: 0.3166
9000/60000 [===>..........................] - ETA: 1:15 - loss: 2.0995 - acc: 0.3222
9250/60000 [===>..........................] - ETA: 1:14 - loss: 2.0890 - acc: 0.3266
9500/60000 [===>..........................] - ETA: 1:14 - loss: 2.0798 - acc: 0.3305
9750/60000 [===>..........................] - ETA: 1:13 - loss: 2.0705 - acc: 0.3338
10000/60000 [====>.........................] - ETA: 1:12 - loss: 2.0621 - acc: 0.3376
10250/60000 [====>.........................] - ETA: 1:12 - loss: 2.0518 - acc: 0.3446
10500/60000 [====>.........................] - ETA: 1:11 - loss: 2.0433 - acc: 0.3477
10750/60000 [====>.........................] - ETA: 1:10 - loss: 2.0328 - acc: 0.3527
11000/60000 [====>.........................] - ETA: 1:10 - loss: 2.0246 - acc: 0.3555
11250/60000 [====>.........................] - ETA: 1:09 - loss: 2.0148 - acc: 0.3605
11500/60000 [====>.........................] - ETA: 1:09 - loss: 2.0066 - acc: 0.3631
11750/60000 [====>.........................] - ETA: 1:08 - loss: 1.9972 - acc: 0.3676
12000/60000 [=====>........................] - ETA: 1:08 - loss: 1.9862 - acc: 0.3713
12250/60000 [=====>........................] - ETA: 1:07 - loss: 1.9753 - acc: 0.3763
12500/60000 [=====>........................] - ETA: 1:07 - loss: 1.9653 - acc: 0.3808
12750/60000 [=====>........................] - ETA: 1:06 - loss: 1.9545 - acc: 0.3844
13000/60000 [=====>........................] - ETA: 1:06 - loss: 1.9455 - acc: 0.3875
13250/60000 [=====>........................] - ETA: 1:05 - loss: 1.9363 - acc: 0.3910
13500/60000 [=====>........................] - ETA: 1:05 - loss: 1.9280 - acc: 0.3944
13750/60000 [=====>........................] - ETA: 1:04 - loss: 1.9169 - acc: 0.3995
14000/60000 [======>.......................] - ETA: 1:04 - loss: 1.9085 - acc: 0.4021
14250/60000 [======>.......................] - ETA: 1:03 - loss: 1.8997 - acc: 0.4058
14500/60000 [======>.......................] - ETA: 1:03 - loss: 1.8887 - acc: 0.4108
14750/60000 [======>.......................] - ETA: 1:02 - loss: 1.8819 - acc: 0.4117
15000/60000 [======>.......................] - ETA: 1:02 - loss: 1.8709 - acc: 0.4160
15250/60000 [======>.......................] - ETA: 1:02 - loss: 1.8619 - acc: 0.4189
15500/60000 [======>.......................] - ETA: 1:01 - loss: 1.8541 - acc: 0.4217
15750/60000 [======>.......................] - ETA: 1:01 - loss: 1.8441 - acc: 0.4248
16000/60000 [=======>......................] - ETA: 1:00 - loss: 1.8349 - acc: 0.4283
16250/60000 [=======>......................] - ETA: 1:00 - loss: 1.8267 - acc: 0.4309
16500/60000 [=======>......................] - ETA: 59s - loss: 1.8178 - acc: 0.4341
16750/60000 [=======>......................] - ETA: 59s - loss: 1.8077 - acc: 0.4376
17000/60000 [=======>......................] - ETA: 58s - loss: 1.7994 - acc: 0.4399
17250/60000 [=======>......................] - ETA: 58s - loss: 1.7895 - acc: 0.4435
17500/60000 [=======>......................] - ETA: 58s - loss: 1.7798 - acc: 0.4467
17750/60000 [=======>......................] - ETA: 57s - loss: 1.7714 - acc: 0.4493
18000/60000 [========>.....................] - ETA: 57s - loss: 1.7629 - acc: 0.4514
18250/60000 [========>.....................] - ETA: 56s - loss: 1.7538 - acc: 0.4547
18500/60000 [========>.....................] - ETA: 56s - loss: 1.7445 - acc: 0.4581
18750/60000 [========>.....................] - ETA: 56s - loss: 1.7356 - acc: 0.4605
19000/60000 [========>.....................] - ETA: 55s - loss: 1.7286 - acc: 0.4619
19250/60000 [========>.....................] - ETA: 55s - loss: 1.7198 - acc: 0.4648
19500/60000 [========>.....................] - ETA: 54s - loss: 1.7115 - acc: 0.4679
19750/60000 [========>.....................] - ETA: 54s - loss: 1.7038 - acc: 0.4698
20000/60000 [=========>....................] - ETA: 54s - loss: 1.6975 - acc: 0.4717
20250/60000 [=========>....................] - ETA: 53s - loss: 1.6899 - acc: 0.4745
20500/60000 [=========>....................] - ETA: 53s - loss: 1.6831 - acc: 0.4762
20750/60000 [=========>....................] - ETA: 53s - loss: 1.6745 - acc: 0.4795
21000/60000 [=========>....................] - ETA: 52s - loss: 1.6667 - acc: 0.4823
21250/60000 [=========>....................] - ETA: 52s - loss: 1.6596 - acc: 0.4845
21500/60000 [=========>....................] - ETA: 51s - loss: 1.6521 - acc: 0.4871
21750/60000 [=========>....................] - ETA: 51s - loss: 1.6465 - acc: 0.4884
22000/60000 [==========>...................] - ETA: 51s - loss: 1.6380 - acc: 0.4908
22250/60000 [==========>...................] - ETA: 50s - loss: 1.6310 - acc: 0.4927
22500/60000 [==========>...................] - ETA: 50s - loss: 1.6237 - acc: 0.4948
22750/60000 [==========>...................] - ETA: 50s - loss: 1.6156 - acc: 0.4974
23000/60000 [==========>...................] - ETA: 49s - loss: 1.6079 - acc: 0.4994
23250/60000 [==========>...................] - ETA: 49s - loss: 1.5999 - acc: 0.5017
23500/60000 [==========>...................] - ETA: 48s - loss: 1.5931 - acc: 0.5037
23750/60000 [==========>...................] - ETA: 48s - loss: 1.5847 - acc: 0.5066
24000/60000 [===========>..................] - ETA: 48s - loss: 1.5787 - acc: 0.5084
24250/60000 [===========>..................] - ETA: 47s - loss: 1.5714 - acc: 0.5107
24500/60000 [===========>..................] - ETA: 47s - loss: 1.5652 - acc: 0.5130
24750/60000 [===========>..................] - ETA: 47s - loss: 1.5583 - acc: 0.5151
25000/60000 [===========>..................] - ETA: 46s - loss: 1.5536 - acc: 0.5168
25250/60000 [===========>..................] - ETA: 46s - loss: 1.5467 - acc: 0.5185
25500/60000 [===========>..................] - ETA: 46s - loss: 1.5409 - acc: 0.5201
25750/60000 [===========>..................] - ETA: 45s - loss: 1.5340 - acc: 0.5222
26000/60000 [============>.................] - ETA: 45s - loss: 1.5268 - acc: 0.5246
26250/60000 [============>.................] - ETA: 45s - loss: 1.5195 - acc: 0.5266
26500/60000 [============>.................] - ETA: 44s - loss: 1.5136 - acc: 0.5279
26750/60000 [============>.................] - ETA: 44s - loss: 1.5072 - acc: 0.5298
27000/60000 [============>.................] - ETA: 43s - loss: 1.5000 - acc: 0.5324
27250/60000 [============>.................] - ETA: 43s - loss: 1.4948 - acc: 0.5337
27500/60000 [============>.................] - ETA: 43s - loss: 1.4898 - acc: 0.5349
27750/60000 [============>.................] - ETA: 42s - loss: 1.4829 - acc: 0.5372
28000/60000 [=============>................] - ETA: 42s - loss: 1.4763 - acc: 0.5393
28250/60000 [=============>................] - ETA: 42s - loss: 1.4706 - acc: 0.5409
28500/60000 [=============>................] - ETA: 41s - loss: 1.4643 - acc: 0.5427
28750/60000 [=============>................] - ETA: 41s - loss: 1.4586 - acc: 0.5443
29000/60000 [=============>................] - ETA: 41s - loss: 1.4520 - acc: 0.5463
29250/60000 [=============>................] - ETA: 40s - loss: 1.4466 - acc: 0.5476
29500/60000 [=============>................] - ETA: 40s - loss: 1.4411 - acc: 0.5495
29750/60000 [=============>................] - ETA: 40s - loss: 1.4343 - acc: 0.5516
30000/60000 [==============>...............] - ETA: 39s - loss: 1.4293 - acc: 0.5529
30250/60000 [==============>...............] - ETA: 39s - loss: 1.4225 - acc: 0.5551
30500/60000 [==============>...............] - ETA: 39s - loss: 1.4169 - acc: 0.5568
30750/60000 [==============>...............] - ETA: 38s - loss: 1.4115 - acc: 0.5583
31000/60000 [==============>...............] - ETA: 38s - loss: 1.4074 - acc: 0.5594
31250/60000 [==============>...............] - ETA: 38s - loss: 1.4022 - acc: 0.5610
31500/60000 [==============>...............] - ETA: 37s - loss: 1.3968 - acc: 0.5627
31750/60000 [==============>...............] - ETA: 37s - loss: 1.3922 - acc: 0.5644
32000/60000 [===============>..............] - ETA: 37s - loss: 1.3869 - acc: 0.5661
32250/60000 [===============>..............] - ETA: 36s - loss: 1.3808 - acc: 0.5682
32500/60000 [===============>..............] - ETA: 36s - loss: 1.3757 - acc: 0.5697
32750/60000 [===============>..............] - ETA: 35s - loss: 1.3706 - acc: 0.5714
33000/60000 [===============>..............] - ETA: 35s - loss: 1.3662 - acc: 0.5727
33250/60000 [===============>..............] - ETA: 35s - loss: 1.3614 - acc: 0.5738
33500/60000 [===============>..............] - ETA: 34s - loss: 1.3562 - acc: 0.5750
33750/60000 [===============>..............] - ETA: 34s - loss: 1.3519 - acc: 0.5764
34000/60000 [================>.............] - ETA: 34s - loss: 1.3469 - acc: 0.5782
34250/60000 [================>.............] - ETA: 33s - loss: 1.3415 - acc: 0.5797
34500/60000 [================>.............] - ETA: 33s - loss: 1.3367 - acc: 0.5812
34750/60000 [================>.............] - ETA: 33s - loss: 1.3317 - acc: 0.5827
35000/60000 [================>.............] - ETA: 32s - loss: 1.3269 - acc: 0.5842
35250/60000 [================>.............] - ETA: 32s - loss: 1.3224 - acc: 0.5852
35500/60000 [================>.............] - ETA: 32s - loss: 1.3175 - acc: 0.5866
35750/60000 [================>.............] - ETA: 31s - loss: 1.3127 - acc: 0.5882
36000/60000 [=================>............] - ETA: 31s - loss: 1.3073 - acc: 0.5898
36250/60000 [=================>............] - ETA: 31s - loss: 1.3030 - acc: 0.5909
36500/60000 [=================>............] - ETA: 30s - loss: 1.2981 - acc: 0.5924
36750/60000 [=================>............] - ETA: 30s - loss: 1.2943 - acc: 0.5936
37000/60000 [=================>............] - ETA: 30s - loss: 1.2900 - acc: 0.5949
37250/60000 [=================>............] - ETA: 29s - loss: 1.2850 - acc: 0.5962
37500/60000 [=================>............] - ETA: 29s - loss: 1.2807 - acc: 0.5975
37750/60000 [=================>............] - ETA: 29s - loss: 1.2765 - acc: 0.5987
38000/60000 [==================>...........] - ETA: 28s - loss: 1.2721 - acc: 0.6000
38250/60000 [==================>...........] - ETA: 28s - loss: 1.2687 - acc: 0.6011
38500/60000 [==================>...........] - ETA: 28s - loss: 1.2644 - acc: 0.6023
38750/60000 [==================>...........] - ETA: 27s - loss: 1.2608 - acc: 0.6034
39000/60000 [==================>...........] - ETA: 27s - loss: 1.2576 - acc: 0.6045
39250/60000 [==================>...........] - ETA: 27s - loss: 1.2536 - acc: 0.6056
39500/60000 [==================>...........] - ETA: 26s - loss: 1.2501 - acc: 0.6067
39750/60000 [==================>...........] - ETA: 26s - loss: 1.2461 - acc: 0.6081
40000/60000 [===================>..........] - ETA: 26s - loss: 1.2433 - acc: 0.6091
40250/60000 [===================>..........] - ETA: 25s - loss: 1.2394 - acc: 0.6102
40500/60000 [===================>..........] - ETA: 25s - loss: 1.2359 - acc: 0.6113
40750/60000 [===================>..........] - ETA: 25s - loss: 1.2323 - acc: 0.6124
41000/60000 [===================>..........] - ETA: 24s - loss: 1.2297 - acc: 0.6132
41250/60000 [===================>..........] - ETA: 24s - loss: 1.2258 - acc: 0.6145
41500/60000 [===================>..........] - ETA: 24s - loss: 1.2233 - acc: 0.6152
41750/60000 [===================>..........] - ETA: 23s - loss: 1.2212 - acc: 0.6161
42000/60000 [====================>.........] - ETA: 23s - loss: 1.2178 - acc: 0.6171
42250/60000 [====================>.........] - ETA: 23s - loss: 1.2160 - acc: 0.6177
42500/60000 [====================>.........] - ETA: 22s - loss: 1.2133 - acc: 0.6184
42750/60000 [====================>.........] - ETA: 22s - loss: 1.2103 - acc: 0.6197
43000/60000 [====================>.........] - ETA: 22s - loss: 1.2069 - acc: 0.6206
43250/60000 [====================>.........] - ETA: 21s - loss: 1.2039 - acc: 0.6214
43500/60000 [====================>.........] - ETA: 21s - loss: 1.2006 - acc: 0.6225
43750/60000 [====================>.........] - ETA: 21s - loss: 1.1970 - acc: 0.6237
44000/60000 [=====================>........] - ETA: 20s - loss: 1.1937 - acc: 0.6246
44250/60000 [=====================>........] - ETA: 20s - loss: 1.1905 - acc: 0.6258
44500/60000 [=====================>........] - ETA: 20s - loss: 1.1876 - acc: 0.6268
44750/60000 [=====================>........] - ETA: 19s - loss: 1.1844 - acc: 0.6279
45000/60000 [=====================>........] - ETA: 19s - loss: 1.1818 - acc: 0.6288
45250/60000 [=====================>........] - ETA: 19s - loss: 1.1783 - acc: 0.6298
45500/60000 [=====================>........] - ETA: 18s - loss: 1.1766 - acc: 0.6305
45750/60000 [=====================>........] - ETA: 18s - loss: 1.1735 - acc: 0.6315
46000/60000 [======================>.......] - ETA: 18s - loss: 1.1708 - acc: 0.6322
46250/60000 [======================>.......] - ETA: 17s - loss: 1.1683 - acc: 0.6331
46500/60000 [======================>.......] - ETA: 17s - loss: 1.1645 - acc: 0.6342
46750/60000 [======================>.......] - ETA: 17s - loss: 1.1617 - acc: 0.6350
47000/60000 [======================>.......] - ETA: 16s - loss: 1.1595 - acc: 0.6357
47250/60000 [======================>.......] - ETA: 16s - loss: 1.1564 - acc: 0.6367
47500/60000 [======================>.......] - ETA: 16s - loss: 1.1533 - acc: 0.6378
47750/60000 [======================>.......] - ETA: 15s - loss: 1.1504 - acc: 0.6388
48000/60000 [=======================>......] - ETA: 15s - loss: 1.1473 - acc: 0.6399
48250/60000 [=======================>......] - ETA: 15s - loss: 1.1446 - acc: 0.6407
48500/60000 [=======================>......] - ETA: 14s - loss: 1.1416 - acc: 0.6418
48750/60000 [=======================>......] - ETA: 14s - loss: 1.1387 - acc: 0.6427
49000/60000 [=======================>......] - ETA: 14s - loss: 1.1358 - acc: 0.6437
49250/60000 [=======================>......] - ETA: 13s - loss: 1.1336 - acc: 0.6444
49500/60000 [=======================>......] - ETA: 13s - loss: 1.1310 - acc: 0.6453
49750/60000 [=======================>......] - ETA: 13s - loss: 1.1284 - acc: 0.6463
50000/60000 [========================>.....] - ETA: 12s - loss: 1.1252 - acc: 0.6473
50250/60000 [========================>.....] - ETA: 12s - loss: 1.1227 - acc: 0.6483
50500/60000 [========================>.....] - ETA: 12s - loss: 1.1198 - acc: 0.6493
50750/60000 [========================>.....] - ETA: 12s - loss: 1.1172 - acc: 0.6501
51000/60000 [========================>.....] - ETA: 11s - loss: 1.1142 - acc: 0.6512
51250/60000 [========================>.....] - ETA: 11s - loss: 1.1112 - acc: 0.6520
51500/60000 [========================>.....] - ETA: 11s - loss: 1.1088 - acc: 0.6529
51750/60000 [========================>.....] - ETA: 10s - loss: 1.1058 - acc: 0.6538
52000/60000 [=========================>....] - ETA: 10s - loss: 1.1033 - acc: 0.6548
52250/60000 [=========================>....] - ETA: 10s - loss: 1.1006 - acc: 0.6557
52500/60000 [=========================>....] - ETA: 9s - loss: 1.0977 - acc: 0.6566
52750/60000 [=========================>....] - ETA: 9s - loss: 1.0948 - acc: 0.6575
53000/60000 [=========================>....] - ETA: 9s - loss: 1.0919 - acc: 0.6585
53250/60000 [=========================>....] - ETA: 8s - loss: 1.0900 - acc: 0.6590
53500/60000 [=========================>....] - ETA: 8s - loss: 1.0869 - acc: 0.6599
53750/60000 [=========================>....] - ETA: 8s - loss: 1.0840 - acc: 0.6609
54000/60000 [==========================>...] - ETA: 7s - loss: 1.0809 - acc: 0.6618
54250/60000 [==========================>...] - ETA: 7s - loss: 1.0783 - acc: 0.6627
54500/60000 [==========================>...] - ETA: 7s - loss: 1.0756 - acc: 0.6636
54750/60000 [==========================>...] - ETA: 6s - loss: 1.0731 - acc: 0.6644
55000/60000 [==========================>...] - ETA: 6s - loss: 1.0705 - acc: 0.6651
55250/60000 [==========================>...] - ETA: 6s - loss: 1.0676 - acc: 0.6661
55500/60000 [==========================>...] - ETA: 5s - loss: 1.0651 - acc: 0.6669
55750/60000 [==========================>...] - ETA: 5s - loss: 1.0622 - acc: 0.6678
56000/60000 [===========================>..] - ETA: 5s - loss: 1.0603 - acc: 0.6685
56250/60000 [===========================>..] - ETA: 4s - loss: 1.0582 - acc: 0.6692
56500/60000 [===========================>..] - ETA: 4s - loss: 1.0555 - acc: 0.6700
56750/60000 [===========================>..] - ETA: 4s - loss: 1.0528 - acc: 0.6708
57000/60000 [===========================>..] - ETA: 3s - loss: 1.0505 - acc: 0.6717
57250/60000 [===========================>..] - ETA: 3s - loss: 1.0476 - acc: 0.6726
57500/60000 [===========================>..] - ETA: 3s - loss: 1.0454 - acc: 0.6734
57750/60000 [===========================>..] - ETA: 2s - loss: 1.0430 - acc: 0.6742
58000/60000 [============================>.] - ETA: 2s - loss: 1.0404 - acc: 0.6750
58250/60000 [============================>.] - ETA: 2s - loss: 1.0379 - acc: 0.6757
58500/60000 [============================>.] - ETA: 1s - loss: 1.0355 - acc: 0.6765
58750/60000 [============================>.] - ETA: 1s - loss: 1.0333 - acc: 0.6773
59000/60000 [============================>.] - ETA: 1s - loss: 1.0310 - acc: 0.6780
59250/60000 [============================>.] - ETA: 0s - loss: 1.0286 - acc: 0.6787
59500/60000 [============================>.] - ETA: 0s - loss: 1.0261 - acc: 0.6794
59750/60000 [============================>.] - ETA: 0s - loss: 1.0240 - acc: 0.6802
60000/60000 [==============================] - ETA: 0s - loss: 1.0220 - acc: 0.6809
60000/60000 [==============================] - 78s 1ms/sample - loss: 1.0220 - acc: 0.6809 - val_loss: 0.5134 - val_acc: 0.8517
Epoch 2/3
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/keras/engine/training_v1.py:2333: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  updates = self.state_updates
2022-12-12 10:44:37.590033: W tensorflow/c/c_api.cc:291] Operation '{name:'loss/mul' id:157 op device:{requested: '', assigned: ''} def:{ { {node loss/mul} } = Mul[T=DT_FLOAT, _has_manual_control_dependencies=true](loss/mul/x, loss/dense_1_loss/weighted_loss/Mul)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
250/60000 [..............................] - ETA: 1:15 - loss: 0.4469 - acc: 0.8760
500/60000 [..............................] - ETA: 1:14 - loss: 0.5276 - acc: 0.8540
750/60000 [..............................] - ETA: 1:14 - loss: 0.4979 - acc: 0.8520
1000/60000 [..............................] - ETA: 1:13 - loss: 0.5146 - acc: 0.8520
1250/60000 [..............................] - ETA: 1:13 - loss: 0.5243 - acc: 0.8512
1500/60000 [..............................] - ETA: 1:12 - loss: 0.5116 - acc: 0.8520
1750/60000 [..............................] - ETA: 1:12 - loss: 0.5161 - acc: 0.8526
2000/60000 [>.............................] - ETA: 1:12 - loss: 0.5077 - acc: 0.8505
2250/60000 [>.............................] - ETA: 1:12 - loss: 0.5005 - acc: 0.8484
2500/60000 [>.............................] - ETA: 1:11 - loss: 0.5009 - acc: 0.8508
2750/60000 [>.............................] - ETA: 1:11 - loss: 0.4926 - acc: 0.8545
3000/60000 [>.............................] - ETA: 1:11 - loss: 0.4760 - acc: 0.8580
3250/60000 [>.............................] - ETA: 1:11 - loss: 0.4793 - acc: 0.8557
3500/60000 [>.............................] - ETA: 1:11 - loss: 0.4743 - acc: 0.8551
3750/60000 [>.............................] - ETA: 1:10 - loss: 0.4730 - acc: 0.8544
4000/60000 [=>............................] - ETA: 1:10 - loss: 0.4775 - acc: 0.8545
4250/60000 [=>............................] - ETA: 1:09 - loss: 0.4777 - acc: 0.8558
4500/60000 [=>............................] - ETA: 1:09 - loss: 0.4697 - acc: 0.8587
4750/60000 [=>............................] - ETA: 1:09 - loss: 0.4697 - acc: 0.8585
5000/60000 [=>............................] - ETA: 1:08 - loss: 0.4616 - acc: 0.8604
5250/60000 [=>............................] - ETA: 1:08 - loss: 0.4651 - acc: 0.8592
5500/60000 [=>............................] - ETA: 1:08 - loss: 0.4663 - acc: 0.8595
5750/60000 [=>............................] - ETA: 1:08 - loss: 0.4572 - acc: 0.8624
6000/60000 [==>...........................] - ETA: 1:07 - loss: 0.4637 - acc: 0.8615
6250/60000 [==>...........................] - ETA: 1:07 - loss: 0.4738 - acc: 0.8597
6500/60000 [==>...........................] - ETA: 1:07 - loss: 0.4823 - acc: 0.8575
6750/60000 [==>...........................] - ETA: 1:06 - loss: 0.4845 - acc: 0.8569
7000/60000 [==>...........................] - ETA: 1:06 - loss: 0.4871 - acc: 0.8560
7250/60000 [==>...........................] - ETA: 1:06 - loss: 0.4804 - acc: 0.8581
7500/60000 [==>...........................] - ETA: 1:06 - loss: 0.4800 - acc: 0.8588
7750/60000 [==>...........................] - ETA: 1:05 - loss: 0.4783 - acc: 0.8601
8000/60000 [===>..........................] - ETA: 1:05 - loss: 0.4813 - acc: 0.8599
8250/60000 [===>..........................] - ETA: 1:04 - loss: 0.4816 - acc: 0.8605
8500/60000 [===>..........................] - ETA: 1:04 - loss: 0.4798 - acc: 0.8612
8750/60000 [===>..........................] - ETA: 1:04 - loss: 0.4764 - acc: 0.8619
9000/60000 [===>..........................] - ETA: 1:04 - loss: 0.4884 - acc: 0.8597
9250/60000 [===>..........................] - ETA: 1:03 - loss: 0.4905 - acc: 0.8592
9500/60000 [===>..........................] - ETA: 1:03 - loss: 0.4948 - acc: 0.8582
9750/60000 [===>..........................] - ETA: 1:03 - loss: 0.4971 - acc: 0.8591
10000/60000 [====>.........................] - ETA: 1:02 - loss: 0.4980 - acc: 0.8589
10250/60000 [====>.........................] - ETA: 1:02 - loss: 0.5019 - acc: 0.8585
10500/60000 [====>.........................] - ETA: 1:02 - loss: 0.4999 - acc: 0.8585
10750/60000 [====>.........................] - ETA: 1:01 - loss: 0.5012 - acc: 0.8589
11000/60000 [====>.........................] - ETA: 1:01 - loss: 0.5043 - acc: 0.8583
11250/60000 [====>.........................] - ETA: 1:01 - loss: 0.5061 - acc: 0.8580
11500/60000 [====>.........................] - ETA: 1:00 - loss: 0.5058 - acc: 0.8589
11750/60000 [====>.........................] - ETA: 1:00 - loss: 0.5049 - acc: 0.8594
12000/60000 [=====>........................] - ETA: 1:00 - loss: 0.5061 - acc: 0.8595
12250/60000 [=====>........................] - ETA: 59s - loss: 0.5023 - acc: 0.8602
12500/60000 [=====>........................] - ETA: 59s - loss: 0.5004 - acc: 0.8607
12750/60000 [=====>........................] - ETA: 59s - loss: 0.4998 - acc: 0.8606
13000/60000 [=====>........................] - ETA: 58s - loss: 0.4972 - acc: 0.8608
13250/60000 [=====>........................] - ETA: 58s - loss: 0.5000 - acc: 0.8598
13500/60000 [=====>........................] - ETA: 58s - loss: 0.4991 - acc: 0.8599
13750/60000 [=====>........................] - ETA: 57s - loss: 0.4968 - acc: 0.8604
14000/60000 [======>.......................] - ETA: 57s - loss: 0.4977 - acc: 0.8607
14250/60000 [======>.......................] - ETA: 57s - loss: 0.4980 - acc: 0.8609
14500/60000 [======>.......................] - ETA: 56s - loss: 0.4970 - acc: 0.8614
14750/60000 [======>.......................] - ETA: 56s - loss: 0.4968 - acc: 0.8612
15000/60000 [======>.......................] - ETA: 56s - loss: 0.4968 - acc: 0.8613
15250/60000 [======>.......................] - ETA: 55s - loss: 0.4985 - acc: 0.8609
15500/60000 [======>.......................] - ETA: 55s - loss: 0.5015 - acc: 0.8604
15750/60000 [======>.......................] - ETA: 55s - loss: 0.5023 - acc: 0.8599
16000/60000 [=======>......................] - ETA: 55s - loss: 0.5051 - acc: 0.8591
16250/60000 [=======>......................] - ETA: 54s - loss: 0.5056 - acc: 0.8589
16500/60000 [=======>......................] - ETA: 54s - loss: 0.5072 - acc: 0.8586
16750/60000 [=======>......................] - ETA: 54s - loss: 0.5050 - acc: 0.8592
17000/60000 [=======>......................] - ETA: 53s - loss: 0.5039 - acc: 0.8588
17250/60000 [=======>......................] - ETA: 53s - loss: 0.5025 - acc: 0.8595
17500/60000 [=======>......................] - ETA: 53s - loss: 0.5008 - acc: 0.8599
17750/60000 [=======>......................] - ETA: 52s - loss: 0.4997 - acc: 0.8599
18000/60000 [========>.....................] - ETA: 52s - loss: 0.4985 - acc: 0.8605
18250/60000 [========>.....................] - ETA: 52s - loss: 0.4988 - acc: 0.8606
18500/60000 [========>.....................] - ETA: 51s - loss: 0.4979 - acc: 0.8610
18750/60000 [========>.....................] - ETA: 51s - loss: 0.4982 - acc: 0.8612
19000/60000 [========>.....................] - ETA: 51s - loss: 0.4977 - acc: 0.8616
19250/60000 [========>.....................] - ETA: 51s - loss: 0.4976 - acc: 0.8617
19500/60000 [========>.....................] - ETA: 50s - loss: 0.4971 - acc: 0.8615
19750/60000 [========>.....................] - ETA: 50s - loss: 0.4982 - acc: 0.8615
20000/60000 [=========>....................] - ETA: 50s - loss: 0.4984 - acc: 0.8615
20250/60000 [=========>....................] - ETA: 49s - loss: 0.4999 - acc: 0.8614
20500/60000 [=========>....................] - ETA: 49s - loss: 0.5009 - acc: 0.8612
20750/60000 [=========>....................] - ETA: 49s - loss: 0.5004 - acc: 0.8612
21000/60000 [=========>....................] - ETA: 48s - loss: 0.5003 - acc: 0.8614
21250/60000 [=========>....................] - ETA: 48s - loss: 0.4982 - acc: 0.8621
21500/60000 [=========>....................] - ETA: 48s - loss: 0.4989 - acc: 0.8621
21750/60000 [=========>....................] - ETA: 47s - loss: 0.4990 - acc: 0.8624
22000/60000 [==========>...................] - ETA: 47s - loss: 0.4987 - acc: 0.8622
22250/60000 [==========>...................] - ETA: 47s - loss: 0.5011 - acc: 0.8622
22500/60000 [==========>...................] - ETA: 46s - loss: 0.5012 - acc: 0.8624
22750/60000 [==========>...................] - ETA: 46s - loss: 0.5024 - acc: 0.8622
23000/60000 [==========>...................] - ETA: 46s - loss: 0.5033 - acc: 0.8623
23250/60000 [==========>...................] - ETA: 46s - loss: 0.5020 - acc: 0.8626
23500/60000 [==========>...................] - ETA: 45s - loss: 0.5016 - acc: 0.8625
23750/60000 [==========>...................] - ETA: 45s - loss: 0.5003 - acc: 0.8627
24000/60000 [===========>..................] - ETA: 45s - loss: 0.5010 - acc: 0.8627
24250/60000 [===========>..................] - ETA: 44s - loss: 0.5016 - acc: 0.8628
24500/60000 [===========>..................] - ETA: 44s - loss: 0.5019 - acc: 0.8627
24750/60000 [===========>..................] - ETA: 44s - loss: 0.5021 - acc: 0.8625
25000/60000 [===========>..................] - ETA: 43s - loss: 0.5020 - acc: 0.8627
25250/60000 [===========>..................] - ETA: 43s - loss: 0.5042 - acc: 0.8625
25500/60000 [===========>..................] - ETA: 43s - loss: 0.5051 - acc: 0.8623
25750/60000 [===========>..................] - ETA: 42s - loss: 0.5039 - acc: 0.8626
26000/60000 [============>.................] - ETA: 42s - loss: 0.5045 - acc: 0.8628
26250/60000 [============>.................] - ETA: 42s - loss: 0.5041 - acc: 0.8629
26500/60000 [============>.................] - ETA: 41s - loss: 0.5038 - acc: 0.8630
26750/60000 [============>.................] - ETA: 41s - loss: 0.5034 - acc: 0.8632
27000/60000 [============>.................] - ETA: 41s - loss: 0.5043 - acc: 0.8631
27250/60000 [============>.................] - ETA: 41s - loss: 0.5032 - acc: 0.8633
27500/60000 [============>.................] - ETA: 40s - loss: 0.5034 - acc: 0.8634
27750/60000 [============>.................] - ETA: 40s - loss: 0.5023 - acc: 0.8637
28000/60000 [=============>................] - ETA: 40s - loss: 0.5012 - acc: 0.8640
28250/60000 [=============>................] - ETA: 39s - loss: 0.5013 - acc: 0.8640
28500/60000 [=============>................] - ETA: 39s - loss: 0.5020 - acc: 0.8639
28750/60000 [=============>................] - ETA: 39s - loss: 0.5024 - acc: 0.8640
29000/60000 [=============>................] - ETA: 38s - loss: 0.5005 - acc: 0.8646
29250/60000 [=============>................] - ETA: 38s - loss: 0.5004 - acc: 0.8645
29500/60000 [=============>................] - ETA: 38s - loss: 0.4992 - acc: 0.8647
29750/60000 [=============>................] - ETA: 37s - loss: 0.4980 - acc: 0.8650
30000/60000 [==============>...............] - ETA: 37s - loss: 0.4979 - acc: 0.8651
30250/60000 [==============>...............] - ETA: 37s - loss: 0.4982 - acc: 0.8651
30500/60000 [==============>...............] - ETA: 36s - loss: 0.4990 - acc: 0.8651
30750/60000 [==============>...............] - ETA: 36s - loss: 0.4991 - acc: 0.8653
31000/60000 [==============>...............] - ETA: 36s - loss: 0.4996 - acc: 0.8653
31250/60000 [==============>...............] - ETA: 35s - loss: 0.4987 - acc: 0.8655
31500/60000 [==============>...............] - ETA: 35s - loss: 0.4987 - acc: 0.8655
31750/60000 [==============>...............] - ETA: 35s - loss: 0.4981 - acc: 0.8656
32000/60000 [===============>..............] - ETA: 35s - loss: 0.4975 - acc: 0.8658
32250/60000 [===============>..............] - ETA: 34s - loss: 0.4969 - acc: 0.8660
32500/60000 [===============>..............] - ETA: 34s - loss: 0.4962 - acc: 0.8662
32750/60000 [===============>..............] - ETA: 34s - loss: 0.4960 - acc: 0.8662
33000/60000 [===============>..............] - ETA: 33s - loss: 0.4967 - acc: 0.8663
33250/60000 [===============>..............] - ETA: 33s - loss: 0.4957 - acc: 0.8666
33500/60000 [===============>..............] - ETA: 33s - loss: 0.4945 - acc: 0.8669
33750/60000 [===============>..............] - ETA: 32s - loss: 0.4941 - acc: 0.8670
34000/60000 [================>.............] - ETA: 32s - loss: 0.4940 - acc: 0.8672
34250/60000 [================>.............] - ETA: 32s - loss: 0.4938 - acc: 0.8672
34500/60000 [================>.............] - ETA: 31s - loss: 0.4932 - acc: 0.8675
34750/60000 [================>.............] - ETA: 31s - loss: 0.4918 - acc: 0.8677
35000/60000 [================>.............] - ETA: 31s - loss: 0.4924 - acc: 0.8679
35250/60000 [================>.............] - ETA: 30s - loss: 0.4909 - acc: 0.8682
35500/60000 [================>.............] - ETA: 30s - loss: 0.4896 - acc: 0.8683
35750/60000 [================>.............] - ETA: 30s - loss: 0.4897 - acc: 0.8683
36000/60000 [=================>............] - ETA: 30s - loss: 0.4896 - acc: 0.8684
36250/60000 [=================>............] - ETA: 29s - loss: 0.4902 - acc: 0.8683
36500/60000 [=================>............] - ETA: 29s - loss: 0.4898 - acc: 0.8684
36750/60000 [=================>............] - ETA: 29s - loss: 0.4906 - acc: 0.8682
37000/60000 [=================>............] - ETA: 28s - loss: 0.4902 - acc: 0.8682
37250/60000 [=================>............] - ETA: 28s - loss: 0.4901 - acc: 0.8683
37500/60000 [=================>............] - ETA: 28s - loss: 0.4901 - acc: 0.8683
37750/60000 [=================>............] - ETA: 27s - loss: 0.4894 - acc: 0.8686
38000/60000 [==================>...........] - ETA: 27s - loss: 0.4886 - acc: 0.8687
38250/60000 [==================>...........] - ETA: 27s - loss: 0.4888 - acc: 0.8685
38500/60000 [==================>...........] - ETA: 26s - loss: 0.4893 - acc: 0.8687
38750/60000 [==================>...........] - ETA: 26s - loss: 0.4895 - acc: 0.8686
39000/60000 [==================>...........] - ETA: 26s - loss: 0.4883 - acc: 0.8689
39250/60000 [==================>...........] - ETA: 25s - loss: 0.4879 - acc: 0.8690
39500/60000 [==================>...........] - ETA: 25s - loss: 0.4875 - acc: 0.8690
39750/60000 [==================>...........] - ETA: 25s - loss: 0.4869 - acc: 0.8691
40000/60000 [===================>..........] - ETA: 25s - loss: 0.4861 - acc: 0.8693
40250/60000 [===================>..........] - ETA: 24s - loss: 0.4863 - acc: 0.8694
40500/60000 [===================>..........] - ETA: 24s - loss: 0.4864 - acc: 0.8694
40750/60000 [===================>..........] - ETA: 24s - loss: 0.4860 - acc: 0.8695
41000/60000 [===================>..........] - ETA: 23s - loss: 0.4852 - acc: 0.8698
41250/60000 [===================>..........] - ETA: 23s - loss: 0.4855 - acc: 0.8697
41500/60000 [===================>..........] - ETA: 23s - loss: 0.4851 - acc: 0.8699
41750/60000 [===================>..........] - ETA: 22s - loss: 0.4850 - acc: 0.8699
42000/60000 [====================>.........] - ETA: 22s - loss: 0.4846 - acc: 0.8701
42250/60000 [====================>.........] - ETA: 22s - loss: 0.4848 - acc: 0.8701
42500/60000 [====================>.........] - ETA: 21s - loss: 0.4853 - acc: 0.8700
42750/60000 [====================>.........] - ETA: 21s - loss: 0.4840 - acc: 0.8703
43000/60000 [====================>.........] - ETA: 21s - loss: 0.4841 - acc: 0.8703
43250/60000 [====================>.........] - ETA: 20s - loss: 0.4839 - acc: 0.8704
43500/60000 [====================>.........] - ETA: 20s - loss: 0.4829 - acc: 0.8707
43750/60000 [====================>.........] - ETA: 20s - loss: 0.4823 - acc: 0.8708
44000/60000 [=====================>........] - ETA: 20s - loss: 0.4817 - acc: 0.8710
44250/60000 [=====================>........] - ETA: 19s - loss: 0.4814 - acc: 0.8711
44500/60000 [=====================>........] - ETA: 19s - loss: 0.4819 - acc: 0.8713
44750/60000 [=====================>........] - ETA: 19s - loss: 0.4817 - acc: 0.8714
45000/60000 [=====================>........] - ETA: 18s - loss: 0.4819 - acc: 0.8714
45250/60000 [=====================>........] - ETA: 18s - loss: 0.4820 - acc: 0.8716
45500/60000 [=====================>........] - ETA: 18s - loss: 0.4814 - acc: 0.8717
45750/60000 [=====================>........] - ETA: 17s - loss: 0.4814 - acc: 0.8719
46000/60000 [======================>.......] - ETA: 17s - loss: 0.4810 - acc: 0.8720
46250/60000 [======================>.......] - ETA: 17s - loss: 0.4796 - acc: 0.8724
46500/60000 [======================>.......] - ETA: 16s - loss: 0.4796 - acc: 0.8724
46750/60000 [======================>.......] - ETA: 16s - loss: 0.4790 - acc: 0.8725
47000/60000 [======================>.......] - ETA: 16s - loss: 0.4794 - acc: 0.8724
47250/60000 [======================>.......] - ETA: 15s - loss: 0.4804 - acc: 0.8723
47500/60000 [======================>.......] - ETA: 15s - loss: 0.4817 - acc: 0.8720
47750/60000 [======================>.......] - ETA: 15s - loss: 0.4818 - acc: 0.8721
48000/60000 [=======================>......] - ETA: 15s - loss: 0.4810 - acc: 0.8722
48250/60000 [=======================>......] - ETA: 14s - loss: 0.4805 - acc: 0.8723
48500/60000 [=======================>......] - ETA: 14s - loss: 0.4807 - acc: 0.8723
48750/60000 [=======================>......] - ETA: 14s - loss: 0.4810 - acc: 0.8722
49000/60000 [=======================>......] - ETA: 13s - loss: 0.4804 - acc: 0.8725
49250/60000 [=======================>......] - ETA: 13s - loss: 0.4800 - acc: 0.8726
49500/60000 [=======================>......] - ETA: 13s - loss: 0.4804 - acc: 0.8727
49750/60000 [=======================>......] - ETA: 12s - loss: 0.4807 - acc: 0.8727
50000/60000 [========================>.....] - ETA: 12s - loss: 0.4804 - acc: 0.8729
50250/60000 [========================>.....] - ETA: 12s - loss: 0.4797 - acc: 0.8731
50500/60000 [========================>.....] - ETA: 11s - loss: 0.4802 - acc: 0.8731
50750/60000 [========================>.....] - ETA: 11s - loss: 0.4795 - acc: 0.8732
51000/60000 [========================>.....] - ETA: 11s - loss: 0.4792 - acc: 0.8733
51250/60000 [========================>.....] - ETA: 10s - loss: 0.4788 - acc: 0.8734
51500/60000 [========================>.....] - ETA: 10s - loss: 0.4786 - acc: 0.8735
51750/60000 [========================>.....] - ETA: 10s - loss: 0.4788 - acc: 0.8735
52000/60000 [=========================>....] - ETA: 10s - loss: 0.4783 - acc: 0.8737
52250/60000 [=========================>....] - ETA: 9s - loss: 0.4787 - acc: 0.8736
52500/60000 [=========================>....] - ETA: 9s - loss: 0.4785 - acc: 0.8735
52750/60000 [=========================>....] - ETA: 9s - loss: 0.4779 - acc: 0.8736
53000/60000 [=========================>....] - ETA: 8s - loss: 0.4777 - acc: 0.8737
53250/60000 [=========================>....] - ETA: 8s - loss: 0.4770 - acc: 0.8739
53500/60000 [=========================>....] - ETA: 8s - loss: 0.4770 - acc: 0.8740
53750/60000 [=========================>....] - ETA: 7s - loss: 0.4768 - acc: 0.8741
54000/60000 [==========================>...] - ETA: 7s - loss: 0.4768 - acc: 0.8742
54250/60000 [==========================>...] - ETA: 7s - loss: 0.4772 - acc: 0.8741
54500/60000 [==========================>...] - ETA: 6s - loss: 0.4779 - acc: 0.8739
54750/60000 [==========================>...] - ETA: 6s - loss: 0.4779 - acc: 0.8741
55000/60000 [==========================>...] - ETA: 6s - loss: 0.4779 - acc: 0.8741
55250/60000 [==========================>...] - ETA: 5s - loss: 0.4788 - acc: 0.8740
55500/60000 [==========================>...] - ETA: 5s - loss: 0.4784 - acc: 0.8741
55750/60000 [==========================>...] - ETA: 5s - loss: 0.4782 - acc: 0.8742
56000/60000 [===========================>..] - ETA: 5s - loss: 0.4784 - acc: 0.8742
56250/60000 [===========================>..] - ETA: 4s - loss: 0.4783 - acc: 0.8743
56500/60000 [===========================>..] - ETA: 4s - loss: 0.4777 - acc: 0.8745
56750/60000 [===========================>..] - ETA: 4s - loss: 0.4768 - acc: 0.8747
57000/60000 [===========================>..] - ETA: 3s - loss: 0.4765 - acc: 0.8748
57250/60000 [===========================>..] - ETA: 3s - loss: 0.4767 - acc: 0.8748
57500/60000 [===========================>..] - ETA: 3s - loss: 0.4770 - acc: 0.8749
57750/60000 [===========================>..] - ETA: 2s - loss: 0.4770 - acc: 0.8749
58000/60000 [============================>.] - ETA: 2s - loss: 0.4764 - acc: 0.8752
58250/60000 [============================>.] - ETA: 2s - loss: 0.4759 - acc: 0.8753
58500/60000 [============================>.] - ETA: 1s - loss: 0.4756 - acc: 0.8753
58750/60000 [============================>.] - ETA: 1s - loss: 0.4755 - acc: 0.8754
59000/60000 [============================>.] - ETA: 1s - loss: 0.4747 - acc: 0.8756
59250/60000 [============================>.] - ETA: 0s - loss: 0.4744 - acc: 0.8756
59500/60000 [============================>.] - ETA: 0s - loss: 0.4748 - acc: 0.8756
59750/60000 [============================>.] - ETA: 0s - loss: 0.4749 - acc: 0.8755
60000/60000 [==============================] - ETA: 0s - loss: 0.4747 - acc: 0.8756
60000/60000 [==============================] - 75s 1ms/sample - loss: 0.4747 - acc: 0.8756 - val_loss: 0.4018 - val_acc: 0.9025
Epoch 3/3
250/60000 [..............................] - ETA: 1:16 - loss: 0.3690 - acc: 0.8920
500/60000 [..............................] - ETA: 1:15 - loss: 0.3675 - acc: 0.9120
750/60000 [..............................] - ETA: 1:15 - loss: 0.4147 - acc: 0.9027
1000/60000 [..............................] - ETA: 1:14 - loss: 0.4131 - acc: 0.8990
1250/60000 [..............................] - ETA: 1:14 - loss: 0.4028 - acc: 0.8984
1500/60000 [..............................] - ETA: 1:14 - loss: 0.3796 - acc: 0.9020
1750/60000 [..............................] - ETA: 1:14 - loss: 0.4121 - acc: 0.8971
2000/60000 [>.............................] - ETA: 1:13 - loss: 0.4218 - acc: 0.8950
2250/60000 [>.............................] - ETA: 1:13 - loss: 0.4357 - acc: 0.8956
2500/60000 [>.............................] - ETA: 1:12 - loss: 0.4190 - acc: 0.8976
2750/60000 [>.............................] - ETA: 1:12 - loss: 0.4179 - acc: 0.8985
3000/60000 [>.............................] - ETA: 1:12 - loss: 0.4174 - acc: 0.8997
3250/60000 [>.............................] - ETA: 1:12 - loss: 0.4087 - acc: 0.9006
3500/60000 [>.............................] - ETA: 1:11 - loss: 0.4062 - acc: 0.9014
3750/60000 [>.............................] - ETA: 1:11 - loss: 0.4076 - acc: 0.9011
4000/60000 [=>............................] - ETA: 1:10 - loss: 0.4051 - acc: 0.9007
4250/60000 [=>............................] - ETA: 1:10 - loss: 0.4028 - acc: 0.9024
4500/60000 [=>............................] - ETA: 1:10 - loss: 0.4068 - acc: 0.8984
4750/60000 [=>............................] - ETA: 1:09 - loss: 0.3993 - acc: 0.9006
5000/60000 [=>............................] - ETA: 1:09 - loss: 0.4139 - acc: 0.8992
5250/60000 [=>............................] - ETA: 1:09 - loss: 0.4130 - acc: 0.8987
5500/60000 [=>............................] - ETA: 1:08 - loss: 0.4116 - acc: 0.8985
5750/60000 [=>............................] - ETA: 1:08 - loss: 0.4080 - acc: 0.8990
6000/60000 [==>...........................] - ETA: 1:08 - loss: 0.4163 - acc: 0.8972
6250/60000 [==>...........................] - ETA: 1:07 - loss: 0.4216 - acc: 0.8958
6500/60000 [==>...........................] - ETA: 1:07 - loss: 0.4248 - acc: 0.8958
6750/60000 [==>...........................] - ETA: 1:07 - loss: 0.4282 - acc: 0.8948
7000/60000 [==>...........................] - ETA: 1:07 - loss: 0.4285 - acc: 0.8946
7250/60000 [==>...........................] - ETA: 1:06 - loss: 0.4263 - acc: 0.8956
7500/60000 [==>...........................] - ETA: 1:06 - loss: 0.4347 - acc: 0.8937
7750/60000 [==>...........................] - ETA: 1:06 - loss: 0.4336 - acc: 0.8942
8000/60000 [===>..........................] - ETA: 1:06 - loss: 0.4315 - acc: 0.8944
8250/60000 [===>..........................] - ETA: 1:05 - loss: 0.4336 - acc: 0.8944
8500/60000 [===>..........................] - ETA: 1:05 - loss: 0.4307 - acc: 0.8947
8750/60000 [===>..........................] - ETA: 1:05 - loss: 0.4272 - acc: 0.8951
9000/60000 [===>..........................] - ETA: 1:04 - loss: 0.4255 - acc: 0.8958
9250/60000 [===>..........................] - ETA: 1:04 - loss: 0.4241 - acc: 0.8962
9500/60000 [===>..........................] - ETA: 1:04 - loss: 0.4249 - acc: 0.8963
9750/60000 [===>..........................] - ETA: 1:03 - loss: 0.4240 - acc: 0.8962
10000/60000 [====>.........................] - ETA: 1:03 - loss: 0.4233 - acc: 0.8959
10250/60000 [====>.........................] - ETA: 1:03 - loss: 0.4208 - acc: 0.8964
10500/60000 [====>.........................] - ETA: 1:02 - loss: 0.4194 - acc: 0.8966
10750/60000 [====>.........................] - ETA: 1:02 - loss: 0.4199 - acc: 0.8957
11000/60000 [====>.........................] - ETA: 1:02 - loss: 0.4177 - acc: 0.8963
11250/60000 [====>.........................] - ETA: 1:01 - loss: 0.4156 - acc: 0.8965
11500/60000 [====>.........................] - ETA: 1:01 - loss: 0.4136 - acc: 0.8975
11750/60000 [====>.........................] - ETA: 1:01 - loss: 0.4091 - acc: 0.8985
12000/60000 [=====>........................] - ETA: 1:00 - loss: 0.4096 - acc: 0.8988
12250/60000 [=====>........................] - ETA: 1:00 - loss: 0.4079 - acc: 0.8995
12500/60000 [=====>........................] - ETA: 1:00 - loss: 0.4115 - acc: 0.8987
12750/60000 [=====>........................] - ETA: 1:00 - loss: 0.4101 - acc: 0.8988
13000/60000 [=====>........................] - ETA: 59s - loss: 0.4094 - acc: 0.8989
13250/60000 [=====>........................] - ETA: 59s - loss: 0.4110 - acc: 0.8988
13500/60000 [=====>........................] - ETA: 59s - loss: 0.4116 - acc: 0.8988
13750/60000 [=====>........................] - ETA: 58s - loss: 0.4121 - acc: 0.8988
14000/60000 [======>.......................] - ETA: 58s - loss: 0.4097 - acc: 0.8994
14250/60000 [======>.......................] - ETA: 58s - loss: 0.4121 - acc: 0.8993
14500/60000 [======>.......................] - ETA: 57s - loss: 0.4118 - acc: 0.8994
14750/60000 [======>.......................] - ETA: 57s - loss: 0.4130 - acc: 0.8991
15000/60000 [======>.......................] - ETA: 57s - loss: 0.4147 - acc: 0.8988
15250/60000 [======>.......................] - ETA: 57s - loss: 0.4161 - acc: 0.8986
15500/60000 [======>.......................] - ETA: 56s - loss: 0.4163 - acc: 0.8988
15750/60000 [======>.......................] - ETA: 56s - loss: 0.4171 - acc: 0.8990
16000/60000 [=======>......................] - ETA: 56s - loss: 0.4168 - acc: 0.8989
16250/60000 [=======>......................] - ETA: 55s - loss: 0.4151 - acc: 0.8993
16500/60000 [=======>......................] - ETA: 55s - loss: 0.4126 - acc: 0.8996
16750/60000 [=======>......................] - ETA: 55s - loss: 0.4141 - acc: 0.8994
17000/60000 [=======>......................] - ETA: 54s - loss: 0.4168 - acc: 0.8990
17250/60000 [=======>......................] - ETA: 54s - loss: 0.4166 - acc: 0.8990
17500/60000 [=======>......................] - ETA: 54s - loss: 0.4180 - acc: 0.8987
17750/60000 [=======>......................] - ETA: 53s - loss: 0.4175 - acc: 0.8989
18000/60000 [========>.....................] - ETA: 53s - loss: 0.4192 - acc: 0.8986
18250/60000 [========>.....................] - ETA: 53s - loss: 0.4206 - acc: 0.8982
18500/60000 [========>.....................] - ETA: 52s - loss: 0.4215 - acc: 0.8984
18750/60000 [========>.....................] - ETA: 52s - loss: 0.4188 - acc: 0.8989
19000/60000 [========>.....................] - ETA: 52s - loss: 0.4179 - acc: 0.8992
19250/60000 [========>.....................] - ETA: 51s - loss: 0.4172 - acc: 0.8993
19500/60000 [========>.....................] - ETA: 51s - loss: 0.4167 - acc: 0.8990
19750/60000 [========>.....................] - ETA: 51s - loss: 0.4162 - acc: 0.8991
20000/60000 [=========>....................] - ETA: 50s - loss: 0.4175 - acc: 0.8996
20250/60000 [=========>....................] - ETA: 50s - loss: 0.4158 - acc: 0.9001
20500/60000 [=========>....................] - ETA: 50s - loss: 0.4154 - acc: 0.9004
20750/60000 [=========>....................] - ETA: 49s - loss: 0.4162 - acc: 0.9003
21000/60000 [=========>....................] - ETA: 49s - loss: 0.4167 - acc: 0.9004
21250/60000 [=========>....................] - ETA: 49s - loss: 0.4166 - acc: 0.9002
21500/60000 [=========>....................] - ETA: 49s - loss: 0.4163 - acc: 0.9006
21750/60000 [=========>....................] - ETA: 48s - loss: 0.4172 - acc: 0.9007
22000/60000 [==========>...................] - ETA: 48s - loss: 0.4162 - acc: 0.9010
22250/60000 [==========>...................] - ETA: 48s - loss: 0.4162 - acc: 0.9009
22500/60000 [==========>...................] - ETA: 47s - loss: 0.4156 - acc: 0.9008
22750/60000 [==========>...................] - ETA: 47s - loss: 0.4140 - acc: 0.9012
23000/60000 [==========>...................] - ETA: 47s - loss: 0.4137 - acc: 0.9013
23250/60000 [==========>...................] - ETA: 46s - loss: 0.4124 - acc: 0.9016
23500/60000 [==========>...................] - ETA: 46s - loss: 0.4124 - acc: 0.9017
23750/60000 [==========>...................] - ETA: 46s - loss: 0.4129 - acc: 0.9016
24000/60000 [===========>..................] - ETA: 45s - loss: 0.4121 - acc: 0.9018
24250/60000 [===========>..................] - ETA: 45s - loss: 0.4150 - acc: 0.9015
24500/60000 [===========>..................] - ETA: 45s - loss: 0.4148 - acc: 0.9016
24750/60000 [===========>..................] - ETA: 44s - loss: 0.4146 - acc: 0.9018
25000/60000 [===========>..................] - ETA: 44s - loss: 0.4138 - acc: 0.9021
25250/60000 [===========>..................] - ETA: 44s - loss: 0.4138 - acc: 0.9021
25500/60000 [===========>..................] - ETA: 43s - loss: 0.4153 - acc: 0.9018
25750/60000 [===========>..................] - ETA: 43s - loss: 0.4167 - acc: 0.9018
26000/60000 [============>.................] - ETA: 43s - loss: 0.4157 - acc: 0.9018
26250/60000 [============>.................] - ETA: 42s - loss: 0.4163 - acc: 0.9018
26500/60000 [============>.................] - ETA: 42s - loss: 0.4172 - acc: 0.9021
26750/60000 [============>.................] - ETA: 42s - loss: 0.4159 - acc: 0.9022
27000/60000 [============>.................] - ETA: 42s - loss: 0.4172 - acc: 0.9020
27250/60000 [============>.................] - ETA: 41s - loss: 0.4164 - acc: 0.9021
27500/60000 [============>.................] - ETA: 41s - loss: 0.4173 - acc: 0.9021
27750/60000 [============>.................] - ETA: 41s - loss: 0.4176 - acc: 0.9022
28000/60000 [=============>................] - ETA: 40s - loss: 0.4175 - acc: 0.9023
28250/60000 [=============>................] - ETA: 40s - loss: 0.4178 - acc: 0.9022
28500/60000 [=============>................] - ETA: 40s - loss: 0.4189 - acc: 0.9022
28750/60000 [=============>................] - ETA: 39s - loss: 0.4177 - acc: 0.9024
29000/60000 [=============>................] - ETA: 39s - loss: 0.4183 - acc: 0.9024
29250/60000 [=============>................] - ETA: 39s - loss: 0.4179 - acc: 0.9025
29500/60000 [=============>................] - ETA: 38s - loss: 0.4175 - acc: 0.9026
29750/60000 [=============>................] - ETA: 38s - loss: 0.4176 - acc: 0.9025
30000/60000 [==============>...............] - ETA: 38s - loss: 0.4176 - acc: 0.9026
30250/60000 [==============>...............] - ETA: 37s - loss: 0.4162 - acc: 0.9028
30500/60000 [==============>...............] - ETA: 37s - loss: 0.4181 - acc: 0.9027
30750/60000 [==============>...............] - ETA: 37s - loss: 0.4177 - acc: 0.9028
31000/60000 [==============>...............] - ETA: 36s - loss: 0.4174 - acc: 0.9029
31250/60000 [==============>...............] - ETA: 36s - loss: 0.4171 - acc: 0.9030
31500/60000 [==============>...............] - ETA: 36s - loss: 0.4169 - acc: 0.9032
31750/60000 [==============>...............] - ETA: 35s - loss: 0.4168 - acc: 0.9033
32000/60000 [===============>..............] - ETA: 35s - loss: 0.4170 - acc: 0.9034
32250/60000 [===============>..............] - ETA: 35s - loss: 0.4170 - acc: 0.9034
32500/60000 [===============>..............] - ETA: 34s - loss: 0.4167 - acc: 0.9036
32750/60000 [===============>..............] - ETA: 34s - loss: 0.4160 - acc: 0.9036
33000/60000 [===============>..............] - ETA: 34s - loss: 0.4159 - acc: 0.9037
33250/60000 [===============>..............] - ETA: 33s - loss: 0.4165 - acc: 0.9036
33500/60000 [===============>..............] - ETA: 33s - loss: 0.4162 - acc: 0.9036
33750/60000 [===============>..............] - ETA: 33s - loss: 0.4166 - acc: 0.9034
34000/60000 [================>.............] - ETA: 33s - loss: 0.4151 - acc: 0.9036
34250/60000 [================>.............] - ETA: 32s - loss: 0.4155 - acc: 0.9036
34500/60000 [================>.............] - ETA: 32s - loss: 0.4147 - acc: 0.9037
34750/60000 [================>.............] - ETA: 32s - loss: 0.4144 - acc: 0.9038
35000/60000 [================>.............] - ETA: 31s - loss: 0.4135 - acc: 0.9040
35250/60000 [================>.............] - ETA: 31s - loss: 0.4126 - acc: 0.9040
35500/60000 [================>.............] - ETA: 31s - loss: 0.4120 - acc: 0.9042
35750/60000 [================>.............] - ETA: 30s - loss: 0.4112 - acc: 0.9042
36000/60000 [=================>............] - ETA: 30s - loss: 0.4120 - acc: 0.9041
36250/60000 [=================>............] - ETA: 30s - loss: 0.4125 - acc: 0.9043
36500/60000 [=================>............] - ETA: 29s - loss: 0.4126 - acc: 0.9043
36750/60000 [=================>............] - ETA: 29s - loss: 0.4117 - acc: 0.9044
37000/60000 [=================>............] - ETA: 29s - loss: 0.4112 - acc: 0.9044
37250/60000 [=================>............] - ETA: 28s - loss: 0.4123 - acc: 0.9043
37500/60000 [=================>............] - ETA: 28s - loss: 0.4121 - acc: 0.9043
37750/60000 [=================>............] - ETA: 28s - loss: 0.4116 - acc: 0.9044
38000/60000 [==================>...........] - ETA: 27s - loss: 0.4109 - acc: 0.9045
38250/60000 [==================>...........] - ETA: 27s - loss: 0.4107 - acc: 0.9045
38500/60000 [==================>...........] - ETA: 27s - loss: 0.4118 - acc: 0.9044
38750/60000 [==================>...........] - ETA: 26s - loss: 0.4113 - acc: 0.9044
39000/60000 [==================>...........] - ETA: 26s - loss: 0.4109 - acc: 0.9045
39250/60000 [==================>...........] - ETA: 26s - loss: 0.4115 - acc: 0.9044
39500/60000 [==================>...........] - ETA: 26s - loss: 0.4120 - acc: 0.9044
39750/60000 [==================>...........] - ETA: 25s - loss: 0.4111 - acc: 0.9046
40000/60000 [===================>..........] - ETA: 25s - loss: 0.4103 - acc: 0.9047
40250/60000 [===================>..........] - ETA: 25s - loss: 0.4094 - acc: 0.9049
40500/60000 [===================>..........] - ETA: 24s - loss: 0.4079 - acc: 0.9051
40750/60000 [===================>..........] - ETA: 24s - loss: 0.4069 - acc: 0.9053
41000/60000 [===================>..........] - ETA: 24s - loss: 0.4067 - acc: 0.9053
41250/60000 [===================>..........] - ETA: 23s - loss: 0.4072 - acc: 0.9052
41500/60000 [===================>..........] - ETA: 23s - loss: 0.4067 - acc: 0.9052
41750/60000 [===================>..........] - ETA: 23s - loss: 0.4069 - acc: 0.9052
42000/60000 [====================>.........] - ETA: 22s - loss: 0.4061 - acc: 0.9053
42250/60000 [====================>.........] - ETA: 22s - loss: 0.4070 - acc: 0.9051
42500/60000 [====================>.........] - ETA: 22s - loss: 0.4077 - acc: 0.9048
42750/60000 [====================>.........] - ETA: 21s - loss: 0.4077 - acc: 0.9049
43000/60000 [====================>.........] - ETA: 21s - loss: 0.4068 - acc: 0.9051
43250/60000 [====================>.........] - ETA: 21s - loss: 0.4070 - acc: 0.9050
43500/60000 [====================>.........] - ETA: 20s - loss: 0.4060 - acc: 0.9052
43750/60000 [====================>.........] - ETA: 20s - loss: 0.4065 - acc: 0.9051
44000/60000 [=====================>........] - ETA: 20s - loss: 0.4072 - acc: 0.9051
44250/60000 [=====================>........] - ETA: 20s - loss: 0.4078 - acc: 0.9049
44500/60000 [=====================>........] - ETA: 19s - loss: 0.4073 - acc: 0.9050
44750/60000 [=====================>........] - ETA: 19s - loss: 0.4075 - acc: 0.9049
45000/60000 [=====================>........] - ETA: 19s - loss: 0.4075 - acc: 0.9050
45250/60000 [=====================>........] - ETA: 18s - loss: 0.4069 - acc: 0.9051
45500/60000 [=====================>........] - ETA: 18s - loss: 0.4065 - acc: 0.9053
45750/60000 [=====================>........] - ETA: 18s - loss: 0.4053 - acc: 0.9054
46000/60000 [======================>.......] - ETA: 17s - loss: 0.4053 - acc: 0.9054
46250/60000 [======================>.......] - ETA: 17s - loss: 0.4055 - acc: 0.9054
46500/60000 [======================>.......] - ETA: 17s - loss: 0.4051 - acc: 0.9054
46750/60000 [======================>.......] - ETA: 16s - loss: 0.4047 - acc: 0.9054
47000/60000 [======================>.......] - ETA: 16s - loss: 0.4054 - acc: 0.9053
47250/60000 [======================>.......] - ETA: 16s - loss: 0.4046 - acc: 0.9055
47500/60000 [======================>.......] - ETA: 15s - loss: 0.4041 - acc: 0.9056
47750/60000 [======================>.......] - ETA: 15s - loss: 0.4034 - acc: 0.9058
48000/60000 [=======================>......] - ETA: 15s - loss: 0.4034 - acc: 0.9058
48250/60000 [=======================>......] - ETA: 14s - loss: 0.4038 - acc: 0.9057
48500/60000 [=======================>......] - ETA: 14s - loss: 0.4028 - acc: 0.9059
48750/60000 [=======================>......] - ETA: 14s - loss: 0.4032 - acc: 0.9059
49000/60000 [=======================>......] - ETA: 13s - loss: 0.4030 - acc: 0.9060
49250/60000 [=======================>......] - ETA: 13s - loss: 0.4027 - acc: 0.9060
49500/60000 [=======================>......] - ETA: 13s - loss: 0.4031 - acc: 0.9061
49750/60000 [=======================>......] - ETA: 13s - loss: 0.4028 - acc: 0.9062
50000/60000 [========================>.....] - ETA: 12s - loss: 0.4026 - acc: 0.9063
50250/60000 [========================>.....] - ETA: 12s - loss: 0.4032 - acc: 0.9061
50500/60000 [========================>.....] - ETA: 12s - loss: 0.4025 - acc: 0.9063
50750/60000 [========================>.....] - ETA: 11s - loss: 0.4028 - acc: 0.9062
51000/60000 [========================>.....] - ETA: 11s - loss: 0.4019 - acc: 0.9064
51250/60000 [========================>.....] - ETA: 11s - loss: 0.4013 - acc: 0.9065
51500/60000 [========================>.....] - ETA: 10s - loss: 0.4011 - acc: 0.9064
51750/60000 [========================>.....] - ETA: 10s - loss: 0.4010 - acc: 0.9064
52000/60000 [=========================>....] - ETA: 10s - loss: 0.4016 - acc: 0.9064
52250/60000 [=========================>....] - ETA: 9s - loss: 0.4017 - acc: 0.9064
52500/60000 [=========================>....] - ETA: 9s - loss: 0.4011 - acc: 0.9065
52750/60000 [=========================>....] - ETA: 9s - loss: 0.4011 - acc: 0.9065
53000/60000 [=========================>....] - ETA: 8s - loss: 0.4011 - acc: 0.9065
53250/60000 [=========================>....] - ETA: 8s - loss: 0.4012 - acc: 0.9064
53500/60000 [=========================>....] - ETA: 8s - loss: 0.4003 - acc: 0.9066
53750/60000 [=========================>....] - ETA: 7s - loss: 0.4003 - acc: 0.9066
54000/60000 [==========================>...] - ETA: 7s - loss: 0.4001 - acc: 0.9066
54250/60000 [==========================>...] - ETA: 7s - loss: 0.4003 - acc: 0.9067
54500/60000 [==========================>...] - ETA: 6s - loss: 0.3996 - acc: 0.9068
54750/60000 [==========================>...] - ETA: 6s - loss: 0.3994 - acc: 0.9068
55000/60000 [==========================>...] - ETA: 6s - loss: 0.4000 - acc: 0.9068
55250/60000 [==========================>...] - ETA: 6s - loss: 0.4000 - acc: 0.9068
55500/60000 [==========================>...] - ETA: 5s - loss: 0.3995 - acc: 0.9068
55750/60000 [==========================>...] - ETA: 5s - loss: 0.3991 - acc: 0.9068
56000/60000 [===========================>..] - ETA: 5s - loss: 0.3987 - acc: 0.9070
56250/60000 [===========================>..] - ETA: 4s - loss: 0.3988 - acc: 0.9069
56500/60000 [===========================>..] - ETA: 4s - loss: 0.3984 - acc: 0.9069
56750/60000 [===========================>..] - ETA: 4s - loss: 0.3990 - acc: 0.9067
57000/60000 [===========================>..] - ETA: 3s - loss: 0.3989 - acc: 0.9068
57250/60000 [===========================>..] - ETA: 3s - loss: 0.3994 - acc: 0.9067
57500/60000 [===========================>..] - ETA: 3s - loss: 0.3993 - acc: 0.9068
57750/60000 [===========================>..] - ETA: 2s - loss: 0.3995 - acc: 0.9069
58000/60000 [============================>.] - ETA: 2s - loss: 0.3995 - acc: 0.9068
58250/60000 [============================>.] - ETA: 2s - loss: 0.3994 - acc: 0.9069
58500/60000 [============================>.] - ETA: 1s - loss: 0.3992 - acc: 0.9069
58750/60000 [============================>.] - ETA: 1s - loss: 0.3988 - acc: 0.9070
59000/60000 [============================>.] - ETA: 1s - loss: 0.3987 - acc: 0.9071
59250/60000 [============================>.] - ETA: 0s - loss: 0.3987 - acc: 0.9071
59500/60000 [============================>.] - ETA: 0s - loss: 0.3980 - acc: 0.9072
59750/60000 [============================>.] - ETA: 0s - loss: 0.3975 - acc: 0.9073
60000/60000 [==============================] - ETA: 0s - loss: 0.3969 - acc: 0.9075
60000/60000 [==============================] - 76s 1ms/sample - loss: 0.3969 - acc: 0.9075 - val_loss: 0.3698 - val_acc: 0.9160
<keras.callbacks.History at 0x7f2abb6b4fd0>

Measure the differential privacy guarantee

Perform a privacy analysis to measure the DP guarantee achieved by a training algorithm. Knowing the level of DP achieved enables the objective comparison of two training runs to determine which of the two is more privacy-preserving. At a high level, the privacy analysis measures how much a potential adversary can improve their guess about properties of any individual training point by observing the outcome of the training procedure (e.g., model updates and parameters).

This guarantee is sometimes referred to as the privacy budget. A lower privacy budget bounds more tightly an adversary's ability to improve their guess. This ensures a stronger privacy guarantee. Intuitively, this is because it is harder for a single training point to affect the outcome of learning: for instance, the information contained in the training point cannot be memorized by the ML algorithm and the privacy of the individual who contributed this training point to the dataset is preserved.

In this tutorial, the privacy analysis is performed in the framework of Rényi Differential Privacy (RDP), which is a relaxation of pure DP based on this paper that is particularly well suited for DP-SGD.

Two metrics are used to express the DP guarantee of an ML algorithm:

Delta (\(\delta\)) - Bounds the probability of the privacy guarantee not holding. A rule of thumb is to set it to be less than the inverse of the size of the training dataset. In this tutorial, it is set to 10^-5 as the MNIST dataset has 60,000 training points.
Epsilon (\(\epsilon\)) - This is the privacy budget. It measures the strength of the privacy guarantee by bounding how much the probability of a particular model output can vary by including (or excluding) a single training point. A smaller value for \(\epsilon\) implies a better privacy guarantee. However, the \(\epsilon\) value is only an upper bound and a large value could still mean good privacy in practice.

Tensorflow Privacy provides a tool, compute_dp_sgd_privacy, to compute the value of \(\epsilon\) given a fixed value of \(\delta\) and the following hyperparameters from the training process:

The total number of points in the training data, n.
The batch_size.
The noise_multiplier.
The number of epochs of training.

compute_dp_sgd_privacy.compute_dp_sgd_privacy(n=train_data.shape[0],
                                              batch_size=batch_size,
                                              noise_multiplier=noise_multiplier,
                                              epochs=epochs,
                                              delta=1e-5)

DP-SGD with sampling rate = 0.417% and noise_multiplier = 1.3 iterated over 720 steps satisfies differential privacy with eps = 0.563 and delta = 1e-05.
The optimal RDP order is 18.0.
(0.5631726490328062, 18.0)

The tool reports that for the hyperparameters chosen above, the trained model has an \(\epsilon\) value of 1.18.

Summary

In this tutorial, you learned about differential privacy (DP) and how you can implement DP principles in existing ML algorithms to provide privacy guarantees for training data. In particular, you learned how to:

Wrap existing optimizers (e.g., SGD, Adam) into their differentially private counterparts using TensorFlow Privacy
Tune hyperparameters introduced by differentially private machine learning
Measure the privacy guarantee provided using analysis tools included in TensorFlow Privacy

Implement Differential Privacy with TensorFlow Privacy Stay organized with collections Save and categorize content based on your preferences.