tf_privacy.DPSequential

DP subclass of tf.keras.Sequential.

tf_privacy.DPSequential(
    l2_norm_clip,
    noise_multiplier,
    num_microbatches=None,
    use_xla=True,
    layer_registry=None,
    *args,
    **kwargs
)

This can be used as a differentially private replacement for tf.keras.Sequential. This class implements DP-SGD using the standard Gaussian mechanism.

This class also utilizes a faster gradient clipping algorithm if the following two conditions hold: (i) the trainable layers of the model are keys in the input layer_registry, (ii) the loss tf.Tensor for a given batch of examples is either a scalar or a 2D tf.Tensor that has only one column (i.e., tf.shape(loss)[1] == 1) and whose i-th row corresponds to the loss of the i-th example. This clipping algorithm specifically computes clipped gradients at the per-example or per microbatch (when num_microbatches is not None) level using the layer registry functions in layer_registry (see clip_grads.py for more information about the algorithm).

Warning: with faster gradient clipping, and when num_microbatches is not None, the per microbatch loss is assumed to be computed as the mean of the loss over the microbatch, or effectively, by reshaping the loss from the shape [batch_size, ...] to the shape [num_microbatches, batch_size/num_microbatches, ...] and computing the mean of the loss over the microbatches. This would require that the loss function behaves accordingly. This is true for multiple common predefined keras loss functions (e.g. mean_squared_loss, binary_crossentropy) but may not hold for custom losses (and how such aggregation is done is not exposed by the loss function, unfortunately). It is the caller's responsibility to make sure that the loss function does behave this way.

Warning: This API does not have privacy guarantees for custom layer-level losses created by the layer.add_loss() API. It does, however, support layer regularization losses. All of these layer-level losses are found in model.losses.

When instantiating this class, you need to supply several DP-related arguments followed by the standard arguments for Sequential.

Examples:

# Create Model instance.
model = DPSequential(l2_norm_clip=1.0, noise_multiplier=0.5, use_xla=True,
         <standard arguments>)

You should use your DPSequential instance with a standard instance of tf.keras.Optimizer as the optimizer, and a standard reduced loss. You do not need to use a differentially private optimizer.

# Use a standard (non-DP) optimizer.
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)

# Use a standard reduced loss.
loss = tf.keras.losses.MeanSquaredError()

model.compile(optimizer=optimizer, loss=loss)
model.fit(train_data, train_labels, epochs=1, batch_size=32)

Args
l2_norm_clip	Clipping norm (max L2 norm of per microbatch gradients).
noise_multiplier	Ratio of the standard deviation to the clipping norm.
num_microbatches	Number of microbatches.
use_xla	If True, compiles train_step to XLA.
layer_registry	A LayerRegistry instance containing functions that help compute gradient norms quickly. See tensorflow_privacy.privacy.fast_gradient_clipping.layer_registry for more details.
*args	These will be passed on to the base class __init__ method.
**kwargs	These will be passed on to the base class __init__ method.

Methods

add

add(
    layer
)

Adds a layer instance on top of the layer stack.

Args
layer	layer instance.

Raises
TypeError	If layer is not a layer instance.
ValueError	In case the layer argument does not know its input shape.
ValueError	In case the layer argument has multiple output tensors, or is already connected somewhere else (forbidden in Sequential models).

pop

pop()

Removes the last layer in the model.

Raises
TypeError	if there are no layers in the model.