|View source on GitHub|
Custom getter that performs Spectral Normalization on a weight tensor.
tf.contrib.gan.features.spectral_normalization_custom_getter( name_filter=_default_name_filter, power_iteration_rounds=1 )
Specifically it divides the weight tensor by its largest singular value. This is intended to stabilize GAN training, by making the discriminator satisfy a local 1-Lipschitz constraint.
To reproduce an SN-GAN, apply this custom_getter to every weight tensor of your discriminator. The last dimension of the weight tensor must be the number of output channels.
Apply this to layers by supplying this as the
custom_getter of a
tf.compat.v1.variable_scope. For example:
with tf.compat.v1.variable_scope('discriminator', custom_getter=spectral_norm_getter()): net = discriminator_fn(net)
IMPORTANT: Keras does not respect the custom_getter supplied by the
VariableScope, so Keras users should use
instead of (or in addition to) this approach.
It is important to carefully select to which weights you want to apply
Spectral Normalization. In general you want to normalize the kernels of
convolution and dense layers, but you do not want to normalize biases. You
also want to avoid normalizing batch normalization (and similar) variables,
but in general such layers play poorly with Spectral Normalization, since the
gamma can cancel out the normalization in other layers. By default we supply a
filter that matches the kernel variable names of the dense and convolution
layers of the tf.layers, tf.contrib.layers, tf.keras and tf.contrib.slim
libraries. If you are using anything else you'll need a custom
This custom getter internally creates a variable used to compute the spectral norm by power iteration. It will update every time the variable is accessed, which means the normalized discriminator weights may change slightly whilst training the generator. Whilst unusual, this matches how the paper's authors implement it, and in general additional rounds of power iteration can't hurt.
name_filter: Optionally, a method that takes a Variable name as input and returns whether this Variable should be normalized.
power_iteration_rounds: The number of iterations of the power method to perform per step. A higher number yields a better approximation of the true spectral norm.
A custom getter function that applies Spectral Normalization to all
Variables whose names match
ValueError: If name_filter is not callable.