归一化

概述

• 层归一化（TensorFlow Core）

$$y_{i} = \frac{\gamma ( x_{i} - \mu )}{\sigma }+ \beta$$

$$y$$：输出

$$x$$：输入

$$\gamma$$：比例因子

$$\mu$$：均值

$$\sigma$$：标准差

$$\beta$$：修正因子

设置

pip install -q -U tensorflow-addons

import tensorflow as tf


准备数据集

mnist = tf.keras.datasets.mnist

x_train, x_test = x_train / 255.0, x_test / 255.0

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step


组归一化教程

下面的示例在 Conv2D 层之后将 10 个通道按标准的“最后一个通道”设置分为 5 个子组：

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# Groupnorm Layer
tfa.layers.GroupNormalization(groups=5, axis=3),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.4978 - accuracy: 0.8549
<tensorflow.python.keras.callbacks.History at 0x7f116cc009e8>


实例归一化教程

示例

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tfa.layers.InstanceNormalization(axis=3,
center=True,
scale=True,
beta_initializer="random_uniform",
gamma_initializer="random_uniform"),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.5717 - accuracy: 0.8241
<tensorflow.python.keras.callbacks.History at 0x7f116c97ae80>


层归一化教程

示例

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tf.keras.layers.LayerNormalization(axis=1 , center=True , scale=True),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.4242 - accuracy: 0.8794
<tensorflow.python.keras.callbacks.History at 0x7f116c7c1860>


文献

Layer norm

Instance norm

Group Norm

Complete Normalizations Overview

[]
[]