ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more

tf.keras.optimizers.schedules.LearningRateSchedule

The learning rate schedule base class.

Used in the notebooks

Used in the tutorials

You can use a learning rate schedule to modulate how the learning rate of your optimizer changes over time.

Several built-in learning rate schedules are available, such as tf.keras.optimizers.schedules.ExponentialDecay or tf.keras.optimizers.schedules.PiecewiseConstantDecay:

lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=10000,
    decay_rate=0.9)
optimizer = keras.optimizers.SGD(learning_rate=lr_schedule)

A LearningRateSchedule instance can be passed in as the learning_rate argument of any optimizer.

To implement your own schedule object, you should implement the __call__ method, which takes a step argument (scalar integer tensor, the current training step count). Like for any other Keras object, you can also optionally make your object serializable by implementing the get_config and from_config methods.

Example:

class MyLRSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):

  def __init__(self, initial_learning_rate):
    self.initial_learning_rate = initial_learning_rate

  def __call__(self, step):
     return self.initial_learning_rate / (step + 1)

optimizer = tf.keras.optimizers.SGD(learning_rate=MyLRSchedule(0.1))

Methods

from_config

View source

Instantiates a LearningRateSchedule from its config.

Args
config Output of get_config().

Returns
A LearningRateSchedule instance.

get_config

View source

__call__

View source

Call self as a function.