tf.mixed_precision.experimental.DynamicLossScale
Stay organized with collections
Save and categorize content based on your preferences.
Loss scale that dynamically adjusts itself.
Inherits From: LossScale
tf.mixed_precision.experimental.DynamicLossScale(
initial_loss_scale=(2 ** 15), increment_period=2000, multiplier=2.0
)
Dynamic loss scaling works by adjusting the loss scale as training progresses.
The goal is to keep the loss scale as high as possible without overflowing the
gradients. As long as the gradients do not overflow, raising the loss scale
never hurts.
The algorithm starts by setting the loss scale to an initial value. Every N
steps that the gradients are finite, the loss scale is increased by some
factor. However, if a NaN or Inf gradient is found, the gradients for that
step are not applied, and the loss scale is decreased by the factor. This
process tends to keep the loss scale as high as possible without gradients
overflowing.
Args |
|---|
initial_loss_scale
|
A Python float. The loss scale to use at the
beginning. It's better to start this at a very high number, because a
loss scale that is too high gets lowered far more quickly than a loss
scale that is too low gets raised. The default is 2 ** 15, which is
approximately half the maximum float16 value.
|
increment_period
|
Increases loss scale every increment_period
consecutive steps that finite gradients are encountered. If a nonfinite
gradient is encountered, the count is reset back to zero.
|
multiplier
|
The multiplier to use when increasing or decreasing the loss
scale.
|
Attributes |
|---|
increment_period
|
|
initial_loss_scale
|
|
multiplier
|
|
Methods
from_config
View source
@classmethod
from_config(
config
)
Creates the LossScale from its config.
get_config
View source
get_config()
Returns the config of this loss scale.
update
View source
update(
grads
)
Updates loss scale based on if gradients are finite in current step.
__call__
View source
__call__()
Returns the current loss scale as a scalar float32 tensor.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2023-03-17 UTC.
[null,null,["Last updated 2023-03-17 UTC."],[],[],null,["# tf.mixed_precision.experimental.DynamicLossScale\n\n\u003cbr /\u003e\n\n|-----------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.8.4/tensorflow/python/training/experimental/loss_scale.py#L307-L457) |\n\nLoss scale that dynamically adjusts itself.\n\nInherits From: [`LossScale`](../../../tf/mixed_precision/experimental/LossScale)\n\n#### View aliases\n\n\n**Main aliases**\n\n\\`tf.train.experimental.DynamicLossScale\\`\n**Compat aliases for migration**\n\nSee\n[Migration guide](https://www.tensorflow.org/guide/migrate) for\nmore details.\n\n[`tf.compat.v1.mixed_precision.DynamicLossScale`](https://www.tensorflow.org/api_docs/python/tf/compat/v1/mixed_precision/DynamicLossScale), [`tf.compat.v1.mixed_precision.experimental.DynamicLossScale`](https://www.tensorflow.org/api_docs/python/tf/compat/v1/mixed_precision/DynamicLossScale), [`tf.compat.v1.train.experimental.DynamicLossScale`](https://www.tensorflow.org/api_docs/python/tf/compat/v1/mixed_precision/DynamicLossScale)\n\n\u003cbr /\u003e\n\n tf.mixed_precision.experimental.DynamicLossScale(\n initial_loss_scale=(2 ** 15), increment_period=2000, multiplier=2.0\n )\n\n| **Warning:** This class is deprecated and will be unexposed from the TF 2 namespace in a future version of TensorFlow. Once this occurs, this class will only be accessible as [`tf.compat.v1.mixed_precision.DynamicLossScale`](../../../tf/mixed_precision/experimental/DynamicLossScale). All the functionality in this class has been merged into [`tf.keras.mixed_precision.LossScaleOptimizer`](../../../tf/keras/mixed_precision/LossScaleOptimizer), so this class is no longer needed.\n\nDynamic loss scaling works by adjusting the loss scale as training progresses.\nThe goal is to keep the loss scale as high as possible without overflowing the\ngradients. As long as the gradients do not overflow, raising the loss scale\nnever hurts.\n\nThe algorithm starts by setting the loss scale to an initial value. Every N\nsteps that the gradients are finite, the loss scale is increased by some\nfactor. However, if a NaN or Inf gradient is found, the gradients for that\nstep are not applied, and the loss scale is decreased by the factor. This\nprocess tends to keep the loss scale as high as possible without gradients\noverflowing.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `initial_loss_scale` | A Python float. The loss scale to use at the beginning. It's better to start this at a very high number, because a loss scale that is too high gets lowered far more quickly than a loss scale that is too low gets raised. The default is 2 \\*\\* 15, which is approximately half the maximum float16 value. |\n| `increment_period` | Increases loss scale every `increment_period` consecutive steps that finite gradients are encountered. If a nonfinite gradient is encountered, the count is reset back to zero. |\n| `multiplier` | The multiplier to use when increasing or decreasing the loss scale. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Attributes ---------- ||\n|----------------------|---------------|\n| `increment_period` | \u003cbr /\u003e \u003cbr /\u003e |\n| `initial_loss_scale` | \u003cbr /\u003e \u003cbr /\u003e |\n| `multiplier` | \u003cbr /\u003e \u003cbr /\u003e |\n\n\u003cbr /\u003e\n\nMethods\n-------\n\n### `from_config`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.8.4/tensorflow/python/training/experimental/loss_scale.py#L201-L204) \n\n @classmethod\n from_config(\n config\n )\n\nCreates the LossScale from its config.\n\n### `get_config`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.8.4/tensorflow/python/training/experimental/loss_scale.py#L452-L457) \n\n get_config()\n\nReturns the config of this loss scale.\n\n### `update`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.8.4/tensorflow/python/training/experimental/loss_scale.py#L392-L439) \n\n update(\n grads\n )\n\nUpdates loss scale based on if gradients are finite in current step.\n\n### `__call__`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.8.4/tensorflow/python/training/experimental/loss_scale.py#L389-L390) \n\n __call__()\n\nReturns the current loss scale as a scalar `float32` tensor."]]
View source on GitHub