Layer-wise adaptive rate scaling config.

Inherits From: BaseOptimizerConfig, Config, ParamsDict

name 'str', name of the optimizer.
momentum float hyperparameter >= 0 that accelerates gradient descent in the relevant direction and dampens oscillations. Defaults to 0.9.
eeta float LARS coefficient as used in the paper. Default set to LARS coefficient from the paper. (eeta / weight_decay) determines the highest scaling factor in LARS..
weight_decay_rate float for weight decay.
nesterov 'boolean' for whether to use nesterov momentum.
classic_momentum boolean for whether to use classic (or popular) momentum. The learning rate is applied during momentum update in classic momentum, but after momentum for popular momentum.
exclude_from_weight_decay A list of string for variable screening, if any of the string appears in a variable's name, the variable will be excluded for computing weight decay. For example, one could specify the list like ['batch_normalization', 'bias'] to exclude BN and bias from weight decay.
exclude_from_layer_adaptation Similar to exclude_from_weight_decay, but for layer adaptation. If it is None, it will be defaulted the same as exclude_from_weight_decay.

default_params Dataclass field
restrictions Dataclass field
clipnorm Dataclass field
clipvalue Dataclass field
global_clipnorm Dataclass field



View source

Returns a dict representation of params_dict.ParamsDict.

For the nested params_dict.ParamsDict, a nested dict will be returned.


View source

Builds a config from the given list of arguments.


View source

Wrapper for from_yaml.


View source


View source

Accesses through built-in dictionary get method.


View source

Makes the ParamsDict immutable.


View source

Override the ParamsDict with a set of given params.

override_params a dict or a ParamsDict specifying the parameters to be overridden.
is_strict a boolean specifying whether override is strict or not. If True, keys in override_params must be present in the ParamsDict. If False, keys in override_params can be different from what is currently defined in the ParamsDict. In this case, the ParamsDict will be extended to include the new keys.


View source

Overrides/returns a unlocked copy with the current config unchanged.


View source

Validate the parameters consistency based on the restrictions.

This method validates the internal consistency using the pre-defined list of restrictions. A restriction is defined as a string which specifies a binary operation. The supported binary operations are {'==', '!=', '<', '<=', '>', '>='}. Note that the meaning of these operators are consistent with the underlying Python immplementation. Users should make sure the define restrictions on their type make sense.

For example, for a ParamsDict like the following

  a1: 1
  a2: 2
    bb1: 10
    bb2: 20
    a1: 1
    a3: 3

one can define two restrictions like this ['a.a1 == b.ccc.a1', 'a.a2 <=']

What it enforces are

  • a.a1 = 1 == b.ccc.a1 = 1
  • a.a2 = 2 <= = 20

KeyError if any of the following happens (1) any of parameters in any of restrictions is not defined in ParamsDict, (2) any inconsistency violating the restriction is found.
ValueError if the restriction defined in the string is not supported.


View source

Implements the membership test operator.


IMMUTABLE_TYPES (<class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>, <class 'NoneType'>)
RESERVED_ATTR ['_locked', '_restrictions']
SEQUENCE_TYPES (<class 'list'>, <class 'tuple'>)
classic_momentum True
clipnorm None
clipvalue None
default_params None
eeta 0.001
exclude_from_layer_adaptation None
exclude_from_weight_decay None
global_clipnorm None
momentum 0.9
name 'LARS'
nesterov False
restrictions None
weight_decay_rate 0.0