tfm.optimization.AdafactorKerasConfig

Configuration for AdafactorKeras optimizer.

Inherits From: BaseOptimizerConfig, Config, ParamsDict

tfm.optimization.AdafactorKerasConfig(
    default_params: dataclasses.InitVar[Optional[Mapping[str, Any]]] = None,
    restrictions: dataclasses.InitVar[Optional[List[str]]] = None,
    clipnorm: Optional[float] = None,
    clipvalue: Optional[float] = None,
    global_clipnorm: Optional[float] = None,
    name: str = 'Adafactor',
    learning_rate: float = 0.001,
    beta_2_decay: float = -0.8,
    epsilon_1: float = 1e-30,
    epsilon_2: float = 0.001,
    clip_threshold: float = 1.0,
    relative_step: bool = True
)

The attributes for this class matches the arguments of the Adafactor implementation provided by keras.

default_params a Python dict or another ParamsDict object including the default parameters to initialize.
restrictions a list of strings, which define a list of restrictions to ensure the consistency of different parameters internally. Each restriction string is defined as a binary relation with a set of operators, including {'==', '!=', '<', '<=', '>', '>='}.

learning_rate: Initial value for the learning rate: either a floating point value, or a tf_keras.optimizers.schedules.LearningRateSchedule instance. Defaults to 0.001.
beta_2_decay float, defaults to -0.8. The decay rate of beta_2.
epsilon_1 float, defaults to 1e-30. A small offset to keep denominator away from 0.
epsilon_2 float, defaults to 1e-3. A small offset to avoid learning rate becoming too small by time.
clip_threshold float, defaults to 1.0. Clipping threshold. This is a part of Adafactor algorithm, independent from clipnorm, clipvalue and global_clipnorm.
relative_step bool, defaults to True. If learning_rate is a constant and relative_step=True, learning rate will be adjusted based on current iterations. This is a default learning rate decay in Adafactor.
BUILDER

default_params Dataclass field
restrictions Dataclass field
clipnorm Dataclass field
clipvalue Dataclass field
global_clipnorm Dataclass field
name Dataclass field
learning_rate Dataclass field

Methods

as_dict

View source

as_dict()

Returns a dict representation of params_dict.ParamsDict.

For the nested params_dict.ParamsDict, a nested dict will be returned.

from_args

View source

@classmethod
from_args(
    *args, **kwargs
)

Builds a config from the given list of arguments.

from_json

View source

@classmethod
from_json(
    file_path: str
)

Wrapper for from_yaml.

from_yaml

View source

@classmethod
from_yaml(
    file_path: str
)

get

View source

get(
    key, value=None
)

Accesses through built-in dictionary get method.

lock

View source

lock()

Makes the ParamsDict immutable.

override

View source

override(
    override_params, is_strict=True
)

Override the ParamsDict with a set of given params.

Args
override_params a dict or a ParamsDict specifying the parameters to be overridden.
is_strict a boolean specifying whether override is strict or not. If True, keys in override_params must be present in the ParamsDict. If False, keys in override_params can be different from what is currently defined in the ParamsDict. In this case, the ParamsDict will be extended to include the new keys.

replace

View source

replace(
    **kwargs
)

Overrides/returns a unlocked copy with the current config unchanged.

validate

View source

validate()

Validate the parameters consistency based on the restrictions.

This method validates the internal consistency using the pre-defined list of restrictions. A restriction is defined as a string which specifies a binary operation. The supported binary operations are {'==', '!=', '<', '<=', '>', '>='}. Note that the meaning of these operators are consistent with the underlying Python immplementation. Users should make sure the define restrictions on their type make sense.

For example, for a ParamsDict like the following

a:
  a1: 1
  a2: 2
b:
  bb:
    bb1: 10
    bb2: 20
  ccc:
    a1: 1
    a3: 3

one can define two restrictions like this ['a.a1 == b.ccc.a1', 'a.a2 <= b.bb.bb2']

What it enforces are

  • a.a1 = 1 == b.ccc.a1 = 1
  • a.a2 = 2 <= b.bb.bb2 = 20

Raises
KeyError if any of the following happens (1) any of parameters in any of restrictions is not defined in ParamsDict, (2) any inconsistency violating the restriction is found.
ValueError if the restriction defined in the string is not supported.

__contains__

View source

__contains__(
    key
)

Implements the membership test operator.

__eq__

__eq__(
    other
)

Return self==value.

IMMUTABLE_TYPES (<class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>, <class 'NoneType'>)
RESERVED_ATTR ['_locked', '_restrictions']
SEQUENCE_TYPES (<class 'list'>, <class 'tuple'>)
beta_2_decay -0.8
clip_threshold 1.0
clipnorm None
clipvalue None
default_params None
epsilon_1 1e-30
epsilon_2 0.001
global_clipnorm None
learning_rate 0.001
name 'Adafactor'
relative_step True
restrictions None