SparseMixer encoder configuration.

Inherits From: Config, ParamsDict


default_params Dataclass field
restrictions Dataclass field
vocab_size Dataclass field
hidden_size Dataclass field
num_layers Dataclass field
moe_layers Dataclass field
attention_layers Dataclass field
num_experts Dataclass field
train_capacity_factor Dataclass field
eval_capacity_factor Dataclass field
examples_per_group Dataclass field
use_fft Dataclass field
num_attention_heads Dataclass field
max_sequence_length Dataclass field
type_vocab_size Dataclass field
inner_dim Dataclass field
inner_activation Dataclass field
output_dropout Dataclass field
attention_dropout Dataclass field
initializer_range Dataclass field
output_range Dataclass field
embedding_width Dataclass field
norm_first Dataclass field



View source

Returns a dict representation of params_dict.ParamsDict.

For the nested params_dict.ParamsDict, a nested dict will be returned.


View source

Builds a config from the given list of arguments.


View source

Wrapper for from_yaml.


View source


View source

Accesses through built-in dictionary get method.


View source

Makes the ParamsDict immutable.


View source

Override the ParamsDict with a set of given params.

override_params a dict or a ParamsDict specifying the parameters to be overridden.
is_strict a boolean specifying whether override is strict or not. If True, keys in override_params must be present in the ParamsDict. If False, keys in override_params can be different from what is currently defined in the ParamsDict. In this case, the ParamsDict will be extended to include the new keys.


View source

Overrides/returns a unlocked copy with the current config unchanged.


View source

Validate the parameters consistency based on the restrictions.

This method validates the internal consistency using the pre-defined list of restrictions. A restriction is defined as a string which specifies a binary operation. The supported binary operations are {'==', '!=', '<', '<=', '>', '>='}. Note that the meaning of these operators are consistent with the underlying Python immplementation. Users should make sure the define restrictions on their type make sense.

For example, for a ParamsDict like the following

  a1: 1
  a2: 2
    bb1: 10
    bb2: 20
    a1: 1
    a3: 3

one can define two restrictions like this ['a.a1 == b.ccc.a1', 'a.a2 <=']

What it enforces are

  • a.a1 = 1 == b.ccc.a1 = 1
  • a.a2 = 2 <= = 20

KeyError if any of the following happens (1) any of parameters in any of restrictions is not defined in ParamsDict, (2) any inconsistency violating the restriction is found.
ValueError if the restriction defined in the string is not supported.


View source

Implements the membership test operator.


IMMUTABLE_TYPES (<class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>, <class 'NoneType'>)
RESERVED_ATTR ['_locked', '_restrictions']
SEQUENCE_TYPES (<class 'list'>, <class 'tuple'>)
attention_dropout 0.1
attention_layers (10, 11, 12, 13)
default_params None
embedding_width None
eval_capacity_factor 1.0
examples_per_group 1.0
hidden_size 768
initializer_range 0.02
inner_activation 'gelu'
inner_dim 3072
max_sequence_length 512
moe_layers (5, 6, 7, 8)
norm_first False
num_attention_heads 8
num_experts 16
num_layers 14
output_dropout 0.1
output_range None
restrictions None
train_capacity_factor 1.0
type_vocab_size 2
use_fft False
vocab_size 30522