![]() |
An agent that mixes a set of agents with a given mixture.
Inherits From: TFAgent
tf_agents.bandits.agents.mixture_agent.MixtureAgent(
mixture_distribution: types.Distribution,
agents: Sequence[tf_agents.agents.TFAgent
],
name: Optional[Text] = None
)
For every data sample, the agent updates the sub-agent that was used to make
the action choice in that sample. For this update to happen, the mixture agent
needs to have the information on which sub-agent is "responsible" for the
action. This information is in a policy info field mixture_agent_id
.
Note that this agent makes use of tf.dynamic_partition
, and thus it is not
compatible with XLA.
Args | |
---|---|
mixture_distribution
|
An instance of tfd.Categorical distribution. This
distribution is used to draw sub-policies by the mixture policy. The
parameters of the distribution is trained by the mixture agent.
|
agents
|
List of instances of TF-Agents bandit agents. These agents will be
trained and used to select actions. The length of this list should match
that of mixture_weights .
|
name
|
The name of this instance of MixtureAgent .
|
Attributes | |
---|---|
action_spec
|
TensorSpec describing the action produced by the agent. |
collect_data_context
|
|
collect_data_spec
|
Returns a Trajectory spec, as expected by the collect_policy .
|
collect_policy
|
Return a policy that can be used to collect data from the environment. |
data_context
|
|
debug_summaries
|
|
policy
|
Return the current policy held by the agent. |
summaries_enabled
|
|
summarize_grads_and_vars
|
|
time_step_spec
|
Describes the TimeStep tensors expected by the agent.
|
train_sequence_length
|
The number of time steps needed in experience tensors passed to train .
Train requires experience to be a For example, for non-RNN DQN training, If this value is |
train_step_counter
|
|
training_data_spec
|
Returns a trajectory spec, as expected by the train() function. |
Methods
initialize
initialize() -> Optional[tf.Operation]
Initializes the agent.
Returns | |
---|---|
An operation that can be used to initialize the agent. |
Raises | |
---|---|
RuntimeError
|
If the class was not initialized properly (super.__init__
was not called).
|
loss
loss(
experience: tf_agents.typing.types.NestedTensor
,
weights: Optional[types.Tensor] = None,
training: bool = False,
**kwargs
) -> tf_agents.agents.tf_agent.LossInfo
Gets loss from the agent.
If the user calls this from _train, it must be in a tf.GradientTape
scope
in order to apply gradients to trainable variables.
If intermediate gradient steps are needed, _loss and _train will return
different values since _loss only supports updating all gradients at once
after all losses have been calculated.
Args | |
---|---|
experience
|
A batch of experience data in the form of a Trajectory . The
structure of experience must match that of self.training_data_spec .
All tensors in experience must be shaped [batch, time, ...] where
time must be equal to self.train_step_length if that
property is not None .
|
weights
|
(optional). A Tensor , either 0-D or shaped [batch] ,
containing weights to be used when calculating the total train loss.
Weights are typically multiplied elementwise against the per-batch loss,
but the implementation is up to the Agent.
|
training
|
Explicit argument to pass to loss . This typically affects
network computation paths like dropout and batch normalization.
|
**kwargs
|
Any additional data as args to loss .
|
Returns | |
---|---|
A LossInfo loss tuple containing loss and info tensors.
|
Raises | |
---|---|
RuntimeError
|
If the class was not initialized properly (super.__init__
was not called).
|
post_process_policy
post_process_policy() -> tf_agents.policies.TFPolicy
Post process policies after training.
The policies of some agents require expensive post processing after training before they can be used. e.g. A Recommender agent might require rebuilding an index of actions. For such agents, this method will return a post processed version of the policy. The post processing may either update the existing policies in place or create a new policy, depnding on the agent. The default implementation for agents that do not want to override this method is to return agent.policy.
Returns | |
---|---|
The post processed policy. |
preprocess_sequence
preprocess_sequence(
experience: tf_agents.typing.types.NestedTensor
) -> tf_agents.typing.types.NestedTensor
Defines preprocess_sequence function to be fed into replay buffers.
This defines how we preprocess the collected data before training.
Defaults to pass through for most agents.
Structure of experience
must match that of self.collect_data_spec
.
Args | |
---|---|
experience
|
a Trajectory shaped [batch, time, ...] or [time, ...] which
represents the collected experience data.
|
Returns | |
---|---|
A post processed Trajectory with the same shape as the input.
|
train
train(
experience: tf_agents.typing.types.NestedTensor
,
weights: Optional[types.Tensor] = None,
**kwargs
) -> tf_agents.agents.tf_agent.LossInfo
Trains the agent.
Args | |
---|---|
experience
|
A batch of experience data in the form of a Trajectory . The
structure of experience must match that of self.training_data_spec .
All tensors in experience must be shaped [batch, time, ...] where
time must be equal to self.train_step_length if that
property is not None .
|
weights
|
(optional). A Tensor , either 0-D or shaped [batch] ,
containing weights to be used when calculating the total train loss.
Weights are typically multiplied elementwise against the per-batch loss,
but the implementation is up to the Agent.
|
**kwargs
|
Any additional data to pass to the subclass. |
Returns | |
---|---|
A LossInfo loss tuple containing loss and info tensors.
|
Raises | |
---|---|
RuntimeError
|
If the class was not initialized properly (super.__init__
was not called).
|