tf.contrib.rnn.IntersectionRNNCell

View source on GitHub

Class IntersectionRNNCell

Intersection Recurrent Neural Network (+RNN) cell.

Inherits From: RNNCell

Architecture with coupled recurrent gate as well as coupled depth gate, designed to improve information flow through stacked RNNs. As the architecture uses depth gating, the dimensionality of the depth output (y) also should not change through depth (input size == output size). To achieve this, the first layer of a stacked Intersection RNN projects the inputs to N (num units) dimensions. Therefore when initializing an IntersectionRNNCell, one should set num_in_proj = N for the first layer and use default settings for subsequent layers.

This implements the recurrent cell from the paper:

https://arxiv.org/abs/1611.09913

Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. "Capacity and Trainability in Recurrent Neural Networks" Proc. ICLR 2017.

The Intersection RNN is built for use in deeply stacked RNNs so it may not achieve best performance with depth 1.

__init__

View source

__init__(
    num_units,
    num_in_proj=None,
    initializer=None,
    forget_bias=1.0,
    y_activation=tf.nn.relu,
    reuse=None
)

Initialize the parameters for an +RNN cell.

Args:

  • num_units: int, The number of units in the +RNN cell
  • num_in_proj: (optional) int, The input dimensionality for the RNN. If creating the first layer of an +RNN, this should be set to num_units. Otherwise, this should be set to None (default). If None, dimensionality of inputs should be equal to num_units, otherwise ValueError is thrown.
  • initializer: (optional) The initializer to use for the weight matrices.
  • forget_bias: (optional) float, default 1.0, The initial bias of the forget gates, used to reduce the scale of forgetting at the beginning of the training.
  • y_activation: (optional) Activation function of the states passed through depth. Default is 'tf.nn.relu`.
  • reuse: (optional) Python boolean describing whether to reuse variables in an existing scope. If not True, and the existing scope already has the given variables, an error is raised.

Properties

graph

DEPRECATED FUNCTION

output_size

Integer or TensorShape: size of outputs produced by this cell.

scope_name

state_size

size(s) of state(s) used by this cell.

It can be represented by an Integer, a TensorShape or a tuple of Integers or TensorShapes.

Methods

get_initial_state

View source

get_initial_state(
    inputs=None,
    batch_size=None,
    dtype=None
)

zero_state

View source

zero_state(
    batch_size,
    dtype
)

Return zero-filled state tensor(s).

Args:

  • batch_size: int, float, or unit Tensor representing the batch size.
  • dtype: the data type to use for the state.

Returns:

If state_size is an int or TensorShape, then the return value is a N-D tensor of shape [batch_size, state_size] filled with zeros.

If state_size is a nested list or tuple, then the return value is a nested list or tuple (of the same structure) of 2-D tensors with the shapes [batch_size, s] for each s in state_size.