Google I/O is a wrap! Catch up on TensorFlow sessions View sessions

tf.linalg.LinearOperatorCirculant

TensorFlow 1 version View source on GitHub

LinearOperator acting like a circulant matrix.

tf.linalg.LinearOperatorCirculant(
    spectrum, input_output_dtype=tf.dtypes.complex64, is_non_singular=None,
    is_self_adjoint=None, is_positive_definite=None, is_square=True,
    name='LinearOperatorCirculant'
)

This operator acts like a circulant matrix A with shape [B1,...,Bb, N, N] for some b >= 0. The first b indices index a batch member. For every batch index (i1,...,ib), A[i1,...,ib, : :] is an N x N matrix. This matrix A is not materialized, but for purposes of broadcasting this shape will be relevant.

Description in terms of circulant matrices

Circulant means the entries of A are generated by a single vector, the convolution kernel h: A_{mn} := h_{m-n mod N}. With h = [w, x, y, z],

A = |w z y x|
    |x w z y|
    |y x w z|
    |z y x w|

This means that the result of matrix multiplication v = Au has Lth column given circular convolution between h with the Lth column of u.

Description in terms of the frequency spectrum

There is an equivalent description in terms of the [batch] spectrum H and Fourier transforms. Here we consider A.shape = [N, N] and ignore batch dimensions. Define the discrete Fourier transform (DFT) and its inverse by

DFT[ h[n] ] = H[k] := sum_{n = 0}^{N - 1} h_n e^{-i 2pi k n / N}
IDFT[ H[k] ] = h[n] = N^{-1} sum_{k = 0}^{N - 1} H_k e^{i 2pi k n / N}

From these definitions, we see that

H[0] = sum_{n = 0}^{N - 1} h_n
H[1] = "the first positive frequency"
H[N - 1] = "the first negative frequency"

Loosely speaking, with * element-wise multiplication, matrix multiplication is equal to the action of a Fourier multiplier: A u = IDFT[ H * DFT[u] ]. Precisely speaking, given [N, R] matrix u, let DFT[u] be the [N, R] matrix with rth column equal to the DFT of the rth column of u. Define the IDFT similarly. Matrix multiplication may be expressed columnwise:

Operator properties deduced from the spectrum.

Letting U be the kth Euclidean basis vector, and U = IDFT[u]. The above formulas show thatA U = H_k * U. We conclude that the elements of H are the eigenvalues of this operator. Therefore

  • This operator is positive definite if and only if Real{H} > 0.

A general property of Fourier transforms is the correspondence between Hermitian functions and real valued transforms.

Suppose H.shape = [B1,...,Bb, N]. We say that H is a Hermitian spectrum if, with % meaning modulus division,
  • This operator corresponds to a real matrix if and only if H is Hermitian.
  • This operator is self-adjoint if and only if H is real.

See e.g. "Discrete-Time Signal Processing", Oppenheim and Schafer.

Example of a self-adjoint positive definite operator

# spectrum is real ==> operator is self-adjoint
# spectrum is positive ==> operator is positive definite
spectrum = [6., 4, 2]

operator = LinearOperatorCirculant(spectrum)

# IFFT[spectrum]
operator.convolution_kernel()
==> [4 + 0j, 1 + 0.58j, 1 - 0.58j]

operator.to_dense()
==> [[4 + 0.0j, 1 - 0.6j, 1 + 0.6j],
     [1 + 0.6j, 4 + 0.0j, 1 - 0.6j],
     [1 - 0.6j, 1 + 0.6j, 4 + 0.0j]]

Example of defining in terms of a real convolution kernel

# convolution_kernel is real ==> spectrum is Hermitian.
convolution_kernel = [1., 2., 1.]]
spectrum = tf.signal.fft(tf.cast(convolution_kernel, tf.complex64))

# spectrum is Hermitian ==> operator is real.
# spectrum is shape [3] ==> operator is shape [3, 3]
# We force the input/output type to be real, which allows this to operate
# like a real matrix.
operator = LinearOperatorCirculant(spectrum, input_output_dtype=tf.float32)

operator.to_dense()
==> [[ 1, 1, 2],
     [ 2, 1, 1],
     [ 1, 2, 1]]

Example of Hermitian spectrum

# spectrum is shape [3] ==> operator is shape [3, 3]
# spectrum is Hermitian ==> operator is real.
spectrum = [1, 1j, -1j]

operator = LinearOperatorCirculant(spectrum)

operator.to_dense()
==> [[ 0.33 + 0j,  0.91 + 0j, -0.24 + 0j],
     [-0.24 + 0j,  0.33 + 0j,  0.91 + 0j],
     [ 0.91 + 0j, -0.24 + 0j,  0.33 + 0j]

Example of forcing real dtype when spectrum is Hermitian

# spectrum is shape [4] ==> operator is shape [4, 4]
# spectrum is real ==> operator is self-adjoint
# spectrum is Hermitian ==> operator is real
# spectrum has positive real part ==> operator is positive-definite.
spectrum = [6., 4, 2, 4]

# Force the input dtype to be float32.
# Cast the output to float32.  This is fine because the operator will be
# real due to Hermitian spectrum.
operator = LinearOperatorCirculant(spectrum, input_output_dtype=tf.float32)

operator.shape
==> [4, 4]

operator.to_dense()
==> [[4, 1, 0, 1],
     [1, 4, 1, 0],
     [0, 1, 4, 1],
     [1, 0, 1, 4]]

# convolution_kernel = tf.signal.ifft(spectrum)
operator.convolution_kernel()
==> [4, 1, 0, 1]

Performance

Suppose operator is a LinearOperatorCirculant of shape [N, N], and x.shape = [N, R]. Then

  • operator.matmul(x) is O(R*N*Log[N])
  • operator.solve(x) is O(R*N*Log[N])
  • operator.determinant() involves a size N reduce_prod.

If instead operator and x have shape [B1,...,Bb, N, N] and [B1,...,Bb, N, R], every operation increases in complexity by B1*...*Bb.

Matrix property hints

This LinearOperator is initialized with boolean flags of the form is_X, for X = non_singular, self_adjoint, positive_definite, square. These have the following meaning:

  • If is_X == True, callers should expect the operator to have the property X. This is a promise that should be fulfilled, but is not a runtime assert. For example, finite floating point precision may result in these promises being violated.
  • If is_X == False, callers should expect the operator to not have X.
  • If is_X == None (the default), callers should have no expectation either way.

References:

Toeplitz and Circulant Matrices - A Review: Gray, 2006 (pdf)

spectrum Shape [B1,...,Bb, N] Tensor. Allowed dtypes: float16, float32, float64, complex64, complex128. Type can be different than input_output_dtype
input_output_dtype dtype for input/output.
is_non_singular Expect that this operator is non-singular.
is_self_adjoint Expect that this operator is equal to its hermitian transpose. If spectrum is real, this will always be true.
is_positive_definite Expect that this operator is positive definite, meaning the quadratic form x^H A x has positive real part for all nonzero x. Note that we do not require the operator to be self-adjoint to be positive-definite. See: https://en.wikipedia.org/wiki/Positive-definite_matrix\

Extension_for_non_symmetric_matrices
is_square Expect that this operator acts like square [batch] matrices.
name A name to prepend to all ops created by this class.

H Returns the adjoint of the current LinearOperator.

Given A representing this LinearOperator, return A*. Note that calling self.adjoint() and self.H are equivalent.

batch_shape TensorShape of batch dimensions of this LinearOperator.

If this operator acts like the batch matrix A with A.shape = [B1,...,Bb, M, N], then this returns TensorShape([B1,...,Bb]), equivalent to A.shape[:-2]

block_depth Depth of recursively defined circulant blocks defining this Operator.

With A the dense representation of this Operator,

block_depth = 1 means A is symmetric circulant. For example,

A = |w z y x|
|x w z y|
|y x w z|
|z y x w|

block_depth = 2 means A is block symmetric circulant with symmetric circulant blocks. For example, with W, X, Y, Z symmetric circulant,

A = |W Z Y X|
|X W Z Y|
|Y X W Z|
|Z Y X W|

block_depth = 3 means A is block symmetric circulant with block symmetric circulant blocks.

block_shape

domain_dimension Dimension (in the sense of vector spaces) of the domain of this operator.

If this operator acts like the batch matrix A with A.shape = [B1,...,Bb, M, N], then this returns N.

dtype The DType of Tensors handled by this LinearOperator.
graph_parents List of graph dependencies of this LinearOperator. (deprecated)
is_non_singular

is_positive_definite

is_self_adjoint

is_square Return True/False depending on if this operator is square.
range_dimension Dimension (in the sense of vector spaces) of the range of this operator.

If this operator acts like the batch matrix A with A.shape = [B1,...,Bb, M, N], then this returns M.

shape TensorShape of this LinearOperator.

If this operator acts like the batch matrix A with A.shape = [B1,...,Bb, M, N], then this returns TensorShape([B1,...,Bb, M, N]), equivalent to A.shape.

spectrum

tensor_rank Rank (in the sense of tensors) of matrix corresponding to this operator.

If this operator acts like the batch matrix A with A.shape = [B1,...,Bb, M, N], then this returns b + 2.

Methods

add_to_tensor

View source

add_to_tensor(
    x, name='add_to_tensor'
)

Add matrix represented by this operator to x. Equivalent to A + x.

Args
x Tensor with same dtype and shape broadcastable to self.shape.
name A name to give this Op.

Returns
A Tensor with broadcast shape and same dtype as self.

adjoint

View source

adjoint(
    name='adjoint'
)

Returns the adjoint of the current LinearOperator.

Given A representing this LinearOperator, return A*. Note that calling self.adjoint() and self.H are equivalent.

Args
name A name for this Op.

Returns
LinearOperator which represents the adjoint of this LinearOperator.

assert_hermitian_spectrum

View source

assert_hermitian_spectrum(
    name='assert_hermitian_spectrum'
)

Returns an Op that asserts this operator has Hermitian spectrum.

This operator corresponds to a real-valued matrix if and only if its spectrum is Hermitian.

Args
name A name to give this Op.

Returns
An Op that asserts this operator has Hermitian spectrum.

assert_non_singular

View source

assert_non_singular(
    name='assert_non_singular'
)

Returns an Op that asserts this operator is non singular.

This operator is considered non-singul