MoViNet for streaming action recognition

View on TensorFlow.org Run in Google Colab View on GitHub Download notebook See TF Hub models

This tutorial demonstrates how to use a pretrained video classification model to classify an activity (such as dancing, swimming, biking etc) in the given video.

The model architecture used in this tutorial is called MoViNet (Mobile Video Networks). MoVieNets are a family of efficient video classification models trained on huge dataset (Kinetics 600).

In contrast to the i3d models available on TF Hub, MoViNets also support frame-by-frame inference on streaming video.

The pretrained models are available from TF Hub. The TF Hub collection also includes quantized models optimized for TFLite.

The source for these models is available in the TensorFlow Model Garden. This includes a longer version of this tutorial that also covers building and fine-tuning a MoViNet model.

This MoViNet tutorial is part of a series of TensorFlow video tutorials. Here are the other three tutorials:

jumping jacks plot

Setup

For inference on smaller models (A0-A2), CPU is sufficient for this Colab.

sudo apt install -y ffmpeg
pip install -q mediapy
pip uninstall -q -y opencv-python-headless
pip install -q "opencv-python-headless<4.3"
# Import libraries
import pathlib

import matplotlib as mpl
import matplotlib.pyplot as plt
import mediapy as media
import numpy as np
import PIL

import tensorflow as tf
import tensorflow_hub as hub
import tqdm

mpl.rcParams.update({
    'font.size': 10,
})
2024-02-01 15:01:34.199338: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-01 15:01:34.199395: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-01 15:01:34.200940: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Get the kinetics 600 label list, and print the first few labels:

labels_path = tf.keras.utils.get_file(
    fname='labels.txt',
    origin='https://raw.githubusercontent.com/tensorflow/models/f8af2291cced43fc9f1d9b41ddbf772ae7b0d7d2/official/projects/movinet/files/kinetics_600_labels.txt'
)
labels_path = pathlib.Path(labels_path)

lines = labels_path.read_text().splitlines()
KINETICS_600_LABELS = np.array([line.strip() for line in lines])
KINETICS_600_LABELS[:20]
Downloading data from https://raw.githubusercontent.com/tensorflow/models/f8af2291cced43fc9f1d9b41ddbf772ae7b0d7d2/official/projects/movinet/files/kinetics_600_labels.txt
9209/9209 [==============================] - 0s 0us/step
array(['abseiling', 'acting in play', 'adjusting glasses', 'air drumming',
       'alligator wrestling', 'answering questions', 'applauding',
       'applying cream', 'archaeological excavation', 'archery',
       'arguing', 'arm wrestling', 'arranging flowers',
       'assembling bicycle', 'assembling computer',
       'attending conference', 'auctioning', 'backflip (human)',
       'baking cookies', 'bandaging'], dtype='<U49')

To provide a simple example video for classification, we can load a short gif of jumping jacks being performed.

jumping jacks

Attribution: Footage shared by Coach Bobby Bluford on YouTube under the CC-BY license.

Download the gif.

jumpingjack_url = 'https://github.com/tensorflow/models/raw/f8af2291cced43fc9f1d9b41ddbf772ae7b0d7d2/official/projects/movinet/files/jumpingjack.gif'
jumpingjack_path = tf.keras.utils.get_file(
    fname='jumpingjack.gif',
    origin=jumpingjack_url,
    cache_dir='.', cache_subdir='.',
)
Downloading data from https://github.com/tensorflow/models/raw/f8af2291cced43fc9f1d9b41ddbf772ae7b0d7d2/official/projects/movinet/files/jumpingjack.gif
783318/783318 [==============================] - 0s 0us/step

Define a function to read a gif file into a tf.Tensor:

The video's shape is (frames, height, width, colors)

jumpingjack=load_gif(jumpingjack_path)
jumpingjack.shape
2024-02-01 15:01:37.376831: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:274] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
TensorShape([13, 224, 224, 3])

How to use the model

This section contains a walkthrough showing how to use the models from TensorFlow Hub. If you just want to see the models in action, skip to the next section.

There are two versions of each model: base and streaming.

  • The base version takes a video as input, and returns the probabilities averaged over the frames.
  • The streaming version takes a video frame and an RNN state as input, and returns the predictions for that frame, and the new RNN state.

The base model

Download the pretrained model from TensorFlow Hub.

%%time
id = 'a2'
mode = 'base'
version = '3'
hub_url = f'https://tfhub.dev/tensorflow/movinet/{id}/{mode}/kinetics-600/classification/{version}'
model = hub.load(hub_url)
CPU times: user 17.2 s, sys: 853 ms, total: 18.1 s
Wall time: 18.3 s

This version of the model has one signature. It takes an image argument which is a tf.float32 with shape (batch, frames, height, width, colors). It returns a dictionary containing one output: A tf.float32 tensor of logits with shape (batch, classes).

sig = model.signatures['serving_default']
print(sig.pretty_printed_signature())
Input Parameters:
  image (KEYWORD_ONLY): TensorSpec(shape=(None, None, None, None, 3), dtype=tf.float32, name='image')
Output Type:
  Dict[['classifier_head', TensorSpec(shape=(None, 600), dtype=tf.float32, name='classifier_head')]]
Captures:
  140544822433744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822046208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822045856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822045152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822045504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822433040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822044800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822043744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822044096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822044448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822433392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822043392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042354352: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822043040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822042688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822432336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822431984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822447968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822447616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822432688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042353648: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042353296: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042352944: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042354000: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822446912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822447264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821768432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821759840: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821759136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821759488: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822446560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042351536: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042351888: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042352592: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042352240: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822446208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042341888: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042351184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042350832: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042342240: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822445504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822445152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822444800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822444448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822445856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042341184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042340832: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042340480: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042341536: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822444096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822406832: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042339776: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042339424: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042340128: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042339072: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822406480: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042362544: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042362192: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042338720: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042338368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822405776: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822405424: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822405072: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822404720: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822406128: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042361840: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042361488: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042361136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042360784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822404368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822404016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042360432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042360080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042359728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042359376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822403664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042366464: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042366112: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042359024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042366816: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822316896: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822316544: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822316192: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822315840: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822403312: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042365056: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042364704: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042365760: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042365408: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822315136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822315488: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821758784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821758432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821758080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821757728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822313024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042364000: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042363648: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042363296: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042364352: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822324912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042386768: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042386416: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042362944: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042387120: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822314784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822314432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822314080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822313728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822324560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042385008: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042386064: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042385712: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042385360: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822313376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822324208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042383600: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042384656: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042384304: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042383952: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822323856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042391040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042390688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042390336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042391392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822323152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822322800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822322448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822322096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822323504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042389280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042388928: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042389984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042389632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822321744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822321392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042387872: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042387520: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042388576: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042388224: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822366048: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042423984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042423632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042423280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042422928: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822365344: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822364992: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822364640: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822364288: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822365696: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042422224: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042421872: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042421520: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042422576: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822363936: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822363584: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042420464: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042415968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042421168: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042420816: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822363232: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042414912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042414560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042415616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042415264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822362528: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822362176: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822267568: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822267216: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822362880: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042414208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042413856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042413504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042413152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822266864: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822266512: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042412448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042412096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042440368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042412800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822266160: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042439312: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042438960: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042440016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042439664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822265456: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822265104: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822264752: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822264400: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822265808: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042437552: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042438608: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042438256: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042437904: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822312800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822264048: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821757376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821756320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821757024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821756672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822312448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042436848: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042436448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042436096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042437200: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822312096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042435040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042434688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042435744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042435392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822311392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822311040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822310688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822310336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822311744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042434336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042433984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042433632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042433280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822309984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822309632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042432928: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042432576: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042473136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042472784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822309280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042472080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042471728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042471376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042472432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822279856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822279504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822279152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822278800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822308928: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042470320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042469968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042471024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042470672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822278448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822278096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042460672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042460320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042469616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042461024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822277744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042459968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042459616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042459264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042458912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822277040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822276688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822276336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822234976: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822277392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042458208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042457856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042457504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042458560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822234624: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822234272: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042476880: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042476528: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042457152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042477232: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822233920: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042475120: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042476176: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042475824: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042475472: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822233216: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822232864: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822232512: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822232160: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822233568: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042474768: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042474416: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042474064: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042473712: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822231808: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822205072: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042489344: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042488992: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042488640: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042489696: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822204720: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042487584: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042487232: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042488288: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042487936: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822231456: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822231104: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822206128: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822205776: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822204368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042486880: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042486528: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042486176: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042485824: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822205424: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822204016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042501808: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042501456: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042501104: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042500752: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822203664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042500048: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042499696: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042499344: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042500400: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822202960: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822202608: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822185824: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822185472: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822203312: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042498288: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042514272: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042498992: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042498640: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822185120: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822184768: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042513216: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042512864: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042513920: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042513568: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822184416: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042512512: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042512160: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042511808: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042511456: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822183008: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822182656: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822183712: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822183360: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822184064: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042510752: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042510400: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042526384: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042511104: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822182304: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822181952: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042525328: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042524976: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042526032: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042525680: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822116016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042523568: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042524624: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042524272: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042523920: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822115312: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822114960: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822114608: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822114256: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822115664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042523216: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042522864: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042538848: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042538496: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822113904: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822113552: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042537792: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042537440: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042537088: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042538144: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822113200: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042536032: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042535680: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042536736: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042536384: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822112496: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822165344: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822164992: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822164640: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822112848: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042535328: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540042534976: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822746800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822746448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822164288: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822163936: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822746096: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822745744: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822745392: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822745040: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822163584: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822744336: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822743984: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822743632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822744688: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822162880: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822162528: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822162176: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822161824: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822163232: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822771200: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822770848: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822743280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822771552: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822161472: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822103728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822769792: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822769440: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822770496: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822770144: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822103376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822769088: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822768736: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822768384: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822768032: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822102672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822102320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822101968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822101616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822103024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822759088: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822758736: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822758384: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822767680: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822100912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822101264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540046826656: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540046825776: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821755968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140540046827184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822100560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822756976: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822758032: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822757680: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822757328: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822100208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822756624: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822756272: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822755920: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822755568: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822136320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822135968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822135616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822135264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822136672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822689280: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822688928: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822688576: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822689632: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822134912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822134560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822687520: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822687168: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822688224: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822687872: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822134208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822686816: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822686464: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822686112: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822685760: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822133504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822133152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822132800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822083248: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822133856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822726320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822725968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822725616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822725264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822082896: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822080432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822724560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822724208: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822723856: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822724912: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822080080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822722800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822751072: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822723504: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822723152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822082192: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822081840: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822081488: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822081136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822082544: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822750720: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822750368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822750016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822749664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822080784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822079728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822749312: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822748960: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822748608: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822748256: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822071136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822747552: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822747200: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822742704: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822747904: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822070432: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822070080: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822069728: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822069376: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822070784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822741296: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822742352: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822742000: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822741648: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822069024: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822068672: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822739888: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822740944: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822740592: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822740240: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822068320: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822739536: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822739184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822685536: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822685184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822067616: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822067264: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822030000: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822029648: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822067968: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822684480: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822684128: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822683776: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822684832: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822029296: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822028944: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822682720: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822682368: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822683424: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822683072: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822028592: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822682016: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822681664: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821771952: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821771600: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822027888: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822027536: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822027184: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822026832: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822028240: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821771248: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821770896: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821770544: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821770192: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822026480: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822046560: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821769488: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821769136: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821768784: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544821769840: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822435152: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822434800: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822434448: TensorSpec(shape=(), dtype=tf.resource, name=None)
  140544822434096: TensorSpec(shape=(), dtype=tf.resource, name=None)

To run this signature on the video you need to add the outer batch dimension to the video first.

#warmup
sig(image = jumpingjack[tf.newaxis, :1]);
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1706799716.996619   36069 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
2024-02-01 15:01:56.996884: E external/local_xla/xla/stream_executor/stream_executor_internal.h:177] SetPriority unimplemented for this stream.
2024-02-01 15:01:58.081007: E external/local_xla/xla/stream_executor/stream_executor_internal.h:177] SetPriority unimplemented for this stream.
%%time
logits = sig(image = jumpingjack[tf.newaxis, ...])
logits = logits['classifier_head'][0]

print(logits.shape)
print()
(600,)

CPU times: user 24 s, sys: 368 ms, total: 24.4 s
Wall time: 14.2 s

Define a get_top_k function that packages the above output processing for later.

Convert the logits to probabilities, and look up the top 5 classes for the video. The model confirms that the video is probably of jumping jacks.

probs = tf.nn.softmax(logits, axis=-1)
for label, p in get_top_k(probs):
  print(f'{label:20s}: {p:.3f}')
jumping jacks       : 0.834
zumba               : 0.008
lunge               : 0.003
doing aerobics      : 0.003
polishing metal     : 0.002

The streaming model

The previous section used a model that runs over a whole video. Often when processing a video you don't want a single prediction at the end, you want to update predictions frame by frame. The stream versions of the model allow you to do this.

Load the stream version of the model.

%%time
id = 'a2'
mode = 'stream'
version = '3'
hub_url = f'https://tfhub.dev/tensorflow/movinet/{id}/{mode}/kinetics-600/classification/{version}'
model = hub.load(hub_url)
WARNING:absl:`state/b3/l0/pool_frame_count` is not a valid tf.function parameter name. Sanitizing to `state_b3_l0_pool_frame_count`.
WARNING:absl:`state/b3/l5/pool_frame_count` is not a valid tf.function parameter name. Sanitizing to `state_b3_l5_pool_frame_count`.
WARNING:absl:`state/b2/l1/stream_buffer` is not a valid tf.function parameter name. Sanitizing to `state_b2_l1_stream_buffer`.
WARNING:absl:`state/b4/l5/stream_buffer` is not a valid tf.function parameter name. Sanitizing to `state_b4_l5_stream_buffer`.
WARNING:absl:`state/b2/l4/stream_buffer` is not a valid tf.function parameter name. Sanitizing to `state_b2_l4_stream_buffer`.
CPU times: user 49.5 s, sys: 2.26 s, total: 51.7 s
Wall time: 52 s

Using this model is slightly more complex than the base model. You have to keep track of the internal state of the model's RNNs.

list(model.signatures.keys())
['call', 'init_states']

The init_states signature takes the video's shape (batch, frames, height, width, colors) as input, and returns a large dictionary of tensors containing the initial RNN states:

lines = model.signatures['init_states'].pretty_printed_signature().splitlines()
lines = lines[:10]
lines.append('      ...')
print('.\n'.join(lines))
Input Parameters:.
  input_shape (KEYWORD_ONLY): TensorSpec(shape=(5,), dtype=tf.int32, name='input_shape').
Output Type:.
  Dict[['state/b4/l5/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 480), dtype=tf.float32, name='state/b4/l5/pool_buffer')], ['state/b0/l1/stream_buffer', TensorSpec(shape=(None, 2, None, None, 40), dtype=tf.float32, name='state/b0/l1/stream_buffer')], ['state/b2/l0/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b2/l0/pool_frame_count')], ['state/b2/l4/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b2/l4/pool_buffer')], ['state/b1/l4/stream_buffer', TensorSpec(shape=(None, 2, None, None, 120), dtype=tf.float32, name='state/b1/l4/stream_buffer')], ['state/b4/l3/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 480), dtype=tf.float32, name='state/b4/l3/pool_buffer')], ['state/b3/l5/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b3/l5/stream_buffer')], ['state/b1/l4/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b1/l4/pool_frame_count')], ['state/b4/l5/stream_buffer', TensorSpec(shape=(None, 2, None, None, 480), dtype=tf.float32, name='state/b4/l5/stream_buffer')], ['state/b1/l2/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b1/l2/pool_frame_count')], ['state/b1/l0/stream_buffer', TensorSpec(shape=(None, 2, None, None, 96), dtype=tf.float32, name='state/b1/l0/stream_buffer')], ['state/b2/l3/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 192), dtype=tf.float32, name='state/b2/l3/pool_buffer')], ['state/b4/l2/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 384), dtype=tf.float32, name='state/b4/l2/pool_buffer')], ['state/b1/l1/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b1/l1/pool_frame_count')], ['state/b3/l3/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b3/l3/pool_buffer')], ['state/b4/l0/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 480), dtype=tf.float32, name='state/b4/l0/pool_buffer')], ['state/b4/l4/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l4/pool_frame_count')], ['state/b1/l0/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b1/l0/pool_frame_count')], ['state/b2/l4/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b2/l4/pool_frame_count')], ['state/b2/l3/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b2/l3/pool_frame_count')], ['state/b4/l0/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l0/pool_frame_count')], ['state/b2/l1/stream_buffer', TensorSpec(shape=(None, 2, None, None, 160), dtype=tf.float32, name='state/b2/l1/stream_buffer')], ['state/b2/l1/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b2/l1/pool_frame_count')], ['state/b1/l3/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 96), dtype=tf.float32, name='state/b1/l3/pool_buffer')], ['state/b1/l0/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 96), dtype=tf.float32, name='state/b1/l0/pool_buffer')], ['state/b2/l2/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b2/l2/pool_buffer')], ['state/b1/l3/stream_buffer', TensorSpec(shape=(None, 2, None, None, 96), dtype=tf.float32, name='state/b1/l3/stream_buffer')], ['state/b3/l2/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b3/l2/pool_buffer')], ['state/b1/l1/stream_buffer', TensorSpec(shape=(None, 2, None, None, 120), dtype=tf.float32, name='state/b1/l1/stream_buffer')], ['state/b3/l4/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l4/pool_frame_count')], ['state/b1/l1/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 120), dtype=tf.float32, name='state/b1/l1/pool_buffer')], ['state/b3/l0/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b3/l0/pool_buffer')], ['state/b2/l3/stream_buffer', TensorSpec(shape=(None, 2, None, None, 192), dtype=tf.float32, name='state/b2/l3/stream_buffer')], ['state/head/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/head/pool_frame_count')], ['state/b1/l2/stream_buffer', TensorSpec(shape=(None, 2, None, None, 96), dtype=tf.float32, name='state/b1/l2/stream_buffer')], ['state/b3/l0/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l0/pool_frame_count')], ['state/b3/l2/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b3/l2/stream_buffer')], ['state/b0/l1/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b0/l1/pool_frame_count')], ['state/b0/l2/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b0/l2/pool_frame_count')], ['state/b2/l0/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b2/l0/pool_buffer')], ['state/b3/l3/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l3/pool_frame_count')], ['state/b3/l2/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l2/pool_frame_count')], ['state/b2/l2/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b2/l2/pool_frame_count')], ['state/head/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 640), dtype=tf.float32, name='state/head/pool_buffer')], ['state/b4/l4/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 480), dtype=tf.float32, name='state/b4/l4/pool_buffer')], ['state/b3/l0/stream_buffer', TensorSpec(shape=(None, 4, None, None, 240), dtype=tf.float32, name='state/b3/l0/stream_buffer')], ['state/b0/l2/stream_buffer', TensorSpec(shape=(None, 2, None, None, 64), dtype=tf.float32, name='state/b0/l2/stream_buffer')], ['state/b3/l1/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b3/l1/stream_buffer')], ['state/b3/l3/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b3/l3/stream_buffer')], ['state/b4/l6/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l6/pool_frame_count')], ['state/b0/l0/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b0/l0/pool_frame_count')], ['state/b3/l5/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l5/pool_frame_count')], ['state/b3/l4/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 144), dtype=tf.float32, name='state/b3/l4/pool_buffer')], ['state/b3/l1/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b3/l1/pool_buffer')], ['state/b4/l2/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l2/pool_frame_count')], ['state/b4/l1/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 384), dtype=tf.float32, name='state/b4/l1/pool_buffer')], ['state/b4/l0/stream_buffer', TensorSpec(shape=(None, 4, None, None, 480), dtype=tf.float32, name='state/b4/l0/stream_buffer')], ['state/b2/l1/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 160), dtype=tf.float32, name='state/b2/l1/pool_buffer')], ['state/b3/l1/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b3/l1/pool_frame_count')], ['state/b0/l2/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 64), dtype=tf.float32, name='state/b0/l2/pool_buffer')], ['state/b1/l4/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 120), dtype=tf.float32, name='state/b1/l4/pool_buffer')], ['state/b2/l0/stream_buffer', TensorSpec(shape=(None, 4, None, None, 240), dtype=tf.float32, name='state/b2/l0/stream_buffer')], ['state/b2/l4/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b2/l4/stream_buffer')], ['state/b1/l3/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b1/l3/pool_frame_count')], ['state/b4/l3/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l3/pool_frame_count')], ['state/b0/l0/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 40), dtype=tf.float32, name='state/b0/l0/pool_buffer')], ['state/b2/l2/stream_buffer', TensorSpec(shape=(None, 2, None, None, 240), dtype=tf.float32, name='state/b2/l2/stream_buffer')], ['state/b4/l5/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l5/pool_frame_count')], ['state/b0/l1/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 40), dtype=tf.float32, name='state/b0/l1/pool_buffer')], ['state/b3/l5/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 240), dtype=tf.float32, name='state/b3/l5/pool_buffer')], ['state/b1/l2/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 96), dtype=tf.float32, name='state/b1/l2/pool_buffer')], ['state/b4/l6/pool_buffer', TensorSpec(shape=(None, 1, 1, 1, 576), dtype=tf.float32, name='state/b4/l6/pool_buffer')], ['state/b4/l1/pool_frame_count', TensorSpec(shape=(1,), dtype=tf.int32, name='state/b4/l1/pool_frame_count')]].
Captures:.
  None.
      ...
initial_state = model.init_states(jumpingjack[tf.newaxis, ...].shape)
type(initial_state)
dict
list(sorted(initial_state.keys()))[:5]
['state/b0/l0/pool_buffer',
 'state/b0/l0/pool_frame_count',
 'state/b0/l1/pool_buffer',
 'state/b0/l1/pool_frame_count',
 'state/b0/l1/stream_buffer']

Once you have the initial state for the RNNs, you can pass the state and a video frame as input (keeping the (batch, frames, height, width, colors) shape for the video frame). The model returns a (logits, state) pair.

After just seeing the first frame, the model is not convinced that the video is of "jumping jacks":

inputs = initial_state.copy()

# Add the batch axis, take the first frme, but keep the frame-axis.
inputs['image'] = jumpingjack[tf.newaxis, 0:1, ...]
# warmup
model(inputs);
logits, new_state = model(inputs)
logits = logits[0]
probs = tf.nn.softmax(logits, axis=-1)

for label, p in get_top_k(probs):
  print(f'{label:20s}: {p:.3f}')

print()
golf chipping       : 0.427
tackling            : 0.134
lunge               : 0.056
stretching arm      : 0.053
passing american football (not in game): 0.039

If you run the model in a loop, passing the updated state with each frame, the model quickly converges to the correct result:

%%time
state = initial_state.copy()
all_logits = []

for n in range(len(jumpingjack)):
  inputs = state
  inputs['image'] = jumpingjack[tf.newaxis, n:n+1, ...]
  result, state = model(inputs)
  all_logits.append(logits)

probabilities = tf.nn.softmax(all_logits, axis=-1)
CPU times: user 1.46 s, sys: 472 ms, total: 1.93 s
Wall time: 728 ms
for label, p in get_top_k(probabilities[-1]):
  print(f'{label:20s}: {p:.3f}')
golf chipping       : 0.427
tackling            : 0.134
lunge               : 0.056
stretching arm      : 0.053
passing american football (not in game): 0.039
id = tf.argmax(probabilities[-1])
plt.plot(probabilities[:, id])
plt.xlabel('Frame #')
plt.ylabel(f"p('{KINETICS_600_LABELS[id]}')");

png

You may notice that the final probability is much more certain than in the previous section where you ran the base model. The base model returns an average of the predictions over the frames.

for label, p in get_top_k(tf.reduce_mean(probabilities, axis=0)):
  print(f'{label:20s}: {p:.3f}')
golf chipping       : 0.427
tackling            : 0.134
lunge               : 0.056
stretching arm      : 0.053
passing american football (not in game): 0.039

Animate the predictions over time

The previous section went into some details about how to use these models. This section builds on top of that to produce some nice inference animations.

The hidden cell below to defines helper functions used in this section.

Start by running the streaming model across the frames of the video, and collecting the logits:

init_states = model.init_states(jumpingjack[tf.newaxis].shape)
# Insert your video clip here
video = jumpingjack
images = tf.split(video[tf.newaxis], video.shape[0], axis=1)

all_logits = []

# To run on a video, pass in one frame at a time
states = init_states
for image in tqdm.tqdm(images):
  # predictions for each frame
  logits, states = model({**states, 'image': image})
  all_logits.append(logits)

# concatenating all the logits
logits = tf.concat(all_logits, 0)
# estimating probabilities
probs = tf.nn.softmax(logits, axis=-1)
100%|██████████| 13/13 [00:00<00:00, 18.23it/s]
final_probs = probs[-1]
print('Top_k predictions and their probablities\n')
for label, p in get_top_k(final_probs):
  print(f'{label:20s}: {p:.3f}')
Top_k predictions and their probablities

jumping jacks       : 0.999
zumba               : 0.000
doing aerobics      : 0.000
dancing charleston  : 0.000
slacklining         : 0.000

Convert the sequence of probabilities into a video:

# Generate a plot and output to a video tensor
plot_video = plot_streaming_top_preds(probs, video, video_fps=8.)
0%|          | 0/13 [00:00<?, ?it/s]/tmpfs/tmp/ipykernel_35048/567636217.py:112: MatplotlibDeprecationWarning: The tostring_rgb function was deprecated in Matplotlib 3.8 and will be removed two minor releases later. Use buffer_rgba instead.
  data = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8)
100%|██████████| 13/13 [00:06<00:00,  1.89it/s]
# For gif format, set codec='gif'
media.show_video(plot_video, fps=3)

Resources

The pretrained models are available from TF Hub. The TF Hub collection also includes quantized models optimized for TFLite.

The source for these models is available in the TensorFlow Model Garden. This includes a longer version of this tutorial that also covers building and fine-tuning a MoViNet model.

Next Steps

To learn more about working with video data in TensorFlow, check out the following tutorials: