Super resolution with TensorFlow Lite

View on Run in Google Colab View source on GitHub Download notebook See TF Hub model


The task of recovering a high resolution (HR) image from its low resolution counterpart is commonly referred to as Single Image Super Resolution (SISR).

The model used here is ESRGAN (ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks). And we are going to use TensorFlow Lite to run inference on the pretrained model.

The TFLite model is converted from this implementation hosted on TF Hub. Note that the model we converted upsamples a 50x50 low resolution image to a 200x200 high resolution image (scale factor=4). If you want a different input size or scale factor, you need to re-convert or re-train the original model.


Let's install required libraries first.

pip install matplotlib tensorflow tensorflow-hub

Import dependencies.

import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
2022-08-31 11:14:42.737109: E tensorflow/stream_executor/cuda/] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-08-31 11:14:43.481023: W tensorflow/stream_executor/platform/default/] Could not load dynamic library ''; dlerror: cannot open shared object file: No such file or directory
2022-08-31 11:14:43.481281: W tensorflow/stream_executor/platform/default/] Could not load dynamic library ''; dlerror: cannot open shared object file: No such file or directory
2022-08-31 11:14:43.481296: W tensorflow/compiler/tf2tensorrt/utils/] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

Download and convert the ESRGAN model

model = hub.load("")
concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]

@tf.function(input_signature=[tf.TensorSpec(shape=[1, 50, 50, 3], dtype=tf.float32)])
def f(input):
  return concrete_func(input);

converter = tf.lite.TFLiteConverter.from_concrete_functions([f.get_concrete_function()], model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Save the TF Lite model.
with'ESRGAN.tflite', 'wb') as f:

esrgan_model_path = './ESRGAN.tflite'
WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 134). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /tmpfs/tmp/tmpvlxdiayo/assets
INFO:tensorflow:Assets written to: /tmpfs/tmp/tmpvlxdiayo/assets
2022-08-31 11:15:03.795559: W tensorflow/compiler/mlir/lite/python/] Ignored output_format.
2022-08-31 11:15:03.795632: W tensorflow/compiler/mlir/lite/python/] Ignored drop_control_dependency.

Download a test image (insect head).

test_img_path = tf.keras.utils.get_file('lr.jpg', '')
Downloading data from
6432/6432 [==============================] - 0s 0us/step

Generate a super resolution image using TensorFlow Lite

lr =
lr = tf.image.decode_jpeg(lr)
lr = tf.expand_dims(lr, axis=0)
lr = tf.cast(lr, tf.float32)

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path=esrgan_model_path)

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Run the model
interpreter.set_tensor(input_details[0]['index'], lr)

# Extract the output and postprocess it
output_data = interpreter.get_tensor(output_details[0]['index'])
sr = tf.squeeze(output_data, axis=0)
sr = tf.clip_by_value(sr, 0, 255)
sr = tf.round(sr)
sr = tf.cast(sr, tf.uint8)
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Visualize the result

lr = tf.cast(tf.squeeze(lr, axis=0), tf.uint8)
plt.figure(figsize = (1, 1))

plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)        
plt.title(f'ESRGAN (x4)')

bicubic = tf.image.resize(lr, [200, 200], tf.image.ResizeMethod.BICUBIC)
bicubic = tf.cast(bicubic, tf.uint8)
plt.subplot(1, 2, 2)   



Performance Benchmarks

Performance benchmark numbers are generated with the tool described here.

Model Name Model Size Device CPU GPU
super resolution (ESRGAN) 4.8 Mb Pixel 3 586.8ms* 128.6ms
Pixel 4 385.1ms* 130.3ms

*4 threads used