Image segmenters predict whether each pixel of an image is associated with a certain class. This is in contrast to object detection, which detects objects in rectangular regions, and image classification, which classifies the overall image. See the introduction of image segmentation for more information about image segmenters.
Use the Task Library ImageSegmenter
API to deploy your custom image segmenters
or pretrained ones into your model apps.
Key features of the ImageSegmenter API
Input image processing, including rotation, resizing, and color space conversion.
Label map locale.
Two output types, category mask and confidence masks.
Colored label for display purpose.
Supported image segmenter models
The following models are guaranteed to be compatible with the ImageSegmenter
API.
Custom models that meet the model compatibility requirements.
Run inference in Java
See the
Image Segmentation reference app
for an example of how to use ImageSegmenter
in an Android app.
Step 1: Import Gradle dependency and other settings
Copy the .tflite
model file to the assets directory of the Android module
where the model will be run. Specify that the file should not be compressed, and
add the TensorFlow Lite library to the module’s build.gradle
file:
android {
// Other settings
// Specify tflite file should not be compressed for the app apk
aaptOptions {
noCompress "tflite"
}
}
dependencies {
// Other dependencies
// Import the Task Vision Library dependency
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.0.0-nightly'
}
Step 2: Using the model
// Initialization
ImageSegmenterOptions options = ImageSegmenterOptions.builder().setOutputType(OutputType.CONFIDENCE_MASK).build();
ImageSegmenter imageSegmenter = ImageSegmenter.createFromFileAndOptions(context, modelFile, options);
// Run inference
List<Segmentation> results = imageSegmenter.segment(image);
See the
source code and javadoc
for more options to configure ImageSegmenter
.
Run inference in C++
// Initialization
ImageSegmenterOptions options;
options.mutable_model_file_with_metadata()->set_file_name(model_file);
std::unique_ptr<ImageSegmenter> image_segmenter = ImageSegmenter::CreateFromOptions(options).value();
// Run inference
const SegmentationResult result = image_segmenter->Segment(*frame_buffer).value();
See the
source code
for more options to configure ImageSegmenter
.
Example results
Here is an example of the segmentation results of deeplab_v3, a generic segmentation model available on TensorFlow Hub.
Color Legend:
(r: 000, g: 000, b: 000):
index : 0
class name : background
(r: 128, g: 000, b: 000):
index : 1
class name : aeroplane
# (omitting multiple lines for conciseness) ...
(r: 128, g: 192, b: 000):
index : 19
class name : train
(r: 000, g: 064, b: 128):
index : 20
class name : tv
Tip: use a color picker on the output PNG file to inspect the output mask with
this legend.
The segmentation category mask should looks like:
Try out the simple CLI demo tool for ImageSegmenter with your own model and test data.
Model compatibility requirements
The ImageSegmenter
API expects a TFLite model with mandatory
TFLite Model Metadata.
Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
- image input of size
[batch x height x width x channels]
. - batch inference is not supported (
batch
is required to be 1). - only RGB inputs are supported (
channels
is required to be 3). - if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
- image input of size
Output masks tensor: (kTfLiteUInt8/kTfLiteFloat32)
tensor of size
[batch x mask_height x mask_width x num_classes]
, wherebatch
is required to be 1,mask_width
andmask_height
are the dimensions of the segmentation masks produced by the model, andnum_classes
is the number of classes supported by the model.optional (but recommended) label map(s) can be attached as AssociatedFile-s with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the
label
field (named asclass_name
in C++) of the results. Thedisplay_name
field is filled from the AssociatedFile (if any) whose locale matches thedisplay_names_locale
field of theImageSegmenterOptions
used at creation time ("en" by default, i.e. English). If none of these are available, only theindex
field of the results will be filled.