Performs segmentation on images.
The API expects a TFLite model with TFLite Model Metadata..
The API supports models with one image input tensor and one output tensor. To be more specific, here are the requirements.
- Input image tensor (
kTfLiteUInt8
/kTfLiteFloat32
)- image input of size
[batch x height x width x channels]
. - batch inference is not supported (
batch
is required to be 1). - only RGB inputs are supported (
channels
is required to be 3). - if type is
kTfLiteFloat32
, NormalizationOptions are required to be attached to the metadata for input normalization.
- image input of size
kTfLiteUInt8
/kTfLiteFloat32
)
- tensor of size
[batch x mask_height x mask_width x num_classes]
, wherebatch
is required to be 1,mask_width
andmask_height
are the dimensions of the segmentation masks produced by the model, andnum_classes
is the number of classes supported by the model. - optional (but recommended) label map(s) can be attached as AssociatedFile-s with type
TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if
any) is used to fill the class name, i.e.
ColoredLabel.getlabel()
of the results. The display name, i.e.ColoredLabel.getDisplayName()
, is filled from the AssociatedFile (if any) whose locale matches the `display_names_locale` field of the `ImageSegmenterOptions` used at creation time ("en" by default, i.e. English). If none of these are available, only the `index` field of the results will be filled.
An example of such model can be found on TensorFlow Hub..
Nested Classes
class | ImageSegmenter.ImageSegmenterOptions | Options for setting up an ImageSegmenter . |
Public Methods
Inherited Methods
Public Methods
public static ImageSegmenter createFromBuffer (ByteBuffer modelBuffer)
Creates an ImageSegmenter
instance with a model buffer and the default ImageSegmenter.ImageSegmenterOptions
.
Parameters
modelBuffer | a direct ByteBuffer or a MappedByteBuffer of the
segmentation model |
---|
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the model buffer is not a direct ByteBuffer or a
MappedByteBuffer
|
public static ImageSegmenter createFromBufferAndOptions (ByteBuffer modelBuffer, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter
instance with a model buffer and ImageSegmenter.ImageSegmenterOptions
.
Parameters
modelBuffer | a direct ByteBuffer or a MappedByteBuffer of the
segmentation model |
---|---|
options |
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the model buffer is not a direct ByteBuffer or a
MappedByteBuffer
|
public static ImageSegmenter createFromFile (Context context, String modelPath)
Creates an ImageSegmenter
instance from the default ImageSegmenter.ImageSegmenterOptions
.
Parameters
context | |
---|---|
modelPath | path of the segmentation model with metadata in the assets |
Throws
IOException | if an I/O error occurs when loading the tflite model |
---|---|
IllegalArgumentException | if an argument is invalid |
IllegalStateException | if there is an internal error |
RuntimeException | if there is an otherwise unspecified error |
public static ImageSegmenter createFromFile (File modelFile)
Creates an ImageSegmenter
instance from the default ImageSegmenter.ImageSegmenterOptions
.
Parameters
modelFile | the segmentation model File instance |
---|
Throws
IOException | if an I/O error occurs when loading the tflite model |
---|---|
IllegalArgumentException | if an argument is invalid |
IllegalStateException | if there is an internal error |
RuntimeException | if there is an otherwise unspecified error |
public static ImageSegmenter createFromFileAndOptions (File modelFile, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter
instance from ImageSegmenter.ImageSegmenterOptions
.
Parameters
modelFile | the segmentation model File instance |
---|---|
options |
Throws
IOException | if an I/O error occurs when loading the tflite model |
---|---|
IllegalArgumentException | if an argument is invalid |
IllegalStateException | if there is an internal error |
RuntimeException | if there is an otherwise unspecified error |
public static ImageSegmenter createFromFileAndOptions (Context context, String modelPath, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter
instance from ImageSegmenter.ImageSegmenterOptions
.
Parameters
context | |
---|---|
modelPath | path of the segmentation model with metadata in the assets |
options |
Throws
IOException | if an I/O error occurs when loading the tflite model |
---|---|
IllegalArgumentException | if an argument is invalid |
IllegalStateException | if there is an internal error |
RuntimeException | if there is an otherwise unspecified error |
public List<Segmentation> segment (long frameBufferHandle, ImageProcessingOptions options)
Parameters
frameBufferHandle | |
---|---|
options |
public List<Segmentation> segment (TensorImage image)
Performs actual segmentation on the provided image.
ImageSegmenter
supports the following TensorImage
color space types:
Parameters
image | a UINT8 TensorImage object that represents an RGB or YUV image |
---|
Returns
- results of performing image segmentation. Note that at the time, a single
Segmentation
element is expected to be returned. The result is stored in aList
for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the color space type of image is unsupported |
public List<Segmentation> segment (MlImage image)
Performs actual segmentation on the provided MlImage
.
Parameters
image | an MlImage to segment. |
---|
Returns
- results of performing image segmentation. Note that at the time, a single
Segmentation
element is expected to be returned. The result is stored in aList
for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the storage type or format of the image is unsupported |
public List<Segmentation> segment (TensorImage image, ImageProcessingOptions options)
Performs actual segmentation on the provided image with ImageProcessingOptions
.
ImageSegmenter
supports the following TensorImage
color space types:
ImageSegmenter
supports the following options:
- image rotation (through
ImageProcessingOptions.Builder.setOrientation(ImageProcessingOptions.Orientation)
). It defaults toImageProcessingOptions.Orientation.TOP_LEFT
Parameters
image | a UINT8 TensorImage object that represents an RGB or YUV image |
---|---|
options | the options configure how to preprocess the image |
Returns
- results of performing image segmentation. Note that at the time, a single
Segmentation
element is expected to be returned. The result is stored in aList
for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the color space type of image is unsupported |
public List<Segmentation> segment (MlImage image, ImageProcessingOptions options)
Performs actual segmentation on the provided MlImage
with ImageProcessingOptions
.
ImageSegmenter
supports the following options:
- image rotation (through
ImageProcessingOptions.Builder.setOrientation(ImageProcessingOptions.Orientation)
). It defaults toImageProcessingOptions.Orientation.TOP_LEFT
.MlImage.getRotation()
is not effective.
Parameters
image | an MlImage to segment. |
---|---|
options | the options configure how to preprocess the image. |
Returns
- results of performing image segmentation. Note that at the time, a single
Segmentation
element is expected to be returned. The result is stored in aList
for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException | if there is an internal error |
---|---|
RuntimeException | if there is an otherwise unspecified error |
IllegalArgumentException | if the color space type of image is unsupported |