TensorFlow Lite in Google Play services can also be accessed using Java APIs, in addition to the Native API. In particular, TensorFlow Lite in Google Play services is available through the TensorFlow Lite Task API and the TensorFlow Lite Interpreter API. The Task Library provides optimized out-of-the-box model interfaces for common machine learning tasks using visual, audio, and text data. The TensorFlow Lite Interpreter API, provided by the TensorFlow runtime, provides a more general-purpose interface for building and running ML models.
The following sections provide instructions on how to use the Interpreter and Task Library APIs with TensorFlow Lite in Google Play services. While it is possible for an app to use both the Interpreter APIs and Task Library APIs, most apps should only use one set of APIs.
Using the Task Library APIs
The TensorFlow Lite Task API wraps the Interpreter API and provides a high-level programming interface for common machine learning tasks that use visual, audio, and text data. You should use the Task API if your application requires one of the supported tasks.
1. Add project dependencies
Your project dependency depends on your machine learning use case. The Task APIs contain the following libraries:
- Vision library:
org.tensorflow:tensorflow-lite-task-vision-play-services
- Audio library:
org.tensorflow:tensorflow-lite-task-audio-play-services
- Text library:
org.tensorflow:tensorflow-lite-task-text-play-services
Add one of the dependencies to your app project code to access the Play services API for TensorFlow Lite. For example, use the following to implement a vision task:
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-vision-play-services:0.4.2'
...
}
2. Add initialization of TensorFlow Lite
Initialize the TensorFlow Lite component of the Google Play services API before using the TensorFlow Lite APIs. The following example initializes the vision library:
Kotlin
init { TfLiteVision.initialize(context) }
3. Run inferences
After initializing the TensorFlow Lite component, call the detect()
method to
generate inferences. The exact code within the detect()
method varies
depending on the library and use case. The following is for a simple object
detection use case with the TfLiteVision
library:
Kotlin
fun detect(...) { if (!TfLiteVision.isInitialized()) { Log.e(TAG, "detect: TfLiteVision is not initialized yet") return } if (objectDetector == null) { setupObjectDetector() } ... }
Depending on the data format, you may also need to preprocess and convert your
data within the detect()
method before generating inferences. For example,
image data for an object detector requires the following:
val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
val results = objectDetector?.detect(tensorImage)
Using the Interpreter APIs
The Interpreter APIs offer more control and flexibility than the Task Library APIs. You should use the Interpreter APIs if your machine learning task is not supported by the Task library, or if you require a more general-purpose interface for building and running ML models.
1. Add project dependencies
Add the following dependencies to your app project code to access the Play services API for TensorFlow Lite:
dependencies {
...
// Tensorflow Lite dependencies for Google Play services
implementation 'com.google.android.gms:play-services-tflite-java:16.0.1'
// Optional: include Tensorflow Lite Support Library
implementation 'com.google.android.gms:play-services-tflite-support:16.0.1'
...
}
2. Add initialization of TensorFlow Lite
Initialize the TensorFlow Lite component of the Google Play services API before using the TensorFlow Lite APIs:
Kotlin
val initializeTask: Task<Void> by lazy { TfLite.initialize(this) }
Java
Task<Void> initializeTask = TfLite.initialize(context);
3. Create an Interpreter and set runtime option
Create an interpreter using InterpreterApi.create()
and configure it to use
Google Play services runtime, by calling InterpreterApi.Options.setRuntime()
,
as shown in the following example code:
Kotlin
import org.tensorflow.lite.InterpreterApi import org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime ... private lateinit var interpreter: InterpreterApi ... initializeTask.addOnSuccessListener { val interpreterOption = InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY) interpreter = InterpreterApi.create( modelBuffer, interpreterOption )} .addOnFailureListener { e -> Log.e("Interpreter", "Cannot initialize interpreter", e) }
Java
import org.tensorflow.lite.InterpreterApi import org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime ... private InterpreterApi interpreter; ... initializeTask.addOnSuccessListener(a -> { interpreter = InterpreterApi.create(modelBuffer, new InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)); }) .addOnFailureListener(e -> { Log.e("Interpreter", String.format("Cannot initialize interpreter: %s", e.getMessage())); });
You should use the implementation above because it avoids blocking the Android
user interface thread. If you need to manage thread execution more closely, you
can add a Tasks.await()
call to interpreter creation:
Kotlin
import androidx.lifecycle.lifecycleScope ... lifecycleScope.launchWhenStarted { // uses coroutine initializeTask.await() }
Java
@BackgroundThread InterpreterApi initializeInterpreter() { Tasks.await(initializeTask); return InterpreterApi.create(...); }
4. Run inferences
Using the interpreter
object you created, call the run()
method to generate
an inference.
Kotlin
interpreter.run(inputBuffer, outputBuffer)
Java
interpreter.run(inputBuffer, outputBuffer);
Hardware acceleration
TensorFlow Lite allows you to accelerate the performance of your model using specialized hardware processors, such as graphics processing units (GPUs). You can take advantage of these specialized processors using hardware drivers called delegates. You can use the following hardware acceleration delegates with TensorFlow Lite in Google Play services:
GPU delegate (recommended) - This delegate is provided through Google Play services and is dynamically loaded, just like the Play services versions of the Task API and Interpreter API.
NNAPI delegate - This delegate is available as an included library dependency in your Android development project, and is bundled into your app.
For more information about hardware acceleration with TensorFlow Lite, see the TensorFlow Lite Delegates page.
Checking device compatibility
Not all devices support GPU hardware acceleration with TFLite. In order to
mitigate errors and potential crashes, use the
TfLiteGpu.isGpuDelegateAvailable
method to check whether a device is
compatible with the GPU delegate.
Use this method to confirm whether a device is compatible with GPU, and use CPU or the NNAPI delegate as a fallback for when GPU is not supported.
useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
Once you have a variable like useGpuTask
, you can use it to determine whether
devices use the GPU delegate. The following examples show how this can be done
with both the Task Library and Interpreter APIs.
With the Task Api
Kotlin
lateinit val optionsTask = useGpuTask.continueWith { task -> val baseOptionsBuilder = BaseOptions.builder() if (task.result) { baseOptionsBuilder.useGpu() } ObjectDetectorOptions.builder() .setBaseOptions(baseOptionsBuilder.build()) .setMaxResults(1) .build() }
Java
Task<ObjectDetectorOptions> optionsTask = useGpuTask.continueWith({ task -> BaseOptions baseOptionsBuilder = BaseOptions.builder(); if (task.getResult()) { baseOptionsBuilder.useGpu(); } return ObjectDetectorOptions.builder() .setBaseOptions(baseOptionsBuilder.build()) .setMaxResults(1) .build() });
With the Interpreter Api
Kotlin
val interpreterTask = useGpuTask.continueWith { task -> val interpreterOptions = InterpreterApi.Options() .setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY) if (task.result) { interpreterOptions.addDelegateFactory(GpuDelegateFactory()) } InterpreterApi.create(FileUtil.loadMappedFile(context, MODEL_PATH), interpreterOptions) }
Java
Task<InterpreterApi.Options> interpreterOptionsTask = useGpuTask.continueWith({ task -> InterpreterApi.Options options = new InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY); if (task.getResult()) { options.addDelegateFactory(new GpuDelegateFactory()); } return options; });
GPU with Task Library APIs
To use the GPU delegate with the Task APIs:
Update the project dependencies to use the GPU delegate from Play services:
implementation 'com.google.android.gms:play-services-tflite-gpu:16.1.0'
Initialize the GPU delegate with
setEnableGpuDelegateSupport
. For example, you can initialize the GPU delegate forTfLiteVision
with the following:Kotlin
TfLiteVision.initialize(context, TfLiteInitializationOptions.builder().setEnableGpuDelegateSupport(true).build())
Java
TfLiteVision.initialize(context, TfLiteInitializationOptions.builder().setEnableGpuDelegateSupport(true).build());
Enable the GPU delegate option with
BaseOptions
:Kotlin
val baseOptions = BaseOptions.builder().useGpu().build()
Java
BaseOptions baseOptions = BaseOptions.builder().useGpu().build();
Configure the options using
.setBaseOptions
. For example, you can set up GPU inObjectDetector
with the following:Kotlin
val options = ObjectDetectorOptions.builder() .setBaseOptions(baseOptions) .setMaxResults(1) .build()
Java
ObjectDetectorOptions options = ObjectDetectorOptions.builder() .setBaseOptions(baseOptions) .setMaxResults(1) .build();
GPU with Interpreter APIs
To use the GPU delegate with the Interpreter APIs:
Update the project dependencies to use the GPU delegate from Play services:
implementation 'com.google.android.gms:play-services-tflite-gpu:16.1.0'
Enable the GPU delegate option in the TFlite initialization:
Kotlin
TfLite.initialize(context, TfLiteInitializationOptions.builder() .setEnableGpuDelegateSupport(true) .build())
Java
TfLite.initialize(context, TfLiteInitializationOptions.builder() .setEnableGpuDelegateSupport(true) .build());
Enable GPU delegate in the interpreter options: set the delegate factory to GpuDelegateFactory by calling
addDelegateFactory() within
InterpreterApi.Options()`:Kotlin
val interpreterOption = InterpreterApi.Options() .setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY) .addDelegateFactory(GpuDelegateFactory())
Java
Options interpreterOption = InterpreterApi.Options() .setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY) .addDelegateFactory(new GpuDelegateFactory());
Migrating from stand-alone TensorFlow Lite
If you are planning to migrate your app from stand-alone TensorFlow Lite to the Play services API, review the following additional guidance for updating your app project code:
- Review the Limitations section of this page to ensure your use case is supported.
- Prior to updating your code, do performance and accuracy checks for your models, particularly if you are using versions of TensorFlow Lite earlier than version 2.1, so you have a baseline to compare against the new implementation.
- If you have migrated all of your code to use the Play services API for
TensorFlow Lite, you should remove the existing TensorFlow Lite runtime
library dependencies (entries with
org.tensorflow:tensorflow-lite:*
) from your build.gradle file so that you can reduce your app size. - Identify all occurrences of
new Interpreter
object creation in your code, and modify each one so that it uses the InterpreterApi.create() call. The new TfLite.initialize is asynchronous, which means in most cases it's not a drop-in replacement: you must register a listener for when the call completes. Refer to the code snippet in Step 3 code. - Add
import org.tensorflow.lite.InterpreterApi;
andimport org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime;
to any source files using theorg.tensorflow.lite.Interpreter
ororg.tensorflow.lite.InterpreterApi
classes. - If any of the resulting calls to
InterpreterApi.create()
have only a single argument, appendnew InterpreterApi.Options()
to the argument list. - Append
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
to the last argument of any calls toInterpreterApi.create()
. - Replace all other occurrences of the
org.tensorflow.lite.Interpreter
class withorg.tensorflow.lite.InterpreterApi
.
If you want to use stand-alone TensorFlow Lite and the Play services API side-by-side, you must use TensorFlow Lite 2.9 (or later). TensorFlow Lite 2.8 and earlier versions are not compatible with the Play services API version.