Installing using Docker
The easiest and most straight-forward way of using TensorFlow Serving is with Docker images. We highly recommend this route unless you have specific needs that are not addressed by running in a container.
TIP: This is also the easiest way to get TensorFlow Serving working with GPU support.
Installing using APT
The TensorFlow Serving ModelServer binary is available in two variants:
tensorflow-model-server: Fully optimized server that uses some platform specific compiler optimizations like SSE4 and AVX instructions. This should be the preferred option for most users, but may not work on some older machines.
tensorflow-model-server-universal: Compiled with basic optimizations, but
doesn't include platform specific instruction sets, so should work on most if
not all machines out there. Use this if
tensorflow-model-server does not work
for you. Note that the binary name is the same for both packages, so if you
already installed tensorflow-model-server, you should first uninstall it using
apt-get remove tensorflow-model-server
Add TensorFlow Serving distribution URI as a package source (one time setup)
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \ curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
Install and update TensorFlow ModelServer
apt-get update && apt-get install tensorflow-model-server
Once installed, the binary can be invoked using the command
You can upgrade to a newer version of tensorflow-model-server with:
apt-get upgrade tensorflow-model-server
Building from source
The recommended approach to building from source is to use Docker. The TensorFlow Serving Docker development images encapsulate all the dependencies you need to build your own version of TensorFlow Serving.
For a listing of what these dependencies are, see the TensorFlow Serving Development Dockerfiles [CPU, GPU].
General installation instructions are on the Docker site.
Clone the build script
After installing Docker, we need to get the source we want to build from. We will use Git to clone the master branch of TensorFlow Serving:
git clone https://github.com/tensorflow/serving.git cd serving
In order to build in a hermetic environment with all dependencies taken care of,
we will use the
run_in_docker.sh script. This script passes build commands
through to a Docker container. By default, the script will build with the latest
nightly Docker development image.
TensorFlow Serving uses Bazel as its build tool. You can use Bazel commands to build individual targets or the entire source tree.
To build the entire tree, execute:
tools/run_in_docker.sh bazel build -c opt tensorflow_serving/...
Binaries are placed in the bazel-bin directory, and can be run using a command like:
To test your build, execute:
tools/run_in_docker.sh bazel test -c opt tensorflow_serving/...
See the basic tutorial and advanced tutorial for more in-depth examples of running TensorFlow Serving.
Building specific versions of TensorFlow Serving
If you want to build from a specific branch (such as a release branch), pass
<branchname> to the
git clone command.
We will also want to match the build environment for that branch of code, by
run_in_docker.sh script the Docker development image we'd like to
For example, to build version 1.10 of TensorFlow Serving:
$ git clone -b r1.10 https://github.com/tensorflow/serving.git ... $ cd serving $ tools/run_in_docker.sh -d tensorflow/serving:1.10-devel \ bazel build tensorflow_serving/... ...
If you'd like to apply generally recommended optimizations, including utilizing
platform-specific instruction sets for your processor, you can add
--config=nativeopt to Bazel build commands when building TensorFlow Serving.
tools/run_in_docker.sh bazel build --config=nativeopt tensorflow_serving/...
It's also possible to compile using specific instruction sets (e.g. AVX).
Wherever you see
bazel build in the documentation, simply add the
|All supported by processor||
tools/run_in_docker.sh bazel build --copt=-mavx2 tensorflow_serving/...
Building with GPU Support
In order to build a custom version of TensorFlow Serving with GPU support, we recommend either building with the provided Docker images, or following the approach in the GPU Dockerfile.
TensorFlow Serving Python API PIP package
To run Python client code without the need to build the API, you can install the
tensorflow-serving-api PIP package using:
pip install tensorflow-serving-api