Creating the TensorFlow Hub pip package using Linux

If you make changes to TensorFlow Hub pip package, you will likely want to rebuild the pip package from source to try out your changes.

This requires:

  • Python
  • TensorFlow
  • Git
  • Bazel

Alternatively, if you install the protobuf compiler you can try out your changes without using bazel.

Setup a virtualenv

Activate virtualenv

Install virtualenv if it's not installed already:

~$ sudo apt-get install python-virtualenv

Create a virtual environment for the package creation:

~$ virtualenv --system-site-packages tensorflow_hub_env

And activate it:

~$ source ~/tensorflow_hub_env/bin/activate  # bash, sh, ksh, or zsh
~$ source ~/tensorflow_hub_env/bin/activate.csh  # csh or tcsh

Clone the TensorFlow Hub repository.

(tensorflow_hub_env)~/$ git clone https://github.com/tensorflow/hub
(tensorflow_hub_env)~/$ cd hub

Test your changes

Run TensorFlow Hub's tests

(tensorflow_hub_env)~/hub/$ bazel test tensorflow_hub:all

Build and install the package

Build TensorFlow Hub pip packaging script

To build a pip package for TensorFlow Hub:

(tensorflow_hub_env)~/hub/$ bazel build tensorflow_hub/pip_package:build_pip_package

Create the TensorFlow Hub pip package

(tensorflow_hub_env)~/hub/$ bazel-bin/tensorflow_hub/pip_package/build_pip_package \
/tmp/tensorflow_hub_pkg

Install and test the pip package (optional)

Run the following commands to install the pip package.

(tensorflow_hub_env)~/hub/$ pip install /tmp/tensorflow_hub_pkg/*.whl

Test import TensorFlow Hub:

(tensorflow_hub_env)~/hub/$ cd ..  # exit the directory to avoid confusion
(tensorflow_hub_env)~/$ python -c "import tensorflow_hub as hub"

"Developer" install (experimental)

Building the package with bazel is the only officially supported method. However if you are unfamiliar with bazel simpler to work with open source tools. For that you can do a "developer install" of the package.

This installation method allows you to install the working directory into your python environment, so that ongoing changes are reflected when you import the package.

Setup the repository

First setup the virtualenv and repository, as described above.

Install protoc

Because TensorFlow Hub uses protobufs you will need the protobuf compiler to create the necessary python _pb2.py files from the .proto files.

On a Mac:

(tensorflow_hub_env)~/hub/$ brew install protobuf

On Linux

(tensorflow_hub_env)~/hub/$ sudo apt install protobuf-compiler

Compile the .proto files

Initially there are no _pb2.py files in the directory:

(tensorflow_hub_env)~/hub/$ ls -1 tensorflow_hub/*_pb2.py

Run protoc to create them:

(tensorflow_hub_env)~/hub/$ protoc -I=tensorflow_hub --python_out=tensorflow_hub tensorflow_hub/*.proto
(tensorflow_hub_env)~/hub/$ ls -1 tensorflow_hub/*_pb2.py

tensorflow_hub/image_module_info_pb2.py
tensorflow_hub/module_attachment_pb2.py
tensorflow_hub/module_def_pb2.py

Import directly from the repository

With the _pb2.py files in place, you can use try out your modifications directly from the TensorFlow Hub directory:

(tensorflow_hub_env)~/$ python -c "import tensorflow_hub as hub"

Install in "developer" mode

Or to use this from outside the repository root, you can use the setup.py develop installation:

(tensorflow_hub_env)~/hub/$ python tensorflow_hub/pip_package/setup.py develop

Now you can use your local changes in a regular python virtualenv, without the need to rebuild and install the pip package for each new change:

(tensorflow_hub_env)~/hub/$ cd ..  # exit the directory to avoid confusion
(tensorflow_hub_env)~/$ python -c "import tensorflow_hub as hub"

De-activate the virtualenv

(tensorflow_hub_env)~/hub/$ deactivate