diff options
author | Jan Eilers <jan.eilers@arm.com> | 2020-12-15 10:42:38 +0000 |
---|---|---|
committer | Jim Flynn <jim.flynn@arm.com> | 2021-01-22 11:48:34 +0000 |
commit | 2cd184763ff7f8767e751f2fe0c461714350aae6 (patch) | |
tree | 048a95b2b571bfa3ad03eceb5dd4ccd4d0a70c06 /delegate/IntegrateDelegateIntoPython.md | |
parent | d672f5d4386dc0545d2e484ce85b76d53edb6bc9 (diff) | |
download | armnn-2cd184763ff7f8767e751f2fe0c461714350aae6.tar.gz |
IVGCVSW-5571 Expose the TfLite Delegate to the TfLite python API
* Implemented external delegate adaptor interface for TfLite
* Activated armnn logging for delegate
* Added logging info to indicate if gpu tuning is turned on
* Added pytests to ensure functionality of the external delegate adaptor
* Included the delegate directory into doxygen
* Added documentation on how to use the external delegate in python
Signed-off-by: Finn Williams <Finn.Williams@arm.com>
Signed-off-by: Jan Eilers <jan.eilers@arm.com>
Change-Id: Id3b4588fb0b9ac7e3f47ba2c19feead7beb58e18
Diffstat (limited to 'delegate/IntegrateDelegateIntoPython.md')
-rw-r--r-- | delegate/IntegrateDelegateIntoPython.md | 121 |
1 files changed, 121 insertions, 0 deletions
diff --git a/delegate/IntegrateDelegateIntoPython.md b/delegate/IntegrateDelegateIntoPython.md new file mode 100644 index 0000000000..69a5ca00e2 --- /dev/null +++ b/delegate/IntegrateDelegateIntoPython.md @@ -0,0 +1,121 @@ +# Integrate the TfLite delegate into a python script +If you have built the TfLite delegate as a separate dynamic library then this tutorial will show you how you can +integrate it in TfLite to run models using python. + +Here is an example python script showing how to do this. In this script we are making use of the +[external adaptor](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate) +tool of TfLite that allows you to load delegates at runtime. +```python +import numpy as np +import tflite_runtime.interpreter as tflite + +# Load TFLite model and allocate tensors. +# (if you are using the complete tensorflow package you can find load_delegate in tf.experimental.load_delegate) +armnn_delegate = tflite.load_delegate( library="<your-armnn-build-dir>/delegate/libarmnnDelegate.so", + options={"backends": "CpuAcc,GpuAcc,CpuRef", "logging-severity":"info"}) +# Delegates/Executes all operations supported by ArmNN to/with ArmNN +interpreter = tflite.Interpreter(model_path="<your-armnn-repo-dir>/delegate/python/test/test_data/mock_model.tflite", + experimental_delegates=[armnn_delegate]) +interpreter.allocate_tensors() + +# Get input and output tensors. +input_details = interpreter.get_input_details() +output_details = interpreter.get_output_details() + +# Test model on random input data. +input_shape = input_details[0]['shape'] +input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8) +interpreter.set_tensor(input_details[0]['index'], input_data) + +interpreter.invoke() + +# Print out result +output_data = interpreter.get_tensor(output_details[0]['index']) +print(output_data) +``` + +# Prepare the environment +Pre-requisites: + * Dynamically build ArmNN Delegate library + * python3 (Depends on TfLite version) + * virtualenv + * numpy (Depends on TfLite version) + * tflite_runtime (>=2.0, depends on ArmNN Delegate) + +If you haven't built the delegate yet then take a look at the [build guide](BuildBuideNative.md). + +We recommend creating a virtual environment for this tutorial. For the following code to work python3 is needed. Please +also check the documentation of the TfLite version you want to use. There might be additional prerequisites for the python +version. +```bash +# Install python3 (We ended up with python3.5.3) and virtualenv +sudo apt-get install python3-pip +sudo pip3 install virtualenv + +# create a virtual environment +cd your/tutorial/dir +# creates a directory myenv at the current location +virtualenv -p python3 myenv +# activate the environment +source myenv/bin/activate +``` + +Now that the environment is active we can install additional packages we need for our example script. As you can see +in the python script at the start of this page, this tutorial uses the `tflite_runtime` rather than the whole tensorflow +package. The `tflite_runtime` is a package that wraps the TfLite Interpreter. Therefore it can only be used to run inferences of +TfLite models. But since ArmNN is only an inference engine itself this is a perfect match. The +`tflite_runtime` is also much smaller than the whole tensorflow package and better suited to run models on +mobile and embedded devices. + +At the time of writing, there are no packages of either `tensorflow` or `tflite_runtime` available on `pypi` that +are built for an arm architecture. That means installing them using `pip` on your development board is currently not +possible. The TfLite [website](https://www.tensorflow.org/lite/guide/python) points you at prebuilt `tflite_runtime` +packages. However, that limits you to specific TfLite and Python versions. For this reason we will build the +`tflite_runtime` from source. + +You will have downloaded the tensorflow repository in order to build the ArmNN delegate. In there you can find further +instructions on how to build the `tflite_runtime` under `tensorflow/lite/tools/pip_package/README.md`. This tutorial +uses bazel to build it natively but there are scripts for cross-compilation available as well. +```bash +# Add the directory where bazel is built to your PATH so that the script can find it +PATH=$PATH:your/build/dir/bazel/output +# Run the following script to build tflite_runtime natively. +tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh +``` +The execution of the script creates a `.whl` file which can be used by `pip` to install the TfLite Runtime package. +The build-script produces some output in which you can find the location where the `.whl` file was created. Then all that is +left to do is to install all necessary python packages with `pip`. +```bash +pip install tensorflow/lite/tools/pip_package/gen/tflite_pip/python3/dist/tflite_runtime-2.3.1-py3-none-any.whl numpy +``` + +Your virtual environment is now all setup. Copy the final python script into a python file e.g. +`ExternalDelegatePythonTutorial.py`. Modify the python script above and replace `<your-armnn-build-dir>` and +`<your-armnn-repo-dir>` with the directories you have set up. If you've been using the [native build guide](BuildGuideNative.md) +this will be `$BASEDIR/armnn/build` and `$BASEDIR/armnn`. + +Finally, execute the script: +```bash +python ExternalDelegatePythonTutorial.py +``` +The output should look similar to this: +```bash +Info: ArmNN v23.0.0 + +Info: Initialization time: 0.56 ms + +INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate. +[[ 12 123 16 12 11 14 20 16 20 12]] +Info: Shutdown time: 0.28 ms +``` + +For more details on what kind of options you can pass to the armnn delegate please check +[armnn_delegate_adaptor.cpp](src/armnn_external_delegate.cpp). + +You can also test the functionality of the external delegate adaptor by running some unit tests: +```bash +pip install pytest +cd armnn/delegate/python/test +# You can deselect tests that require backends that your hardware doesn't support using markers e.g. `-m "not GpuAccTest` +pytest --delegate-dir="<your-armnn-build-dir>/armnn/delegate/libarmnnDelegate.so" -m "not GpuAccTest" +``` |