1 # Delegate build guide introduction
3 The Arm NN Delegate can be found within the Arm NN repository but it is a standalone piece of software. However,
4 it makes use of the Arm NN library. For this reason we have added two options to build the delegate. The first option
5 allows you to build the delegate together with the Arm NN library, the second option is a standalone build
8 This tutorial uses an Aarch64 machine with Ubuntu 18.04 installed that can build all components
9 natively (no cross-compilation required). This is to keep this guide simple.
12 - [Delegate build guide introduction](#delegate-build-guide-introduction)
13 - [Dependencies](#dependencies)
14 * [Build Tensorflow Lite for C++](#build-tensorflow-lite-for-c--)
15 * [Build Flatbuffers](#build-flatbuffers)
16 * [Build the Arm Compute Library](#build-the-arm-compute-library)
17 * [Build the Arm NN Library](#build-the-arm-nn-library)
18 - [Build the TfLite Delegate (Stand-Alone)](#build-the-tflite-delegate--stand-alone-)
19 - [Build the Delegate together with Arm NN](#build-the-delegate-together-with-arm-nn)
20 - [Integrate the Arm NN TfLite Delegate into your project](#integrate-the-arm-nn-tflite-delegate-into-your-project)
26 * Tensorflow Lite: this guide uses version 2.3.1 . Other versions may work.
28 * Arm NN 20.11 or higher
31 * Git. This guide uses version 2.17.1 . Other versions might work.
32 * pip. This guide uses version 20.3.3 . Other versions might work.
33 * wget. This guide uses version 1.17.1 . Other versions might work.
34 * zip. This guide uses version 3.0 . Other versions might work.
35 * unzip. This guide uses version 6.00 . Other versions might work.
36 * cmake 3.7.0 or higher. This guide uses version 3.7.2
37 * scons. This guide uses version 2.4.1 . Other versions might work.
38 * bazel. This guide uses version 3.1.0 . Other versions might work.
40 Our first step is to build all the build dependencies I have mentioned above. We will have to create quite a few
41 directories. To make navigation a bit easier define a base directory for the project. At this stage we can also
42 install all the tools that are required during the build. This guide assumes you are using a Bash shell.
44 export BASEDIR=~/ArmNNDelegate
47 apt-get update && apt-get install git wget unzip zip python git cmake scons
49 ## Build Tensorflow Lite for C++
50 Tensorflow has a few dependencies on it's own. It requires the python packages pip3, numpy, wheel,
51 and also bazel which is used to compile Tensoflow. A description on how to build bazel can be
52 found [here](https://docs.bazel.build/versions/master/install-compile-source.html). There are multiple ways.
53 I decided to compile from source because that should work for any platform and therefore adds the most value
54 to this guide. Depending on your operating system and architecture there might be an easier way.
56 # Install the required python packages
57 pip3 install -U pip numpy wheel
59 # Bazel has a dependency on JDK (The specific JDK version depends on the bazel version but default-jdk tends to work.)
60 sudo apt-get install default-jdk
62 wget -O bazel-3.1.0-dist.zip https://github.com/bazelbuild/bazel/releases/download/3.1.0/bazel-3.1.0-dist.zip
63 unzip -d bazel bazel-3.1.0-dist.zip
65 env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
66 # This creates an "output" directory where the bazel binary can be found
69 ### Download and build Tensorflow Lite
73 git clone https://github.com/tensorflow/tensorflow.git
75 git checkout tags/v2.3.1 # Minimum version required for the delegate
77 Before we build, a target for tensorflow lite needs to be defined in the `BUILD` file. This can be
78 found in the root directory of Tensorflow. Append the following target to the file:
81 name = "libtensorflow_lite_all.so",
84 "//tensorflow/lite:framework",
85 "//tensorflow/lite/kernels:builtin_ops",
89 Now the build process can be started. When calling "configure", as below, a dialog shows up that asks the
90 user to specify additional options. If you don't have any particular needs to your build, decline all
91 additional options and choose default values.
93 PATH="$BASEDIR/bazel/output:$PATH" ./configure
94 $BASEDIR/bazel/output/bazel build --config=opt --config=monolithic --strip=always libtensorflow_lite_all.so
98 Flatbuffers is a memory efficient cross-platform serialization library as
99 described [here](https://google.github.io/flatbuffers/). It is used in tflite to store models and is also a dependency
100 of the delegate. After downloading the right version it can be built and installed using cmake.
103 wget -O flatbuffers-1.12.0.zip https://github.com/google/flatbuffers/archive/v1.12.0.zip
104 unzip -d . flatbuffers-1.12.0.zip
105 cd flatbuffers-1.12.0
106 mkdir install && mkdir build && cd build
107 # I'm using a different install directory but that is not required
108 cmake .. -DCMAKE_INSTALL_PREFIX:PATH=$BASEDIR/flatbuffers-1.12.0/install
112 ## Build the Arm Compute Library
114 The Arm NN library depends on the Arm Compute Library (ACL). It provides a set of functions that are optimized for
115 both Arm CPUs and GPUs. The Arm Compute Library is used directly by Arm NN to run machine learning workloads on
118 It is important to have the right version of ACL and Arm NN to make it work. Luckily, Arm NN and ACL are developed
119 very closely and released together. If you would like to use the Arm NN version "20.11" you should use the same "20.11"
122 To build the Arm Compute Library on your platform, download the Arm Compute Library and checkout the tag
123 that contains the version you want to use. Build it using `scons`.
126 git clone https://review.mlplatform.org/ml/ComputeLibrary
128 git checkout <tag_name> # e.g. v20.11
129 # The machine used for this guide only has a Neon CPU which is why I only have "neon=1" but if
130 # your machine has an arm Gpu you can enable that by adding `opencl=1 embed_kernels=1 to the command below
131 scons arch=arm64-v8a neon=1 extra_cxx_flags="-fPIC" benchmark_tests=0 validation_tests=0
134 ## Build the Arm NN Library
136 With ACL built we can now continue to building Arm NN. To do so, download the repository and checkout the matching
137 version as you did for ACL. Create a build directory and use `cmake` to build it.
140 git clone "https://review.mlplatform.org/ml/armnn"
142 git checkout <branch_name> # e.g. branches/armnn_20_11
143 mkdir build && cd build
144 # if you've got an arm Gpu add `-DARMCOMPUTECL=1` to the command below
145 cmake .. -DARMCOMPUTE_ROOT=$BASEDIR/ComputeLibrary -DARMCOMPUTENEON=1 -DBUILD_UNIT_TESTS=0
149 # Build the TfLite Delegate (Stand-Alone)
151 The delegate as well as Arm NN is built using `cmake`. Create a build directory as usual and build the delegate
152 with the additional cmake arguments shown below
154 cd $BASEDIR/armnn/delegate && mkdir build && cd build
155 cmake .. -DCMAKE_BUILD_TYPE=release # A release build rather than a debug build.
156 -DTENSORFLOW_ROOT=$BASEDIR/tensorflow \ # The root directory where tensorflow can be found.
157 -DTFLITE_LIB_ROOT=$BASEDIR/tensorflow/bazel-bin \ # Directory where tensorflow libraries can be found.
158 -DFLATBUFFERS_ROOT=$BASEDIR/flatbuffers-1.12.0/install \ # Flatbuffers install directory.
159 -DArmnn_DIR=$BASEDIR/armnn/build \ # Directory where the Arm NN library can be found
160 -DARMNN_SOURCE_DIR=$BASEDIR/armnn # The top directory of the Arm NN repository.
161 # Required are the includes for Arm NN
165 To ensure that the build was successful you can run the unit tests for the delegate that can be found in
166 the build directory for the delegate. [Doctest](https://github.com/onqtam/doctest) was used to create those tests. Using test filters you can
167 filter out tests that your build is not configured for. In this case, because Arm NN was only built for Cpu
168 acceleration (CpuAcc), we filter for all test suites that have `CpuAcc` in their name.
170 cd $BASEDIR/armnn/delegate/build
171 ./DelegateUnitTests --test-suite=*CpuAcc*
173 If you have built for Gpu acceleration as well you might want to change your test-suite filter:
175 ./DelegateUnitTests --test-suite=*CpuAcc*,*GpuAcc*
178 # Build the Delegate together with Arm NN
180 In the introduction it was mentioned that there is a way to integrate the delegate build into Arm NN. This is
181 pretty straight forward. The cmake arguments that were previously used for the delegate have to be added
182 to the Arm NN cmake arguments. Also another argument `BUILD_ARMNN_TFLITE_DELEGATE` needs to be added to
183 instruct Arm NN to build the delegate as well. The new commands to build Arm NN are as follows:
185 Download Arm NN if you have not already done so:
188 git clone "https://review.mlplatform.org/ml/armnn"
190 git checkout <branch_name> # e.g. branches/armnn_20_11
192 Build Arm NN with the delegate included
196 rm -rf build # Remove any previous cmake build.
197 mkdir build && cd build
198 # if you've got an arm Gpu add `-DARMCOMPUTECL=1` to the command below
199 cmake .. -DARMCOMPUTE_ROOT=$BASEDIR/ComputeLibrary \
201 -DBUILD_UNIT_TESTS=0 \
202 -DBUILD_ARMNN_TFLITE_DELEGATE=1 \
203 -DTENSORFLOW_ROOT=$BASEDIR/tensorflow \
204 -DTFLITE_LIB_ROOT=$BASEDIR/tensorflow/bazel-bin \
205 -DFLATBUFFERS_ROOT=$BASEDIR/flatbuffers-1.12.0/install
208 The delegate library can then be found in `build/armnn/delegate`.
210 # Test the Arm NN delegate using the [TFLite Model Benchmark Tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark)
212 The TFLite Model Benchmark Tool has a useful command line interface to test delegates. We can use this to demonstrate the use of the Arm NN delegate and its options.
214 Some examples of this can be viewed in this [YouTube demonstration](https://www.youtube.com/watch?v=NResQ1kbm-M&t=920s).
216 ## Download the TFLite Model Benchmark Tool
218 Binary builds of the benchmarking tool for various platforms are available [here](https://www.tensorflow.org/lite/performance/measurement#native_benchmark_binary). In this example I will target an aarch64 Linux environment. I will also download a sample uint8 tflite model from the [Arm ML Model Zoo](https://github.com/ARM-software/ML-zoo).
221 mkdir $BASEDIR/benchmarking
222 cd $BASEDIR/benchmarking
223 # Get the benchmarking binary.
224 wget https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model -O benchmark_model
225 # Make it executable.
226 chmod +x benchmark_model
227 # and a sample model from model zoo.
228 wget https://github.com/ARM-software/ML-zoo/blob/master/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/mobilenet_v2_1.0_224_quantized_1_default_1.tflite?raw=true -O mobilenet_v2_1.0_224_quantized_1_default_1.tflite
231 ## Execute the benchmarking tool with the Arm NN delegate
233 cd $BASEDIR/benchmarking
234 LD_LIBRARY_PATH=../armnn/build ./benchmark_model --graph=mobilenet_v2_1.0_224_quantized_1_default_1.tflite --external_delegate_path="../armnn/build/delegate/libarmnnDelegate.so" --external_delegate_options="backends:CpuAcc;logging-severity:info"
236 The "external_delegate_options" here are specific to the Arm NN delegate. They are used to specify a target Arm NN backend or to enable/disable various options in Arm NN. A full description can be found in the parameters of function [tflite_plugin_create_delegate](namespacetflite.xhtml).
238 # Integrate the Arm NN TfLite Delegate into your project
240 The delegate can be integrated into your c++ project by creating a TfLite Interpreter and
241 instructing it to use the Arm NN delegate for the graph execution. This should look similiar
242 to the following code snippet.
244 // Create TfLite Interpreter
245 std::unique_ptr<Interpreter> armnnDelegateInterpreter;
246 InterpreterBuilder(tfLiteModel, ::tflite::ops::builtin::BuiltinOpResolver())
247 (&armnnDelegateInterpreter)
249 // Create the Arm NN Delegate
250 armnnDelegate::DelegateOptions delegateOptions(backends);
251 std::unique_ptr<TfLiteDelegate, decltype(&armnnDelegate::TfLiteArmnnDelegateDelete)>
252 theArmnnDelegate(armnnDelegate::TfLiteArmnnDelegateCreate(delegateOptions),
253 armnnDelegate::TfLiteArmnnDelegateDelete);
255 // Instruct the Interpreter to use the armnnDelegate
256 armnnDelegateInterpreter->ModifyGraphWithDelegate(theArmnnDelegate.get());
259 For further information on using TfLite Delegates please visit the [tensorflow website](https://www.tensorflow.org/lite/guide)
261 For more details of the kind of options you can pass to the Arm NN delegate please check the parameters of function [tflite_plugin_create_delegate](namespacetflite.xhtml).