1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
<!-- Copyright (c) 2020 ARM Limited. -->
<!-- -->
<!-- SPDX-License-Identifier: MIT -->
<!-- -->
<!-- HTML header for doxygen 1.8.13-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<meta name="robots" content="NOINDEX, NOFOLLOW" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>ArmNN: Delegate build guide introduction</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
</script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js"],
jax: ["input/TeX","output/HTML-CSS"],
});
</script><script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<img alt="ArmNN" src="Arm_NN_horizontal_blue.png" style="max-width: 10rem; margin-top: .5rem; margin-left 10px"/>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">21.08</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
$(function() {
initMenu('',true,false,'search.php','Search');
$(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('md_delegate__build_guide_native.xhtml','');});
</script>
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0"
name="MSearchResults" id="MSearchResults">
</iframe>
</div>
<div class="header">
<div class="headertitle">
<div class="title">Delegate build guide introduction </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>The Arm NN Delegate can be found within the Arm NN repository but it is a standalone piece of software. However, it makes use of the Arm NN library. For this reason we have added two options to build the delegate. The first option allows you to build the delegate together with the Arm NN library, the second option is a standalone build of the delegate.</p>
<p>This tutorial uses an Aarch64 machine with Ubuntu 18.04 installed that can build all components natively (no cross-compilation required). This is to keep this guide simple.</p>
<p><b>Table of content:</b></p><ul>
<li><a href="#delegate-build-guide-introduction">Delegate build guide introduction</a></li>
<li><a href="#dependencies">Dependencies</a><ul>
<li><a href="#build-tensorflow-lite-for-c--">Build Tensorflow Lite for C++</a></li>
<li><a href="#build-flatbuffers">Build Flatbuffers</a></li>
<li><a href="#build-the-arm-compute-library">Build the Arm Compute Library</a></li>
<li><a href="#build-the-arm-nn-library">Build the Arm NN Library</a></li>
</ul>
</li>
<li><a href="#build-the-tflite-delegate--stand-alone-">Build the TfLite Delegate (Stand-Alone)</a></li>
<li><a href="#build-the-delegate-together-with-arm-nn">Build the Delegate together with Arm NN</a></li>
<li><a href="#integrate-the-arm-nn-tflite-delegate-into-your-project">Integrate the Arm NN TfLite Delegate into your project</a></li>
</ul>
<h1>Dependencies</h1>
<p>Build Dependencies:</p><ul>
<li>Tensorflow Lite: this guide uses version 2.3.1 . Other versions may work.</li>
<li>Flatbuffers 1.12.0</li>
<li>Arm NN 20.11 or higher</li>
</ul>
<p>Required Tools:</p><ul>
<li>Git. This guide uses version 2.17.1 . Other versions might work.</li>
<li>pip. This guide uses version 20.3.3 . Other versions might work.</li>
<li>wget. This guide uses version 1.17.1 . Other versions might work.</li>
<li>zip. This guide uses version 3.0 . Other versions might work.</li>
<li>unzip. This guide uses version 6.00 . Other versions might work.</li>
<li>cmake 3.7.0 or higher. This guide uses version 3.7.2</li>
<li>scons. This guide uses version 2.4.1 . Other versions might work.</li>
<li>bazel. This guide uses version 3.1.0 . Other versions might work.</li>
</ul>
<p>Our first step is to build all the build dependencies I have mentioned above. We will have to create quite a few directories. To make navigation a bit easier define a base directory for the project. At this stage we can also install all the tools that are required during the build. This guide assumes you are using a Bash shell. </p><div class="fragment"><div class="line">export BASEDIR=~/ArmNNDelegate</div><div class="line">mkdir $BASEDIR</div><div class="line">cd $BASEDIR</div><div class="line">apt-get update && apt-get install git wget unzip zip python git cmake scons</div></div><!-- fragment --> <h2>Build Tensorflow Lite for C++</h2>
<p>Tensorflow has a few dependencies on it's own. It requires the python packages pip3, numpy, wheel, and also bazel which is used to compile Tensoflow. A description on how to build bazel can be found <a href="https://docs.bazel.build/versions/master/install-compile-source.html">here</a>. There are multiple ways. I decided to compile from source because that should work for any platform and therefore adds the most value to this guide. Depending on your operating system and architecture there might be an easier way. </p><div class="fragment"><div class="line"># Install the required python packages</div><div class="line">pip3 install -U pip numpy wheel</div><div class="line"></div><div class="line"># Bazel has a dependency on JDK (The specific JDK version depends on the bazel version but default-jdk tends to work.)</div><div class="line">sudo apt-get install default-jdk</div><div class="line"># Build Bazel</div><div class="line">wget -O bazel-3.1.0-dist.zip https://github.com/bazelbuild/bazel/releases/download/3.1.0/bazel-3.1.0-dist.zip</div><div class="line">unzip -d bazel bazel-3.1.0-dist.zip</div><div class="line">cd bazel</div><div class="line">env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh </div><div class="line"># This creates an "output" directory where the bazel binary can be found</div></div><!-- fragment --><h3>Download and build Tensorflow Lite</h3>
<div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">git clone https://github.com/tensorflow/tensorflow.git</div><div class="line">cd tensorflow/</div><div class="line">git checkout tags/v2.3.1 # Minimum version required for the delegate</div></div><!-- fragment --><p> Before we build, a target for tensorflow lite needs to be defined in the <code>BUILD</code> file. This can be found in the root directory of Tensorflow. Append the following target to the file: </p><div class="fragment"><div class="line">cc_binary(</div><div class="line"> name = "libtensorflow_lite_all.so",</div><div class="line"> linkshared = 1,</div><div class="line"> deps = [</div><div class="line"> "//tensorflow/lite:framework",</div><div class="line"> "//tensorflow/lite/kernels:builtin_ops",</div><div class="line"> ],</div><div class="line">)</div></div><!-- fragment --><p> Now the build process can be started. When calling "configure", as below, a dialog shows up that asks the user to specify additional options. If you don't have any particular needs to your build, decline all additional options and choose default values. </p><div class="fragment"><div class="line">PATH="$BASEDIR/bazel/output:$PATH" ./configure</div><div class="line">$BASEDIR/bazel/output/bazel build --config=opt --config=monolithic --strip=always libtensorflow_lite_all.so</div></div><!-- fragment --><h2>Build Flatbuffers</h2>
<p>Flatbuffers is a memory efficient cross-platform serialization library as described <a href="https://google.github.io/flatbuffers/">here</a>. It is used in tflite to store models and is also a dependency of the delegate. After downloading the right version it can be built and installed using cmake. </p><div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">wget -O flatbuffers-1.12.0.zip https://github.com/google/flatbuffers/archive/v1.12.0.zip</div><div class="line">unzip -d . flatbuffers-1.12.0.zip</div><div class="line">cd flatbuffers-1.12.0 </div><div class="line">mkdir install && mkdir build && cd build</div><div class="line"># I'm using a different install directory but that is not required</div><div class="line">cmake .. -DCMAKE_INSTALL_PREFIX:PATH=$BASEDIR/flatbuffers-1.12.0/install </div><div class="line">make install</div></div><!-- fragment --><h2>Build the Arm Compute Library</h2>
<p>The Arm NN library depends on the Arm Compute Library (ACL). It provides a set of functions that are optimized for both Arm CPUs and GPUs. The Arm Compute Library is used directly by Arm NN to run machine learning workloads on Arm CPUs and GPUs.</p>
<p>It is important to have the right version of ACL and Arm NN to make it work. Luckily, Arm NN and ACL are developed very closely and released together. If you would like to use the Arm NN version "20.11" you should use the same "20.11" version for ACL too.</p>
<p>To build the Arm Compute Library on your platform, download the Arm Compute Library and checkout the tag that contains the version you want to use. Build it using <code>scons</code>. </p><div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">git clone https://review.mlplatform.org/ml/ComputeLibrary </div><div class="line">cd ComputeLibrary/</div><div class="line">git checkout <tag_name> # e.g. v20.11</div><div class="line"># The machine used for this guide only has a Neon CPU which is why I only have "neon=1" but if </div><div class="line"># your machine has an arm Gpu you can enable that by adding `opencl=1 embed_kernels=1 to the command below</div><div class="line">scons arch=arm64-v8a neon=1 extra_cxx_flags="-fPIC" benchmark_tests=0 validation_tests=0 </div></div><!-- fragment --><h2>Build the Arm NN Library</h2>
<p>With ACL built we can now continue to building Arm NN. To do so, download the repository and checkout the matching version as you did for ACL. Create a build directory and use <code>cmake</code> to build it. </p><div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">git clone "https://review.mlplatform.org/ml/armnn" </div><div class="line">cd armnn</div><div class="line">git checkout <branch_name> # e.g. branches/armnn_20_11</div><div class="line">mkdir build && cd build</div><div class="line"># if you've got an arm Gpu add `-DARMCOMPUTECL=1` to the command below</div><div class="line">cmake .. -DARMCOMPUTE_ROOT=$BASEDIR/ComputeLibrary -DARMCOMPUTENEON=1 -DBUILD_UNIT_TESTS=0 </div><div class="line">make</div></div><!-- fragment --><h1>Build the TfLite Delegate (Stand-Alone)</h1>
<p>The delegate as well as Arm NN is built using <code>cmake</code>. Create a build directory as usual and build the delegate with the additional cmake arguments shown below </p><div class="fragment"><div class="line">cd $BASEDIR/armnn/delegate && mkdir build && cd build</div><div class="line">cmake .. -DCMAKE_BUILD_TYPE=release # A release build rather than a debug build.</div><div class="line"> -DTENSORFLOW_ROOT=$BASEDIR/tensorflow \ # The root directory where tensorflow can be found.</div><div class="line"> -DTFLITE_LIB_ROOT=$BASEDIR/tensorflow/bazel-bin \ # Directory where tensorflow libraries can be found.</div><div class="line"> -DFLATBUFFERS_ROOT=$BASEDIR/flatbuffers-1.12.0/install \ # Flatbuffers install directory.</div><div class="line"> -DArmnn_DIR=$BASEDIR/armnn/build \ # Directory where the Arm NN library can be found</div><div class="line"> -DARMNN_SOURCE_DIR=$BASEDIR/armnn # The top directory of the Arm NN repository. </div><div class="line"> # Required are the includes for Arm NN</div><div class="line">make</div></div><!-- fragment --><p>To ensure that the build was successful you can run the unit tests for the delegate that can be found in the build directory for the delegate. <a href="https://github.com/onqtam/doctest">Doctest</a> was used to create those tests. Using test filters you can filter out tests that your build is not configured for. In this case, because Arm NN was only built for Cpu acceleration (CpuAcc), we filter for all test suites that have <code>CpuAcc</code> in their name. </p><div class="fragment"><div class="line">cd $BASEDIR/armnn/delegate/build</div><div class="line">./DelegateUnitTests --test-suite=*CpuAcc* </div></div><!-- fragment --><p> If you have built for Gpu acceleration as well you might want to change your test-suite filter: </p><div class="fragment"><div class="line">./DelegateUnitTests --test-suite=*CpuAcc*,*GpuAcc*</div></div><!-- fragment --><h1>Build the Delegate together with Arm NN</h1>
<p>In the introduction it was mentioned that there is a way to integrate the delegate build into Arm NN. This is pretty straight forward. The cmake arguments that were previously used for the delegate have to be added to the Arm NN cmake arguments. Also another argument <code>BUILD_ARMNN_TFLITE_DELEGATE</code> needs to be added to instruct Arm NN to build the delegate as well. The new commands to build Arm NN are as follows:</p>
<p>Download Arm NN if you have not already done so: </p><div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">git clone "https://review.mlplatform.org/ml/armnn" </div><div class="line">cd armnn</div><div class="line">git checkout <branch_name> # e.g. branches/armnn_20_11</div></div><!-- fragment --><p> Build Arm NN with the delegate included </p><div class="fragment"><div class="line">cd $BASEDIR</div><div class="line">cd armnn</div><div class="line">rm -rf build # Remove any previous cmake build.</div><div class="line">mkdir build && cd build</div><div class="line"># if you've got an arm Gpu add `-DARMCOMPUTECL=1` to the command below</div><div class="line">cmake .. -DARMCOMPUTE_ROOT=$BASEDIR/ComputeLibrary \</div><div class="line"> -DARMCOMPUTENEON=1 \</div><div class="line"> -DBUILD_UNIT_TESTS=0 \</div><div class="line"> -DBUILD_ARMNN_TFLITE_DELEGATE=1 \</div><div class="line"> -DTENSORFLOW_ROOT=$BASEDIR/tensorflow \</div><div class="line"> -DTFLITE_LIB_ROOT=$BASEDIR/tensorflow/bazel-bin \</div><div class="line"> -DFLATBUFFERS_ROOT=$BASEDIR/flatbuffers-1.12.0/install</div><div class="line">make</div></div><!-- fragment --><p> The delegate library can then be found in <code>build/armnn/delegate</code>.</p>
<h1>Test the Arm NN delegate using the <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark">TFLite Model Benchmark Tool</a></h1>
<p>The TFLite Model Benchmark Tool has a useful command line interface to test delegates. We can use this to demonstrate the use of the Arm NN delegate and its options.</p>
<p>Some examples of this can be viewed in this <a href="https://www.youtube.com/watch?v=NResQ1kbm-M&t=920s">YouTube demonstration</a>.</p>
<h2>Download the TFLite Model Benchmark Tool</h2>
<p>Binary builds of the benchmarking tool for various platforms are available <a href="https://www.tensorflow.org/lite/performance/measurement#native_benchmark_binary">here</a>. In this example I will target an aarch64 Linux environment. I will also download a sample uint8 tflite model from the <a href="https://github.com/ARM-software/ML-zoo">Arm ML Model Zoo</a>.</p>
<div class="fragment"><div class="line">mkdir $BASEDIR/benchmarking</div><div class="line">cd $BASEDIR/benchmarking</div><div class="line"># Get the benchmarking binary.</div><div class="line">wget https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model -O benchmark_model</div><div class="line"># Make it executable.</div><div class="line">chmod +x benchmark_model</div><div class="line"># and a sample model from model zoo.</div><div class="line">wget https://github.com/ARM-software/ML-zoo/blob/master/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/mobilenet_v2_1.0_224_quantized_1_default_1.tflite?raw=true -O mobilenet_v2_1.0_224_quantized_1_default_1.tflite</div></div><!-- fragment --><p>## Execute the benchmarking tool with the Arm NN delegate </p><div class="fragment"><div class="line">cd $BASEDIR/benchmarking</div><div class="line">LD_LIBRARY_PATH=../armnn/build ./benchmark_model --graph=mobilenet_v2_1.0_224_quantized_1_default_1.tflite --external_delegate_path="../armnn/build/delegate/libarmnnDelegate.so" --external_delegate_options="backends:CpuAcc;logging-severity:info"</div></div><!-- fragment --><p> The "external_delegate_options" here are specific to the Arm NN delegate. They are used to specify a target Arm NN backend or to enable/disable various options in Arm NN. A full description can be found in the parameters of function <a href="namespacetflite.xhtml">tflite_plugin_create_delegate</a>.</p>
<h1>Integrate the Arm NN TfLite Delegate into your project</h1>
<p>The delegate can be integrated into your c++ project by creating a TfLite Interpreter and instructing it to use the Arm NN delegate for the graph execution. This should look similiar to the following code snippet. </p><div class="fragment"><div class="line">// Create TfLite Interpreter</div><div class="line">std::unique_ptr<Interpreter> armnnDelegateInterpreter;</div><div class="line">InterpreterBuilder(tfLiteModel, ::tflite::ops::builtin::BuiltinOpResolver())</div><div class="line"> (&armnnDelegateInterpreter)</div><div class="line"></div><div class="line">// Create the Arm NN Delegate</div><div class="line">armnnDelegate::DelegateOptions delegateOptions(backends);</div><div class="line">std::unique_ptr<TfLiteDelegate, decltype(&armnnDelegate::TfLiteArmnnDelegateDelete)></div><div class="line"> theArmnnDelegate(armnnDelegate::TfLiteArmnnDelegateCreate(delegateOptions),</div><div class="line"> armnnDelegate::TfLiteArmnnDelegateDelete);</div><div class="line"></div><div class="line">// Instruct the Interpreter to use the armnnDelegate</div><div class="line">armnnDelegateInterpreter->ModifyGraphWithDelegate(theArmnnDelegate.get());</div></div><!-- fragment --><p>For further information on using TfLite Delegates please visit the <a href="https://www.tensorflow.org/lite/guide">tensorflow website</a></p>
<p>For more details of the kind of options you can pass to the Arm NN delegate please check the parameters of function <a href="namespacetflite.xhtml">tflite_plugin_create_delegate</a>. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="buildguides.xhtml">Installation and build Guides</a></li>
<li class="footer">Generated on Tue Aug 24 2021 16:18:47 for ArmNN by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li>
</ul>
</div>
</body>
</html>
|
