1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
<!-- Copyright (c) 2020 ARM Limited. -->
<!-- -->
<!-- SPDX-License-Identifier: MIT -->
<!-- -->
<!-- HTML header for doxygen 1.8.13-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<meta name="robots" content="NOINDEX, NOFOLLOW" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>ArmNN: TfLite Delegate Quick Start Guide</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
</script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js"],
jax: ["input/TeX","output/HTML-CSS"],
});
</script><script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<img alt="ArmNN" src="Arm_NN_horizontal_blue.png" style="max-width: 10rem; margin-top: .5rem; margin-left 10px"/>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">22.11</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
$(function() {
initMenu('',true,false,'search.php','Search');
$(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('md_delegate__delegate_quick_start_guide.xhtml','');});
</script>
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0"
name="MSearchResults" id="MSearchResults">
</iframe>
</div>
<div class="header">
<div class="headertitle">
<div class="title">TfLite Delegate Quick Start Guide </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>If you have downloaded the Arm NN Github binaries or built the TfLite delegate yourself, then this tutorial will show you how you can integrate it into TfLite to run models using python.</p>
<p>Here is an example python script showing how to do this. In this script we are making use of the <a href="https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate">external adaptor</a> tool of TfLite that allows you to load delegates at runtime. </p><div class="fragment"><div class="line">import numpy as np</div><div class="line">import tflite_runtime.interpreter as tflite</div><div class="line"></div><div class="line"># Load TFLite model and allocate tensors.</div><div class="line"># (if you are using the complete tensorflow package you can find load_delegate in tf.experimental.load_delegate)</div><div class="line">armnn_delegate = tflite.load_delegate( library="<path-to-armnn-binaries>/libarmnnDelegate.so",</div><div class="line"> options={"backends": "CpuAcc,GpuAcc,CpuRef", "logging-severity":"info"})</div><div class="line"># Delegates/Executes all operations supported by Arm NN to/with Arm NN</div><div class="line">interpreter = tflite.Interpreter(model_path="<your-armnn-repo-dir>/delegate/python/test/test_data/mock_model.tflite", </div><div class="line"> experimental_delegates=[armnn_delegate])</div><div class="line">interpreter.allocate_tensors()</div><div class="line"></div><div class="line"># Get input and output tensors.</div><div class="line">input_details = interpreter.get_input_details()</div><div class="line">output_details = interpreter.get_output_details()</div><div class="line"></div><div class="line"># Test model on random input data.</div><div class="line">input_shape = input_details[0]['shape']</div><div class="line">input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)</div><div class="line">interpreter.set_tensor(input_details[0]['index'], input_data)</div><div class="line"></div><div class="line">interpreter.invoke()</div><div class="line"></div><div class="line"># Print out result</div><div class="line">output_data = interpreter.get_tensor(output_details[0]['index'])</div><div class="line">print(output_data)</div></div><!-- fragment --><h1>Prepare the environment</h1>
<p>Pre-requisites:</p><ul>
<li>Dynamically build Arm NN Delegate library or download the Arm NN binaries</li>
<li>python3 (Depends on TfLite version)</li>
<li>virtualenv</li>
<li>numpy (Depends on TfLite version)</li>
<li>tflite_runtime (>=2.5, depends on Arm NN Delegate)</li>
</ul>
<p>If you haven't built the delegate yet then take a look at the ./BuildGuideNative.md "build guide". Otherwise, you can download the binaries <a href="https://github.com/ARM-software/armnn/releases/">here</a>. Set the following environment variable to the location of the .so binary files:</p>
<div class="fragment"><div class="line">export LD_LIBRARY_PATH=<path_to_so_binary_files></div></div><!-- fragment --><p>We recommend creating a virtual environment for this tutorial. For the following code to work python3 is needed. Please also check the documentation of the TfLite version you want to use. There might be additional prerequisites for the python version. We will use Tensorflow Lite 2.5.0 for this guide. </p><div class="fragment"><div class="line"># Install python3 (We ended up with python3.5.3) and virtualenv</div><div class="line">sudo apt-get install python3-pip</div><div class="line">sudo pip3 install virtualenv</div><div class="line"></div><div class="line"># create a virtual environment</div><div class="line">cd your/tutorial/dir</div><div class="line"># creates a directory myenv at the current location</div><div class="line">virtualenv -p python3 myenv </div><div class="line"># activate the environment</div><div class="line">source myenv/bin/activate</div></div><!-- fragment --><p>Now that the environment is active we can install additional packages we need for our example script. As you can see in the python script at the start of this page, this tutorial uses the <code>tflite_runtime</code> rather than the whole tensorflow package. The <code>tflite_runtime</code> is a package that wraps the TfLite Interpreter. Therefore it can only be used to run inferences of TfLite models. But since Arm NN is only an inference engine itself this is a perfect match. The <code>tflite_runtime</code> is also much smaller than the whole tensorflow package and better suited to run models on mobile and embedded devices.</p>
<p>The TfLite <a href="https://www.tensorflow.org/lite/guide/python">website</a> shows you two methods to download the <code>tflite_runtime</code> package. In our experience, the use of the pip command works for most systems including debian. However, if you're using an older version of Tensorflow, you may need to build the pip package from source. You can find more information https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/pip_package/README.md "here". But in our case, with Tensorflow Lite 2.5.0, we can install through:</p>
<div class="fragment"><div class="line">pip3 install --extra-index-url https:<span class="comment">//google-coral.github.io/py-repo/ tflite_runtime==2.5.0</span></div></div><!-- fragment --><p>Your virtual environment is now all setup. Copy the final python script into a python file e.g. <code>ExternalDelegatePythonTutorial.py</code>. Modify the python script above and replace <code><path-to-armnn-binaries></code> and <code><your-armnn-repo-dir></code> with the directories you have set up. If you've been using the ./BuildGuideNative.md "native build guide" this will be <code>$BASEDIR/armnn/build</code> and <code>$BASEDIR/armnn</code>.</p>
<p>Finally, execute the script: </p><div class="fragment"><div class="line">python ExternalDelegatePythonTutorial.py</div></div><!-- fragment --><p> The output should look similar to this: </p><div class="fragment"><div class="line">Info: Arm NN v28.0.0</div><div class="line"></div><div class="line">Info: Initialization time: 0.56 ms</div><div class="line"></div><div class="line">INFO: TfLiteArmnnDelegate: Created TfLite Arm NN delegate.</div><div class="line">[[ 12 123 16 12 11 14 20 16 20 12]]</div><div class="line">Info: Shutdown time: 0.28 ms</div></div><!-- fragment --><p>For more details of the kind of options you can pass to the Arm NN delegate please check the parameters of function tflite_plugin_create_delegate.</p>
<p>You can also test the functionality of the external delegate adaptor by running some unit tests: </p><div class="fragment"><div class="line">pip install pytest</div><div class="line">cd armnn/delegate/python/test</div><div class="line"># You can deselect tests that require backends that your hardware doesn't support using markers e.g. -m "not GpuAccTest"</div><div class="line">pytest --delegate-dir="<path-to-armnn-binaries>/libarmnnDelegate.so" -m "not GpuAccTest"</div></div><!-- fragment --> </div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="quickstart.xhtml">Quick Start Guide</a></li>
<li class="footer">Generated on Wed Nov 23 2022 10:57:39 for ArmNN by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li>
</ul>
</div>
</body>
</html>
|
