1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
|
<!-- Copyright (c) 2020 ARM Limited. -->
<!-- -->
<!-- SPDX-License-Identifier: MIT -->
<!-- -->
<!-- HTML header for doxygen 1.8.13-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<meta name="robots" content="NOINDEX, NOFOLLOW" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>ArmNN: Backend Developer Guide</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
</script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js"],
jax: ["input/TeX","output/HTML-CSS"],
});
</script><script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<img alt="ArmNN" src="Arm_NN_horizontal_blue.png" style="max-width: 10rem; margin-top: .5rem; margin-left 10px"/>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">21.02</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
$(function() {
initMenu('',true,false,'search.php','Search');
$(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('md_src_backends__r_e_a_d_m_e.xhtml','');});
</script>
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0"
name="MSearchResults" id="MSearchResults">
</iframe>
</div>
<div class="header">
<div class="headertitle">
<div class="title">Backend Developer Guide </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>Arm NN allows adding new backends through the 'Pluggable Backend' mechanism.</p>
<h2>How to add a new backend</h2>
<p>Backends reside under <a href="./">src/backends</a>, in separate subfolders. For Linux builds they must have a <code>backend.cmake</code> file, which is read automatically by <a href="backends.cmake">src/backends/backends.cmake</a>. The <code>backend.cmake</code> file under the backend-specific folder is then included by the main CMakeLists.txt file at the root of the Arm NN source tree.</p>
<h3>The backend.cmake file</h3>
<p>The <code>backend.cmake</code> has three main purposes:</p>
<ol type="1">
<li>It makes sure the artifact (a cmake OBJECT library) is linked into the Arm NN shared library by appending the name of the library to the <code>armnnLibraries</code> list.</li>
<li>It makes sure that the subdirectory where backend sources reside gets included into the build.</li>
<li>To include backend-specific unit tests, the object library for the unit tests needs to be added to the <code>armnnUnitTestLibraries</code> list.</li>
</ol>
<p>Example <code>backend.cmake</code> file taken from <a href="reference/backend.cmake">reference/backend.cmake</a>:</p>
<div class="fragment"><div class="line">#</div><div class="line"># Make sure the reference backend is included in the build.</div><div class="line"># By adding the subdirectory, cmake requires the presence of CMakeLists.txt</div><div class="line"># in the reference (backend) folder.</div><div class="line">#</div><div class="line">add_subdirectory(${PROJECT_SOURCE_DIR}/src/backends/reference)</div><div class="line"></div><div class="line">#</div><div class="line"># Add the cmake OBJECT libraries built by the reference backend to the</div><div class="line"># list of libraries linked against the Arm NN shared library.</div><div class="line">#</div><div class="line">list(APPEND armnnLibraries armnnRefBackend armnnRefBackendWorkloads)</div><div class="line"></div><div class="line">#</div><div class="line"># Backend specific unit tests can be integrated through the</div><div class="line"># armnnUnitTestLibraries variable. This makes sure that the</div><div class="line"># UnitTests executable can run the backend-specific unit</div><div class="line"># tests.</div><div class="line">#</div><div class="line">list(APPEND armnnUnitTestLibraries armnnRefBackendUnitTests)</div></div><!-- fragment --><h3>The CMakeLists.txt file</h3>
<p>As described in the previous section, adding a new backend will require creating a <code>CMakeLists.txt</code> in the backend folder. This follows the standard cmake conventions, and is required to build a static cmake OBJECT library to be linked into the Arm NN shared library. As with any cmake build, the code can be structured into subfolders and modules as the developer sees fit.</p>
<p>Example can be found under <a href="reference/CMakeLists.txt">reference/CMakeLists.txt</a>.</p>
<h3>The backend.mk file</h3>
<p>Arm NN on Android uses the native Android build system. New backends are integrated by creating a </p><div class="fragment"><div class="line">{backend.mk```}</div><div class="line">files to be built by the Android build system <span class="keywordflow">for</span> the Arm NN shared library.</div><div class="line"></div><div class="line">Optionally, backend-specific unit tests can be added similarly, by</div><div class="line">appending the list of cpp files to the ```BACKEND_TEST_SOURCES``` variable.</div><div class="line"></div><div class="line">Example taken from [reference/backend.mk](reference/backend.mk):</div><div class="line"></div><div class="line">```make</div><div class="line">BACKEND_SOURCES := \</div><div class="line"> RefLayerSupport.cpp \
RefWorkloadFactory.cpp \
workloads/<a class="code" href="namespacearmnn.xhtml#a7636fbbc4f8ea2d0cf9f3ac2d12a4c62">Activation</a>.cpp \
workloads/ElementwiseFunction.cpp \
workloads/Broadcast.cpp \</div><div class="line"> ...</div><div class="line"></div><div class="line">BACKEND_TEST_SOURCES := \</div><div class="line"> test/RefCreateWorkloadTests.cpp \
test/RefEndToEndTests.cpp \
test/RefJsonPrinterTests.cpp \</div><div class="line"> ...</div></div><!-- fragment --><h2>How to add common code across backends</h2>
<p>For multiple backends that need common code, there is support for including them in the build similarly to the backend code. This requires adding three files under a subfolder at the same level as the backends folders. These are:</p>
<ol type="1">
<li>common.cmake</li>
<li>common.mk</li>
<li>CMakeLists.txt</li>
</ol>
<p>They work the same way as the backend files. The only difference between them is that common code is built first, so the backend code can depend on them.</p>
<p>[aclCommon](aclCommon) is an example for this concept and you can find the corresponding files:</p>
<ol type="1">
<li><a href="aclCommon/common.cmake">aclCommon/common.cmake</a></li>
<li><a href="aclCommon/common.mk">aclCommon/common.mk</a></li>
<li><a href="aclCommon/CMakeLists.txt">aclCommon/CMakeLists.txt</a></li>
</ol>
<h2>Identifying backends</h2>
<p>Backends are identified by a string that must be unique across backends. This string is wrapped in the <a href="../../include/armnn/BackendId.hpp">BackendId</a> object for backward compatibility with previous Arm NN versions.</p>
<h2>The IBackendInternal interface</h2>
<p>All backends need to implement the <a href="../../include/armnn/backends/IBackendInternal.hpp">IBackendInternal</a> interface. The interface functions to be implemented are:</p>
<div class="fragment"><div class="line">{c++}</div><div class="line"> <span class="keyword">virtual</span> <a class="code" href="namespacearmnn.xhtml#a12bff6d51d63dac1375c89bc8415dc46">IMemoryManagerUniquePtr</a> CreateMemoryManager() <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> IWorkloadFactoryPtr CreateWorkloadFactory(</div><div class="line"> <span class="keyword">const</span> IMemoryManagerSharedPtr& memoryManager = <span class="keyword">nullptr</span>) <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> IBackendContextPtr CreateBackendContext(<span class="keyword">const</span> IRuntime::CreationOptions&) <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> IBackendProfilingContextPtr CreateBackendProfilingContext(<span class="keyword">const</span> IRuntime::CreationOptions& creationOptions,</div><div class="line"> <a class="code" href="classarmnn_1_1profiling_1_1_i_backend_profiling.xhtml">armnn::profiling::IBackendProfiling</a>& backendProfiling) <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> <a class="code" href="namespacearmnn.xhtml#a11fa919c11fe46aad613b2e960fcfe90">ILayerSupportSharedPtr</a> GetLayerSupport() <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> Optimizations GetOptimizations() <span class="keyword">const</span> = 0;</div><div class="line"> <span class="keyword">virtual</span> SubgraphUniquePtr OptimizeSubgraph(<span class="keyword">const</span> SubgraphView& subgraph, <span class="keywordtype">bool</span>& optimizationAttempted) <span class="keyword">const</span>;</div><div class="line"> <span class="keyword">virtual</span> OptimizationViews OptimizeSubgraphView(<span class="keyword">const</span> SubgraphView& subgraph) <span class="keyword">const</span>;</div></div><!-- fragment --><p>Note that <code>GetOptimizations()</code> and <code>SubgraphViewUniquePtr OptimizeSubgraphView(const SubgraphView& subgraph, bool& optimizationAttempted)</code> have been deprecated. The method <code>OptimizationViews OptimizeSubgraph(const SubgraphView& subgraph)</code> should be used instead to apply specific optimizations to a given sub-graph.</p>
<p>The Arm NN framework then creates instances of the IBackendInternal interface with the help of the <a href="../../include/armnn/BackendRegistry.hpp">BackendRegistry</a> singleton.</p>
<p><b>Important:</b> the <code>IBackendInternal</code> object is not guaranteed to have a longer lifetime than the objects it creates. It is only intended to be a single entry point for the factory functions it has. The best use of this is to be a lightweight, stateless object and make no assumptions between its lifetime and the lifetime of the objects it creates.</p>
<p>For each backend one needs to register a factory function that can be retrieved using a <a href="../../include/armnn/BackendId.hpp">BackendId</a>. The Arm NN framework creates the backend interfaces dynamically when it sees fit and it keeps these objects for a short period of time. Examples:</p>
<ul>
<li>During optimization Arm NN needs to decide which layers are supported by the backend. To do this, it creates a backends and calls the <code>GetLayerSupport()</code> function and creates an <code>ILayerSupport</code> object to help deciding this.</li>
<li>During optimization Arm NN can run backend-specific optimizations. After splitting the graph into sub-graphs based on backends, it calls the <code>OptimizeSubgraphView()</code> function on each of them and, if possible, substitutes the corresponding sub-graph in the original graph with its optimized version.</li>
<li>When the Runtime is initialized it creates an optional <code>IBackendContext</code> object and keeps this context alive for the Runtime's lifetime. It notifies this context object before and after a network is loaded or unloaded.</li>
<li>When the LoadedNetwork creates the backend-specific workloads for the layers, it creates a backend specific workload factory and calls this to create the workloads.</li>
</ul>
<h2>The BackendRegistry</h2>
<p>As mentioned above, all backends need to be registered through the BackendRegistry so Arm NN knows about them. Registration requires a unique backend ID string and a lambda function that returns a unique pointer to the <a href="../../include/armnn/backends/IBackendInternal.hpp">IBackendInternal interface</a>.</p>
<p>For registering a backend only this lambda function needs to exist, not the actual backend. This allows dynamically creating the backend objects when they are needed.</p>
<p>The BackendRegistry has a few convenience functions, like we can query the registered backends and are able to tell if a given backend is registered or not.</p>
<p>Dynamic backends are registered during the runtime creation.</p>
<h2>The ILayerSupport interface</h2>
<p>Arm NN uses the <a href="../../include/armnn/ILayerSupport.hpp">ILayerSupport</a> interface to decide if a layer with a set of parameters (i.e. input and output tensors, descriptor, weights, filter, kernel if any) are supported on a given backend. The backends need a way to communicate this information by implementing the <code>GetLayerSupport()</code> function on the <code>IBackendInternal</code> interface.</p>
<p>Examples of this can be found in the <a href="reference/RefLayerSupport.hpp">RefLayerSupport header</a> and the <a href="reference/RefLayerSupport.cpp">RefLayerSupport implementation</a>.</p>
<h2>The IWorkloadFactory interface</h2>
<p>The <a href="backendsCommon/WorkloadFactory.hpp">IWorkloadFactory interface</a> is used for creating the backend specific workloads. The factory function that creates the IWorkloadFactory object in the IBackendInterface takes an IMemoryManager object.</p>
<p>To create a workload object the <code>IWorkloadFactory</code> takes a <code>WorkloadInfo</code> object that holds the input and output tensor information and a workload specific queue descriptor.</p>
<h2>The IMemoryManager interface</h2>
<p>Backends may choose to implement custom memory management. Arm NN supports this concept through the following mechanism:</p>
<ul>
<li>the <code>IBackendInternal</code> interface has a <code>CreateMemoryManager()</code> function, which is called before creating the workload factory</li>
<li>the memory manager is passed to the <code>CreateWorkloadFactory(...)</code> function so the workload factory can use it for creating the backend-specific workloads</li>
<li>the LoadedNetwork calls <code>Acquire()</code> on the memory manager before it starts executing the network and it calls <code>Release()</code> in its destructor</li>
</ul>
<h2>The Optimizations</h2>
<p>The backends may choose to implement backend-specific optimizations. This is supported through the method <code>OptimizationViews OptimizeSubgraph(const SubgraphView& subgraph)</code> of the backend interface that allows the backends to apply their specific optimizations to a given sub-graph.</p>
<p>The <code>OptimizeSubgraph(...)</code> method returns an OptimizationViews object containing three lists:</p>
<ul>
<li>A list of the sub-graph substitutions: a "substitution" is a pair of sub-graphs, the first is the "substitutable" sub-graph, representing the part of the original graph that has been optimized by the backend, while the second is the "replacement" sub-graph, containing the actual optimized layers that will be replaced in the original graph correspondingly to the "substitutable" sub-graph</li>
<li>A list of the failed sub-graphs: these are the parts of the original sub-graph that are not supported by the backend, thus have been rejected. Arm NN will try to re-allocate these parts on other backends if available.</li>
<li>A list of the untouched sub-graphs: these are the parts of the original sub-graph that have not been optimized, but that can run (unoptimized) on the backend.</li>
</ul>
<p>The previous way backends had to provide a list optimizations to the Optimizer (through the <code>GetOptimizations()</code> method) is still in place for backward compatibility, but it's now considered deprecated and will be remove in a future release.</p>
<h2>The IBackendContext interface</h2>
<p>Backends may need to be notified whenever a network is loaded or unloaded. To support that, one can implement the optional <a href="../../include/armnn/backends/IBackendContext.hpp">IBackendContext</a> interface. The framework calls the <code>CreateBackendContext(...)</code> method for each backend in the Runtime. If the backend returns a valid unique pointer to a backend context, then the runtime will hold this for its entire lifetime. It then calls the following interface functions for each stored context:</p>
<ul>
<li><code>BeforeLoadNetwork(NetworkId networkId)</code></li>
<li><code>AfterLoadNetwork(NetworkId networkId)</code></li>
<li><code>BeforeUnloadNetwork(NetworkId networkId)</code></li>
<li><code>AfterUnloadNetwork(NetworkId networkId)</code></li>
</ul>
<h2>Dynamic backends</h2>
<p>Backends can also be loaded by Arm NN dynamically at runtime. To be properly loaded and used, the backend instances must comply to the standard interface for dynamic backends and to the versioning rules that enforce ABI compatibility.</p>
<h2>Dynamic backends base interface</h2>
<p>The dynamic backend shared object must expose the following interface for Arm NN to handle it correctly:</p>
<div class="fragment"><div class="line">{c++}</div><div class="line"><span class="keyword">extern</span> <span class="stringliteral">"C"</span></div><div class="line">{</div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span>* <a class="code" href="_test_dynamic_backend_8cpp.xhtml#adaff295134ed2825ae43a8e9281b6f2a">GetBackendId</a>();</div><div class="line"><span class="keywordtype">void</span> <a class="code" href="namespacearmnn.xhtml#aa09a8bb02eed50715082d8b7fccd2f8d">GetVersion</a>(uint32_t* outMajor, uint32_t* outMinor);</div><div class="line"><span class="keywordtype">void</span>* <a class="code" href="_test_dynamic_backend_8cpp.xhtml#a6a075b7c32d5511f95903749eef44b22">BackendFactory</a>();</div><div class="line">}</div></div><!-- fragment --><p>Interface details:</p>
<ul>
<li><code>extern "C"</code> is needed to use avoid C++ name mangling, necessary to allow Arm NN to dynamically load the symbols.</li>
<li><code><a class="el" href="_test_dynamic_backend_8cpp.xhtml#adaff295134ed2825ae43a8e9281b6f2a">GetBackendId()</a></code>: must return the unique id of the dynamic backends. If at the time of the loading the id already exists in the internal Arm NN's backend registry, the backend will be skipped and not loaded in Arm NN</li>
<li><code><a class="el" href="namespacearmnn.xhtml#aa09a8bb02eed50715082d8b7fccd2f8d">GetVersion()</a></code>: must return the version of the dynamic backend. The version must indicate the version of the Backend API the dynamic backend has been built with. The current Backend API version can be found by inspecting the IBackendInternal interface. At the time of loading, the version of the backend will be checked against the version of the Backend API Arm NN is built with. If the backend version is not compatible with the current Backend API, the backend will not be loaded as it will be assumed that it is not ABI compatible with the current Arm NN build.</li>
<li><code><a class="el" href="_test_dynamic_backend_8cpp.xhtml#a6a075b7c32d5511f95903749eef44b22">BackendFactory()</a></code>: must return a valid instance of the backend. The backend instance is an object that must inherit from the version of the IBackendInternal interface declared by <a class="el" href="namespacearmnn.xhtml#aa09a8bb02eed50715082d8b7fccd2f8d">GetVersion()</a>. It is the backend developer's responsibility to ensure that the backend implementation correctly reflects the version declared by <a class="el" href="namespacearmnn.xhtml#aa09a8bb02eed50715082d8b7fccd2f8d">GetVersion()</a>, and that the object returned by the <a class="el" href="_test_dynamic_backend_8cpp.xhtml#a6a075b7c32d5511f95903749eef44b22">BackendFactory()</a> function is a valid and well-formed instance of the IBackendInternal interface.</li>
</ul>
<h2>Dynamic backend versioning and ABI compatibility</h2>
<p>Dynamic backend versioning policy:</p>
<p>Updates to Arm NN's Backend API follow these rules: changes to the Backend API (the IBackendInternal interface) that break ABI compatibility with the previous API version will be indicated by a change of the API's major version, while changes that guarantee ABI compatibility with the previous API version will be indicated by a change in API's the minor version.</p>
<p>For example:</p>
<ul>
<li>Dynamic backend version 2.4 (i.e. built with Backend API version 2.4) is compatible with Arm NN's Backend API version 2.4 (same version, backend built against the same Backend API)</li>
<li>Dynamic backend version 2.1 (i.e. built with Backend API version 2.1) is compatible with Arm NN's Backend API version 2.4 (same major version, backend built against earlier compatible API)</li>
<li>Dynamic backend version 2.5 (i.e. built with Backend API version 2.5) is not compatible with Arm NN's Backend API version 2.4 (same major version, backend built against later incompatible API, backend might require update to the latest compatible backend API)</li>
<li>Dynamic backend version 2.0 (i.e. built with Backend API version 2.0) is not compatible with Arm NN's Backend API version 1.0 (backend requires a completely new API version)</li>
<li>Dynamic backend version 2.0 (i.e. built with Backend API version 2.0) is not compatible with Arm NN's Backend API version 3.0 (backward compatibility in the Backend API is broken)</li>
</ul>
<h2>Dynamic backend loading paths</h2>
<p>During the creation of the Runtime, Arm NN will scan a given set of paths searching for suitable dynamic backend objects to load. A list of (absolute) paths can be specified at compile-time by setting a define named <code>DYNAMIC_BACKEND_PATHS</code> in the form of a colon-separated list of strings.</p>
<div class="fragment"><div class="line">-DDYNAMIC_BACKEND_PATHS="PATH_1:PATH_2...:PATH_N"</div></div><!-- fragment --><p>The paths will be processed in the same order as they are indicated in the macro.</p>
<p>It is also possible to override those paths at runtime when creating the Runtime object by setting the value of the <code>m_DynamicBackendsPath</code> member in the CreationOptions class. Only one path is allowed for the override via the CreationOptions class. By setting the value of the <code>m_DynamicBackendsPath</code> to a path in the filesystem, Arm NN will entirely ignore the list of paths passed via the </p><div class="fragment"><div class="line">{<a class="code" href="_dynamic_backend_utils_8hpp.xhtml#a6100170f71dc5f7f2558234843db82b3">DYNAMIC_BACKEND_PATHS</a>```}</div><div class="line"></div><div class="line">All the specified paths are validated before processing (they must exist, must be directories, and must be absolute paths),</div><div class="line">in <span class="keywordflow">case</span> of <a class="code" href="namespacearmnn.xhtml#a4dc0adc6737b5944e7671bee71788407acb5e100e5a9a3e7f6d1fd97512215282">error</a> a <a class="code" href="namespacearmnn.xhtml#a4dc0adc6737b5944e7671bee71788407a7b83d3f08fa392b79e3f553b585971cd">warning</a> message will be added to the log, but Arm NN<span class="stringliteral">'s execution will not be stopped.</span></div><div class="line"><span class="stringliteral">If all paths are not valid, then no dynamic backends will be loaded in the Arm NN'</span>s runtime.</div><div class="line"></div><div class="line">Passing an empty list of paths at compile-time and providing no path <span class="keyword">override</span> at runtime will effectively disable the</div><div class="line">dynamic backend loading feature, and no dynamic backends will be loaded into Arm NN<span class="stringliteral">'s runtime.</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">## Dynamic backend file naming convention</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">During the creation of a Runtime object, Arm NN will scan the paths specified for dynamic backend loading searching for suitable backend objects.</span></div><div class="line"><span class="stringliteral">Arm NN will try to load only the files that match the following accepted naming scheme:</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">```shell</span></div><div class="line"><span class="stringliteral"><vendor>_<name>_backend.so[<version>] (e.g. "Arm_GpuAcc_backend.so" or "Arm_GpuAcc_backend.so.1.2.3")</span></div></div><!-- fragment --><p>Only alphanumeric characters are allowed for both the <code><vendor></code> and the <code><name></code> fields, namely lowercase and/or uppercase characters, and/or numerical digits (see the table below for examples). Only dots and numbers are allowed for the optional <code><version></code> field.</p>
<p>Symlinks to other files are allowed to support the standard linux shared object versioning:</p>
<div class="fragment"><div class="line">Arm_GpuAcc_backend.so -> Arm_GpuAcc_backend.so.1.2.3</div><div class="line">Arm_GpuAcc_backend.so.1 -> Arm_GpuAcc_backend.so.1.2.3</div><div class="line">Arm_GpuAcc_backend.so.1.2 -> Arm_GpuAcc_backend.so.1.2.3</div><div class="line">Arm_GpuAcc_backend.so.1.2.3</div></div><!-- fragment --><p>Files are identified by their full canonical path, so it is allowed to have files with the same name in different directories. However, if those are actually the same dynamic backend, only the first in order of parsing will be loaded.</p>
<p>Examples:</p>
<table class="doxtable">
<tr>
<th>Filename </th><th>Description </th></tr>
<tr>
<td>Arm_GpuAcc_backend.so </td><td>valid: basic backend name </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.1 </td><td>valid: single field version number </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.1.2 </td><td>valid: multiple field version number </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.1.2.3 </td><td>valid: multiple field version number </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.10.1.27 </td><td>valid: Multiple digit version </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.10.1.33. </td><td>not valid: dot not followed by version number </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.3.4..5 </td><td>not valid: dot not followed by version number </td></tr>
<tr>
<td>Arm_GpuAcc_backend.so.1,1.1 </td><td>not valid: comma instead of dot in the version </td></tr>
<tr>
<td>Arm123_GpuAcc_backend.so </td><td>valid: digits in vendor name are allowed </td></tr>
<tr>
<td>Arm_GpuAcc456_backend.so </td><td>valid: digits in backend id are allowed </td></tr>
<tr>
<td>ArmCo_GpuAcc_backend.so </td><td>not valid: invalid character in vendor name </td></tr>
<tr>
<td>Arm_Gpu.Acc_backend.so </td><td>not valid: invalid character in backend id </td></tr>
<tr>
<td>GpuAcc_backend.so </td><td>not valid: missing vendor name </td></tr>
<tr>
<td>_GpuAcc_backend.so </td><td>not valid: missing vendor name </td></tr>
<tr>
<td>Arm__backend.so </td><td>not valid: missing backend id </td></tr>
<tr>
<td>Arm_GpuAcc.so </td><td>not valid: missing "backend" at the end </td></tr>
<tr>
<td>__backend.so </td><td>not valid: missing vendor name and backend id </td></tr>
<tr>
<td>__.so </td><td>not valid: missing all fields </td></tr>
<tr>
<td>Arm_GpuAcc_backend </td><td>not valid: missing at least ".so" at the end </td></tr>
<tr>
<td>Arm_GpuAcc_backend_v1.2.so </td><td>not valid: extra version info at the end </td></tr>
<tr>
<td>Arm_CpuAcc_backend.so </td><td>valid: basic backend name </td></tr>
<tr>
<td>Arm_CpuAcc_backend.so.1 -> Arm_CpuAcc_backend.so </td><td>valid: symlink to valid backend file </td></tr>
<tr>
<td>Arm_CpuAcc_backend.so.1.2 -> Arm_CpuAcc_backend.so.1 </td><td>valid: symlink to valid symlink </td></tr>
<tr>
<td>Arm_CpuAcc_backend.so.1.2.3 -> Arm_CpuAcc_backend.so.1.2 </td><td>valid: symlink to valid symlink </td></tr>
<tr>
<td>Arm_no_backend.so -> nothing </td><td>not valid: symlink resolves to non-existent file </td></tr>
<tr>
<td>pathA/Arm_GpuAcc_backend.so </td><td>valid: basic backend name </td></tr>
<tr>
<td>pathB/Arm_GpuAcc_backend.so </td><td>valid: but duplicated from pathA/ </td></tr>
</table>
<p>Arm NN will try to load the dynamic backends in the same order as they are parsed from the filesystem.</p>
<h2>Dynamic backend examples</h2>
<p>The source code includes an example that is used to generate some mock dynamic backends for testing purposes. The source files are:</p>
<p><a href="backendsCommon/test/TestDynamicBackend.hpp">TestDynamicBackend.hpp</a> <a href="backendsCommon/test/TestDynamicBackend.cpp">TestDynamicBackend.cpp</a></p>
<p>This example is useful for going through all the use cases that constitute an invalid dynamic backend object, such as an invalid/malformed implementation of the shared object interface, or an invalid value returned by any of the interface methods that would prevent Arm NN from making use of the dynamic backend.</p>
<p>A dynamic implementation of the reference backend is also provided. The source files are:</p>
<p><a href="dynamic/reference/RefDynamicBackend.hpp">RefDynamicBackend.hpp</a> <a href="dynamic/reference/RefDynamicBackend.cpp">RefDynamicBackend.cpp</a></p>
<p>The implementation itself is quite simple and straightforward. Since an implementation of this particular backend was already available, the dynamic version is just a wrapper around the original code that simply returns the backend id, version and an instance of the backend itself via the factory function. For the sake of the example, the source code of the reference backend is used to build the dynamic version (as you would for any new dynamic backend), while all the other symbols needed are provided by linking the dynamic backend against Arm NN.</p>
<p>The makefile used for building the reference dynamic backend is also provided: <a href="dynamic/reference/CMakeLists.txt">CMakeLists.txt</a></p>
<p>A unit test that loads the reference backend dynamically and that exercises it is also included in the file <a href="dynamic/backendsCommon/test/DynamicBackendTests.cpp">DynamicBackendTests.cpp</a>, by the test case <code>CreateReferenceDynamicBackend</code>. In the test, a path on the filesystem is scanned for valid dynamic backend files (using the override option in <code>CreationOptions</code>) where only the reference dynamic backend is. In this example the file is named <code>Arm_CpuRef_backend.so</code>, which is compliant with the expected naming scheme for dynamic backends. A <code>DynamicBackend</code> is created in the runtime to represent the newly loaded backend, then the backend is registered in the Backend Registry with the id "CpuRef" (returned by <code><a class="el" href="_test_dynamic_backend_8cpp.xhtml#adaff295134ed2825ae43a8e9281b6f2a">GetBackendId()</a></code>). The unit test makes sure that the backend is actually registered in Arm NN, before trying to create an instance of the backend by calling the factory function provided through the shared object interface (<code><a class="el" href="_test_dynamic_backend_8cpp.xhtml#a6a075b7c32d5511f95903749eef44b22">BackendFactory()</a></code>). The backend instance is used to verify that everything is in order, testing basic 2D convolution support by making use of the Layer Support API and the Workload Factory. At the end of test, the runtime object goes out of scope and the dynamic backend instance is automatically destroyed, and the handle to the shared object is closed. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="contribguides.xhtml">Contribution Guides</a></li>
<li class="footer">Generated on Thu Feb 25 2021 17:27:56 for ArmNN by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li>
</ul>
</div>
</body>
</html>
|
