1 # How to Cross-Compile Arm NN on x86_64 for arm64
3 - [Introduction](#introduction)
4 - [Cross-compiling ToolChain](#cross-compiling-toolchain)
5 - [Build and install Google's Protobuf library](#build-and-install-google-s-protobuf-library)
6 - [Build Boost library for arm64](#build-boost-library-for-arm64)
7 - [Build Compute Library](#build-compute-library)
8 - [Download ArmNN](#download-armnn)
9 - [Build Flatbuffer](#build-flatbuffer)
10 - [Build Onnx](#build-onnx)
11 - [Build TfLite](#build-tflite)
12 - [Build Arm NN](#build-armnn)
13 - [Build Standalone Sample Dynamic Backend](#build-standalone-sample-dynamic-backend)
14 - [Run Unit Tests](#run-unit-tests)
15 - [Troubleshooting and Errors:](#troubleshooting-and-errors-)
19 These are the step by step instructions on Cross-Compiling Arm NN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
20 The instructions assume you are using a bash shell and show how to build the Arm NN core library, Boost, Protobuf, Tflite, Flatbuffer and Compute Libraries.
21 Start by creating a directory to contain all components:
24 mkdir $HOME/armnn-devenv
28 #####Note: We are currently in the process of removing boost as a dependency to Arm NN. This process is finished for everything apart from our unit tests. This means you don't need boost to build and use Arm NN but you need it to execute our unit tests. Boost will soon be removed from Arm NN entirely. We also are deprecating support for Caffe and Tensorflow parsers in 21.02. This will be removed in 21.05.
30 ## Cross-compiling ToolChain
31 * Install the standard cross-compilation libraries for arm64:
33 sudo apt install crossbuild-essential-arm64
36 ## Build and install Google's Protobuf library
38 We support protobuf version 3.12.0
39 * Get protobuf from here: https://github.com/protocolbuffers/protobuf :
41 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf
43 git submodule update --init --recursive
46 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
47 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
51 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
55 * Build the arm64 version of the protobuf libraries:
59 CC=aarch64-linux-gnu-gcc \
60 CXX=aarch64-linux-gnu-g++ \
61 ../configure --host=aarch64-linux \
62 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
63 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
68 ## Build Boost library for arm64
69 * Build Boost library for arm64
70 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
71 Using any version of Boost greater than 1.64 will fail to build Arm NN, due to different dependency issues.
74 tar -zxvf boost_1_64_0.tar.gz
76 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
77 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
78 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
81 ## Build Compute Library
82 * Building the Arm Compute Library:
85 git clone https://github.com/ARM-software/ComputeLibrary.git
87 git checkout <tag_name>
88 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j4 internal_only=0
91 For example, if you want to checkout release tag of 21.02:
100 git clone https://github.com/ARM-software/armnn.git
102 git checkout <branch_name>
106 For example, if you want to checkout release branch of 21.02:
108 git checkout branches/armnn_21_02
113 * Building Flatbuffer version 1.12.0
115 cd $HOME/armnn-devenv
116 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
117 tar xf flatbuffers-1.12.0.tar.gz
118 cd flatbuffers-1.12.0
122 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
123 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers \
124 -DFLATBUFFERS_BUILD_TESTS=0
128 * Build arm64 version of flatbuffer
133 # Add -fPIC to allow us to use the libraries in shared objects.
134 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
135 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
136 -DFLATBUFFERS_BUILD_FLATC=1 \
137 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers-arm64 \
138 -DFLATBUFFERS_BUILD_TESTS=0
145 cd $HOME/armnn-devenv
146 git clone https://github.com/onnx/onnx.git
148 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD
149 LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib:$LD_LIBRARY_PATH \
150 $HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc \
151 onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out $HOME/armnn-devenv/onnx
155 * Building TfLite (Tensorflow version 2.3.1)
157 cd $HOME/armnn-devenv
158 git clone https://github.com/tensorflow/tensorflow.git
160 git checkout fcc4b966f1265f466e82617020af93670141b009
163 cp ../tensorflow/tensorflow/lite/schema/schema.fbs .
164 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
168 * Compile Arm NN for arm64:
170 cd $HOME/armnn-devenv/armnn
175 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the Arm NN build:
178 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
179 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
180 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
181 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
182 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
183 -DONNX_GENERATED_SOURCES=$HOME/armnn-devenv/onnx \
184 -DBUILD_ONNX_PARSER=1 \
185 -DBUILD_TF_LITE_PARSER=1 \
186 -DTF_LITE_GENERATED_PATH=$HOME/armnn-devenv/tflite \
187 -DFLATBUFFERS_ROOT=$HOME/armnn-devenv/flatbuffers-arm64 \
188 -DFLATC_DIR=$HOME/armnn-devenv/flatbuffers-1.12.0/build \
189 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install \
190 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
191 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 \
192 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0
195 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
197 -DSAMPLE_DYNAMIC_BACKEND=1 \
198 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
205 ## Build Standalone Sample Dynamic Backend
206 * The sample dynamic backend is located in armnn/src/dynamic/sample
208 cd $HOME/armnn-devenv/armnn/src/dynamic/sample
213 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the Arm NN build:
216 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
217 -DCMAKE_CXX_FLAGS=--std=c++14 \
218 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
227 * Copy the build folder to an arm64 linux machine
228 * Copy the libprotobuf.so.23.0.0 library file to the build folder
229 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build Arm NN
230 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
236 * Create a symbolic link to libprotobuf.so.23.0.0:
239 ln -s libprotobuf.so.23.0.0 ./libprotobuf.so.23
245 LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./UnitTests
246 Running 4493 test cases...
248 *** No errors detected
251 ## Troubleshooting and Errors:
252 ### Error adding symbols: File in wrong format
253 * When building Arm NN:
255 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
256 collect2: error: ld returned 1 exit status
257 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
258 make[2]: *** [libarmnn.so] Error 1
259 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
260 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
261 Makefile:127: recipe for target 'all' failed
262 make: *** [all] Error 2
264 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
267 ### Virtual memory exhausted
268 * When compiling the boost libraries:
270 virtual memory exhausted: Cannot allocate memory
272 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
275 ### Unrecognized command line option '-m64'
276 * When compiling the boost libraries:
278 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
280 * Clean the boost library directory before trying to build with a different architecture:
284 * It should show the following for arm64:
292 ### Missing libz.so.1
293 * When compiling armNN:
295 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.23.0.0, not found (try using -rpath or -rpath-link)
298 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
300 sudo dpkg --add-architecture arm64
301 sudo apt-get install zlib1g:arm64
305 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
306 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
307 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
309 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
313 ### Unable to install arm64 packages after adding arm64 architecture
314 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
315 * From stackoverflow:
316 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
317 * Open /etc/apt/sources.list with your preferred text editor.
319 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
321 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
323 * Then add the following:
325 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
326 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
327 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
328 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
329 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
330 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
331 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
333 * Update and install again:
335 sudo apt-get install zlib1g:arm64
341 ### Undefined references to google::protobuf:: functions
342 * Missing or out of date protobuf compilation libraries.
343 Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required).
344 Follow the instructions above to install protobuf 3.12.0
347 ### Errors on strict-aliasing rules when compiling the Compute Library
348 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
350 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
352 * Add Werror=0 to the scons command:
354 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0