1 # How to Cross-Compile Arm NN on x86_64 for arm64
3 - [Introduction](#introduction)
4 - [Cross-compiling ToolChain](#cross-compiling-toolchain)
5 - [Install Cmake](#build-cmake)
6 - [Build and install Google's Protobuf library](#build-and-install-google-s-protobuf-library)
7 - [Download Arm NN](#download-arm-nn)
8 - [Build Arm Compute Library](#build-arm-compute-library)
9 - [Build Flatbuffer](#build-flatbuffer)
10 - [Build Onnx](#build-onnx)
11 - [Build TfLite](#build-tflite)
12 - [Build Arm NN](#build-armnn)
13 - [Generate TF Lite Schema](#generate-tflite-schema)
14 - [Build Standalone Sample Dynamic Backend](#build-standalone-sample-dynamic-backend)
15 - [Run Unit Tests](#run-unit-tests)
16 - [Troubleshooting and Errors:](#troubleshooting-and-errors-)
18 ### NOTE: This build guide is being replaced by the [Arm NN Build Tool](build-tool/README.md) and will be removed soon. Please use the [Arm NN Build Tool](build-tool/README.md) for a user-friendly way to build Arm NN and its dependencies.
21 These are the step by step instructions on Cross-Compiling Arm NN under an x86_64 system to target an Arm64 Ubuntu Linux system. This build flow has been tested with Ubuntu 18.04 and 20.04 and it depends on the same version of Ubuntu or Debian being installed on both the build host and target machines. The instructions assume you are using a bash shell and show how to build the Arm NN core library, Protobuf, Tflite, Flatbuffer and Compute Libraries.
22 Start by creating a directory to contain all components:
25 mkdir $HOME/armnn-devenv
29 ## Cross-compiling ToolChain
30 * Install the standard cross-compilation libraries for arm64:
32 sudo apt install crossbuild-essential-arm64
36 Cmake 3.19rc3 is required to build TF Lite Delegate.
39 sudo apt-get install libssl-dev
40 wget https://github.com/Kitware/CMake/releases/download/v3.19.0-rc3/cmake-3.19.0-rc3.tar.gz
41 tar -zxvf cmake-3.19.0-rc3.tar.gz
43 ./bootstrap --prefix=$HOME/armnn-devenv/cmake/install
49 ## Build and install Google's Protobuf library
51 We support protobuf version 3.12.0
52 * Get protobuf from here: https://github.com/protocolbuffers/protobuf:
53 (Requires Git if not previously installed: `sudo apt install git`)
55 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf
57 git submodule update --init --recursive
60 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
61 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
65 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
69 * Build the arm64 version of the protobuf libraries:
73 CC=aarch64-linux-gnu-gcc \
74 CXX=aarch64-linux-gnu-g++ \
75 ../configure --host=aarch64-linux \
76 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
77 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
86 git clone https://github.com/ARM-software/armnn.git
89 * Checkout Arm NN branch:
92 git checkout <branch_name>
95 For example, if you want to check out the 21.11 release branch:
97 git checkout branches/armnn_21_11
101 ## Build Arm Compute Library
102 * Clone Arm Compute Library:
105 cd $HOME/armnn-devenv
106 git clone https://github.com/ARM-software/ComputeLibrary.git
108 * Checkout Arm Compute Library release tag:
111 git checkout <tag_name>
113 Arm NN and Arm Compute Library are developed closely together. If you would like to use the Arm NN 21.11 release you will need the 21.11 release of ACL too. For example, if you want to checkout the 21.11 release tag:
117 Arm NN provides a script that downloads the version of Arm Compute Library that Arm NN was tested with:
119 git checkout $(../armnn/scripts/get_compute_library.sh -p)
121 * Build the Arm Compute Library:
122 (Requires SCons if not previously installed: `sudo apt install scons`)
124 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j4
128 * Building Flatbuffer version 1.12.0
130 cd $HOME/armnn-devenv
131 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
132 tar xf flatbuffers-1.12.0.tar.gz
133 cd flatbuffers-1.12.0
137 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
138 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers \
139 -DFLATBUFFERS_BUILD_TESTS=0
143 * Build arm64 version of flatbuffer
148 # Add -fPIC to allow us to use the libraries in shared objects.
149 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
150 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
151 -DFLATBUFFERS_BUILD_FLATC=1 \
152 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers-arm64 \
153 -DFLATBUFFERS_BUILD_TESTS=0
160 cd $HOME/armnn-devenv
161 git clone https://github.com/onnx/onnx.git
163 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD
164 LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib:$LD_LIBRARY_PATH \
165 $HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc \
166 onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out $HOME/armnn-devenv/onnx
170 * Arm NN provides a script, armnn/scripts/get_tensorflow.sh, that can be used to check out the version of TensorFlow that Arm NN was tested with:
172 cd $HOME/armnn-devenv
173 git clone https://github.com/tensorflow/tensorflow.git
175 git checkout $(../armnn/scripts/get_tensorflow.sh -p) # Checks out the latest tested version of TF
179 * You will need to download gcc-arm-8.3-2019.03 toolchain and continue building TF Lite as following:
181 curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz
182 mkdir tflite-toolchains
183 tar xvf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz -C tflite-toolchains
184 mkdir -p tflite/build
186 ARMCC_PREFIX=$HOME/armnn-devenv/tflite-toolchains/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu- \
187 ARMCC_FLAGS="-funsafe-math-optimizations" \
188 cmake -DCMAKE_C_COMPILER=${ARMCC_PREFIX}gcc \
189 -DCMAKE_CXX_COMPILER=${ARMCC_PREFIX}g++ \
190 -DCMAKE_C_FLAGS="${ARMCC_FLAGS}" -DCMAKE_CXX_FLAGS="${ARMCC_FLAGS}" \
191 -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_SYSTEM_NAME=Linux \
192 -DTFLITE_ENABLE_XNNPACK=OFF \
193 -DCMAKE_SYSTEM_PROCESSOR=aarch64 \
194 $HOME/armnn-devenv/tensorflow/tensorflow/lite/ \
198 ## Generate TF Lite Schema
200 cd $HOME/armnn-devenv/tflite
201 cp $HOME/armnn-devenv/tensorflow/tensorflow/lite/schema/schema.fbs .
202 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
206 * Compile Arm NN for arm64:
208 cd $HOME/armnn-devenv/armnn
213 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the Arm NN build:
216 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
217 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
218 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
219 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
220 -DONNX_GENERATED_SOURCES=$HOME/armnn-devenv/onnx \
221 -DBUILD_ONNX_PARSER=1 \
222 -DBUILD_TF_LITE_PARSER=1 \
223 -DTENSORFLOW_ROOT=$HOME/armnn-devenv/tensorflow \
224 -DTF_LITE_SCHEMA_INCLUDE_PATH=$WORKING_DIR/tflite \
225 -DFLATBUFFERS_ROOT=$HOME/armnn-devenv/flatbuffers-arm64 \
226 -DFLATC_DIR=$HOME/armnn-devenv/flatbuffers-1.12.0/build \
227 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install \
228 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 \
229 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0
232 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
234 -DSAMPLE_DYNAMIC_BACKEND=1 \
235 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
237 * If you want to build Arm NN TF Lite Delegate, add the arguments:
239 -DTFLITE_LIB_ROOT=$HOME/armnn-devenv/tflite/build \
240 -DBUILD_ARMNN_TFLITE_DELEGATE=1
247 ## Build Standalone Sample Dynamic Backend
248 * The sample dynamic backend is located in armnn/src/dynamic/sample
250 cd $HOME/armnn-devenv/armnn/src/dynamic/sample
255 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the Arm NN build:
258 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
259 -DCMAKE_CXX_FLAGS=--std=c++14 \
260 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
269 * Copy the build folder to an arm64 linux machine
270 * Copy the libprotobuf.so.23.0.0 library file to the build folder
271 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build Arm NN
272 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
278 * Create a symbolic link to libprotobuf.so.23.0.0:
281 ln -s libprotobuf.so.23.0.0 ./libprotobuf.so.23
287 LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./UnitTests
288 [doctest] doctest version is "2.4.6"
289 [doctest] run with "--help" for options
290 ===============================================================================
291 [doctest] test cases: 4817 | 4817 passed | 0 failed | 0 skipped
292 [doctest] assertions: 807634 | 807634 passed | 0 failed |
293 [doctest] Status: SUCCESS!
296 * Run the Delegate UnitTests:
299 LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./delegate/DelegateUnitTests
302 ## Troubleshooting and Errors:
304 * When Cross-compiling Arm NN if you are getting error
306 [ 62%] Generating SchemaText.cpp
307 /bin/sh: 1: xxd: not found
308 make[2]: *** [armnn/CMakeFiles/UnitTests.dir/build.make:63: armnn/SchemaText.cpp] Error 127
309 make[1]: *** [CMakeFiles/Makefile2:674: armnn/CMakeFiles/UnitTests.dir/all] Error 2
310 make[1]: *** Waiting for unfinished jobs....
312 * Missing xxd, this can be fixed by installing it:
314 sudo apt-get install xxd
319 ### Missing libz.so.1
320 * When compiling armNN:
322 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.23.0.0, not found (try using -rpath or -rpath-link)
325 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
327 sudo dpkg --add-architecture arm64
328 sudo apt-get install zlib1g:arm64
332 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
333 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
334 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
336 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
340 ### Unable to install arm64 packages after adding arm64 architecture
341 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
342 * From stackoverflow:
343 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
344 * Open /etc/apt/sources.list with your preferred text editor.
346 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
348 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
350 * Then add the following:
352 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
353 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
354 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
355 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
356 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
357 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
358 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
360 * Update and install again:
362 sudo apt-get install zlib1g:arm64
368 ### Undefined references to google::protobuf:: functions
369 * Missing or out of date protobuf compilation libraries.
370 Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required).
371 Follow the instructions above to install protobuf 3.12.0
374 ### Errors on strict-aliasing rules when compiling the Compute Library
375 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
377 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
379 * Add Werror=0 to the scons command:
381 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 Werror=0