1 # How to Cross-Compile Arm NN on x86_64 for arm64
3 - [Introduction](#introduction)
4 - [Cross-compiling ToolChain](#cross-compiling-toolchain)
5 - [Build and install Google's Protobuf library](#build-and-install-google-s-protobuf-library)
6 - [Build Caffe for x86_64](#build-caffe-for-x86-64)
7 - [Build Boost library for arm64](#build-boost-library-for-arm64)
8 - [Build Compute Library](#build-compute-library)
9 - [Download ArmNN](#download-armnn)
10 - [Build Tensorflow](#build-tensorflow)
11 - [Build Flatbuffer](#build-flatbuffer)
12 - [Build Onnx](#build-onnx)
13 - [Build TfLite](#build-tflite)
14 - [Build Arm NN](#build-armnn)
15 - [Build Standalone Sample Dynamic Backend](#build-standalone-sample-dynamic-backend)
16 - [Run Unit Tests](#run-unit-tests)
17 - [Troubleshooting and Errors:](#troubleshooting-and-errors-)
21 These are the step by step instructions on Cross-Compiling Arm NN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
22 The instructions assume you are using a bash shell and show how to build the Arm NN core library, Boost, Protobuf, Caffe, Tensorflow, Tflite, Flatbuffer and Compute Libraries.
23 Start by creating a directory to contain all components:
26 mkdir $HOME/armnn-devenv
30 #####Note: We are currently in the process of removing boost as a dependency to Arm NN. This process is finished for everything apart from our unit tests. This means you don't need boost to build and use Arm NN but you need it to execute our unit tests. Boost will soon be removed from Arm NN entirely. We also are deprecating support for Caffe and Tensorflow parsers in 21.02. This will be removed in 21.05.
32 ## Cross-compiling ToolChain
33 * Install the standard cross-compilation libraries for arm64:
35 sudo apt install crossbuild-essential-arm64
38 ## Build and install Google's Protobuf library
40 We support protobuf version 3.12.0
41 * Get protobuf from here: https://github.com/protocolbuffers/protobuf :
43 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf
45 git submodule update --init --recursive
48 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
49 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
53 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
57 * Build the arm64 version of the protobuf libraries:
61 CC=aarch64-linux-gnu-gcc \
62 CXX=aarch64-linux-gnu-g++ \
63 ../configure --host=aarch64-linux \
64 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
65 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
70 ## Build Caffe for x86_64
71 * Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html
72 * Install dependencies:
74 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev
75 sudo apt-get install --no-install-recommends libboost-all-dev
76 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
77 sudo apt-get install libopenblas-dev
78 sudo apt-get install libatlas-base-dev
80 * Download Caffe from: https://github.com/BVLC/caffe. We have tested using tag 1.0
83 git clone https://github.com/BVLC/caffe.git
85 git checkout eeebdab16155d34ff8f5f42137da7df4d1c7eab0
86 cp Makefile.config.example Makefile.config
88 * Adjust Makefile.config as necessary for your environment, for example:
93 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir):
94 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $(HOME)/armnn-devenv/google/x86_64_pb_install/include/
95 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $(HOME)/armnn-devenv/google/x86_64_pb_install/lib/
99 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH
100 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH
102 * Compilation with Make:
107 # These should all run without errors
110 * caffe.pb.h and caffe.pb.cc will be needed when building Arm NN's Caffe Parser
112 ## Build Boost library for arm64
113 * Build Boost library for arm64
114 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
115 Using any version of Boost greater than 1.64 will fail to build Arm NN, due to different dependency issues.
117 cd $HOME/armnn-devenv
118 tar -zxvf boost_1_64_0.tar.gz
120 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
121 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
122 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
125 ## Build Compute Library
126 * Building the Arm Compute Library:
128 cd $HOME/armnn-devenv
129 git clone https://github.com/ARM-software/ComputeLibrary.git
131 git checkout <tag_name>
132 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j4 internal_only=0
135 For example, if you want to checkout release tag of 21.02:
143 cd $HOME/armnn-devenv
144 git clone https://github.com/ARM-software/armnn.git
146 git checkout <branch_name>
150 For example, if you want to checkout release branch of 21.02:
152 git checkout branches/armnn_21_02
157 * Building Tensorflow version 2.3.1:
159 cd $HOME/armnn-devenv
160 git clone https://github.com/tensorflow/tensorflow.git
162 git checkout fcc4b966f1265f466e82617020af93670141b009
163 ../armnn/scripts/generate_tensorflow_protobuf.sh ../tensorflow-protobuf ../google/x86_64_pb_install
167 * Building Flatbuffer version 1.12.0
169 cd $HOME/armnn-devenv
170 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
171 tar xf flatbuffers-1.12.0.tar.gz
172 cd flatbuffers-1.12.0
176 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
177 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers \
178 -DFLATBUFFERS_BUILD_TESTS=0
182 * Build arm64 version of flatbuffer
187 # Add -fPIC to allow us to use the libraries in shared objects.
188 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
189 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
190 -DFLATBUFFERS_BUILD_FLATC=1 \
191 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers-arm64 \
192 -DFLATBUFFERS_BUILD_TESTS=0
199 cd $HOME/armnn-devenv
200 git clone https://github.com/onnx/onnx.git
202 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD
203 LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib:$LD_LIBRARY_PATH \
204 $HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc \
205 onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out $HOME/armnn-devenv/onnx
211 cd $HOME/armnn-devenv
214 cp ../tensorflow/tensorflow/lite/schema/schema.fbs .
215 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
219 * Compile Arm NN for arm64:
221 cd $HOME/armnn-devenv/armnn
226 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the Arm NN build:
229 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
230 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
231 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
232 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
233 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
234 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \
235 -DBUILD_CAFFE_PARSER=1 \
236 -DONNX_GENERATED_SOURCES=$HOME/armnn-devenv/onnx \
237 -DBUILD_ONNX_PARSER=1 \
238 -DTF_GENERATED_SOURCES=$HOME/armnn-devenv/tensorflow-protobuf \
239 -DBUILD_TF_PARSER=1 \
240 -DBUILD_TF_LITE_PARSER=1 \
241 -DTF_LITE_GENERATED_PATH=$HOME/armnn-devenv/tflite \
242 -DFLATBUFFERS_ROOT=$HOME/armnn-devenv/flatbuffers-arm64 \
243 -DFLATC_DIR=$HOME/armnn-devenv/flatbuffers-1.12.0/build \
244 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install \
245 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
246 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 \
247 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0
250 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
252 -DSAMPLE_DYNAMIC_BACKEND=1 \
253 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
260 ## Build Standalone Sample Dynamic Backend
261 * The sample dynamic backend is located in armnn/src/dynamic/sample
263 cd $HOME/armnn-devenv/armnn/src/dynamic/sample
268 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the Arm NN build:
271 CXX=aarch64-linux-gnu-g++ CC=aarch64-linux-gnu-gcc cmake .. \
272 -DCMAKE_CXX_FLAGS=--std=c++14 \
273 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
282 * Copy the build folder to an arm64 linux machine
283 * Copy the libprotobuf.so.23.0.0 library file to the build folder
284 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build Arm NN
285 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
291 * Create a symbolic link to libprotobuf.so.24.0.0:
294 ln -s libprotobuf.so.23.0.0 ./libprotobuf.so.23
300 LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./UnitTests
301 Running 4493 test cases...
303 *** No errors detected
306 ## Troubleshooting and Errors:
307 ### Error adding symbols: File in wrong format
308 * When building Arm NN:
310 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
311 collect2: error: ld returned 1 exit status
312 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
313 make[2]: *** [libarmnn.so] Error 1
314 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
315 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
316 Makefile:127: recipe for target 'all' failed
317 make: *** [all] Error 2
319 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
322 ### Virtual memory exhausted
323 * When compiling the boost libraries:
325 virtual memory exhausted: Cannot allocate memory
327 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
330 ### Unrecognized command line option '-m64'
331 * When compiling the boost libraries:
333 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
335 * Clean the boost library directory before trying to build with a different architecture:
339 * It should show the following for arm64:
347 ### Missing libz.so.1
348 * When compiling armNN:
350 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.24.0.0, not found (try using -rpath or -rpath-link)
353 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
355 sudo dpkg --add-architecture arm64
356 sudo apt-get install zlib1g:arm64
360 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
361 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
362 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
364 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
368 ### Unable to install arm64 packages after adding arm64 architecture
369 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
370 * From stackoverflow:
371 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
372 * Open /etc/apt/sources.list with your preferred text editor.
374 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
376 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
378 * Then add the following:
380 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
381 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
382 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
383 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
384 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
385 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
386 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
388 * Update and install again:
390 sudo apt-get install zlib1g:arm64
396 ### Undefined references to google::protobuf:: functions
397 * When compiling Arm NN there are multiple errors of the following type:
399 libarmnnCaffeParser.so: undefined reference to `google::protobuf:*
401 * Missing or out of date protobuf compilation libraries.
402 Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required).
403 Follow the instructions above to install protobuf 3.12.0
404 Note this will require you to recompile Caffe for x86_64
407 ### Errors on strict-aliasing rules when compiling the Compute Library
408 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
410 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
412 * Add Werror=0 to the scons command:
414 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0