1 # How to Cross-Compile Arm NN on x86_64 for arm64
3 - [Introduction](#introduction)
4 - [Cross-compiling ToolChain](#cross-compiling-toolchain)
5 - [Build and install Google's Protobuf library](#build-and-install-google-s-protobuf-library)
6 - [Build Caffe for x86_64](#build-caffe-for-x86-64)
7 - [Build Boost library for arm64](#build-boost-library-for-arm64)
8 - [Build Compute Library](#build-compute-library)
9 - [Build Tensorflow](#build-tensorflow)
10 - [Build Flatbuffer](#build-flatbuffer)
11 - [Build Onnx](#build-onnx)
12 - [Build TfLite](#build-tflite)
13 - [Build Arm NN](#build-armnn)
14 - [Build Standalone Sample Dynamic Backend](#build-standalone-sample-dynamic-backend)
15 - [Run Unit Tests](#run-unit-tests)
16 - [Troubleshooting and Errors:](#troubleshooting-and-errors-)
20 These are the step by step instructions on Cross-Compiling Arm NN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04.
21 The instructions show how to build the Arm NN core library and the Boost, Protobuf, Caffe, Tensorflow, Tflite, Flatbuffer and Compute Libraries for compilation.
23 #####Note: We are currently in the process of removing boost as a dependency to Arm NN. This process is finished for everything apart from our unit tests. This means you don't need boost to build and use Arm NN but you need it to execute our unit tests. Boost will soon be removed from Arm NN entirely. We also are deprecating support for Caffe and Tensorflow parsers in 21.02. This will be removed in 21.05.
25 ## Cross-compiling ToolChain
26 * Install the standard cross-compilation libraries for arm64:
28 sudo apt install crossbuild-essential-arm64
31 ## Build and install Google's Protobuf library
33 We support protobuf version 3.12.0
34 * Get protobuf from here: https://github.com/protocolbuffers/protobuf :
36 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf
38 git submodule update --init --recursive
41 * Build a native (x86_64) version of the protobuf libraries and compiler (protoc):
42 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++)
46 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install
50 * Build the arm64 version of the protobuf libraries:
54 export CC=aarch64-linux-gnu-gcc \
55 export CXX=aarch64-linux-gnu-g++ \
56 ../configure --host=aarch64-linux \
57 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \
58 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc
63 ## Build Caffe for x86_64
64 * Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html
65 * Install dependencies:
67 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev
68 sudo apt-get install --no-install-recommends libboost-all-dev
69 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
70 sudo apt-get install libopenblas-dev
71 sudo apt-get install libatlas-base-dev
73 * Download Caffe from: https://github.com/BVLC/caffe. We have tested using tag 1.0
75 git clone https://github.com/BVLC/caffe.git
77 git checkout eeebdab16155d34ff8f5f42137da7df4d1c7eab0
78 cp Makefile.config.example Makefile.config
80 * Adjust Makefile.config as necessary for your environment, for example:
85 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir):
86 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/include/
87 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/lib/
91 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH
92 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH
94 * Compilation with Make:
99 # These should all run without errors
102 * caffe.pb.h and caffe.pb.cc will be needed when building Arm NN's Caffe Parser
104 ## Build Boost library for arm64
105 * Build Boost library for arm64
106 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html
107 Using any version of Boost greater than 1.64 will fail to build Arm NN, due to different dependency issues.
109 tar -zxvf boost_1_64_0.tar.gz
111 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam
112 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install
113 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam
116 ## Build Compute Library
117 * Building the Arm Compute Library:
119 git clone https://github.com/ARM-software/ComputeLibrary.git
121 git checkout <branch_name>
123 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j4 internal_only=0
126 For example, if you want to checkout release branch of 20.02:
128 git checkout branches/arm_compute_20_02
133 * Building Tensorflow version 2.3.1:
135 git clone https://github.com/tensorflow/tensorflow.git
137 git checkout fcc4b966f1265f466e82617020af93670141b009
138 ../armnn/scripts/generate_tensorflow_protobuf.sh ../tensorflow-protobuf ../google/x86_64_pb_install
142 * Building Flatbuffer version 1.12.0
144 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz
145 tar xf flatbuffers-1.12.0.tar.gz
146 cd flatbuffers-1.12.0
150 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \
151 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers \
152 -DFLATBUFFERS_BUILD_TESTS=0
156 * Build arm64 version of flatbuffer
160 # Add -fPIC to allow us to use the libraries in shared objects.
161 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
162 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
163 -DFLATBUFFERS_BUILD_FLATC=1 \
164 -DCMAKE_INSTALL_PREFIX:PATH=$HOME/armnn-devenv/flatbuffers-arm64 \
165 -DFLATBUFFERS_BUILD_TESTS=0
172 git clone https://github.com/onnx/onnx.git
174 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD
175 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib:$LD_LIBRARY_PATH
176 $HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out $HOME/armnn-devenv/onnx
184 cp ../tensorflow/tensorflow/lite/schema/schema.fbs .
185 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs
189 * Compile Arm NN for arm64:
191 git clone https://github.com/ARM-software/armnn.git
193 git checkout <branch_name>
199 For example, if you want to checkout release branch of 20.02:
201 git checkout branches/armnn_20_02
205 * Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the Arm NN build:
208 export CXX=aarch64-linux-gnu-g++ \
209 export CC=aarch64-linux-gnu-gcc \
211 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \
212 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \
213 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
214 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \
215 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \
216 -DBUILD_CAFFE_PARSER=1 \
217 -DONNX_GENERATED_SOURCES=$HOME/armnn-devenv/onnx \
218 -DBUILD_ONNX_PARSER=1 \
219 -DTF_GENERATED_SOURCES=$HOME/armnn-devenv/tensorflow-protobuf \
220 -DBUILD_TF_PARSER=1 \
221 -DBUILD_TF_LITE_PARSER=1 \
222 -DTF_LITE_GENERATED_PATH=$HOME/armnn-devenv/tflite \
223 -DFLATBUFFERS_ROOT=$HOME/armnn-devenv/flatbuffers-arm64 \
224 -DFLATC_DIR=$HOME/armnn-devenv/flatbuffers-1.12.0/build \
225 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install \
226 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \
227 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.24.0.0 \
228 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.24.0.0
231 * If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command:
233 -DSAMPLE_DYNAMIC_BACKEND=1 \
234 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH
241 ## Build Standalone Sample Dynamic Backend
242 * The sample dynamic backend is located in armnn/src/dynamic/sample
248 * Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the Arm NN build:
251 export CXX=aarch64-linux-gnu-g++ \
252 export CC=aarch64-linux-gnu-gcc \
254 -DCMAKE_CXX_FLAGS=--std=c++14 \
255 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \
256 -DBoost_SYSTEM_LIBRARY=$HOME/armnn-devenv/boost_arm64_install/lib/libboost_system.a \
257 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so
266 * Copy the build folder to an arm64 linux machine
267 * Copy the libprotobuf.so.15.0.1 library file to the build folder
268 * If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build Arm NN
269 * cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location:
273 export LD_LIBRARY_PATH=<current_working_directory>
276 * Create a symbolic link to libprotobuf.so.24.0.0:
279 ln -s libprotobuf.so.24.0.0 ./libprotobuf.so.24
286 Running 567 test cases...
288 *** No errors detected
291 ## Troubleshooting and Errors:
292 ### Error adding symbols: File in wrong format
293 * When building Arm NN:
295 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format
296 collect2: error: ld returned 1 exit status
297 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed
298 make[2]: *** [libarmnn.so] Error 1
299 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed
300 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2
301 Makefile:127: recipe for target 'all' failed
302 make: *** [all] Error 2
304 * Boost libraries are not compiled for the correct architecture, try recompiling for arm64
307 ### Virtual memory exhausted
308 * When compiling the boost libraries:
310 virtual memory exhausted: Cannot allocate memory
312 * Not enough memory available to compile. Increase the amount of RAM or swap space available.
315 ### Unrecognized command line option '-m64'
316 * When compiling the boost libraries:
318 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’
320 * Clean the boost library directory before trying to build with a different architecture:
324 * It should show the following for arm64:
332 ### Missing libz.so.1
333 * When compiling armNN:
335 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.24.0.0, not found (try using -rpath or -rpath-link)
338 * Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them:
340 sudo dpkg --add-architecture arm64
341 sudo apt-get install zlib1g:arm64
345 * If apt-get update returns 404 errors for arm64 repos refer to section 5 below.
346 * Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here:
347 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4
349 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb
353 ### Unable to install arm64 packages after adding arm64 architecture
354 * Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually:
355 * From stackoverflow:
356 https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718
357 * Open /etc/apt/sources.list with your preferred text editor.
359 * Mark all the current (default) repos as \[arch=<current_os_arch>], e.g.
361 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
363 * Then add the following:
365 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted
366 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted
367 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe
368 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe
369 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse
370 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse
371 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
373 * Update and install again:
375 sudo apt-get install zlib1g:arm64
381 ### Undefined references to google::protobuf:: functions
382 * When compiling Arm NN there are multiple errors of the following type:
384 libarmnnCaffeParser.so: undefined reference to `google::protobuf:*
386 * Missing or out of date protobuf compilation libraries.
387 Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required).
388 Follow the instructions above to install protobuf 3.12.0
389 Note this will require you to recompile Caffe for x86_64
392 ### Errors on strict-aliasing rules when compiling the Compute Library
393 * When compiling the Compute Library there are multiple errors on strict-aliasing rules:
395 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror]
397 * Add Werror=0 to the scons command:
399 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0