MLECO-1944: Minor documentation clean-up done and script to download TPIP added21.05-rc1

Signed-off-by: alexander <alexander.efremov@arm.com>
Change-Id: Id4a9b220ce753a5ecf3483e86d837c1814dc7fb9

4 files changed, 162 insertions, 21 deletions

diff --git a/docs/documentation.md b/docs/documentation.md
index 7f8fbf9..d08e313 100644
--- a/docs/documentation.md
+++ b/docs/documentation.md
@@ -68,6 +68,9 @@ The repository has the following structure:
 .
 ├── dependencies
 ├── docs
+├── model_conditioning_examples
+├── resources
+├── /resources_downloaded/
 ├── scripts
 │   └── ...
 ├── source
@@ -77,9 +80,9 @@ The repository has the following structure:
 │   │ └── tensorflow-lite-micro
 │   └── use_case
 │     └── <usecase_name>
-│          ├── include
-│          ├── src
-│          └── usecase.cmake
+│                ├── include
+│                ├── src
+│                └── usecase.cmake
 ├── tests
 │   └── ...
 └── CMakeLists.txt
@@ -91,6 +94,14 @@ Where:
 
 - `docs`: contains the documentation for this ML applications.
 
+- `model_conditioning_examples`: contains short example scripts that demonstrate some methods available in TensorFlow 
+    to condition your model in preparation for deployment on Arm Ethos NPU.
+
+- `resources`: contains ML use cases applications resources such as input data, label files, etc.
+
+- `resources_downloaded`: created by `set_up_default_resources.py`, contains downloaded resources for ML use cases 
+    applications such as models, test data, etc.
+
 - `scripts`: contains build related and source generation scripts.
 
 - `source`: contains C/C++ sources for the platform and ML applications.
diff --git a/docs/sections/building.md b/docs/sections/building.md
index 98cb5e8..9e0d422 100644
--- a/docs/sections/building.md
+++ b/docs/sections/building.md
@@ -7,6 +7,7 @@
     - [Preparing build environment](#preparing-build-environment)
       - [Fetching submodules](#fetching-submodules)
       - [Fetching resource files](#fetching-resource-files)
+    - [Building for default configuration](#building-for-default-configuration)
     - [Create a build directory](#create-a-build-directory)
     - [Configuring the build for MPS3 SSE-300](#configuring-the-build-for-mps3-sse-300)
       - [Using GNU Arm Embedded Toolchain](#using-gnu-arm-embedded-toolchain)
@@ -253,6 +254,10 @@ repository to link against.
 These are part of the [ethos-u repository](https://git.mlplatform.org/ml/ethos-u/ethos-u.git/about/) and set as
 submodules of this project.
 
+> **NOTE**: If you are using non git project sources, run `python3 ./download_dependencies.py` and ignore further git
+> instructions. Proceed to [Fetching resource files](#fetching-resource-files) section.
+>
+
 To pull the submodules:
 
 ```sh
@@ -263,11 +268,10 @@ This will download all the required components and place them in a tree like:
 
 ```tree
 dependencies
- └── ethos-u
-     ├── cmsis
-     ├── core_driver
-     ├── tensorflow
-     └── ...
+    ├── cmsis
+    ├── core-driver
+    ├── core-software
+    └── tensorflow
 ```
 
 > **NOTE**: The default source paths for the TPIP sources assume the above directory structure, but all of the relevant
@@ -288,9 +292,33 @@ This will fetch all the models into `resources_downloaded` directory. It will
 also optimize the models using the Vela compiler for default 128 MAC configuration
 of Arm® Ethos™-U55 NPU.
 
+### Building for default configuration
+
+A helper script `build_default.py` is provided to configure and build all the
+applications. It configures the project with default settings i.e., for `mps3`
+target and `sse-300` subsystem. Under the hood, it invokes all the necessary
+CMake commands that are described in the next sections.
+
+If using the `Arm GNU embedded toolchain`, execute:
+
+```commandline
+./build_default.py
+```
+
+If using the `Arm Compiler`, execute:
+
+```commandline
+./build_default.py --toolchain arm
+```
+
+Additional command line arguments supported by this script are:
+ - `--skip-download`: Do not download resources: models and test vectors
+ - `--skip-vela`: Do not run Vela optimizer on downloaded models.
+
 ### Create a build directory
 
-Create a build directory in the root of the project and navigate inside:
+To configure and build the project manually, create a build directory in the
+root of the project and navigate inside:
 
 ```commandline
 mkdir build && cd build
diff --git a/download_dependencies.py b/download_dependencies.py
new file mode 100755
index 0000000..b22c63f
--- /dev/null
+++ b/download_dependencies.py
@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+
+#  Copyright (c) 2021 Arm Limited. All rights reserved.
+#  SPDX-License-Identifier: Apache-2.0
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# The script does effectively the same as "git submodule update --init" command.
+
+import logging
+import os
+import sys
+import tarfile
+import tempfile
+from urllib.request import urlopen
+from zipfile import ZipFile
+
+logging.basicConfig(filename='download_dependencies.log', level=logging.DEBUG)
+logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))
+
+tf = "https://github.com/tensorflow/tensorflow/archive/6cff09aee1f832d495b3cae40cab0de58155a0af.zip"
+cmsis = "https://github.com/ARM-software/CMSIS_5/archive/0d7e4fa7131241a17e23dfae18140e0b2e77728f.zip"
+ethos_u_core_sw = "https://git.mlplatform.org/ml/ethos-u/ethos-u-core-software.git/snapshot/ethos-u-core-software-7f3c1c92732b611a53968b14e70a2b116e43b980.tar.gz"
+ethos_u_core_driver = "https://git.mlplatform.org/ml/ethos-u/ethos-u-core-driver.git/snapshot/ethos-u-core-driver-effc7aa8b9272fb20cdd1a7d1097818af70acc93.tar.gz"
+
+
+def download(url_file: str, post_process=None):
+    with urlopen(url_file) as response, tempfile.NamedTemporaryFile() as temp:
+        logging.info(f"Downloading {url_file} ...")
+        temp.write(response.read())
+        temp.seek(0)
+        logging.info(f"Finished downloading {url_file}.")
+        if post_process:
+            post_process(temp)
+
+
+def unzip(file, to_path):
+    with ZipFile(file) as z:
+        for archive_path in z.infolist():
+            archive_path.filename = archive_path.filename[archive_path.filename.find("/") + 1:]
+            if archive_path.filename:
+                z.extract(archive_path, to_path)
+                target_path = os.path.join(to_path, archive_path.filename)
+                attr = archive_path.external_attr >> 16
+                if attr != 0:
+                    os.chmod(target_path, attr)
+
+
+def untar(file, to_path):
+    with tarfile.open(file) as z:
+        for archive_path in z.getmembers():
+            index = archive_path.name.find("/")
+            if index < 0:
+                continue
+            archive_path.name = archive_path.name[index + 1:]
+            if archive_path.name:
+                z.extract(archive_path, to_path)
+
+
+def main(dependencies_path: str):
+
+    download(cmsis,
+             lambda file: unzip(file.name,
+                                to_path=os.path.join(dependencies_path, "cmsis")))
+    download(ethos_u_core_sw,
+             lambda file: untar(file.name,
+                                to_path=os.path.join(dependencies_path, "core-software")))
+    download(ethos_u_core_driver,
+             lambda file: untar(file.name,
+                                to_path=os.path.join(dependencies_path, "core-driver")))
+    download(tf,
+             lambda file: unzip(file.name,
+                                to_path=os.path.join(dependencies_path, "tensorflow")))
+
+
+if __name__ == '__main__':
+    download_dir = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "dependencies"))
+
+    if os.path.isdir(download_dir):
+        logging.info(f'{download_dir} exists. Skipping download.')
+    else:
+        main(download_dir)
diff --git a/set_up_default_resources.py b/set_up_default_resources.py
index 7639364..362552a 100755
--- a/set_up_default_resources.py
+++ b/set_up_default_resources.py
@@ -176,14 +176,18 @@ def set_up_resources(run_vela_on_models=False):
                 res_dst = os.path.join(download_dir,
                                        uc["use_case_name"],
                                        res_name)
-            try:
-                g = urllib.request.urlopen(res_url)
-                with open(res_dst, 'b+w') as f:
-                    f.write(g.read())
-                    logging.info(f"- Downloaded {res_url} to {res_dst}.")
-            except URLError:
-                logging.error(f"URLError while downloading {res_url}.")
-                raise
+
+            if os.path.isfile(res_dst):
+                logging.info(f"File {res_dst} exists, skipping download.")
+            else:
+                try:
+                    g = urllib.request.urlopen(res_url)
+                    with open(res_dst, 'b+w') as f:
+                        f.write(g.read())
+                        logging.info(f"- Downloaded {res_url} to {res_dst}.")
+                except URLError:
+                    logging.error(f"URLError while downloading {res_url}.")
+                    raise
 
     # 3. Run vela on models in resources_downloaded
     # New models will have same name with '_vela' appended.
@@ -201,6 +205,15 @@ def set_up_resources(run_vela_on_models=False):
 
         for model in models:
             output_dir = os.path.dirname(model)
+            # model name after compiling with vela is an initial model name + _vela suffix
+            vela_optimised_model_path = str(model).replace(".tflite", "_vela.tflite")
+            # we want it to be initial model name + _vela_H128 suffix which indicates selected MAC config.
+            new_vela_optimised_model_path = vela_optimised_model_path.replace("_vela.tflite", "_vela_H128.tflite")
+
+            if os.path.isfile(new_vela_optimised_model_path):
+                logging.info(f"File {new_vela_optimised_model_path} exists, skipping optimisation.")
+                continue
+
             command = (f". {env_activate} && vela {model} " +
                        "--accelerator-config=ethos-u55-128 " +
                        "--block-config-limit=0 " +
@@ -209,10 +222,7 @@ def set_up_resources(run_vela_on_models=False):
                        "--system-config=Ethos_U55_High_End_Embedded " +
                        f"--output-dir={output_dir}")
             call_command(command)
-            # model name after compiling with vela is an initial model name + _vela suffix
-            vela_optimised_model_path = str(model).replace(".tflite", "_vela.tflite")
-            # we want it to be initial model name + _vela_H128 suffix which indicates selected MAC config.
-            new_vela_optimised_model_path = vela_optimised_model_path.replace("_vela.tflite", "_vela_H128.tflite")
+
             # rename default vela model
             os.rename(vela_optimised_model_path, new_vela_optimised_model_path)
             logging.info(f"Renaming {vela_optimised_model_path} to {new_vela_optimised_model_path}.")