MLBEDSW-4783: Add config file directory structure

Add directory structure to support third party config files. Config
files should now be placed in an appropriately named directory under
the config_files directory, but can also be accessed by providing its
absolute path to vela --config.

Signed-off-by: Rickard Bolin <rickard.bolin@arm.com>
Change-Id: I2fcf52e7b2ddd2c4491dc370c85c0b3937d18062

5 files changed, 92 insertions, 13 deletions

diff --git a/OPTIONS.md b/OPTIONS.md
index e24f963..b4132d3 100644
--- a/OPTIONS.md
+++ b/OPTIONS.md
@@ -66,6 +66,19 @@ the required Network argument.
 vela --supported-ops-report
 ```
 
+### List Configuration Files
+
+Displays the configuration files in the `ethosu/config_files` directory. All
+configuration files must have the .ini extension and be placed in an
+appropriately named directory under `ethosu/config_files`. Note that the file
+depth from `ethosu/config_files` must be exactly 2 for the file to be
+discovered (e.g. `config_files/directory_name/my_config_file.ini`). Can be
+used without the required Network argument.
+
+```bash
+vela --list-config-files
+```
+
 ### Output Directory
 
 Specifies the output directory of the optimised network model as well as the
@@ -93,12 +106,15 @@ vela network.tflite --enable-debug-db
 Specifies the path to the Vela configuration file.  The format of the file is a
 Python ConfigParser `.ini` file.  This option can be specified multiple times to
 allow multiple files to be searched for the required system config and memory
-mode.  More details can be found in the Configuration File section below.  
+mode.  Custom configuration files can be used by adding a .ini file in an
+appropriate directory under the `ethosu/config_files` directory or by providing
+its absolute path. More details can be found in the Configuration File and List
+Configuration Files sections.
 **Type: POSIX path**  
 **Default: use default configuration**  
 
 ```bash
-vela network.tflite --config my_vela_cfg1.ini --config my_vela_cfg2.ini --system-config My_Sys_Cfg --memory-mode My_Mem_Mode
+vela network.tflite --config DirectoryName/my_vela_cfg1.ini --config absolute/path/to/my_vela_cfg2.ini --system-config My_Sys_Cfg --memory-mode My_Mem_Mode
 ```
 
 ### Timing
@@ -400,11 +416,25 @@ To see the configuration values being used by Vela use the `--verbose_config`
 CLI option.  This can also be used to display the internal-default values and to
 see a full list of all the available options.
 
-An example Vela configuration file, called `vela_cfg.ini`, is included in the
-directory containing this file.  Example usage based on this file is:  
+An example Vela configuration file, called `vela.ini`, is included in the
+`ethosu/config_files/Arm` directory. Example usage based on this file is:
+
+```bash
+vela network.tflite --accelerator-config ethos-u55-256 --config Arm/vela.ini --system-config Ethos_U55_High_End_Embedded --memory-mode Shared_Sram
+```
+
+Hardware vendors and/or users may wish to contribute their own configuration
+files for various SoC platforms by adding a .ini file in an appropriate
+directory under the ethosu/config_files directory.  This can be done by
+following the process outlined in CONTRIBUTIONS.md. These can then be accessed
+with `--config <DirectoryName>/config.ini` as in the example above.
+
+To use configuration files located outside the config_files directory, provide
+its absolute path to `--config`. The `--list-config-files` option can be used to
+view all available configuration files:
 
 ```bash
-vela network.tflite --accelerator-config ethos-u55-256 --config vela_cfg.ini --system-config Ethos_U55_High_End_Embedded --memory-mode Shared_Sram
+vela --list-config-files
 ```
 
 The following is an in-line explanation of the Vela configuration file format:
diff --git a/README.md b/README.md
index eb8871e..9ed3602 100644
--- a/README.md
+++ b/README.md
@@ -163,13 +163,20 @@ vela --optimise Performance --arena-cache-size 300000 my_model.tflite
 
 7) Compile a network using a particular embedded system configuration defined in
 Vela's configuration file.  The following command selects the `My_Sys_Config`
-system configuration along with the `My_Mem_Mode` memory mode from the `vela_cfg.ini` configuration file.
+system configuration along with the `My_Mem_Mode` memory mode from the `vela.ini`
+configuration file located in the config_files directory.
 
 ```bash
-vela --config vela_cfg.ini --system-config My_Sys_Config --memory-mode My_Mem_Mode my_model.tflite
+vela --config Arm/vela.ini --system-config My_Sys_Config --memory-mode My_Mem_Mode my_model.tflite
 ```
 
-8) To get a list of all available options (see CLI Options section below):
+8) To get a list of all available configuration files in the config_files directory:
+
+```bash
+vela --list-config-files
+```
+
+9) To get a list of all available options (see CLI Options section below):
 
 ```bash
 vela --help
diff --git a/vela.ini b/ethosu/config_files/Arm/vela.ini
index bcc3357..bcc3357 100644
--- a/vela.ini
+++ b/ethosu/config_files/Arm/vela.ini
diff --git a/ethosu/vela/vela.py b/ethosu/vela/vela.py
index 2108e20..b5f9f98 100644
--- a/ethosu/vela/vela.py
+++ b/ethosu/vela/vela.py
@@ -18,6 +18,7 @@
 #
 # Provides command line interface, options parsing, and network loading. Before calling the compiler driver.
 import argparse
+import glob
 import os
 import sys
 import time
@@ -49,6 +50,8 @@ from .tosa_model_semantic import TosaSemantic
 from .tosa_supported_operators import TosaSupportedOperators
 from ethosu.vela.architecture_features import ArchitectureFeatures
 
+CONFIG_FILES_PATH = os.path.abspath(f"{__file__}/../../config_files")
+
 
 def process(input_name, enable_debug_db, arch, model_reader_options, compiler_options, scheduler_options):
     if compiler_options.timing:
@@ -282,6 +285,12 @@ def generate_supported_ops():
         print(f"Report file: {filepath}")
 
 
+def list_config_files():
+    print("\nAvailable config files:\n")
+    for config in glob.glob(os.path.join(CONFIG_FILES_PATH, "*", "*.ini")):
+        print(config.lstrip(CONFIG_FILES_PATH + os.path.sep))
+
+
 def main(args=None):
     try:
         if args is None:
@@ -298,6 +307,15 @@ def main(args=None):
             help="Generate the SUPPORTED_OPS.md file in the current working directory and exit",
         )
 
+        parser.add_argument(
+            "--list-config-files",
+            action="store_true",
+            help=(
+                "Display all available configurations in the `config_files` folder and exit. To select config file, "
+                "use the --config argument with one of the listed config files (For example: --config Arm/vela.ini )"
+            ),
+        )
+
         # set network nargs to be optional to allow the support-ops-report CLI option to be used standalone
         parser.add_argument(
             "network",
@@ -428,14 +446,36 @@ def main(args=None):
             generate_supported_ops()
             return 0
 
+        if args.list_config_files:
+            list_config_files()
+            return 0
+
         if args.network is None:
             parser.error("the following argument is required: NETWORK")
 
+        def _parse_config(config):
+            if not config.endswith(".ini"):
+                raise InputFileError(config, "Configuration files must use the .ini extension")
+
+            if len(config.split(os.path.sep)) == 2 and not config.startswith(os.path.sep):
+                config_path = os.path.join(CONFIG_FILES_PATH, config)
+            else:
+                print(
+                    f"Warning: Configuration file `{config}` not located in a folder directly under the "
+                    "`config_files` directory. Note that the file depth from the `config_files` directory must be "
+                    "exactly 2 to be discovered via the --list-config-files mechanism "
+                    "(e.g. `config_files/directory_name/my_config_file.ini`). This config file will still be "
+                    "parsed however, and treated as an absolute path config instead."
+                )
+                config_path = config
+
+            if not os.access(config_path, os.R_OK):
+                raise InputFileError(config_path, "File not found or is not readable")
+
+            return config_path
+
         # check all config files exist because they will be read as a group
-        if args.config is not None:
-            for filename in args.config:
-                if not os.access(filename, os.R_OK):
-                    raise InputFileError(filename, "File not found or is not readable")
+        config_files = [_parse_config(cfg) for cfg in args.config] if args.config else None
 
         if args.cpu_tensor_alignment < 16 or args.cpu_tensor_alignment & (args.cpu_tensor_alignment - 1) != 0:
             parser.error(
@@ -457,7 +497,7 @@ def main(args=None):
         sys.setrecursionlimit(args.recursion_limit)
 
         arch = architecture_features.ArchitectureFeatures(
-            vela_config_files=args.config,
+            vela_config_files=config_files,
             system_config=args.system_config,
             memory_mode=args.memory_mode,
             accelerator_config=args.accelerator_config,
diff --git a/setup.py b/setup.py
index 244fc4e..d6e03b1 100644
--- a/setup.py
+++ b/setup.py
@@ -83,12 +83,14 @@ setup(
     ],
     keywords=["ethos-u", "vela compiler", "tflite", "npu"],
     packages=[
+        "ethosu",
         "ethosu.vela",
         "ethosu.vela.ethos_u55_regs",
         "ethosu.vela.tflite",
         "ethosu.vela.tosa",
         "ethosu.mlw_codec",
     ],
+    package_data={"ethosu": ["config_files/*/*.ini"]},
     python_requires="~=3.7",
     install_requires=[
         "flatbuffers==1.12.0",