IVGCVSW-5571 Expose the TfLite Delegate to the TfLite python API

 * Implemented external delegate adaptor interface for TfLite
 * Activated armnn logging for delegate
 * Added logging info to indicate if gpu tuning is turned on
 * Added pytests to ensure functionality of the external delegate adaptor
 * Included the delegate directory into doxygen
 * Added documentation on how to use the external delegate in python

Signed-off-by: Finn Williams <Finn.Williams@arm.com>
Signed-off-by: Jan Eilers <jan.eilers@arm.com>
Change-Id: Id3b4588fb0b9ac7e3f47ba2c19feead7beb58e18

11 files changed, 432 insertions, 6 deletions

diff --git a/delegate/CMakeLists.txt b/delegate/CMakeLists.txt
index 8383722628..495c1e3d77 100644
--- a/delegate/CMakeLists.txt
+++ b/delegate/CMakeLists.txt
@@ -15,6 +15,7 @@ list(APPEND armnnDelegate_sources
         include/armnn_delegate.hpp
         include/DelegateOptions.hpp
         src/armnn_delegate.cpp
+        src/armnn_external_delegate.cpp
         src/DelegateOptions.cpp
         src/Activation.hpp
         src/ArgMinMax.hpp
diff --git a/delegate/IntegrateDelegateIntoPython.md b/delegate/IntegrateDelegateIntoPython.md
new file mode 100644
index 0000000000..69a5ca00e2
--- /dev/null
+++ b/delegate/IntegrateDelegateIntoPython.md
@@ -0,0 +1,121 @@
+# Integrate the TfLite delegate into a python script
+If you have built the TfLite delegate as a separate dynamic library then this tutorial will show you how you can
+integrate it in TfLite to run models using python.
+
+Here is an example python script showing how to do this. In this script we are making use of the 
+[external adaptor](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate) 
+tool of TfLite that allows you to load delegates at runtime.
+```python
+import numpy as np
+import tflite_runtime.interpreter as tflite
+
+# Load TFLite model and allocate tensors.
+# (if you are using the complete tensorflow package you can find load_delegate in tf.experimental.load_delegate)
+armnn_delegate = tflite.load_delegate( library="<your-armnn-build-dir>/delegate/libarmnnDelegate.so",
+                                       options={"backends": "CpuAcc,GpuAcc,CpuRef", "logging-severity":"info"})
+# Delegates/Executes all operations supported by ArmNN to/with ArmNN
+interpreter = tflite.Interpreter(model_path="<your-armnn-repo-dir>/delegate/python/test/test_data/mock_model.tflite", 
+                                 experimental_delegates=[armnn_delegate])
+interpreter.allocate_tensors()
+
+# Get input and output tensors.
+input_details = interpreter.get_input_details()
+output_details = interpreter.get_output_details()
+
+# Test model on random input data.
+input_shape = input_details[0]['shape']
+input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
+interpreter.set_tensor(input_details[0]['index'], input_data)
+
+interpreter.invoke()
+
+# Print out result
+output_data = interpreter.get_tensor(output_details[0]['index'])
+print(output_data)
+```
+
+# Prepare the environment
+Pre-requisites:
+ * Dynamically build ArmNN Delegate library
+ * python3 (Depends on TfLite version)
+ * virtualenv
+ * numpy (Depends on TfLite version)
+ * tflite_runtime (>=2.0, depends on ArmNN Delegate)
+
+If you haven't built the delegate yet then take a look at the [build guide](BuildBuideNative.md).
+
+We recommend creating a virtual environment for this tutorial. For the following code to work python3 is needed. Please
+also check the documentation of the TfLite version you want to use. There might be additional prerequisites for the python
+version.
+```bash
+# Install python3 (We ended up with python3.5.3) and virtualenv
+sudo apt-get install python3-pip
+sudo pip3 install virtualenv
+
+# create a virtual environment
+cd your/tutorial/dir
+# creates a directory myenv at the current location
+virtualenv -p python3 myenv 
+# activate the environment
+source myenv/bin/activate
+```
+
+Now that the environment is active we can install additional packages we need for our example script. As you can see 
+in the python script at the start of this page, this tutorial uses the `tflite_runtime` rather than the whole tensorflow 
+package. The `tflite_runtime` is a package that wraps the TfLite Interpreter. Therefore it can only be used to run inferences of 
+TfLite models. But since ArmNN is only an inference engine itself this is a perfect match. The 
+`tflite_runtime` is also much smaller than the whole tensorflow package and better suited to run models on 
+mobile and embedded devices.
+
+At the time of writing, there are no packages of either `tensorflow` or `tflite_runtime` available on `pypi` that 
+are built for an arm architecture. That means installing them using `pip` on your development board is currently not 
+possible. The TfLite [website](https://www.tensorflow.org/lite/guide/python) points you at prebuilt `tflite_runtime` 
+packages. However, that limits you to specific TfLite and Python versions. For this reason we will build the 
+`tflite_runtime` from source.
+
+You will have downloaded the tensorflow repository in order to build the ArmNN delegate. In there you can find further 
+instructions on how to build the `tflite_runtime` under `tensorflow/lite/tools/pip_package/README.md`. This tutorial 
+uses bazel to build it natively but there are scripts for cross-compilation available as well.
+```bash
+# Add the directory where bazel is built to your PATH so that the script can find it
+PATH=$PATH:your/build/dir/bazel/output
+# Run the following script to build tflite_runtime natively.
+tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
+```
+The execution of the script creates a `.whl` file which can be used by `pip` to install the TfLite Runtime package. 
+The build-script produces some output in which you can find the location where the `.whl` file was created. Then all that is 
+left to do is to install all necessary python packages with `pip`.
+```bash
+pip install tensorflow/lite/tools/pip_package/gen/tflite_pip/python3/dist/tflite_runtime-2.3.1-py3-none-any.whl numpy
+```
+
+Your virtual environment is now all setup. Copy the final python script into a python file e.g. 
+`ExternalDelegatePythonTutorial.py`. Modify the python script above and replace `<your-armnn-build-dir>` and 
+`<your-armnn-repo-dir>` with the directories you have set up. If you've been using the [native build guide](BuildGuideNative.md) 
+this will be `$BASEDIR/armnn/build` and `$BASEDIR/armnn`.
+
+Finally, execute the script:
+```bash
+python ExternalDelegatePythonTutorial.py
+```
+The output should look similar to this:
+```bash
+Info: ArmNN v23.0.0
+
+Info: Initialization time: 0.56 ms
+
+INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
+[[ 12 123  16  12  11  14  20  16  20  12]]
+Info: Shutdown time: 0.28 ms
+```
+
+For more details on what kind of options you can pass to the armnn delegate please check 
+[armnn_delegate_adaptor.cpp](src/armnn_external_delegate.cpp).
+
+You can also test the functionality of the external delegate adaptor by running some unit tests:
+```bash
+pip install pytest
+cd armnn/delegate/python/test
+# You can deselect tests that require backends that your hardware doesn't support using markers e.g. `-m "not GpuAccTest`
+pytest --delegate-dir="<your-armnn-build-dir>/armnn/delegate/libarmnnDelegate.so" -m "not GpuAccTest" 
+```
diff --git a/delegate/include/DelegateOptions.hpp b/delegate/include/DelegateOptions.hpp
index daf20150be..6058061b3d 100644
--- a/delegate/include/DelegateOptions.hpp
+++ b/delegate/include/DelegateOptions.hpp
@@ -6,6 +6,8 @@
 #pragma once
 
 #include <armnn/ArmNN.hpp>
+#include <armnn/Logging.hpp>
+#include <armnn/Optional.hpp>
 
 #include <set>
 #include <string>
@@ -17,10 +19,13 @@ namespace armnnDelegate
 class DelegateOptions
 {
 public:
-    DelegateOptions(armnn::Compute computeDevice, const std::vector<armnn::BackendOptions>& backendOptions = {});
+    DelegateOptions(armnn::Compute computeDevice,
+                    const std::vector<armnn::BackendOptions>& backendOptions = {},
+                    armnn::Optional<armnn::LogSeverity> logSeverityLevel = armnn::EmptyOptional());
 
     DelegateOptions(const std::vector<armnn::BackendId>& backends,
-                    const std::vector<armnn::BackendOptions>& backendOptions = {});
+                    const std::vector<armnn::BackendOptions>& backendOptions = {},
+                    armnn::Optional<armnn::LogSeverity> logSeverityLevel = armnn::EmptyOptional());
 
     const std::vector<armnn::BackendId>& GetBackends() const { return m_Backends; }
 
@@ -28,6 +33,18 @@ public:
 
     const std::vector<armnn::BackendOptions>& GetBackendOptions() const { return m_BackendOptions; }
 
+    /// Appends a backend option to the list of backend options
+    void AddBackendOption(const armnn::BackendOptions& option) { m_BackendOptions.push_back(option); }
+
+    /// Sets the severity level for logging within ArmNN that will be used on creation of the delegate
+    void SetLoggingSeverity(const armnn::LogSeverity& level) { m_LoggingSeverity = level; }
+    void SetLoggingSeverity(const std::string& level) { m_LoggingSeverity = armnn::StringToLogLevel(level); }
+
+    /// Returns the severity level for logging within ArmNN
+    armnn::LogSeverity GetLoggingSeverity() { return m_LoggingSeverity.value(); }
+
+    bool IsLoggingEnabled() { return m_LoggingSeverity.has_value(); }
+
 private:
     /// Which backend to run Delegate on.
     /// Examples of possible values are: CpuRef, CpuAcc, GpuAcc.
@@ -52,6 +69,9 @@ private:
     ///   "TuningFile" : string [filenameString]
     ///   "KernelProfilingEnabled" : bool [true | false]
     std::vector<armnn::BackendOptions> m_BackendOptions;
+
+    /// Severity level for logging within ArmNN that will be used on creation of the delegate
+    armnn::Optional<armnn::LogSeverity> m_LoggingSeverity;
 };
 
 } // namespace armnnDelegate
diff --git a/delegate/python/test/conftest.py b/delegate/python/test/conftest.py
new file mode 100644
index 0000000000..780a0492e1
--- /dev/null
+++ b/delegate/python/test/conftest.py
@@ -0,0 +1,30 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+import pytest
+import os
+
+
+@pytest.fixture(scope="module")
+def test_data_folder(request):
+    """
+    This fixture returns path to the folder with the shared test resources
+    """
+    return str(os.path.join(request.fspath.dirname, "test_data"))
+
+
+def pytest_addoption(parser):
+    """
+    Adds the program option 'delegate-dir' to pytest
+    """
+    parser.addoption("--delegate-dir",
+                     action="append",
+                     help="Directory of the armnn tflite delegate library",
+                     required=True)
+
+
+def pytest_generate_tests(metafunc):
+    """
+    Makes the program option 'delegate-dir' available to all tests as a function fixture
+    """
+    if "delegate_dir" in metafunc.fixturenames:
+        metafunc.parametrize("delegate_dir", metafunc.config.getoption("delegate_dir"))
\ No newline at end of file
diff --git a/delegate/python/test/pytest.ini b/delegate/python/test/pytest.ini
new file mode 100644
index 0000000000..719af3ce05
--- /dev/null
+++ b/delegate/python/test/pytest.ini
@@ -0,0 +1,9 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+[pytest]
+addopts = --strict-markers
+markers =
+    CpuRefTest: marks tests that require the CpuRef backend
+    CpuAccTest: marks tests that require the CpuAcc backend
+    GpuAccTest: marks tests that require the GpuAcc backend
\ No newline at end of file
diff --git a/delegate/python/test/test_data/mock_model.tflite b/delegate/python/test/test_data/mock_model.tflite
new file mode 100644
index 0000000000..0b8944d3ed
--- /dev/null
+++ b/delegate/python/test/test_data/mock_model.tflite
diff --git a/delegate/python/test/test_external_delegate.py b/delegate/python/test/test_external_delegate.py
new file mode 100644
index 0000000000..995de8cd70
--- /dev/null
+++ b/delegate/python/test/test_external_delegate.py
@@ -0,0 +1,65 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+import pytest
+import tflite_runtime.interpreter as tflite
+import os
+from utils import run_mock_model
+
+
+def test_external_delegate_unknown_options(delegate_dir):
+    print(delegate_dir)
+    with pytest.raises(ValueError):
+        tflite.load_delegate(
+            delegate_dir,
+            options={"wrong": "wrong"})
+
+
+def test_external_delegate_options_multiple_backends(delegate_dir):
+    tflite.load_delegate(
+        delegate_dir,
+        options={"backends": "GpuAcc,CpuAcc,CpuRef,Unknown"})
+
+
+@pytest.mark.GpuAccTest
+def test_external_delegate_options_gpu_tuning(delegate_dir, test_data_folder, tmp_path):
+
+    tuning_file = os.path.join(str(tmp_path), "test_gpu.tuning")
+    # cleanup previous test run if necessary
+    if os.path.exists(tuning_file):
+        os.remove(tuning_file)
+
+    # with tuning level 2 a tuning file should be created
+    armnn_delegate = tflite.load_delegate(
+        delegate_dir,
+        options={
+            "backends": "GpuAcc",
+            "gpu-tuning-level": "2",
+            "gpu-tuning-file": tuning_file,
+            "logging-severity": "info"})
+
+    run_mock_model(armnn_delegate, test_data_folder)
+
+    # destroy delegate, otherwise tuning file won't be written to file
+    armnn_delegate.__del__()
+    assert (os.path.exists(tuning_file))
+
+    # if no tuning level is provided it defaults to 0 which means it will use the tuning parameters from a tuning
+    # file if one is provided
+    armnn_delegate2 = tflite.load_delegate(
+        delegate_dir,
+        options={
+            "backends": "GpuAcc",
+            "gpu-tuning-file": tuning_file,
+            "logging-severity": "info"})
+
+    run_mock_model(armnn_delegate2, test_data_folder)
+
+    # cleanup
+    os.remove(tuning_file)
+
+def test_external_delegate_options_wrong_logging_level(delegate_dir):
+    with pytest.raises(ValueError):
+        tflite.load_delegate(
+            delegate_dir,
+            options={"logging-severity": "wrong"})
diff --git a/delegate/python/test/utils.py b/delegate/python/test/utils.py
new file mode 100644
index 0000000000..3adc24fe35
--- /dev/null
+++ b/delegate/python/test/utils.py
@@ -0,0 +1,24 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+import tflite_runtime.interpreter as tflite
+import numpy as np
+import os
+
+
+def run_mock_model(delegate, test_data_folder):
+    model_path = os.path.join(test_data_folder, 'mock_model.tflite')
+    interpreter = tflite.Interpreter(model_path=model_path,
+                                     experimental_delegates=[delegate])
+    interpreter.allocate_tensors()
+
+    # Get input and output tensors.
+    input_details = interpreter.get_input_details()
+    output_details = interpreter.get_output_details()
+
+    # Test model on random input data.
+    input_shape = input_details[0]['shape']
+    input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
+    interpreter.set_tensor(input_details[0]['index'], input_data)
+
+    interpreter.invoke()
\ No newline at end of file
diff --git a/delegate/src/DelegateOptions.cpp b/delegate/src/DelegateOptions.cpp
index af78685fa6..3ec2d20d77 100644
--- a/delegate/src/DelegateOptions.cpp
+++ b/delegate/src/DelegateOptions.cpp
@@ -9,14 +9,16 @@ namespace armnnDelegate
 {
 
 DelegateOptions::DelegateOptions(armnn::Compute computeDevice,
-                                 const std::vector<armnn::BackendOptions>& backendOptions)
-    : m_Backends({computeDevice}), m_BackendOptions(backendOptions)
+                                 const std::vector<armnn::BackendOptions>& backendOptions,
+                                 const armnn::Optional<armnn::LogSeverity> logSeverityLevel)
+    : m_Backends({computeDevice}), m_BackendOptions(backendOptions), m_LoggingSeverity(logSeverityLevel)
 {
 }
 
 DelegateOptions::DelegateOptions(const std::vector<armnn::BackendId>& backends,
-                                 const std::vector<armnn::BackendOptions>& backendOptions)
-    : m_Backends(backends), m_BackendOptions(backendOptions)
+                                 const std::vector<armnn::BackendOptions>& backendOptions,
+                                 const armnn::Optional<armnn::LogSeverity> logSeverityLevel)
+    : m_Backends(backends), m_BackendOptions(backendOptions), m_LoggingSeverity(logSeverityLevel)
 {
 }
 
diff --git a/delegate/src/armnn_delegate.cpp b/delegate/src/armnn_delegate.cpp
index 5139adbf75..6250a5f638 100644
--- a/delegate/src/armnn_delegate.cpp
+++ b/delegate/src/armnn_delegate.cpp
@@ -120,6 +120,12 @@ Delegate::Delegate(armnnDelegate::DelegateOptions options)
   : m_Runtime(nullptr, nullptr),
     m_Options(std::move(options))
 {
+    // Configures logging for ARMNN
+    if (options.IsLoggingEnabled())
+    {
+        armnn::ConfigureLogging(true, true, options.GetLoggingSeverity());
+    }
+
     // Create ArmNN Runtime
     armnn::IRuntime::CreationOptions runtimeOptions;
 
diff --git a/delegate/src/armnn_external_delegate.cpp b/delegate/src/armnn_external_delegate.cpp
new file mode 100644
index 0000000000..53b17256af
--- /dev/null
+++ b/delegate/src/armnn_external_delegate.cpp
@@ -0,0 +1,148 @@
+//
+// Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+// SPDX-License-Identifier: MIT
+//
+#include "armnn_delegate.hpp"
+#include <armnn/Logging.hpp>
+
+#include <iostream>
+#include <tensorflow/lite/minimal_logging.h>
+
+namespace tflite
+{
+
+/**
+ * This file defines two symbols that need to be exported to use the TFLite external delegate provider. This is a plugin
+ * that can be used for fast integration of delegates into benchmark tests and other tools. It allows loading of
+ * a dynamic delegate library at runtime.
+ *
+ * The external delegate also has Tensorflow Lite Python bindings. Therefore the dynamic external delegate
+ * can be directly used with Tensorflow Lite Python APIs.
+ *
+ * See tensorflow/lite/delegates/external for details or visit the tensorflow guide
+ * [here](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate)
+ */
+
+extern "C"
+{
+std::vector<std::string> gpu_options {"gpu-tuning-level",
+                                      "gpu-tuning-file",
+                                      "gpu-kernel-profiling-enabled"};
+
+
+/**
+ * Create an ArmNN delegate plugin
+ *
+ * Available options:
+ *
+ *    Option key: "backends" \n
+ *    Possible values: ["EthosNPU"/"GpuAcc"/"CpuAcc"/"CpuRef"] \n
+ *    Descriptions: A comma separated list without whitespaces of
+ *                  backends which should be used for execution. Falls
+ *                  back to next backend in list if previous doesn't
+ *                  provide support for operation. e.g. "GpuAcc,CpuAcc"
+ *
+ *    Option key: "logging-severity" \n
+ *    Possible values: ["trace"/"debug"/"info"/"warning"/"error"/"fatal"] \n
+ *    Description: Sets the logging severity level for ArmNN. Logging
+ *                 is turned off if this option is not provided.
+ *
+ *    Option key: "gpu-tuning-level" \n
+ *    Possible values: ["0"/"1"/"2"/"3"] \n
+ *    Description: 0=UseOnly(default), 1=RapidTuning, 2=NormalTuning,
+ *                 3=ExhaustiveTuning. Requires option gpu-tuning-file.
+ *                 1,2 and 3 will create a tuning-file, 0 will apply the
+ *                 tunings from an existing file
+ *
+ *    Option key: "gpu-tuning-file" \n
+ *    Possible values: [filenameString] \n
+ *    Description: File name for the tuning file.
+ *
+ *    Option key: "gpu-kernel-profiling-enabled" \n
+ *    Possible values: ["true"/"false"] \n
+ *    Description: Enables GPU kernel profiling
+ *
+ *
+ * @param[in]     option_keys     Delegate option names
+ * @param[in]     options_values  Delegate option values
+ * @param[in]     num_options     Number of delegate options
+ * @param[in,out] report_error    Error callback function
+ *
+ * @return An ArmNN delegate if it succeeds else NULL
+ */
+TfLiteDelegate* tflite_plugin_create_delegate(char** options_keys,
+                                              char** options_values,
+                                              size_t num_options,
+                                              void (*report_error)(const char*))
+{
+    // Returning null indicates an error during delegate creation so we initialize with that
+    TfLiteDelegate* delegate = nullptr;
+    try
+    {
+        // (Initializes with CpuRef backend)
+        armnnDelegate::DelegateOptions options = armnnDelegate::TfLiteArmnnDelegateOptionsDefault();
+        for (size_t i = 0; i < num_options; ++i)
+        {
+            // Process backends
+            if (std::string(options_keys[i]) == std::string("backends"))
+            {
+                // The backend option is a comma separated string of backendIDs that needs to be split
+                std::vector<armnn::BackendId> backends;
+                char* pch;
+                pch = strtok(options_values[i],",");
+                while (pch != NULL)
+                {
+                    backends.push_back(pch);
+                    pch = strtok (NULL, ",");
+                }
+                options.SetBackends(backends);
+            }
+            // Process logging level
+            else if (std::string(options_keys[i]) == std::string("logging-severity"))
+            {
+                options.SetLoggingSeverity(options_values[i]);
+            }
+            // Process GPU backend options
+            else if (std::string(options_keys[i]) == std::string("gpu-tuning-level"))
+            {
+                armnn::BackendOptions option("GpuAcc", {{"TuningLevel", atoi(options_values[i])}});
+                options.AddBackendOption(option);
+            }
+            else if (std::string(options_keys[i]) == std::string("gpu-tuning-file"))
+            {
+                armnn::BackendOptions option("GpuAcc", {{"TuningFile", std::string(options_values[i])}});
+                options.AddBackendOption(option);
+            }
+            else if (std::string(options_keys[i]) == std::string("gpu-kernel-profiling-enabled"))
+            {
+                armnn::BackendOptions option("GpuAcc", {{"KernelProfilingEnabled", (*options_values[i] != '0')}});
+                options.AddBackendOption(option);
+            }
+            else
+            {
+                throw armnn::Exception("Unknown option for the ArmNN Delegate given: " + std::string(options_keys[i]));
+            }
+        }
+        delegate = TfLiteArmnnDelegateCreate(options);
+    }
+    catch (const std::exception& ex)
+    {
+        if(report_error)
+        {
+            report_error(ex.what());
+        }
+    }
+    return delegate;
+}
+
+/** Destroy a given delegate plugin
+ *
+ * @param[in] delegate Delegate to destruct
+ */
+void tflite_plugin_destroy_delegate(TfLiteDelegate* delegate)
+{
+    armnnDelegate::TfLiteArmnnDelegateDelete(delegate);
+}
+
+}  // extern "C"
+}  // namespace tflite
\ No newline at end of file