# TOSA Reference Model

## Introduction

The *Tensor Operator Set Architecture (TOSA) Specification
<https://git.mlplatform.org/tosa/specification.git/>* is a set of operators
with defined accuracy and compatibility constraints that Arm expects
to be implemented on its Neural Processing Units (NPUs).  Most
operators from the common ML frameworks (TensorFlow, PyTorch, etc)
should be expressible in TOSA.  TOSA is focused on inference, leaving
training to the original frameworks.

The *TOSA Reference Model* package provides a reference implementation
and testing infrastructure for TOSA.  The reference model consumes a
FlatBuffers serialization of the network subgraph generated by the
TOSA Serialization Library, along with input tensors for placeholder
nodes in NumPy format.  By default, the model validates and evaluates
the network subgraph, and writes out the resulting output tensors in
NumPy format.

This package also contains a conformance test suite that can be
created and then used to validate an implementation of TOSA.

## Installation Requirements

The *TOSA Reference Model* and testing suite requires the following
tools:

* CMake version 3.4 or later
* GNU Make 4.1 or later
* GCC (tested with 9.4.0) or Clang C++ compiler (tested with clang-10)
  with C++17 support

The model includes the following git submodules:

* TOSA Serialization Library
* JSON for Modern C++ - 3.8.0
* Eigen 3.3.7
* doctest 2.4.9 (When building unit tests)

The model is written using
C++17 and has been primarily tested on Ubuntu x86_64 20.04 LTS Linux
systems.

The testing infrastructure requires:

* Python 3.6 or later
* FlatBuffers 2.0 or later
* NumPy 1.15 or later

Check out the required git submodules with:

``` bash
git submodule update --init --recursive
```

### Versioning

The *TOSA Reference Model* repository has branches (major.minor) and tags
(major.minor.patch) that correspond to each TOSA version. The `main` branch is
used as the active development branch for the next version.

Perform a check-out of a specific version before compilation or installation of
the test infrastructure by using:

```bash
git checkout --recurse-submodules VERSION
```

Where `VERSION` can be for example: `v0.23` or `v0.23.0`


## Compilation

The *TOSA Reference Model* build can be prepared by creating makefiles using CMake:

``` bash
mkdir -p build
cd build
cmake ..
```

Optionally, `-DCMAKE_BUILD_TYPE=Debug` can be used on the `cmake`
command to create a debug build.  Next compile using `make`:

``` bash
make
```

The resulting executable will be named:
`reference_model/tosa_reference_model`. This executable can be disabled with
`-DBUILD_TOSA_REFERENCE_MODEL_EXECUTABLE=NO`.
A static library will also be generated by default as follows:
`reference_model/libtosa_reference_model_lib.a`.
To make this a shared library (.so) add the following option
`-DBUILD_SHARED_LIBS=YES`.

CMake only needs to be re-run if the build environment changes (e.g., new dependencies or source
files). Code changes that do not affect these build rules can be
rebuilt simply using `make`.

## Executable Usage

The inputs to the *TOSA Reference Model* consist of a FlatBuffers file
containing the serialized subgraph, a JSON test descriptor that describes
a sequence of name, shape and NumPy file for each input and output tensor.

The JSON test descriptor must have the following field:
* tosa_file:
    type: string.
    TOSA flatbuffer file location.
* ifm_name:
    type: list(string).
    Input placeholder tensor names.
* ifm_file:
    type: list(string).
    Input NumPy array file location.
* ofm_name:
    type: list(string).
    Output placeholder tensor names.
* ofm_file:
    type: list(string).
    Output NumPy array file location.
* expected_failure:
    type: boolean.
    Is this test expected to fail in runtime.
* expected_return_code:
    type: int
    The expected return code of the reference model i
    (0 = VALID, 1 = ERROR, 2 = UNPREDICTABLE)

Note by default, all the files specified by "tosa_file", "ifm_file",
"ofm_file" are relative to desc.json. This could be overwritten by
--flatbuffer_dir=, if desired.

An example command is shown below:

``` bash
./build/reference_model/tosa_reference_model \
  --test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
```

Instead of drive model by JSON test descriptor, user can also drive model
with --tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file=
options directly.

In case where --test_desc= and other options are specified at the same time,
JSON test descriptor will be initialized first. All other options
(--tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file=) will
overwrite whatever specified by JSON descriptor.

On a successful execution, the output tensors will be written in NumPy
format into output tensors specified by "ofm_file".

For example, you can generate new output .npy by:
``` bash
./build/reference_model/tosa_reference_model \
  --test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json \
  --ofm_file=out.npy
```

In this case, the "ofm_file" field in desc.json will be ignored, and the
one specified by --ofm_file= will be picked.

When using JSON-formatted FlatBuffers input (.json extension), the
FlatBuffers schema file from the TOSA Serialization library must be
specified using --operator_fbs=.  When using the binary FlatBuffers
format (.tosa), the schema is not necessary.

### Examples

The TOSA Reference Model distribution contains several example
networks with inputs and reference outputs generated by
TensorFlow or TensorFlow Lite in the examples directory.

These examples can be run through the TOSA Reference model and should
produce the equivalent TOSA-compliant reference output tensors.
Please note that differences in floating-point ordering and rounding
may cause small differences in output for floating-point tests and
differences in quantized scaling between TensorFlow Lite and the TOSA
Specification may cause differences in quantized integer tests.

## ModelRunner API

As an alternative to the executable described above,
the model_runner.h is provided which can be used to invoke the
TOSA Reference Model easily within C++.
A sample of this class and how it can used can be found in
[model_runner_simple_sample.cpp](reference_model/samples/model_runner_simple_sample.cpp).
This sample can be compiled by adding `-DBUILD_MODEL_RUNNER_SAMPLE=YES` to the CMake command
and executed by running `./build/reference_model/model_runner_sample`.

### ModelRunner API Unit Tests
Unit test are generated by default for the ModelRunner.
This executable can be disabled by adding`-DBUILD_TOSA_REFERENCE_MODEL_TESTS=NO` to the CMake command.
This executable can be run using
`./build/reference_model/unit_tests` and requires the submodule doctest.

## Debugging

The debugging facility can be enabled by setting a debug scope and
debug level on the command line.  For most purposes, the following
flags will work: `-d ALL -l HIGH`.  Debug output can be directed to a
file using the `-o` switch.

## TOSA Unit Test Infrastructure

The TOSA Unit Test infrastructure can create and run self-contained tests
for implementations of the *Tensor Operator Set Architecture (TOSA)
Specification*.  These test tools generate TOSA operators for
verification of the TOSA reference model against existing frameworks
or other operator implementations.

A conformance suite generator is included in the TOSA Unit Test
infrastructure to provide coverage validation of an
implementation of TOSA. Please see the *TOSA Conformance Generator*
section below to find out how to create these.

### Test content

The test building tools by default generate positive tests with random
arguments and reference inputs for each TOSA operator.  Positive tests
are expected to run without error and will produce a result.
The test builder can also generate negative tests for all the ERROR_IF
conditions within the TOSA Specification by using the `--test-type`
options. Negative tests may contain invalid arguments or inputs and
are expected to be loaded and fail without producing a result.
Other errors or unpredictable results are expected to be handled in a
system dependent way and are not tested by the test builder tool.

The unit tests are typically structured as a combination of input
placeholder nodes, const nodes, and attributes feeding into a single
TOSA operator.  The unit tests use a Python copy of the FlatBuffers
schema written by `flatc` to verify TOSA.

Each test has a JSON file which provides machine-readable metadata for
the test, including the TOSA flatbuffer file, names, shapes, and
NumPy filenames for each input and output tensor.  There is also a
boolean value for whether a failure is expected because the test is
expected to trigger an invalid set of operands or attributes.
Extra metadata may be included such as tags for filtering, data
generation or compliance validation.

The test runner tool can execute the unit tests on the TOSA Reference
Model to generate reference output tensor values (for positive tests).
The test runner is a modular tool which can be extended to run the same
tests on additional tools or frameworks - such a tool or framework is
called a System Under Test (SUT).
The reference output NumPy files generated by this step will be
compared with output of SUTs to validate them.

### Installation

The test infrastructure needs installing before being used. It is recommended
to create a [python virtual environment](https://docs.python.org/3/library/venv.html)
and then install the TOSA Unit Test infrastructure from the root of the
reference model:

``` bash
pip install .
```

When installing without a python virtual environment, use the pip
option `--user` to install it for the current user only.

#### Development

If you need an editable environment for adding a System Under Test, then use
one of the following scripts (due to a deficiency in python setup tools).
Create a python virtual environment and then source:

```bash
# For bash
. scripts/py-dev-env.bash

# For other shells
source scripts/py-dev-env.sh
```

### TOSA Conformance Generator

This script enables creation of part or all of the *TOSA conformance tests*, to enable running of these tests against the reference model or implementation.

The Conformance Tests comprise of a wide coverage of tests for all TOSA
operators created by using the Unit Test Builder.

#### Setup

The generation of conformance tests is dependent on the FlatBuffers
command `flatc` - please follow the section on the FlatBuffers compiler below.

#### Usage

These are the main script options for controlling the types of tests produced:

* `--profile` - controls the TOSA profile, `tosa-bi` - for base inference integer tests, `tosa-mi` - for the main inference floating point tests, or `all` - for both.
* `--test-type` - selects `positive`, `negative` or `both` types of test.
* `--output-type` - selects the output file type between `json`, `binary` or `both`.
The default - `json` - converts NumPy data files and flatbuffer files into JSON for
ease in viewing and comparison.

An example to create the TOSA operator unit tests for ADD and SUB:

```bash
tosa_verif_conformance_generator \
  --profile tosa-bi              \
  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
  --operators add sub
```

The above command will create some temporary files in a `conformance_build`
directory, but will output the conformance unit tests into a `conformance`
directory.

If you have a different build directory for the reference model, you may have
to supply one or more of the following options for path locations:

* `--ref-model-path` - path to the `tosa_reference_model` executable.
* `--schema-path` or `--operator-fbs` - path to the TOSA flatbuffer schema file (`tosa.fbs`)
* `--flatc-path` - path to the FlatBuffers compiler `flatc`

This is an example using the default locations:

```bash
tosa_verif_conformance_generator \
  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
  --flatc-path reference_model/build/thirdparty/serialization_lib/third_party/flatbuffers/flatc \
  --schema-path reference_model/thirdparty/serialization_lib/schema/tosa.fbs \
  --operators abs
```

This next example will create all the conformance tests, using different
temporary build and output directories:

```bash
tosa_verif_conformance_generator \
  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
  --build-directory tmp_build    \
  --output-directory conf_tests
```

Use the Unit Test Runner to run the conformance tests on the reference model,
or extend the runner to also test against your implementation (System Under
Test) - see those sections below for more information.

### Unit Test Builder

Unit tests are mostly single operator tests, that can be created and
used to check an implementation. These tests can be used to investigate TOSA
or complement the conformance tests.

The test builder is invoked by `tosa_verif_build_tests`.  The
builder generates test outputs in `./vtest/<operator_name>/` by
default.  To restrict test generation to particular regular expression
wildcard, use the `--filter ` argument.  The tool can be run with no
arguments to generate all tests.

Inputs and certain attributes are created using a random number
generator, while others are exhaustive (within reasonable bounds)
where the combinatorics allow exhaustive tests.  The test generation
is deterministic for a given random seed, but additional tests can be
generated using `--seed`.  As many corner-case error are often
uncovered using creative tensor shapes, the random seed parameter will
help get coverage of additional shapes.

By default only the positive tests will be produced, use the
argument `--test-type both` to build positive and negative tests.

Additional parameters on some operators can be found in the command
line help.

### Unit Test Runner

The unit test running script takes self-contained unit tests from the
builder and runs them on the reference model or on a System Under
Test.

#### Selecting tests

The `--test` or `-t` option is used to specify a directory containing
a test. Shell wildcards can be used to run more than one test at a time.
Tests will be run sequentially by default, but you may control how
many tests are run in parallel using the `--jobs` or `-j` switch.

For example, to run all of the TOSA add operator tests on the reference
model, eight at a time:

``` bash
tosa_verif_run_tests -t vtest/add/add* -j 8
```

The default location that is used for the reference model is
`reference_model/build/reference_model/tosa_reference_model`, use the option
`--ref-model-path` if you run this from a different location.

You can also supply a list of tests in a file, one per line, using the
`--test-list` or `-T` option. Alternatively use `--recursive` or `-r` to
run all the tests that can be found in the specified `-t` directory.

Finally you can choose the type of test to run - positive, negative or both
(default) -using the `test-type` option. To only run the positive tests:

```bash
tosa_verif_run_tests --test-type positive -t vtest/*/*
```

#### Verbosity

The test runner is reasonably quiet by default, so running a large number of
tests without any obvious errors will show only 1 line of output per test
completion. The `-v` switch will show the commands being run in the
background, and display any errors from the reference model.

#### Debugging

To enable debugging on the reference model, shortcut commands have
been provided: `--ref-debug=high` and `--ref-intermediates` to
turn on debugging and dump intermediate tensor values.

### Systems Under Test

Additional Systems Under Test (SUTs), such as reference
implementations of operators, full frameworks, and hardware implementations
can be tested by the test runner.

To do this you need to define an SUT module by extending the
`TosaTestRunner` class found in `verif/runner/tosa_test_runner.py`, and
then supplying this to the TOSA Test Runner.

#### SUT inputs and outputs

With each test is a `desc.json` file that contains input and output filename
information which is read and supplied to the `TosaTestRunner` class.

A TOSA System Under Test will need to be able to read the following input files:

* TOSA FlatBuffers (either JSON or binary format) - use the TOSA
  Serialization Library (<https://git.mlplatform.org/tosa/serialization_lib.git>)
  to do this.
* Tensors from python NumPy array files - see the
  [NumPy documentation](https://NumPy.org/doc/stable/reference/generated/NumPy.load.html)
  for more information. Use the TOSA Serialization Library to help
  (see the link above).

Utilize the `TosaTestRunner` class to convert these test artifacts
into another format before giving them to your SUT.

For positive tests usually at least one results file should be produced by
your SUT containing the resulting tensor in NumPy format. The expected
filenames are supplied in the `desc.json` information.

#### TosaTestRunner class

Your python class extending the `TosaTestRunner` class must contain:

* `__init__(self,...)` function that calls the super() class function.
* `runTestGraph(self)` function that invokes your SUT and then translates the
    return code into a `TosaTestRunner.TosaGraphResult`. It returns this result
    and an optional error message.

Examples of implementations can be found:

* `verif/runner/tosa_refmodel_sut_run.py` - the reference model
* `verif/tests/tosa_mock_sut_run.py` - mock version for testing

There is a helper function `run_sh_command` in `verif/runner/run_command.py`
that provides a robust way of calling shell commands from python that can be used
to invoke your SUT.

#### Testing with the Unit Test Runner

The SUT can then be supplied to the test runner by using the `--sut-module`
flag, the following invokes the reference model as the SUT (default behaviour
when not supplied):

```bash
tosa_verif_run_tests --sut-module runner.tosa_refmodel_sut_run -t TEST
```

You can also pass arguments to your SUT, for example this
will pass an argument called `ARGUMENT` with a value of `VALUE`
to the `mysut.tosa_mysut_sut_run` module:

```bash
tosa_run_tests --sut-module mysut.tosa_mysut_sut_run \
    --sut-module-args mysut.tosa_mysut_sut_run:ARGUMENT=VALUE \
    -t TEST
```

You can repeat this switch multiple times to pass multiple different arguments.

For an example of how to read these arguments in your SUT module, please see the
`tosa_mock_sut_run.py` file.


### TOSA Framework Unit Tests

Included in the TOSA Unit Test infrastructure are scripts to enable the creation
of TOSA unit tests for example frameworks. Included at the moment is support for
TensorFlow and TensorFlow Lite.

#### Setup

Installation (via `pip install`) of the following python package is required to
generate the tests:

* `tensorflow`

A built copy of the tensorflow framework from source is required to compile the
tests to TOSA - see the online documentation <https://www.tensorflow.org/install/source>
on how to do this.
The following tools are used from this build:

* `tensorflow/basel-bin/tensorflow/compiler/mlir/lite/flatbuffer_translate`
* `tensorflow/basel-bin/tensorflow/compiler/mlir/tf-opt`

#### Usage

The command to generate the unit test framework models:

```bash
tosa_verif_framework_generator -o tests
```

Next to convert these models to TOSA and then run them on the reference model:

```bash
tosa_verif_framework_compiler_runner \
  --tf-base-dir tensorflow           \
  --tools-base-dir reference_model   \
  --recursive                        \
  --test tests
```

## Other tools

Included in this repository are some support utilities used by the test runner:

* `tosa_json2numpy` - converts from JSON format to NumPy array or the reverse operation.
* `tosa_json2fbbin` - converts from JSON flatbuffer format to flatbuffer
    binary format or the reverse operation. This is dependent on the FlatBuffers
    command `flatc` - see the section on the FlatBuffers compiler below.
* `tosa_verif_result_check` - compares two results files.
* `tosa_convert2conformance` - converts a unit test into a conformance suitable test.
* `tosa_schemavalidation` - validates a test desc.json against the schema.

Please see the respective `--help` of each utility for more information on using
them standalone.

### FlatBuffers compiler

The FlatBuffers compiler tool (`flatc`) is only needed if you want to use
tosa_json2fbbin to convert the TOSA flatbuffer binary test files (.tosa) to
JSON or from JSON to binary.
It is best to use the flatbuffer version that comes with the reference model.
After following the reference model compilation instructions, you can build
the FlatBuffers tool using:

``` bash
# After compiling the reference model (in the build directory)
cd thirdparty/serialization_lib/third_party/flatbuffers
make flatc
```

## License

The *TOSA Reference Model* and TOSA Unit Tests are licensed under Apache-2.0.

Copyright (c) 2020-2023 Arm Limited.