aboutsummaryrefslogtreecommitdiff

TOSA Reference Model

Introduction

The Tensor Operator Set Architecture (TOSA) Specification https://git.mlplatform.org/tosa/specification.git/ is a set of operators with defined accuracy and compatibility constraints that Arm expects to be implemented on its Neural Processing Units (NPUs). Most operators from the common ML frameworks (TensorFlow, PyTorch, etc) should be expressible in TOSA. TOSA is focused on inference, leaving training to the original frameworks.

The TOSA Reference Model package provides a reference implementation and testing infrastructure for TOSA. The reference model consumes a FlatBuffers serialization of the network subgraph generated by the TOSA Serialization Library, along with input tensors for placeholder nodes in NumPy format. By default, the model validates and evalutes the network subgraph, and writes out the resulting output tensors in NumPy format.

Installation Requirements

The TOSA Reference Model and testing suite requires the following tools:

  • CMake version 3.4 or later
  • GNU Make 4.1 or later
  • GCC (tested with 7.5.0) or Clang C++ compiler (tested with clang-9) with C++17 support

The model includes the following git submodules:

  • TOSA Serialization Library
  • JSON for Modern C++ - 3.8.0
  • Eigen 3.3.7

The model is written using C++17 and has been primarily tested on Ubuntu x86_64 18.04 LTS Linux systems.

The testing infrastructure requires:

  • Python 3.6 or later
  • FlatBuffers 2.0 or later
  • NumPy 1.15 or later

Check out the required git submodules with:

git submodule update --init --recursive

Versioning

The TOSA Reference Model repository has branches (major.minor) and tags (major.minor.patch) that correspond to each TOSA version. The main branch is used as the active development branch for the next version.

Perform a check-out of a specific version before compilation or installation of the test infrastructure by using:

git checkout --recurse-submodules VERSION

Where VERSION can be for example: v0.23 or v0.23.0

Compilation

The TOSA Reference Model build can be prepared by creating makefiles using CMake:

mkdir -p build
cd build
cmake ..

Optionally, -DCMAKE_BUILD_MODE=Debug can be used on the cmake command to create a debug build. Next compile using make:

make

The resulting executable will be named: reference_model/tosa_reference_model. CMake only needs to be re-run if the build environment changes (e.g., new dependencies or source files). Code changes that do not affect these build rules can be rebuilt simply using make.

Usage

The inputs to the TOSA Reference Model consist of a FlatBuffers file containing the serialized subgraph, a JSON test descriptor that describes a sequence of name, shape and numpy file for each input and output tensor.

The JSON test descriptor must have the following field: tosa_file: type: string. TOSA flatbuffer file location. ifm_name: type: list(string). Input placeholder tensor names. ifm_file: type: list(string). Input numpy array file location. ofm_name: type: list(string). Output placeholder tensor names. ofm_file: type: list(string). Output numpy array file location. expected_failure: type: boolean. Is this test expected to fail in runtime. * expected_return_code: type: int The expected return code of the reference model i (0 = VALID, 1 = ERROR, 2 = UNPREDICTABLE)

Note by default, all the files specified by "tosa_file", "ifm_file", "ofm_file" are relative to desc.json. This could be overwritten by -Cflatbuffer_dir=, if desired.

An example command is shown below:

./build/reference_model/tosa_reference_model \
  -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json

Instead of drive model by JSON test descriptor, user can also drive model with -Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file= options directly.

In case where -Ctest_desc= and other options are specified at the same time, JSON test descriptor will be initialized first. All other options (-Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file=) will overwrite whatever specified by JSON descriptor.

On a successful execution, the output tensors will be written in NumPy format into output tensors specified by "ofm_file".

For example, you can generate new output .npy by:

./build/reference_model/tosa_reference_model \
  -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
  -Cofm_file=out.npy

In this case, the "ofm_file" field in desc.json will be ignored, and the one specified by -Cofm_file= will be picked.

When using JSON-formatted FlatBuffers input (.json extension), the FlatBuffers schema file from the TOSA Serialization library must be specified using -Coperator_fbs=. When using the binary FlatBuffers format (.tosa), the schema is not necessary.

Examples

The TOSA Reference Model distribution contains several example networks with inputs and reference outputs generated by TensorFlow or TensorFlow Lite in the examples directory.

These examples can be run through the TOSA Reference model and should produce the equivalent TOSA-compliant reference output tensors. Please note that differences in floating-point ordering and rounding may cause small differences in output for floating-point tests and differences in quantized scaling between TensorFlow Lite and the TOSA Specification may cause differences in quantized integer tests.

Debugging

The debugging facility can be enabled by setting a debug scope and debug level on the command line. For most purposes, the following flags will work: -dALL -lHIGH. Debug output can be directed to a file using the -o switch.

TOSA Unit Test Infrastructure

The TOSA Unit Test infrastruture builds and runs self-contained tests for implementations of the Tensor Operator Set Architecture (TOSA) Specification. These tools directly generate TOSA operators for verification of the TOSA reference model against existing frameworks or other operator implementations.

The test builder tool by default generates positive tests with random arguments and reference inputs for each TOSA operator. Positive tests are expected to run without error and usually produce a result (some control flow operators may not produce a result). The test builder can also generate negative tests for all the ERROR_IF conditions within the TOSA Specification by using the --test-type options. Negative tests may contain invalid arguments or inputs and are expected to run and fail without producing a result. Other errors or unpredictable results are handled in a system dependent way and are not tested by the test builder tool.

The unit tests are typically structured as a combination of input placeholder nodes, const nodes, and attributes feeding into a single TOSA operator. The unit tests use a Python copy of the FlatBuffers schema written by flatc to verify tosa.

Each test has a JSON file which provides machine-readable metadata for the test, including the TOSA flatbuffer file, names, shapes, and NumPy filenames for each input and output tensor. There is also a boolean value for whether a failure is expected because the test is expected to trigger an invalid set of operands or attributes.

The test runner tool can execute the unit tests on the TOSA Reference Model to generate reference output tensor values (for positive tests). The test runner is a modular tool which can be extended to run the same tests on additional tools or frameworks - such a tool or framework is called a System Under Test (SUT). The reference output NumPy files generated by this step can be programatically compared with output of SUTs to validate them.

Installation

The test infrastructure needs installing before being used. It is recommended to create a python virtual environment and then install the TOSA Unit Test infrastruture from the root of the reference model:

pip install .

When installing without a python virtual environment, use the pip option --user to install it for the current user only.

Unit Test Builder

The test builder is invoked by tosa_verif_build_tests. The builder generates test outputs in ./vtest/<operator_name>/ by default. To restrict test generation to particular regular expression wildcard, use the --filter argument. The tool can be run with no arguments to generate all tests.

Inputs and certain attributes are created using a random number generator, while others are exhaustive (within reasonable bounds) where the combinatorics allow exhaustive tests. The test generation is deterministic for a given random seed, but additional tests can be generated using --seed. As many corner-case error are often uncovered using creative tensor shapes, the random seed parameter will help get coverage of additional shapes.

By default only the positive tests will be produced, use the argument --test-type both to build positive and negative tests.

Additional parameters on some operators can be found in the command line help.

Unit Test Runner

The unit test running script takes self-contained unit tests from the builder and runs them on the reference model or on a System Under Test.

Selecting tests

The --test or -t option is used to specify a directory containing a test. Shell wildcards can be used to run more than one test at a time. Tests will be run sequentially by default, but you may control how many tests are run in parallel using the --jobs or -j switch.

For example, to run all of the TOSA add operator tests on the reference model, eight at a time:

tosa_verif_run_tests -t vtest/add/add* -j 8

The default location that is used for the reference model is reference_model/build/reference_model/tosa_reference_model, use the option --ref-model-path if you run this from a different location.

You can also supply a list of tests in a file, one per line, using the --test-list or -T option.

Finally you can choose the type of test to run - positive, negative or both (default) -using the test-type option. To only run the positive tests:

tosa_run_tests --test-type positive -t vtest/*/*

Verbosity

The test runner is reasonably quiet by default, so running a large number of tests without any obvious errors will show only 1 line of output per test completion. The -v switch will show the commands being run in the background.

Debugging

To enable debugging on the reference model, shortcut commands have been provided: --ref-debug=high and --ref-intermediates to turn on debugging and dump intermediate tensor values.

Systems Under Test

Additional Systems Under Test (SUTs), such as reference implementations of operators, full frameworks, and hardware implementations can be tested by the test runner.

To do this you need to define an SUT module by extending the TosaTestRunner class found in verif/runner/tosa_test_runner.py, and then supplying this to the TOSA Test Runner.

SUT inputs and outputs

With each test is a desc.json file that contains input and output filename information which is read and supplied to the TosaTestRunner class.

A TOSA System Under Test will need to be able to read the following input files:

Utilize the TosaTestRunner class to convert these test artifacts into another format before giving them to your SUT.

For positive tests usually at least one results file should be produced by your SUT containing the resulting tensor in numpy format. The expected filenames are supplied in the desc.json information.

TosaTestRunner class

Your python class extending the TosaTestRunner class must contain:

  • __init__(self,...) function that calls the super() class function.
  • runTestGraph(self) function that invokes your SUT and then translates the return code into a TosaTestRunner.TosaGraphResult. It returns this result and an optional error message.

Examples of implementations can be found:

  • verif/runner/tosa_refmodel_sut_run.py - the reference model
  • verif/tests/tosa_mock_sut_run.py - mock version for testing

There is a helper function run_sh_command in verif/runner/run_command.py that provides a robust way of calling shell commands from python that can be used to invoke your SUT.

Testing with the Unit Test Runner

The SUT can then be supplied to the test runner by using the --sut-module flag, the following invokes the reference model as the SUT (default behaviour when not supplied):

tosa_verif_run_tests --sut-module runner.tosa_refmodel_sut_run -t TEST

You can also pass arguments to your SUT, for example this will pass an argument called ARGUMENT with a value of VALUE to the mysut.tosa_mysut_sut_run module:

tosa_run_tests --sut-module mysut.tosa_mysut_sut_run \
    --sut-module-args mysut.tosa_mysut_sut_run:ARGUMENT=VALUE \
    -t TEST

You can repeat this switch multiple times to pass multiple different arguments.

For an example of how to read these arguments in your SUT module, please see the tosa_mock_sut_run.py file.

Other tools

Included in this repository are some support utilities used by the test runner:

  • json2numpy - converts from JSON format to numpy array or the reverse operation.
  • json2fbbin - converts from JSON flatbuffer format to flatbuffer binary format or the reverse operation. This is dependent on the FlatBuffers command flatc - see the section on the FlatBuffers compiler below.
  • tosa_verif_result_check - compares two results files.
  • convert2conformance - converts a unit test into a conformance suitable test.

Please see the respective --help of each utility for more information on using them standalone.

FlatBuffers compiler

The FlatBuffers compiler tool (flatc) is only needed if you want to use json2fbbin to convert the TOSA flatbuffer binary test files (.tosa) to JSON or from JSON to binary. It is best to use the flatbuffer version that comes with the reference model. After following the reference model compilation instructions, you can build the FlatBuffers tool using:

# After compiling the reference model (in the build directory)
cd thirdparty/serialization_lib/third_party/flatbuffers
make

License

The TOSA Reference Model and TOSA Unit Tests are licensed under Apache-2.0.

Copyright (c) 2020-2022 Arm Limited.