TOSA Reference Model
Introduction
The Tensor Operator Set Architecture (TOSA) Specification https://git.mlplatform.org/tosa/specification.git/ is a set of operators with defined accuracy and compatibility constraints that Arm expects to be implemented on its Neural Processing Units (NPUs). Most operators from the common ML frameworks (TensorFlow, PyTorch, etc) should be expressible in TOSA. TOSA is focused on inference, leaving training to the original frameworks.
The TOSA Reference Model package provides a reference implementation and testing infrastructure for TOSA. The reference model consumes a FlatBuffers serialization of the network subgraph generated by the TOSA Serialization Library, along with input tensors for placeholder nodes in NumPy format. By default, the model validates and evaluates the network subgraph, and writes out the resulting output tensors in NumPy format.
This package also contains a conformance test suite that can be created and then used to validate an implementation of TOSA.
Installation Requirements
The TOSA Reference Model and testing suite requires the following tools:
- CMake version 3.4 or later
- GNU Make 4.1 or later
- GCC (tested with 9.4.0) or Clang C++ compiler (tested with clang-10) with C++17 support
The model includes the following git submodules:
- TOSA Serialization Library
- JSON for Modern C++ - 3.8.0
- Eigen 3.3.7
- doctest 2.4.9 (When building unit tests)
The model is written using C++17 and has been primarily tested on Ubuntu x86_64 20.04 LTS Linux systems.
The testing infrastructure requires:
- Python 3.6 or later
- FlatBuffers 2.0 or later
- NumPy 1.15 or later
Check out the required git submodules with:
git submodule update --init --recursive
Versioning
The TOSA Reference Model repository has branches (major.minor) and tags
(major.minor.patch) that correspond to each TOSA version. The main
branch is
used as the active development branch for the next version.
Perform a check-out of a specific version before compilation or installation of the test infrastructure by using:
git checkout --recurse-submodules VERSION
Where VERSION
can be for example: v0.23
or v0.23.0
Compilation
The TOSA Reference Model build can be prepared by creating makefiles using CMake:
mkdir -p build
cd build
cmake ..
Optionally, -DCMAKE_BUILD_TYPE=Debug
can be used on the cmake
command to create a debug build. Next compile using make
:
make
The resulting executable will be named:
reference_model/tosa_reference_model
. This executable can be disabled with
-DBUILD_TOSA_REFERENCE_MODEL_EXECUTABLE=NO
.
A static library will also be generated by default as follows:
reference_model/libtosa_reference_model_lib.a
.
To make this a shared library (.so) add the following option
-DBUILD_SHARED_LIBS=YES
.
CMake only needs to be re-run if the build environment changes (e.g., new dependencies or source
files). Code changes that do not affect these build rules can be
rebuilt simply using make
.
Executable Usage
The inputs to the TOSA Reference Model consist of a FlatBuffers file containing the serialized subgraph, a JSON test descriptor that describes a sequence of name, shape and NumPy file for each input and output tensor.
The JSON test descriptor must have the following field: tosa_file: type: string. TOSA flatbuffer file location. ifm_name: type: list(string). Input placeholder tensor names. ifm_file: type: list(string). Input NumPy array file location. ofm_name: type: list(string). Output placeholder tensor names. ofm_file: type: list(string). Output NumPy array file location. expected_failure: type: boolean. Is this test expected to fail in runtime. * expected_return_code: type: int The expected return code of the reference model i (0 = VALID, 1 = ERROR, 2 = UNPREDICTABLE)
Note by default, all the files specified by "tosa_file", "ifm_file", "ofm_file" are relative to desc.json. This could be overwritten by --flatbuffer_dir=, if desired.
An example command is shown below:
./build/reference_model/tosa_reference_model \
--test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
Instead of drive model by JSON test descriptor, user can also drive model with --tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file= options directly.
In case where --test_desc= and other options are specified at the same time, JSON test descriptor will be initialized first. All other options (--tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file=) will overwrite whatever specified by JSON descriptor.
On a successful execution, the output tensors will be written in NumPy format into output tensors specified by "ofm_file".
For example, you can generate new output .npy by:
./build/reference_model/tosa_reference_model \
--test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json \
--ofm_file=out.npy
In this case, the "ofm_file" field in desc.json will be ignored, and the one specified by --ofm_file= will be picked.
When using JSON-formatted FlatBuffers input (.json extension), the FlatBuffers schema file from the TOSA Serialization library must be specified using --operator_fbs=. When using the binary FlatBuffers format (.tosa), the schema is not necessary.
Examples
The TOSA Reference Model distribution contains several example networks with inputs and reference outputs generated by TensorFlow or TensorFlow Lite in the examples directory.
These examples can be run through the TOSA Reference model and should produce the equivalent TOSA-compliant reference output tensors. Please note that differences in floating-point ordering and rounding may cause small differences in output for floating-point tests and differences in quantized scaling between TensorFlow Lite and the TOSA Specification may cause differences in quantized integer tests.
ModelRunner API
As an alternative to the executable described above,
the model_runner.h is provided which can be used to invoke the
TOSA Reference Model easily within C++.
A sample of this class and how it can used can be found in
model_runner_simple_sample.cpp.
This sample can be compiled by adding -DBUILD_MODEL_RUNNER_SAMPLE=YES
to the CMake command
and executed by running ./build/reference_model/model_runner_sample
.
ModelRunner API Unit Tests
Unit test are generated by default for the ModelRunner.
This executable can be disabled by adding-DBUILD_TOSA_REFERENCE_MODEL_TESTS=NO
to the CMake command.
This executable can be run using
./build/reference_model/unit_tests
and requires the submodule doctest.
Debugging
The debugging facility can be enabled by setting a debug scope and
debug level on the command line. For most purposes, the following
flags will work: -d ALL -l HIGH
. Debug output can be directed to a
file using the -o
switch.
TOSA Unit Test Infrastructure
The TOSA Unit Test infrastructure can create and run self-contained tests for implementations of the Tensor Operator Set Architecture (TOSA) Specification. These test tools generate TOSA operators for verification of the TOSA reference model against existing frameworks or other operator implementations.
A conformance suite generator is included in the TOSA Unit Test infrastructure to provide coverage validation of an implementation of TOSA. Please see the TOSA Conformance Generator section below to find out how to create these.
Test content
The test building tools by default generate positive tests with random
arguments and reference inputs for each TOSA operator. Positive tests
are expected to run without error and will produce a result.
The test builder can also generate negative tests for all the ERROR_IF
conditions within the TOSA Specification by using the --test-type
options. Negative tests may contain invalid arguments or inputs and
are expected to be loaded and fail without producing a result.
Other errors or unpredictable results are expected to be handled in a
system dependent way and are not tested by the test builder tool.
The unit tests are typically structured as a combination of input
placeholder nodes, const nodes, and attributes feeding into a single
TOSA operator. The unit tests use a Python copy of the FlatBuffers
schema written by flatc
to verify TOSA.
Each test has a JSON file which provides machine-readable metadata for the test, including the TOSA flatbuffer file, names, shapes, and NumPy filenames for each input and output tensor. There is also a boolean value for whether a failure is expected because the test is expected to trigger an invalid set of operands or attributes. Extra metadata may be included such as tags for filtering, data generation or compliance validation.
The test runner tool can execute the unit tests on the TOSA Reference Model to generate reference output tensor values (for positive tests). The test runner is a modular tool which can be extended to run the same tests on additional tools or frameworks - such a tool or framework is called a System Under Test (SUT). The reference output NumPy files generated by this step will be compared with output of SUTs to validate them.
Installation
The test infrastructure needs installing before being used. It is recommended to create a python virtual environment and then install the TOSA Unit Test infrastructure from the root of the reference model:
pip install .
When installing without a python virtual environment, use the pip
option --user
to install it for the current user only.
Development
If you need an editable environment for adding a System Under Test, then use one of the following scripts (due to a deficiency in python setup tools). Create a python virtual environment and then source:
# For bash
. scripts/py-dev-env.bash
# For other shells
source scripts/py-dev-env.sh
TOSA Conformance Generator
This script enables creation of part or all of the TOSA conformance tests, to enable running of these tests against the reference model or implementation.
The Conformance Tests comprise of a wide coverage of tests for all TOSA operators created by using the Unit Test Builder.
Setup
The generation of conformance tests is dependent on the FlatBuffers
command flatc
- please follow the section on the FlatBuffers compiler below.
Usage
These are the main script options for controlling the types of tests produced:
--profile
- controls the TOSA profile,tosa-bi
- for base inference integer tests,tosa-mi
- for the main inference floating point tests, orall
- for both.--test-type
- selectspositive
,negative
orboth
types of test.--output-type
- selects the output file type betweenjson
,binary
orboth
. The default -json
- converts NumPy data files and flatbuffer files into JSON for ease in viewing and comparison.
An example to create the TOSA operator unit tests for ADD and SUB:
tosa_verif_conformance_generator \
--profile tosa-bi \
--ref-model-path reference_model/build/reference_model/tosa_reference_model \
--operators add sub
The above command will create some temporary files in a conformance_build
directory, but will output the conformance unit tests into a conformance
directory.
If you have a different build directory for the reference model, you may have to supply one or more of the following options for path locations:
--ref-model-path
- path to thetosa_reference_model
executable.--schema-path
or--operator-fbs
- path to the TOSA flatbuffer schema file (tosa.fbs
)--flatc-path
- path to the FlatBuffers compilerflatc
This is an example using the default locations:
tosa_verif_conformance_generator \
--ref-model-path reference_model/build/reference_model/tosa_reference_model \
--flatc-path reference_model/build/thirdparty/serialization_lib/third_party/flatbuffers/flatc \
--schema-path reference_model/thirdparty/serialization_lib/schema/tosa.fbs \
--operators abs
This next example will create all the conformance tests, using different temporary build and output directories:
tosa_verif_conformance_generator \
--ref-model-path reference_model/build/reference_model/tosa_reference_model \
--build-directory tmp_build \
--output-directory conf_tests
Use the Unit Test Runner to run the conformance tests on the reference model, or extend the runner to also test against your implementation (System Under Test) - see those sections below for more information.
Unit Test Builder
Unit tests are mostly single operator tests, that can be created and used to check an implementation. These tests can be used to investigate TOSA or complement the conformance tests.
The test builder is invoked by tosa_verif_build_tests
. The
builder generates test outputs in ./vtest/<operator_name>/
by
default. To restrict test generation to particular regular expression
wildcard, use the --filter
argument. The tool can be run with no
arguments to generate all tests.
Inputs and certain attributes are created using a random number
generator, while others are exhaustive (within reasonable bounds)
where the combinatorics allow exhaustive tests. The test generation
is deterministic for a given random seed, but additional tests can be
generated using --seed
. As many corner-case error are often
uncovered using creative tensor shapes, the random seed parameter will
help get coverage of additional shapes.
By default only the positive tests will be produced, use the
argument --test-type both
to build positive and negative tests.
Additional parameters on some operators can be found in the command line help.
Unit Test Runner
The unit test running script takes self-contained unit tests from the builder and runs them on the reference model or on a System Under Test.
Selecting tests
The --test
or -t
option is used to specify a directory containing
a test. Shell wildcards can be used to run more than one test at a time.
Tests will be run sequentially by default, but you may control how
many tests are run in parallel using the --jobs
or -j
switch.
For example, to run all of the TOSA add operator tests on the reference model, eight at a time:
tosa_verif_run_tests -t vtest/add/add* -j 8
The default location that is used for the reference model is
reference_model/build/reference_model/tosa_reference_model
, use the option
--ref-model-path
if you run this from a different location.
You can also supply a list of tests in a file, one per line, using the
--test-list
or -T
option. Alternatively use --recursive
or -r
to
run all the tests that can be found in the specified -t
directory.
Finally you can choose the type of test to run - positive, negative or both
(default) -using the test-type
option. To only run the positive tests:
tosa_verif_run_tests --test-type positive -t vtest/*/*
Verbosity
The test runner is reasonably quiet by default, so running a large number of
tests without any obvious errors will show only 1 line of output per test
completion. The -v
switch will show the commands being run in the
background, and display any errors from the reference model.
Debugging
To enable debugging on the reference model, shortcut commands have
been provided: --ref-debug=high
and --ref-intermediates
to
turn on debugging and dump intermediate tensor values.
Systems Under Test
Additional Systems Under Test (SUTs), such as reference implementations of operators, full frameworks, and hardware implementations can be tested by the test runner.
To do this you need to define an SUT module by extending the
TosaTestRunner
class found in verif/runner/tosa_test_runner.py
, and
then supplying this to the TOSA Test Runner.
SUT inputs and outputs
With each test is a desc.json
file that contains input and output filename
information which is read and supplied to the TosaTestRunner
class.
A TOSA System Under Test will need to be able to read the following input files:
- TOSA FlatBuffers (either JSON or binary format) - use the TOSA Serialization Library (https://git.mlplatform.org/tosa/serialization_lib.git) to do this.
- Tensors from python NumPy array files - see the NumPy documentation for more information. Use the TOSA Serialization Library to help (see the link above).
Utilize the TosaTestRunner
class to convert these test artifacts
into another format before giving them to your SUT.
For positive tests usually at least one results file should be produced by
your SUT containing the resulting tensor in NumPy format. The expected
filenames are supplied in the desc.json
information.
TosaTestRunner class
Your python class extending the TosaTestRunner
class must contain:
__init__(self,...)
function that calls the super() class function.runTestGraph(self)
function that invokes your SUT and then translates the return code into aTosaTestRunner.TosaGraphResult
. It returns this result and an optional error message.
Examples of implementations can be found:
verif/runner/tosa_refmodel_sut_run.py
- the reference modelverif/tests/tosa_mock_sut_run.py
- mock version for testing
There is a helper function run_sh_command
in verif/runner/run_command.py
that provides a robust way of calling shell commands from python that can be used
to invoke your SUT.
Testing with the Unit Test Runner
The SUT can then be supplied to the test runner by using the --sut-module
flag, the following invokes the reference model as the SUT (default behaviour
when not supplied):
tosa_verif_run_tests --sut-module runner.tosa_refmodel_sut_run -t TEST
You can also pass arguments to your SUT, for example this
will pass an argument called ARGUMENT
with a value of VALUE
to the mysut.tosa_mysut_sut_run
module:
tosa_run_tests --sut-module mysut.tosa_mysut_sut_run \
--sut-module-args mysut.tosa_mysut_sut_run:ARGUMENT=VALUE \
-t TEST
You can repeat this switch multiple times to pass multiple different arguments.
For an example of how to read these arguments in your SUT module, please see the
tosa_mock_sut_run.py
file.
TOSA Framework Unit Tests
Included in the TOSA Unit Test infrastructure are scripts to enable the creation of TOSA unit tests for example frameworks. Included at the moment is support for TensorFlow and TensorFlow Lite.
Setup
Installation (via pip install
) of the following python package is required to
generate the tests:
tensorflow
A built copy of the tensorflow framework from source is required to compile the tests to TOSA - see the online documentation https://www.tensorflow.org/install/source on how to do this. The following tools are used from this build:
tensorflow/basel-bin/tensorflow/compiler/mlir/lite/flatbuffer_translate
tensorflow/basel-bin/tensorflow/compiler/mlir/tf-opt
Usage
The command to generate the unit test framework models:
tosa_verif_framework_generator -o tests
Next to convert these models to TOSA and then run them on the reference model:
tosa_verif_framework_compiler_runner \
--tf-base-dir tensorflow \
--tools-base-dir reference_model \
--recursive \
--test tests
Other tools
Included in this repository are some support utilities used by the test runner:
tosa_json2numpy
- converts from JSON format to NumPy array or the reverse operation.tosa_json2fbbin
- converts from JSON flatbuffer format to flatbuffer binary format or the reverse operation. This is dependent on the FlatBuffers commandflatc
- see the section on the FlatBuffers compiler below.tosa_verif_result_check
- compares two results files.tosa_convert2conformance
- converts a unit test into a conformance suitable test.tosa_schemavalidation
- validates a test desc.json against the schema.
Please see the respective --help
of each utility for more information on using
them standalone.
FlatBuffers compiler
The FlatBuffers compiler tool (flatc
) is only needed if you want to use
tosa_json2fbbin to convert the TOSA flatbuffer binary test files (.tosa) to
JSON or from JSON to binary.
It is best to use the flatbuffer version that comes with the reference model.
After following the reference model compilation instructions, you can build
the FlatBuffers tool using:
# After compiling the reference model (in the build directory)
cd thirdparty/serialization_lib/third_party/flatbuffers
make flatc
License
The TOSA Reference Model and TOSA Unit Tests are licensed under Apache-2.0.
Copyright (c) 2020-2023 Arm Limited.