The Tensor Operator Set Architecture (TOSA) Specification https://git.mlplatform.org/tosa/specification.git/ is a set of operators with defined accuracy and compatibility constraints that Arm expects to be implemented on its Neural Processing Units (NPUs). Most operators from the common ML frameworks (TensorFlow, PyTorch, etc) should be expressible in TOSA. TOSA is focused on inference, leaving training to the original frameworks.
The TOSA Reference Model package provides a reference implementation and testing infrastructure for TOSA. The reference model consumes a FlatBuffers serialization of the network subgraph generated by the TOSA Serialization Library, along with input tensors for placeholder nodes in NumPy format. By default, the model validates and evalutes the network subgraph, and writes out the resulting output tensors in NumPy format.
The TOSA Reference Model and testing suite requires the following tools:
- CMake version 3.4 or later
- GNU Make 4.1 or later
- GCC (tested with 7.5.0) or Clang C++ compiler (tested with clang-9) with C++17 support
The model includes the following git submodules:
- TOSA Serialization Library
- JSON for Modern C++ - 3.8.0
- Eigen 3.3.7
The model is written using C++17 and has been primarily tested on Ubuntu x86_64 18.04 LTS Linux systems.
The testing infrastructure requires:
- Python 3.6 or later
- FlatBuffers 2.0 or later
- NumPy 1.15 or later
Check out the required git submodules with:
git submodule update --init --recursive
The TOSA Reference Model repository has branches (major.minor) and tags
(major.minor.patch) that correspond to each TOSA version. The
main branch is
used as the active development branch for the next version.
Perform a check-out of a specific version before compilation or installation of the test infrastructure by using:
git checkout --recurse-submodules VERSION
VERSION can be for example:
The TOSA Reference Model build can be prepared by creating makefiles using CMake:
mkdir -p build cd build cmake ..
-DCMAKE_BUILD_MODE=Debug can be used on the
command to create a debug build. Next compile using
The resulting executable will be named:
reference_model/tosa_reference_model. CMake only needs to be re-run
if the build environment changes (e.g., new dependencies or source
files). Code changes that do not affect these build rules can be
rebuilt simply using
The inputs to the TOSA Reference Model consist of a FlatBuffers file containing the serialized subgraph, a JSON test descriptor that describes a sequence of name, shape and numpy file for each input and output tensor.
The JSON test descriptor must have the following field: tosa_file: type: string. TOSA flatbuffer file location. ifm_name: type: list(string). Input placeholder tensor names. ifm_file: type: list(string). Input numpy array file location. ofm_name: type: list(string). Output placeholder tensor names. ofm_file: type: list(string). Output numpy array file location. expected_failure: type: boolean. Is this test expected to fail in runtime. * expected_return_code: type: int The expected return code of the reference model i (0 = VALID, 1 = ERROR, 2 = UNPREDICTABLE)
Note by default, all the files specified by "tosa_file", "ifm_file", "ofm_file" are relative to desc.json. This could be overwritten by -Cflatbuffer_dir=, if desired.
An example command is shown below:
./build/reference_model/tosa_reference_model \ -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
Instead of drive model by JSON test descriptor, user can also drive model with -Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file= options directly.
In case where -Ctest_desc= and other options are specified at the same time, JSON test descriptor will be initialized first. All other options (-Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file=) will overwrite whatever specified by JSON descriptor.
On a successful execution, the output tensors will be written in NumPy format into output tensors specified by "ofm_file".
For example, you can generate new output .npy by:
./build/reference_model/tosa_reference_model \ -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json -Cofm_file=out.npy
In this case, the "ofm_file" field in desc.json will be ignored, and the one specified by -Cofm_file= will be picked.
When using JSON-formatted FlatBuffers input (.json extension), the FlatBuffers schema file from the TOSA Serialization library must be specified using -Coperator_fbs=. When using the binary FlatBuffers format (.tosa), the schema is not necessary.
The TOSA Reference Model distribution contains several example networks with inputs and reference outputs generated by TensorFlow or TensorFlow Lite in the examples directory.
These examples can be run through the TOSA Reference model and should produce the equivalent TOSA-compliant reference output tensors. Please note that differences in floating-point ordering and rounding may cause small differences in output for floating-point tests and differences in quantized scaling between TensorFlow Lite and the TOSA Specification may cause differences in quantized integer tests.
The debugging facility can be enabled by setting a debug scope and
debug level on the command line. For most purposes, the following
flags will work:
-dALL -lHIGH. Debug output can be directed to a
file using the
The TOSA Unit Test infrastruture builds and runs self-contained tests for implementations of the Tensor Operator Set Architecture (TOSA) Specification. These tools directly generate TOSA operators for verification of the TOSA reference model against existing frameworks or other operator implementations.
The test builder tool by default generates positive tests with random
arguments and reference inputs for each TOSA operator. Positive tests
are expected to run without error and usually produce a result (some
control flow operators may not produce a result).
The test builder can also generate negative tests for all the ERROR_IF
conditions within the TOSA Specification by using the
options. Negative tests may contain invalid arguments or inputs and
are expected to run and fail without producing a result. Other errors
or unpredictable results are handled in a system dependent way and
are not tested by the test builder tool.
The unit tests are typically structured as a combination of input
placeholder nodes, const nodes, and attributes feeding into a single
TOSA operator. The unit tests use a Python copy of the FlatBuffers
schema written by
flatc to verify tosa.
Each test has a JSON file which provides machine-readable metadata for the test, including the TOSA flatbuffer file, names, shapes, and NumPy filenames for each input and output tensor. There is also a boolean value for whether a failure is expected because the test is expected to trigger an invalid set of operands or attributes.
The test runner tool can execute the unit tests on the TOSA Reference Model to generate reference output tensor values (for positive tests). The test runner is a modular tool which can be extended to run the same tests on additional tools or frameworks - such a tool or framework is called a System Under Test (SUT). The reference output NumPy files generated by this step can be programatically compared with output of SUTs to validate them.
The test infrastructure needs installing before being used. It is recommended to create a python virtual environment and then install the TOSA Unit Test infrastruture from the root of the reference model:
pip install .
When installing without a python virtual environment, use the pip
--user to install it for the current user only.
The test builder is invoked by
builder generates test outputs in
default. To restrict test generation to particular regular expression
wildcard, use the
--filter argument. The tool can be run with no
arguments to generate all tests.
Inputs and certain attributes are created using a random number
generator, while others are exhaustive (within reasonable bounds)
where the combinatorics allow exhaustive tests. The test generation
is deterministic for a given random seed, but additional tests can be
--seed. As many corner-case error are often
uncovered using creative tensor shapes, the random seed parameter will
help get coverage of additional shapes.
By default only the positive tests will be produced, use the
--test-type both to build positive and negative tests.
Additional parameters on some operators can be found in the command line help.
The unit test running script takes self-contained unit tests from the builder and runs them on the reference model or on a System Under Test.
-t option is used to specify a directory containing
a test. Shell wildcards can be used to run more than one test at a time.
Tests will be run sequentially by default, but you may control how
many tests are run in parallel using the
For example, to run all of the TOSA add operator tests on the reference model, eight at a time:
tosa_verif_run_tests -t vtest/add/add* -j 8
The default location that is used for the reference model is
reference_model/build/reference_model/tosa_reference_model, use the option
--ref-model-path if you run this from a different location.
You can also supply a list of tests in a file, one per line, using the
Finally you can choose the type of test to run - positive, negative or both
(default) -using the
test-type option. To only run the positive tests:
tosa_run_tests --test-type positive -t vtest/*/*
The test runner is reasonably quiet by default, so running a large number of
tests without any obvious errors will show only 1 line of output per test
-v switch will show the commands being run in the
To enable debugging on the reference model, shortcut commands have
turn on debugging and dump intermediate tensor values.
Additional Systems Under Test (SUTs), such as reference implementations of operators, full frameworks, and hardware implementations can be tested by the test runner.
To do this you need to define an SUT module by extending the
TosaTestRunner class found in
then supplying this to the TOSA Test Runner.
With each test is a
desc.json file that contains input and output filename
information which is read and supplied to the
A TOSA System Under Test will need to be able to read the following input files:
- TOSA FlatBuffers (either JSON or binary format) - use the TOSA Serialization Library (https://git.mlplatform.org/tosa/serialization_lib.git) to do this.
- Tensors from python numpy array files - see the numpy documentation for more information. Use the TOSA Serialization Library to help (see the link above).
TosaTestRunner class to convert these test artifacts
into another format before giving them to your SUT.
For positive tests usually at least one results file should be produced by
your SUT containing the resulting tensor in numpy format. The expected
filenames are supplied in the
Your python class extending the
TosaTestRunner class must contain:
__init__(self,...)function that calls the super() class function.
runTestGraph(self)function that invokes your SUT and then translates the return code into a
TosaTestRunner.TosaGraphResult. It returns this result and an optional error message.
Examples of implementations can be found:
verif/runner/tosa_refmodel_sut_run.py- the reference model
verif/tests/tosa_mock_sut_run.py- mock version for testing
There is a helper function
that provides a robust way of calling shell commands from python that can be used
to invoke your SUT.
The SUT can then be supplied to the test runner by using the
flag, the following invokes the reference model as the SUT (default behaviour
when not supplied):
tosa_verif_run_tests --sut-module runner.tosa_refmodel_sut_run -t TEST
You can also pass arguments to your SUT, for example this
will pass an argument called
ARGUMENT with a value of
tosa_run_tests --sut-module mysut.tosa_mysut_sut_run \ --sut-module-args mysut.tosa_mysut_sut_run:ARGUMENT=VALUE \ -t TEST
You can repeat this switch multiple times to pass multiple different arguments.
For an example of how to read these arguments in your SUT module, please see the
Included in this repository are some support utilities used by the test runner:
json2numpy- converts from JSON format to numpy array or the reverse operation.
json2fbbin- converts from JSON flatbuffer format to flatbuffer binary format or the reverse operation. This is dependent on the FlatBuffers command
flatc- see the section on the FlatBuffers compiler below.
tosa_verif_result_check- compares two results files.
convert2conformance- converts a unit test into a conformance suitable test.
Please see the respective
--help of each utility for more information on using
The FlatBuffers compiler tool (
flatc) is only needed if you want to use
json2fbbin to convert the TOSA flatbuffer binary test files (.tosa) to JSON
or from JSON to binary.
It is best to use the flatbuffer version that comes with the reference model.
After following the reference model compilation instructions, you can build
the FlatBuffers tool using:
# After compiling the reference model (in the build directory) cd thirdparty/serialization_lib/third_party/flatbuffers make
The TOSA Reference Model and TOSA Unit Tests are licensed under Apache-2.0.
Copyright (c) 2020-2022 Arm Limited.