Update README for conformance

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I8cd95ed0ae3260eea785e4e0275961534cd1f0c5

1 files changed, 117 insertions, 94 deletions

diff --git a/README.md b/README.md
index 30773d7..32155cf 100644
--- a/README.md
+++ b/README.md
@@ -14,10 +14,13 @@ The *TOSA Reference Model* package provides a reference implementation
 and testing infrastructure for TOSA.  The reference model consumes a
 FlatBuffers serialization of the network subgraph generated by the
 TOSA Serialization Library, along with input tensors for placeholder
-nodes in NumPy format.  By default, the model validates and evalutes
+nodes in NumPy format.  By default, the model validates and evaluates
 the network subgraph, and writes out the resulting output tensors in
 NumPy format.
 
+This package also contains a conformance test suite that can be
+created and then used to validate an implementation of TOSA.
+
 ## Installation Requirements
 
 The *TOSA Reference Model* and testing suite requires the following
@@ -100,7 +103,7 @@ rebuilt simply using `make`.
 
 The inputs to the *TOSA Reference Model* consist of a FlatBuffers file
 containing the serialized subgraph, a JSON test descriptor that describes
-a sequence of name, shape and numpy file for each input and output tensor.
+a sequence of name, shape and NumPy file for each input and output tensor.
 
 The JSON test descriptor must have the following field:
 * tosa_file:
@@ -111,13 +114,13 @@ The JSON test descriptor must have the following field:
     Input placeholder tensor names.
 * ifm_file:
     type: list(string).
-    Input numpy array file location.
+    Input NumPy array file location.
 * ofm_name:
     type: list(string).
     Output placeholder tensor names.
 * ofm_file:
     type: list(string).
-    Output numpy array file location.
+    Output NumPy array file location.
 * expected_failure:
     type: boolean.
     Is this test expected to fail in runtime.
@@ -202,41 +205,49 @@ file using the `-o` switch.
 
 ## TOSA Unit Test Infrastructure
 
-The TOSA Unit Test infrastructure builds and runs self-contained tests
+The TOSA Unit Test infrastructure can create and run self-contained tests
 for implementations of the *Tensor Operator Set Architecture (TOSA)
-Specification*.  These tools directly generate TOSA operators for
+Specification*.  These test tools generate TOSA operators for
 verification of the TOSA reference model against existing frameworks
 or other operator implementations.
 
-The test builder tool by default generates positive tests with random
+A conformance suite generator is included in the TOSA Unit Test
+infrastructure to provide coverage validation of an
+implementation of TOSA. Please see the *TOSA Conformance Generator*
+section below to find out how to create these.
+
+### Test content
+
+The test building tools by default generate positive tests with random
 arguments and reference inputs for each TOSA operator.  Positive tests
-are expected to run without error and usually produce a result (some
-control flow operators may not produce a result).
+are expected to run without error and will produce a result.
 The test builder can also generate negative tests for all the ERROR_IF
 conditions within the TOSA Specification by using the `--test-type`
 options. Negative tests may contain invalid arguments or inputs and
-are expected to run and fail without producing a result. Other errors
-or unpredictable results are handled in a system dependent way and
-are not tested by the test builder tool.
+are expected to be loaded and fail without producing a result.
+Other errors or unpredictable results are expected to be handled in a
+system dependent way and are not tested by the test builder tool.
 
 The unit tests are typically structured as a combination of input
 placeholder nodes, const nodes, and attributes feeding into a single
 TOSA operator.  The unit tests use a Python copy of the FlatBuffers
-schema written by `flatc` to verify tosa.
+schema written by `flatc` to verify TOSA.
 
 Each test has a JSON file which provides machine-readable metadata for
 the test, including the TOSA flatbuffer file, names, shapes, and
 NumPy filenames for each input and output tensor.  There is also a
 boolean value for whether a failure is expected because the test is
 expected to trigger an invalid set of operands or attributes.
+Extra metadata may be included such as tags for filtering, data
+generation or compliance validation.
 
 The test runner tool can execute the unit tests on the TOSA Reference
 Model to generate reference output tensor values (for positive tests).
 The test runner is a modular tool which can be extended to run the same
 tests on additional tools or frameworks - such a tool or framework is
 called a System Under Test (SUT).
-The reference output NumPy files generated by this step can be
-programatically compared with output of SUTs to validate them.
+The reference output NumPy files generated by this step will be
+compared with output of SUTs to validate them.
 
 ### Installation
 
@@ -252,8 +263,92 @@ pip install .
 When installing without a python virtual environment, use the pip
 option `--user` to install it for the current user only.
 
+#### Development
+
+If you need an editable environment for adding a System Under Test, then use
+one of the following scripts (due to a deficiency in python setup tools).
+Create a python virtual environment and then source:
+
+```bash
+# For bash
+. scripts/py-dev-env.bash
+
+# For other shells
+source scripts/py-dev-env.sh
+```
+
+### TOSA Conformance Generator
+
+This script enables creation of part or all of the *TOSA conformance tests*, to enable running of these tests against the reference model or implementation.
+
+The Conformance Tests comprise of a wide coverage of tests for all TOSA
+operators created by using the Unit Test Builder.
+
+#### Setup
+
+The generation of conformance tests is dependent on the FlatBuffers
+command `flatc` - please follow the section on the FlatBuffers compiler below.
+
+#### Usage
+
+These are the main script options for controlling the types of tests produced:
+
+* `--profile` - controls the TOSA profile, `tosa-bi` - for base inference integer tests, `tosa-mi` - for the main inference floating point tests, or `all` - for both.
+* `--test-type` - selects `positive`, `negative` or `both` types of test.
+* `--output-type` - selects the output file type between `json`, `binary` or `both`.
+The default - `json` - converts NumPy data files and flatbuffer files into JSON for
+ease in viewing and comparison.
+
+An example to create the TOSA operator unit tests for ADD and SUB:
+
+```bash
+tosa_verif_conformance_generator \
+  --profile tosa-bi              \
+  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
+  --operators add sub
+```
+
+The above command will create some temporary files in a `conformance_build`
+directory, but will output the conformance unit tests into a `conformance`
+directory.
+
+If you have a different build directory for the reference model, you may have
+to supply one or more of the following options for path locations:
+
+* `--ref-model-path` - path to the `tosa_reference_model` executable.
+* `--schema-path` or `--operator-fbs` - path to the TOSA flatbuffer schema file (`tosa.fbs`)
+* `--flatc-path` - path to the FlatBuffers compiler `flatc`
+
+This is an example using the default locations:
+
+```bash
+tosa_verif_conformance_generator \
+  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
+  --flatc-path reference_model/build/thirdparty/serialization_lib/third_party/flatbuffers/flatc \
+  --schema-path reference_model/thirdparty/serialization_lib/schema/tosa.fbs \
+  --operators abs
+```
+
+This next example will create all the conformance tests, using different
+temporary build and output directories:
+
+```bash
+tosa_verif_conformance_generator \
+  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
+  --build-directory tmp_build    \
+  --output-directory conf_tests
+```
+
+Use the Unit Test Runner to run the conformance tests on the reference model,
+or extend the runner to also test against your implementation (System Under
+Test) - see those sections below for more information.
 
 ### Unit Test Builder
+
+Unit tests are mostly single operator tests, that can be created and
+used to check an implementation. These tests can be used to investigate TOSA
+or complement the conformance tests.
+
 The test builder is invoked by `tosa_verif_build_tests`.  The
 builder generates test outputs in `./vtest/<operator_name>/` by
 default.  To restrict test generation to particular regular expression
@@ -299,7 +394,8 @@ The default location that is used for the reference model is
 `--ref-model-path` if you run this from a different location.
 
 You can also supply a list of tests in a file, one per line, using the
-`--test-list` or `-T` option.
+`--test-list` or `-T` option. Alternatively use `--recursive` or `-r` to
+run all the tests that can be found in the specified `-t` directory.
 
 Finally you can choose the type of test to run - positive, negative or both
 (default) -using the `test-type` option. To only run the positive tests:
@@ -313,7 +409,7 @@ tosa_verif_run_tests --test-type positive -t vtest/*/*
 The test runner is reasonably quiet by default, so running a large number of
 tests without any obvious errors will show only 1 line of output per test
 completion. The `-v` switch will show the commands being run in the
-background.
+background, and display any errors from the reference model.
 
 #### Debugging
 
@@ -341,8 +437,8 @@ A TOSA System Under Test will need to be able to read the following input files:
 * TOSA FlatBuffers (either JSON or binary format) - use the TOSA
   Serialization Library (<https://git.mlplatform.org/tosa/serialization_lib.git>)
   to do this.
-* Tensors from python numpy array files - see the
-  [numpy documentation](https://numpy.org/doc/stable/reference/generated/numpy.load.html)
+* Tensors from python NumPy array files - see the
+  [NumPy documentation](https://NumPy.org/doc/stable/reference/generated/NumPy.load.html)
   for more information. Use the TOSA Serialization Library to help
   (see the link above).
 
@@ -350,7 +446,7 @@ Utilize the `TosaTestRunner` class to convert these test artifacts
 into another format before giving them to your SUT.
 
 For positive tests usually at least one results file should be produced by
-your SUT containing the resulting tensor in numpy format. The expected
+your SUT containing the resulting tensor in NumPy format. The expected
 filenames are supplied in the `desc.json` information.
 
 #### TosaTestRunner class
@@ -436,83 +532,11 @@ tosa_verif_framework_compiler_runner \
   --test tests
 ```
 
-### TOSA Conformance Generator
-
-This script enables creation of part or all of the *TOSA conformance tests*, to enable running or development of these tests.
-
-Currently only the Base Profile of TOSA is fully supported by the generator.
-
-**DEPRECATION NOTES:**
-
-* The repository *TOSA conformance tests - <https://git.mlplatform.org/tosa/conformance_tests.git/>* -
-has been DEPRECATED, tests need to be generated using the script detailed in this section.
-* The framework tests are DEPRECATED as part of the conformance testing, so there
-is no need to follow the TOSA Framework Unit Tests instructions above for this section.
-
-#### Setup
-
-The generation of conformance tests is dependent on the FlatBuffers
-command `flatc` - please follow the section on the FlatBuffers compiler below.
-
-#### Usage
-
-These are the main script options for controlling the types of tests produced:
-
-* `--profile` - controls the TOSA profile, only `tosa-bi` - for base inference tests -
-is fully supported. The other options are `tosa-mi` - for the floating point main
-inference tests - or `all` - for both.
-* `--unit-tests` - choose which tests to produce, only `operator` should be used as
-`framework` (and `both`) tests are DEPRECATED.
-* `--test-type` - selects `positive`, `negative` or `both` types of test.
-* `--output-type` - selects the output file type between `json`, `binary` or `both`.
-The default - `json` - converts numpy data files and flatbuffer files into JSON for
-ease in viewing and comparison.
-
-An example to create the TOSA operator unit tests for ADD and SUB:
-
-```bash
-tosa_verif_conformance_generator \
-  --profile tosa-bi              \
-  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
-  --operators add sub
-```
-
-The above command will create some temporary files in a `conformance_build`
-directory, but will output the conformance unit tests into a `conformance`
-directory.
-
-If you have a different build directory for the reference model, you may have
-to supply one or more of the following options for path locations:
-
-* `--ref-model-path` - path to the `tosa_reference_model` executable.
-* `--schema-path` or `--operator-fbs` - path to the TOSA flatbuffer schema file (`tosa.fbs`)
-* `--flatc-path` - path to the flatbuffers compiler `flatc`
-
-This is an example using the default locations:
-
-```bash
-tosa_verif_conformance_generator \
-  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
-  --flatc-path reference_model/build/thirdparty/serialization_lib/third_party/flatbuffers/flatc \
-  --schema-path referecne_model/thirdparty/serialization_lib/schema/tosa.fbs \
-  --operators abs
-```
-
-This next example will create all the conformance tests, using different
-temporary build and output directories:
-
-```bash
-tosa_verif_conformance_generator \
-  --ref-model-path reference_model/build/reference_model/tosa_reference_model \
-  --build-directory tmp_build    \
-  --output-directory conf_tests
-```
-
 ## Other tools
 
 Included in this repository are some support utilities used by the test runner:
 
-* `tosa_json2numpy` - converts from JSON format to numpy array or the reverse operation.
+* `tosa_json2numpy` - converts from JSON format to NumPy array or the reverse operation.
 * `tosa_json2fbbin` - converts from JSON flatbuffer format to flatbuffer
     binary format or the reverse operation. This is dependent on the FlatBuffers
     command `flatc` - see the section on the FlatBuffers compiler below.
@@ -538,7 +562,6 @@ cd thirdparty/serialization_lib/third_party/flatbuffers
 make flatc
 ```
 
-
 ## License
 
 The *TOSA Reference Model* and TOSA Unit Tests are licensed under Apache-2.0.