diff options
Diffstat (limited to 'docs/02_tests.dox')
| -rw-r--r-- | docs/02_tests.dox | 407 |
1 files changed, 0 insertions, 407 deletions
diff --git a/docs/02_tests.dox b/docs/02_tests.dox deleted file mode 100644 index b636880524..0000000000 --- a/docs/02_tests.dox +++ /dev/null @@ -1,407 +0,0 @@ -/// -/// Copyright (c) 2017-2020 ARM Limited. -/// -/// SPDX-License-Identifier: MIT -/// -/// Permission is hereby granted, free of charge, to any person obtaining a copy -/// of this software and associated documentation files (the "Software"), to -/// deal in the Software without restriction, including without limitation the -/// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -/// sell copies of the Software, and to permit persons to whom the Software is -/// furnished to do so, subject to the following conditions: -/// -/// The above copyright notice and this permission notice shall be included in all -/// copies or substantial portions of the Software. -/// -/// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -/// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -/// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -/// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -/// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -/// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -/// SOFTWARE. -/// -namespace arm_compute -{ -namespace test -{ -/** -@page tests Validation and benchmarks tests - -@tableofcontents - -@section tests_overview Overview - -Benchmark and validation tests are based on the same framework to setup and run -the tests. In addition to running simple, self-contained test functions the -framework supports fixtures and data test cases. The former allows to share -common setup routines between various backends thus reducing the amount of -duplicated code. The latter can be used to parameterize tests or fixtures with -different inputs, e.g. different tensor shapes. One limitation is that -tests/fixtures cannot be parameterized based on the data type if static type -information is needed within the test (e.g. to validate the results). - -@note By default tests are not built. To enable them you need to add validation_tests=1 and / or benchmark_tests=1 to your SCons line. - -@note Tests are not included in the pre-built binary archive, you have to build them from sources. - -@subsection tests_overview_structure Directory structure - - . - `-- tests <- Top level test directory. All files in here are shared among validation and benchmark. - |-- framework <- Underlying test framework. - |-- CL \ - |-- GLES_COMPUTE \ - |-- NEON -> Backend specific files with helper functions etc. - |-- benchmark <- Top level directory for the benchmarking files. - | |-- fixtures <- Fixtures for benchmark tests. - | |-- CL <- OpenCL backend test cases on a function level. - | |-- GLES_COMPUTE <- Same of OpenGL ES - | `-- NEON <- Same for NEON - |-- datasets <- Datasets for benchmark and validation tests. - |-- main.cpp <- Main entry point for the tests. Currently shared between validation and benchmarking. - `-- validation -> Top level directory for validation files. - |-- CPP -> C++ reference code - |-- CL \ - |-- GLES_COMPUTE \ - |-- NEON -> Backend specific test cases - `-- fixtures -> Fixtures shared among all backends. Used to setup target function and tensors. - -@subsection tests_overview_fixtures Fixtures - -Fixtures can be used to share common setup, teardown or even run tasks among -multiple test cases. For that purpose a fixture can define a `setup`, -`teardown` and `run` method. Additionally the constructor and destructor might -also be customized. - -An instance of the fixture is created immediately before the actual test is -executed. After construction the @ref framework::Fixture::setup method is called. Then the test -function or the fixtures `run` method is invoked. After test execution the -@ref framework::Fixture::teardown method is called and lastly the fixture is destructed. - -@subsubsection tests_overview_fixtures_fixture Fixture - -Fixtures for non-parameterized test are straightforward. The custom fixture -class has to inherit from @ref framework::Fixture and choose to implement any of the -`setup`, `teardown` or `run` methods. None of the methods takes any arguments -or returns anything. - - class CustomFixture : public framework::Fixture - { - void setup() - { - _ptr = malloc(4000); - } - - void run() - { - ARM_COMPUTE_ASSERT(_ptr != nullptr); - } - - void teardown() - { - free(_ptr); - } - - void *_ptr; - }; - -@subsubsection tests_overview_fixtures_data_fixture Data fixture - -The advantage of a parameterized fixture is that arguments can be passed to the setup method at runtime. To make this possible the setup method has to be a template with a type parameter for every argument (though the template parameter doesn't have to be used). All other methods remain the same. - - class CustomFixture : public framework::Fixture - { - #ifdef ALTERNATIVE_DECLARATION - template <typename ...> - void setup(size_t size) - { - _ptr = malloc(size); - } - #else - template <typename T> - void setup(T size) - { - _ptr = malloc(size); - } - #endif - - void run() - { - ARM_COMPUTE_ASSERT(_ptr != nullptr); - } - - void teardown() - { - free(_ptr); - } - - void *_ptr; - }; - -@subsection tests_overview_test_cases Test cases - -All following commands can be optionally prefixed with `EXPECTED_FAILURE_` or -`DISABLED_`. - -@subsubsection tests_overview_test_cases_test_case Test case - -A simple test case function taking no inputs and having no (shared) state. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the dataset mode in which the test will be active. - - - TEST_CASE(TestCaseName, DatasetMode::PRECOMMIT) - { - ARM_COMPUTE_ASSERT_EQUAL(1 + 1, 2); - } - -@subsubsection tests_overview_test_cases_fixture_fixture_test_case Fixture test case - -A simple test case function taking no inputs that inherits from a fixture. The -test case will have access to all public and protected members of the fixture. -Only the setup and teardown methods of the fixture will be used. The body of -this function will be used as test function. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the class name of the fixture. -- Third argument is the dataset mode in which the test will be active. - - - class FixtureName : public framework::Fixture - { - public: - void setup() override - { - _one = 1; - } - - protected: - int _one; - }; - - FIXTURE_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT) - { - ARM_COMPUTE_ASSERT_EQUAL(_one + 1, 2); - } - -@subsubsection tests_overview_test_cases_fixture_register_fixture_test_case Registering a fixture as test case - -Allows to use a fixture directly as test case. Instead of defining a new test -function the run method of the fixture will be executed. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the class name of the fixture. -- Third argument is the dataset mode in which the test will be active. - - - class FixtureName : public framework::Fixture - { - public: - void setup() override - { - _one = 1; - } - - void run() override - { - ARM_COMPUTE_ASSERT_EQUAL(_one + 1, 2); - } - - protected: - int _one; - }; - - REGISTER_FIXTURE_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT); - - -@subsubsection tests_overview_test_cases_data_test_case Data test case - -A parameterized test case function that has no (shared) state. The dataset will -be used to generate versions of the test case with different inputs. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the dataset mode in which the test will be active. -- Third argument is the dataset. -- Further arguments specify names of the arguments to the test function. The - number must match the arity of the dataset. - - - DATA_TEST_CASE(TestCaseName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3}), num) - { - ARM_COMPUTE_ASSERT(num < 4); - } - -@subsubsection tests_overview_test_cases_fixture_data_test_case Fixture data test case - -A parameterized test case that inherits from a fixture. The test case will have -access to all public and protected members of the fixture. Only the setup and -teardown methods of the fixture will be used. The setup method of the fixture -needs to be a template and has to accept inputs from the dataset as arguments. -The body of this function will be used as test function. The dataset will be -used to generate versions of the test case with different inputs. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the class name of the fixture. -- Third argument is the dataset mode in which the test will be active. -- Fourth argument is the dataset. - - - class FixtureName : public framework::Fixture - { - public: - template <typename T> - void setup(T num) - { - _num = num; - } - - protected: - int _num; - }; - - FIXTURE_DATA_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3})) - { - ARM_COMPUTE_ASSERT(_num < 4); - } - -@subsubsection tests_overview_test_cases_register_fixture_data_test_case Registering a fixture as data test case - -Allows to use a fixture directly as parameterized test case. Instead of -defining a new test function the run method of the fixture will be executed. -The setup method of the fixture needs to be a template and has to accept inputs -from the dataset as arguments. The dataset will be used to generate versions of -the test case with different inputs. - -- First argument is the name of the test case (has to be unique within the - enclosing test suite). -- Second argument is the class name of the fixture. -- Third argument is the dataset mode in which the test will be active. -- Fourth argument is the dataset. - - - class FixtureName : public framework::Fixture - { - public: - template <typename T> - void setup(T num) - { - _num = num; - } - - void run() override - { - ARM_COMPUTE_ASSERT(_num < 4); - } - - protected: - int _num; - }; - - REGISTER_FIXTURE_DATA_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3})); - -@section writing_tests Writing validation tests - -Before starting a new test case have a look at the existing ones. They should -provide a good overview how test cases are structured. - -- The C++ reference needs to be added to `tests/validation/CPP/`. The - reference function is typically a template parameterized by the underlying - value type of the `SimpleTensor`. This makes it easy to specialise for - different data types. -- If all backends have a common interface it makes sense to share the setup - code. This can be done by adding a fixture in - `tests/validation/fixtures/`. Inside of the `setup` method of a fixture - the tensors can be created and initialised and the function can be configured - and run. The actual test will only have to validate the results. To be shared - among multiple backends the fixture class is usually a template that accepts - the specific types (data, tensor class, function class etc.) as parameters. -- The actual test cases need to be added for each backend individually. - Typically the will be multiple tests for different data types and for - different execution modes, e.g. precommit and nightly. - -@section tests_running_tests Running tests -@subsection tests_running_tests_benchmark_and_validation Benchmarking and validation suites -@subsubsection tests_running_tests_benchmarking_filter Filter tests -All tests can be run by invoking - - ./arm_compute_benchmark ./data - -where `./data` contains the assets needed by the tests. - -If only a subset of the tests has to be executed the `--filter` option takes a -regular expression to select matching tests. - - ./arm_compute_benchmark --filter='^NEON/.*AlexNet' ./data - -@note Filtering will be much faster if the regular expression starts from the start ("^") or end ("$") of the line. - -Additionally each test has a test id which can be used as a filter, too. -However, the test id is not guaranteed to be stable when new tests are added. -Only for a specific build the same the test will keep its id. - - ./arm_compute_benchmark --filter-id=10 ./data - -All available tests can be displayed with the `--list-tests` switch. - - ./arm_compute_benchmark --list-tests - -More options can be found in the `--help` message. - -@subsubsection tests_running_tests_benchmarking_runtime Runtime -By default every test is run once on a single thread. The number of iterations -can be controlled via the `--iterations` option and the number of threads via -`--threads`. - -@subsubsection tests_running_tests_benchmarking_output Output -By default the benchmarking results are printed in a human readable format on -the command line. The colored output can be disabled via `--no-color-output`. -As an alternative output format JSON is supported and can be selected via -`--log-format=json`. To write the output to a file instead of stdout the -`--log-file` option can be used. - -@subsubsection tests_running_tests_benchmarking_mode Mode -Tests contain different datasets of different sizes, some of which will take several hours to run. -You can select which datasets to use by using the `--mode` option, we recommed you use `--mode=precommit` to start with. - -@subsubsection tests_running_tests_benchmarking_instruments Instruments -You can use the `--instruments` option to select one or more instruments to measure the execution time of the benchmark tests. - -`PMU` will try to read the CPU PMU events from the kernel (They need to be enabled on your platform) - -`MALI` will try to collect Mali hardware performance counters. (You need to have a recent enough Mali driver) - -`WALL_CLOCK_TIMER` will measure time using `gettimeofday`: this should work on all platforms. - -You can pass a combinations of these instruments: `--instruments=PMU,MALI,WALL_CLOCK_TIMER` - -@note You need to make sure the instruments have been selected at compile time using the `pmu=1` or `mali=1` scons options. - -@subsubsection tests_running_examples Examples - -To run all the precommit validation tests: - - LD_LIBRARY_PATH=. ./arm_compute_validation --mode=precommit - -To run the OpenCL precommit validation tests: - - LD_LIBRARY_PATH=. ./arm_compute_validation --mode=precommit --filter="^CL.*" - -To run the NEON precommit benchmark tests with PMU and Wall Clock timer in miliseconds instruments enabled: - - LD_LIBRARY_PATH=. ./arm_compute_benchmark --mode=precommit --filter="^NEON.*" --instruments="pmu,wall_clock_timer_ms" --iterations=10 - -To run the OpenCL precommit benchmark tests with OpenCL kernel timers in miliseconds enabled: - - LD_LIBRARY_PATH=. ./arm_compute_benchmark --mode=precommit --filter="^CL.*" --instruments="opencl_timer_ms" --iterations=10 - -@note You might need to export the path to OpenCL library as well in your LD_LIBRARY_PATH if Compute Library was built with OpenCL enabled. -*/ -} // namespace test -} // namespace arm_compute |