COMPMID-417 Minor documentation fixes and error handling in instruments

Fixed some Doxygen issues in the introduction and tests sections
Switched MaliCounter to use ARM_COMPUTE_ERROR in order to get more context in case of failure
Switched to use ARM_COMPUTE_ERROR instead of ERROR_ON in PMU and MaliCounter so that checks work in release mode too. (As these checks are to detect platform issues rather than programming errors)

Change-Id: Ibbefa5a87d93c5c3aa5e190680052e4e99987dfe
Reviewed-on: http://mpd-gerrit.cambridge.arm.com/89662
Reviewed-by: Gian Marco Iodice <gianmarco.iodice@arm.com>
Reviewed-by: Georgios Pinitas <georgios.pinitas@arm.com>
Tested-by: Kaizen <jeremy.johnson+kaizengerrit@arm.com>

6 files changed, 57 insertions, 34 deletions

diff --git a/docs/00_introduction.dox b/docs/00_introduction.dox
index b097128ac1..2b6ddfb045 100644
--- a/docs/00_introduction.dox
+++ b/docs/00_introduction.dox
@@ -417,8 +417,6 @@ Example:
 
 @b validation_tests: Enable the build of the validation suite.
 
-@note You will need the Boost Test and Program options headers and libraries to build the validation tests. See @ref building_boost for more information.
-
 @b benchmark_tests: Enable the build of the benchmark tests
 
 @b pmu: Enable the PMU cycle counter to measure execution time in benchmark tests. (Your device needs to support it)
diff --git a/docs/02_tests.dox b/docs/02_tests.dox
index 212f8f0fae..0eb6cee487 100644
--- a/docs/02_tests.dox
+++ b/docs/02_tests.dox
@@ -298,13 +298,13 @@ the test case with different inputs.
 Before starting a new test case have a look at the existing ones. They should
 provide a good overview how test cases are structured.
 
-- The C++ reference needs to be added to `tests/validation_new/CPP/`. The
+- The C++ reference needs to be added to `tests/validation/CPP/`. The
   reference function is typically a template parameterized by the underlying
   value type of the `SimpleTensor`. This makes it easy to specialise for
   different data types.
 - If all backends have a common interface it makes sense to share the setup
   code. This can be done by adding a fixture in
-  `tests/validation_new/fixtures/`. Inside of the `setup` method of a fixture
+  `tests/validation/fixtures/`. Inside of the `setup` method of a fixture
   the tensors can be created and initialised and the function can be configured
   and run. The actual test will only have to validate the results. To be shared
   among multiple backends the fixture class is usually a template that accepts
@@ -382,6 +382,23 @@ As an alternative output format JSON is supported and can be selected via
 `--log-format=json`. To write the output to a file instead of stdout the
 `--log-file` option can be used.
 
+@subsubsection tests_running_tests_benchmarking_mode Mode
+Tests contain different datasets of different sizes, some of which will take several hours to run.
+You can select which datasets to use by using the `--mode` option, we recommed you use `--mode=precommit` to start with.
+
+@subsubsection tests_running_tests_benchmarking_instruments Instruments
+You can use the `--instruments` option to select one or more instruments to measure the execution time of the benchmark tests.
+
+`PMU` will try to read the CPU PMU events from the kernel (They need to be enabled on your platform)
+
+`MALI` will try to collect Mali hardware performance counters. (You need to have a recent enough Mali driver)
+
+`WALL_CLOCK` will measure time using `gettimeofday`: this should work on all platforms.
+
+You can pass a combinations of these instruments: `--instruments=PMU,MALI,WALL_CLOCK`
+
+@note You need to make sure the instruments have been selected at compile time using the `pmu=1` or `mali=1` scons options.
+
 <!-- FIXME: Remove before release and change above to benchmark and validation -->
 @subsection tests_running_tests_validation Validation
 
@@ -412,6 +429,7 @@ For a complete list of possible selectors please see: http://www.boost.org/doc/l
 
 @subsubsection tests_running_tests_validation_verbosity Verbosity
 There are two separate flags to control the verbosity of the test output. `--report_level` controls the verbosity of the summary produced after all tests have been executed. `--log_level` controls the verbosity of the information generated during the execution of tests. All available settings can be found in the Boost documentation for [--report_level](http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/report_level.html) and [--log_level](http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html), respectively.
+<!-- FIXME: end remove -->
 */
 } // namespace test
 } // namespace arm_compute
diff --git a/tests/framework/instruments/MaliCounter.cpp b/tests/framework/instruments/MaliCounter.cpp
index 887846e88c..6cc3ac5bcb 100644
--- a/tests/framework/instruments/MaliCounter.cpp
+++ b/tests/framework/instruments/MaliCounter.cpp
@@ -23,6 +23,8 @@
  */
 #include "MaliCounter.h"
 
+#include "arm_compute/core/Error.h"
+
 namespace arm_compute
 {
 namespace test
@@ -46,7 +48,7 @@ MaliHWInfo get_mali_hw_info(const char *path)
 
     if(fd < 0)
     {
-        throw std::runtime_error("Failed to get HW info.");
+        ARM_COMPUTE_ERROR("Failed to get HW info.");
     }
 
     {
@@ -57,7 +59,7 @@ MaliHWInfo get_mali_hw_info(const char *path)
 
         if(mali_userspace::mali_ioctl(fd, version_check_args) != 0)
         {
-            throw std::runtime_error("Failed to check version.");
+            ARM_COMPUTE_ERROR("Failed to check version.");
             close(fd);
         }
     }
@@ -70,7 +72,7 @@ MaliHWInfo get_mali_hw_info(const char *path)
 
         if(mali_userspace::mali_ioctl(fd, flags) != 0)
         {
-            throw std::runtime_error("Failed settings flags ioctl.");
+            ARM_COMPUTE_ERROR("Failed settings flags ioctl.");
             close(fd);
         }
     }
@@ -81,7 +83,7 @@ MaliHWInfo get_mali_hw_info(const char *path)
 
         if(mali_ioctl(fd, props) != 0)
         {
-            throw std::runtime_error("Failed settings flags ioctl.");
+            ARM_COMPUTE_ERROR("Failed settings flags ioctl.");
             close(fd);
         }
 
@@ -141,7 +143,7 @@ void MaliCounter::init()
 
     if(_fd < 0)
     {
-        throw std::runtime_error("Failed to open /dev/mali0.");
+        ARM_COMPUTE_ERROR("Failed to open /dev/mali0.");
     }
 
     {
@@ -150,11 +152,11 @@ void MaliCounter::init()
 
         if(mali_userspace::mali_ioctl(_fd, check) != 0)
         {
-            throw std::runtime_error("Failed to get ABI version.");
+            ARM_COMPUTE_ERROR("Failed to get ABI version.");
         }
         else if(check.major < 10)
         {
-            throw std::runtime_error("Unsupported ABI version 10.");
+            ARM_COMPUTE_ERROR("Unsupported ABI version 10.");
         }
     }
 
@@ -166,7 +168,7 @@ void MaliCounter::init()
 
         if(mali_userspace::mali_ioctl(_fd, flags) != 0)
         {
-            throw std::runtime_error("Failed settings flags ioctl.");
+            ARM_COMPUTE_ERROR("Failed settings flags ioctl.");
         }
     }
 
@@ -183,7 +185,7 @@ void MaliCounter::init()
 
         if(mali_userspace::mali_ioctl(_fd, setup) != 0)
         {
-            throw std::runtime_error("Failed setting hwcnt reader ioctl.");
+            ARM_COMPUTE_ERROR("Failed setting hwcnt reader ioctl.");
         }
 
         _hwc_fd = setup.fd;
@@ -194,34 +196,34 @@ void MaliCounter::init()
 
         if(ioctl(_hwc_fd, mali_userspace::KBASE_HWCNT_READER_GET_API_VERSION, &api_version) != 0) // NOLINT
         {
-            throw std::runtime_error("Could not determine hwcnt reader API.");
+            ARM_COMPUTE_ERROR("Could not determine hwcnt reader API.");
         }
         else if(api_version != mali_userspace::HWCNT_READER_API)
         {
-            throw std::runtime_error("Invalid API version.");
+            ARM_COMPUTE_ERROR("Invalid API version.");
         }
     }
 
     if(ioctl(_hwc_fd, static_cast<int>(mali_userspace::KBASE_HWCNT_READER_GET_BUFFER_SIZE), &_buffer_size) != 0) // NOLINT
     {
-        throw std::runtime_error("Failed to get buffer size.");
+        ARM_COMPUTE_ERROR("Failed to get buffer size.");
     }
 
     if(ioctl(_hwc_fd, static_cast<int>(mali_userspace::KBASE_HWCNT_READER_GET_HWVER), &_hw_ver) != 0) // NOLINT
     {
-        throw std::runtime_error("Could not determine HW version.");
+        ARM_COMPUTE_ERROR("Could not determine HW version.");
     }
 
     if(_hw_ver < 5)
     {
-        throw std::runtime_error("Unsupported HW version.");
+        ARM_COMPUTE_ERROR("Unsupported HW version.");
     }
 
     _sample_data = static_cast<uint8_t *>(mmap(nullptr, _buffer_count * _buffer_size, PROT_READ, MAP_PRIVATE, _hwc_fd, 0));
 
     if(_sample_data == MAP_FAILED) // NOLINT
     {
-        throw std::runtime_error("Failed to map sample data.");
+        ARM_COMPUTE_ERROR("Failed to map sample data.");
     }
 
     auto product = std::find_if(std::begin(mali_userspace::products), std::end(mali_userspace::products), [&](const mali_userspace::CounterMapping & cm)
@@ -235,7 +237,7 @@ void MaliCounter::init()
     }
     else
     {
-        throw std::runtime_error("Could not identify GPU.");
+        ARM_COMPUTE_ERROR("Could not identify GPU.");
     }
 
     _raw_counter_buffer.resize(_buffer_size / sizeof(uint32_t));
@@ -279,7 +281,7 @@ void MaliCounter::sample_counters()
 {
     if(ioctl(_hwc_fd, mali_userspace::KBASE_HWCNT_READER_DUMP, 0) != 0)
     {
-        throw std::runtime_error("Could not sample hardware counters.");
+        ARM_COMPUTE_ERROR("Could not sample hardware counters.");
     }
 }
 
@@ -293,7 +295,7 @@ void MaliCounter::wait_next_event()
 
     if(count < 0)
     {
-        throw std::runtime_error("poll() failed.");
+        ARM_COMPUTE_ERROR("poll() failed.");
     }
 
     if((poll_fd.revents & POLLIN) != 0)
@@ -302,7 +304,7 @@ void MaliCounter::wait_next_event()
 
         if(ioctl(_hwc_fd, static_cast<int>(mali_userspace::KBASE_HWCNT_READER_GET_BUFFER), &meta) != 0) // NOLINT
         {
-            throw std::runtime_error("Failed READER_GET_BUFFER.");
+            ARM_COMPUTE_ERROR("Failed READER_GET_BUFFER.");
         }
 
         memcpy(_raw_counter_buffer.data(), _sample_data + _buffer_size * meta.buffer_idx, _buffer_size);
@@ -310,12 +312,12 @@ void MaliCounter::wait_next_event()
 
         if(ioctl(_hwc_fd, mali_userspace::KBASE_HWCNT_READER_PUT_BUFFER, &meta) != 0) // NOLINT
         {
-            throw std::runtime_error("Failed READER_PUT_BUFFER.");
+            ARM_COMPUTE_ERROR("Failed READER_PUT_BUFFER.");
         }
     }
     else if((poll_fd.revents & POLLHUP) != 0)
     {
-        throw std::runtime_error("HWC hung up.");
+        ARM_COMPUTE_ERROR("HWC hung up.");
     }
 }
 
@@ -337,7 +339,7 @@ const uint32_t *MaliCounter::get_counters(mali_userspace::MaliCounterBlockName b
         default:
             if(core < 0)
             {
-                std::runtime_error("Invalid core number.");
+                ARM_COMPUTE_ERROR("Invalid core number.");
             }
 
             return _raw_counter_buffer.data() + mali_userspace::MALI_NAME_BLOCK_SIZE * (3 + _core_index_remap[core]);
diff --git a/tests/framework/instruments/PMU.cpp b/tests/framework/instruments/PMU.cpp
index 0594e96cb2..d0cacbba73 100644
--- a/tests/framework/instruments/PMU.cpp
+++ b/tests/framework/instruments/PMU.cpp
@@ -74,9 +74,10 @@ void PMU::open(const perf_event_attr &perf_config)
     ARM_COMPUTE_ERROR_ON_MSG(_fd < 0, "perf_event_open failed");
 
     const int result = ioctl(_fd, PERF_EVENT_IOC_ENABLE, 0);
-
-    ARM_COMPUTE_ERROR_ON_MSG(result == -1, "Failed to enable PMU counter: %d", errno);
-    ARM_COMPUTE_UNUSED(result);
+    if(result == -1)
+    {
+        ARM_COMPUTE_ERROR("Failed to enable PMU counter: %d", errno);
+    }
 }
 
 void PMU::close()
@@ -91,8 +92,10 @@ void PMU::close()
 void PMU::reset()
 {
     const int result = ioctl(_fd, PERF_EVENT_IOC_RESET, 0);
-    ARM_COMPUTE_ERROR_ON_MSG(result == -1, "Failed to reset PMU counter: %d", errno);
-    ARM_COMPUTE_UNUSED(result);
+    if(result == -1)
+    {
+        ARM_COMPUTE_ERROR("Failed to reset PMU counter: %d", errno);
+    }
 }
 } // namespace framework
 } // namespace test
diff --git a/tests/framework/instruments/PMU.h b/tests/framework/instruments/PMU.h
index d51b2f8ae1..e0a1870539 100644
--- a/tests/framework/instruments/PMU.h
+++ b/tests/framework/instruments/PMU.h
@@ -86,8 +86,10 @@ T PMU::get_value() const
     T             value{};
     const ssize_t result = read(_fd, &value, sizeof(T));
 
-    ARM_COMPUTE_ERROR_ON_MSG(result == -1, "Can't get PMU counter value: %d", errno);
-    ARM_COMPUTE_UNUSED(result);
+    if(result == -1)
+    {
+        ARM_COMPUTE_ERROR("Can't get PMU counter value: %d", errno);
+    }
 
     return value;
 }
diff --git a/tests/main.cpp b/tests/main.cpp
index ee12a38d1a..230d856958 100644
--- a/tests/main.cpp
+++ b/tests/main.cpp
@@ -103,7 +103,7 @@ int main(int argc, char **argv)
 
     auto help = parser.add_option<framework::ToggleOption>("help");
     help->set_help("Show this help message");
-    auto dataset_mode = parser.add_option<framework::EnumOption<framework::DatasetMode>>("mode", allowed_modes, framework::DatasetMode::ALL);
+    auto dataset_mode = parser.add_option<framework::EnumOption<framework::DatasetMode>>("mode", allowed_modes, framework::DatasetMode::PRECOMMIT);
     dataset_mode->set_help("For managed datasets select which group to use");
     auto instruments = parser.add_option<framework::EnumListOption<framework::InstrumentType>>("instruments", allowed_instruments, std::initializer_list<framework::InstrumentType> { framework::InstrumentType::ALL });
     instruments->set_help("Set the profiling instruments to use");