aboutsummaryrefslogtreecommitdiff
path: root/verif
AgeCommit message (Collapse)Author
2023-12-15Add basic framework test cases for dynamic shapesJerry Ge
- Added a basic infrastructure for allowing generate network with dynamic_shapes - Added tests cases for - batch_to_space - depth_to_space, space_to_depth - linear Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Ie3f13231a74485df64b852f554cfe65e995f0d03
2023-12-14Remove inferred dimension from RESHAPEJeremy Johnson
Test generation changed to only produce static reshape tests Reference model changed to produce ERROR_IF on inferred shapes Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92c92a40e7c0e457961bc654630040dff79a750b
2023-12-14Main Compliance testing for SCATTER and GATHERJeremy Johnson
Added indices shuffling and random INT32 support to generate lib with testing of these new random generator modes Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I058d8b092470228075e8fe69c2ededa639163003
2023-12-11Enforce no output rewrite REQUIRE in SCATTERJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3555e7216d403d436bf6e39d4b16bb000645c4bb
2023-12-11Rectify coverage loss due to FP16 complianceJeremy Johnson
Operator coverage restored/improved: * REDUCE_SUM * AVG_POOL2D * FULLY_CONNECTED Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie622b2f97b5c34971fdde5b20df3d616171ac11f
2023-12-04Path fixes for ref model and librariesJeremy Johnson
Support relative desc.json paths in refmodel of just "desc.json". Catch when generate and verify library paths are None and provide help in tosa_verif_check_result. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie52ac399ea002e5fcdcc1eec3d9df6153a778e88
2023-12-04Change TANH, SIGMOID to ABS_ERROR complianceJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icf04afc7fdae8f506ba4710aaa085d6ea53bb5bf
2023-12-04Main Compliance FP16 support - generate and verify.Jeremy Johnson
FP16 support for all existing operators for compliance: * DOT_PRODUCT * ULP * EXACT * ABS_ERROR Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8d25448a793375b53880da3787d8f839767f02cf
2023-12-04Main Compliance testing support for RESHAPEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I750ef5d3c95daca10e7717437ed375a81b434582
2023-11-30Main Compliance testing support for CASTJeremy Johnson
Limit CAST input tensor to maximums of output type to avoid saturation and infinity. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I33350a4ce0ec828da7d2e7aa8cd3183a89a97431
2023-11-30Adjust random data ranges for Main Compliance to avoid FP inf and nanJeremy Johnson
POW - there are now 3 test sets to cover random ranges. Also added ROUND mode to data generator to force integer exponent values. LOG, EXP, RSQRT, REDUCE_SUM & FULLY_CONNECTED - have had their ranges reduced for each test. Fix generate library configuration defaults and checks. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie5d3bd78f690cc787a2ca4eb9b4bd6808bd9238c
2023-11-23Main Compliance testing support for COMPARISON opsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id6229cfaccad866b110630119eb045dbf6453bf5
2023-11-23Main Compliance testing support for LOG and ACTIVATIONsJeremy Johnson
Increase exp2 allowed range to account for denormals. Minor adjustments to verify to match spec updates for pseudo code. Set ranges of activation function inputs to match spec. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6fcf665932ac2c9080e284b865da8f7746801f59
2023-11-23Verify library fix to only accept FP64 ref dataJeremy Johnson
Make test_verif_result_check output to only say result files are non-compliant as it may not be implmenetation result issues. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I14d10a7ee3aeff7802b6b173280d9f2054760a08
2023-11-23Main Compliance testing support for AVG_POOL2DJeremy Johnson
Fix test set 4 to have more symmetrical data set for AVG_POOL2D. Increased size of AVG_POOL2D test shapes to meet MIN_DOT_PRODUCTS. Print out sign of bias magnitude in verify library to help debugging. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icf139cf5788cbc68f2c537567bdfb4cfe31274df
2023-11-16[reference_model] add local_bound attribute to test_genTai Ly
Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I9b87d1ad0f09648d07a6ecfc2075a174a9d04a3d
2023-11-16Fix FP16, BF16 data ranges for conformance testsJeremy Johnson
Enable use of data ranges for old data gen path as well as the new generate library path, so that FP16 and BF16 test data is produced within the correct ranges. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I749870a3112f8c3a75f4d16b8322c813fbf977cd
2023-11-16Main Compliance testing support for EXP & POWJeremy Johnson
Added new ABS_ERROR mode to verify lib and ref model. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ifb78290675833d3df7df91a4d6cef336b02b64a4
2023-11-16Main Compliance testing for simple UNARY opsJeremy Johnson
For RECIPROCAL, RSQRT, CEIL, FLOOR, ABS, NEGATE & IDENTITY. Improved ULP informational output. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I49644573b4c9a30b2b9d6c9624f2a1d46976a378
2023-11-10Main Compliance testing for FULLY_CONNECTEDJeremy Johnson
Updated shapes to meet MIN_DOT_PRODUCTS. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I82297917c009b3120306f8a9bb965209d109ef8d
2023-11-10Main Compliance testing support for ARGMAX, REDUCE_SUM/MAX/MINJeremy Johnson
Add extra tests for FP32 REDUCE_SUM that meet MIN_DOT_PRODUCTS. Plus improved dot product test generation skip information. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia8198a9500ddddfc86c5bb84230b9a4edf5ffd50
2023-11-10Main Compliance testing support for ADD, SUB, MINIMUM and MAXIMUMJeremy Johnson
Added main inference compliance mode data generation and verification. Improved error reporting in tosa_verif_build_tests. Fixed pseudo random invalid range check. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib538b5999444d67704ebc322be3c7becbe5ad206
2023-11-02Main compliance testing support for MULJeremy Johnson
Update verify ULP mode to allow fractions (e.g. 0.5). Update pseudo generator to accept ranges. Fix up pseudo random distribution based on ranges. Change-Id: I9168c5f7d37722678c0f1f9e906953c8cec367b1 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-11-02Compliance mode testing for CONV2DJeremy Johnson
Added CONV2D data generation. Updated verify dot product check to latest specification. Updated test generator and python datagenerator library to create const files during test generation. Add support for compliance test sets to conformance test_select. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5be3b761a1e3ef259c058e493877cd5a89d5778b
2023-10-31Allow more integer types in result check.Jeremy Johnson
Added int8, int16, uint8 and uint16. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6b1174aa7369c34a613b322c4b05bf4fc586ee6e
2023-10-26Compliance testing support for MAX_POOL2D & PADJeremy Johnson
Added Pseudo Random number generator in generate library. Enabled MAX_POOL2D, PAD FP32 tests to use new generator and compliance. Fixed verify library exact mode to expect reference data as FP64. Simplified tosa_verif_build_tests internal interfaces for new tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icc0ffa924cf38107c3a212efd452c47a650c9d98
2023-10-25Improve verfiy and generate library validationJeremy Johnson
Improved libraries validation to catch unknown values. Improved verify output to match generate library. Refactored generate tests to use less code duplication. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9c38745fbc8e70f46c19ddae6c62ee248d33b5f1
2023-10-24Improve conformance backward version supportJeremy Johnson
Turn off ARGMAX rank 6 tests for v0.60.0 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I535b7887a9db099d39ad943e151f870ebc716791
2023-10-16Data generator library python interface addedJeremy Johnson
Added support for using generate library in tosa_verif_build_tests and tosa_verif_run_tests tosa tool scripts. Reduced scope of compliance test creation and verification to the supported type of FP32. Fix missing virtual destructor warning in generate_dot_product.h and add config file for generate library. Simple pytests included to check python interface. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6cdad9b00660d6ddc8bd07fdea813937fb48626a
2023-10-12Conformance simple backward version and binary files supportJeremy Johnson
Add --test-version option to tosa_verif_conformance_generator to select the version for tests to output. Add --output-type to allow json, binary or both files to be created during conformance generation. Fix passing schema_path to test runner. Add explicit verify lib path arg to test runner. Change-Id: I5f1ad137d713fca408a98470ea77bddf8916c5f3 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-10-02Support for compliance checking testingJeremy Johnson
Updated to conformance generator to not generate tests with results for compliance tests. Updated test runner to run compliance mode version (precise & abs mode) of reference model to create test results to use against SUT results. Updated reference model to enable abs_mode on correct desc.json flags. Updated test checker to support compliance checking using verifier lib. Seperated color printing from test checker. Change-Id: I7e2fbfc6883916caa5d94d4ece122c48bf45f530 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-10-02Update verifier library data-type supportJeremy Johnson
Make compliance meta-data data-type required for all. Add data-type checking for all verifier modes. Add initial enum support for new ROUND compliance mode. Improve print out information from library. Use numpy ctypes.data_as to get f16 support compared to ctypes_lib. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie983ba4ea958a88556f30c09b3ebc19cd9ec96b7
2023-09-28Refactor path arguments to tosa-toolsJeremy Johnson
tosa_verif_conformance_generator - Move to using ref-model-path instead of ref-model-dir - Add schema-path and flatc-path - Add model_files module to locate default places for files convert2conformance - Remove default paths verifier - Switch to using exact path of verifier library tosa_verif_run_tests - Use conformance model_files to locate defaults Change-Id: Ieca3b164670e2a7dcb047743667cc4e8317daa97 Signed-off-by: Josef Malmström <josef.malmstrom@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-09-28Add 0-rank tensor support for concat in framework testWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iff77091e4a57f487431ffbf7ac1c89301a153c8b
2023-09-15[reference_model] Support StatefulOps and the tests for CallOnceOpTai Ly
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I03cb878736ccd7e1f5e1f780d7171949a19a9de2
2023-09-13Build compliance verifier library with a python interface.Jeremy Johnson
Build steps to create tosa_reference_verify_lib.so New python class to handle loading and calling verifier with pytests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id42c71c398ed7364742b4a039977412ef3081c4a
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-11Fixes for conformance generator tags and test selectionJeremy Johnson
Incorrectly used "tag" in test.desc instead of "tags" Incorrectly selected all tests rather than just all positive tests depending on operator selection option. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib257bc57accf745beedfa4a3b2ece9539c0972c2
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-09-07Add desc.json schema validation with compliance/data_gen extensionsJeremy Johnson
Rename scripts utlities to tosa_* such as tosa_json2numpy Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie8d584e2afb189fb74cf96b39590c7c27444ba14
2023-09-01[reference model] Fix dim test generationv0.90a0Tai Ly
In tosa_test_gen.py, change output_shape for dim operators to [] instead of [1] Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I8e7901e02831e9a467bce17e55e9379cb5f1b739
2023-08-30Fix conformance regressions due to type and region changes.Jeremy Johnson
Compress flatbuffer JSON files for conformance to allow while_loop tests to fit in the 30MB file size limit. Do not include new SHAPE type in usable DTypes so that selection of ERROR_IF tests are not changed. Also enable strict json mode for flatbuffers. Change-Id: Id89a9963244c34769a4662f9951adde0aa3334d9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-08-16Remove `"` from the search pattern `"tosa.*"`TatWai Chong
The double quotation mark is not always appear in the assembly output of operations. Remove it to easier capture general printing pattern. Change-Id: I59301daa07abd905de7259f8ae527a74b1ce5a6c Signed-off-by: TatWai Chong <tatwai.chong@arm.com>
2023-08-08Re-write json2numpy testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I17227adfca203f353e51ae9297f7e01f8d7edbe9
2023-08-07Deprecation of parts of TOSA conformanceJeremy Johnson
* Add deprecation notice for conformance_tests repo * Add deprecation notice for framework tests in conformance Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92b17f4d6d993e47f30d7996a18fc13b934c4cdf
2023-08-02Update Framework compiler scriptEric Kunze
Reuse the name deduced from the test run in results. Also move most string formatting to f strings. Move path handling to pathlib from os.path Signed-off-by: Eric Kunze <eric.kunze@arm.com> Change-Id: I44c7786d4b7af44306e218ff49608df35d4521a7
2023-07-31Update serialization library submoduleJeremy Johnson
* Update Flatbuffers version * Allows consts to be saved as inputs Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ibb9e38d381f8229bfca65b559e35c9ded889c384
2023-07-26Update framework tools to add name in jsonEric Kunze
Name of the test can now come from the json description rather than inferred from the directory. Old method remains to keep functionality with older tests Change-Id: I79f872103ae6a101b5cc799af91c933839f28dfb
2023-07-26Add Verif Conformance READMEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If30b329905aad9e520e4a9a3ac1dffccd37bb0e9
generated by cgit v1.2.1 (git 2.23.0) at 2024-06-26 14:32:14 +0000