aboutsummaryrefslogtreecommitdiff
path: root/verif/conformance
AgeCommit message (Collapse)Author
2023-11-30Main Compliance testing support for CASTJeremy Johnson
Limit CAST input tensor to maximums of output type to avoid saturation and infinity. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I33350a4ce0ec828da7d2e7aa8cd3183a89a97431
2023-11-30Adjust random data ranges for Main Compliance to avoid FP inf and nanJeremy Johnson
POW - there are now 3 test sets to cover random ranges. Also added ROUND mode to data generator to force integer exponent values. LOG, EXP, RSQRT, REDUCE_SUM & FULLY_CONNECTED - have had their ranges reduced for each test. Fix generate library configuration defaults and checks. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie5d3bd78f690cc787a2ca4eb9b4bd6808bd9238c
2023-11-23Main Compliance testing support for COMPARISON opsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id6229cfaccad866b110630119eb045dbf6453bf5
2023-11-23Main Compliance testing support for LOG and ACTIVATIONsJeremy Johnson
Increase exp2 allowed range to account for denormals. Minor adjustments to verify to match spec updates for pseudo code. Set ranges of activation function inputs to match spec. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6fcf665932ac2c9080e284b865da8f7746801f59
2023-11-23Main Compliance testing support for AVG_POOL2DJeremy Johnson
Fix test set 4 to have more symmetrical data set for AVG_POOL2D. Increased size of AVG_POOL2D test shapes to meet MIN_DOT_PRODUCTS. Print out sign of bias magnitude in verify library to help debugging. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icf139cf5788cbc68f2c537567bdfb4cfe31274df
2023-11-16Main Compliance testing support for EXP & POWJeremy Johnson
Added new ABS_ERROR mode to verify lib and ref model. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ifb78290675833d3df7df91a4d6cef336b02b64a4
2023-11-16Main Compliance testing for simple UNARY opsJeremy Johnson
For RECIPROCAL, RSQRT, CEIL, FLOOR, ABS, NEGATE & IDENTITY. Improved ULP informational output. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I49644573b4c9a30b2b9d6c9624f2a1d46976a378
2023-11-10Main Compliance testing for FULLY_CONNECTEDJeremy Johnson
Updated shapes to meet MIN_DOT_PRODUCTS. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I82297917c009b3120306f8a9bb965209d109ef8d
2023-11-10Main Compliance testing support for ARGMAX, REDUCE_SUM/MAX/MINJeremy Johnson
Add extra tests for FP32 REDUCE_SUM that meet MIN_DOT_PRODUCTS. Plus improved dot product test generation skip information. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia8198a9500ddddfc86c5bb84230b9a4edf5ffd50
2023-11-10Main Compliance testing support for ADD, SUB, MINIMUM and MAXIMUMJeremy Johnson
Added main inference compliance mode data generation and verification. Improved error reporting in tosa_verif_build_tests. Fixed pseudo random invalid range check. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib538b5999444d67704ebc322be3c7becbe5ad206
2023-11-02Main compliance testing support for MULJeremy Johnson
Update verify ULP mode to allow fractions (e.g. 0.5). Update pseudo generator to accept ranges. Fix up pseudo random distribution based on ranges. Change-Id: I9168c5f7d37722678c0f1f9e906953c8cec367b1 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-11-02Compliance mode testing for CONV2DJeremy Johnson
Added CONV2D data generation. Updated verify dot product check to latest specification. Updated test generator and python datagenerator library to create const files during test generation. Add support for compliance test sets to conformance test_select. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5be3b761a1e3ef259c058e493877cd5a89d5778b
2023-10-26Compliance testing support for MAX_POOL2D & PADJeremy Johnson
Added Pseudo Random number generator in generate library. Enabled MAX_POOL2D, PAD FP32 tests to use new generator and compliance. Fixed verify library exact mode to expect reference data as FP64. Simplified tosa_verif_build_tests internal interfaces for new tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icc0ffa924cf38107c3a212efd452c47a650c9d98
2023-10-24Improve conformance backward version supportJeremy Johnson
Turn off ARGMAX rank 6 tests for v0.60.0 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I535b7887a9db099d39ad943e151f870ebc716791
2023-10-16Data generator library python interface addedJeremy Johnson
Added support for using generate library in tosa_verif_build_tests and tosa_verif_run_tests tosa tool scripts. Reduced scope of compliance test creation and verification to the supported type of FP32. Fix missing virtual destructor warning in generate_dot_product.h and add config file for generate library. Simple pytests included to check python interface. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6cdad9b00660d6ddc8bd07fdea813937fb48626a
2023-10-12Conformance simple backward version and binary files supportJeremy Johnson
Add --test-version option to tosa_verif_conformance_generator to select the version for tests to output. Add --output-type to allow json, binary or both files to be created during conformance generation. Fix passing schema_path to test runner. Add explicit verify lib path arg to test runner. Change-Id: I5f1ad137d713fca408a98470ea77bddf8916c5f3 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-10-02Support for compliance checking testingJeremy Johnson
Updated to conformance generator to not generate tests with results for compliance tests. Updated test runner to run compliance mode version (precise & abs mode) of reference model to create test results to use against SUT results. Updated reference model to enable abs_mode on correct desc.json flags. Updated test checker to support compliance checking using verifier lib. Seperated color printing from test checker. Change-Id: I7e2fbfc6883916caa5d94d4ece122c48bf45f530 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-09-28Refactor path arguments to tosa-toolsJeremy Johnson
tosa_verif_conformance_generator - Move to using ref-model-path instead of ref-model-dir - Add schema-path and flatc-path - Add model_files module to locate default places for files convert2conformance - Remove default paths verifier - Switch to using exact path of verifier library tosa_verif_run_tests - Use conformance model_files to locate defaults Change-Id: Ieca3b164670e2a7dcb047743667cc4e8317daa97 Signed-off-by: Josef Malmström <josef.malmstrom@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-11Fixes for conformance generator tags and test selectionJeremy Johnson
Incorrectly used "tag" in test.desc instead of "tags" Incorrectly selected all tests rather than just all positive tests depending on operator selection option. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib257bc57accf745beedfa4a3b2ece9539c0972c2
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-08-07Deprecation of parts of TOSA conformanceJeremy Johnson
* Add deprecation notice for conformance_tests repo * Add deprecation notice for framework tests in conformance Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92b17f4d6d993e47f30d7996a18fc13b934c4cdf
2023-07-26Add Verif Conformance READMEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If30b329905aad9e520e4a9a3ac1dffccd37bb0e9
2023-06-29Add support for ERF operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib42b867287b83a183a0d0fb1f1eb29974f58fae4
2023-05-17Add support for one dimension of size -1 in ReshapeOpJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I0ef7607f4266296a1204c5cccdb5be36f345b5ba
2023-05-168K level: rank 6 ARGMAX testsJeremy Johnson
Change-Id: I81097ae263a64ee8e35b8bd798598671ac089c3f Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-04-268K levels: Tensor op tests kernel/stride at 8192 maximumsJeremy Johnson
Operators updated: AVG_POOL2D, MAX_POOL2D, CONV2D, CONV3D, DEPTHWISE_CONV2D & TRANSPOSE_CONV2D tosa_verif_build_tests argument --level-8k-sizes used to allow kernel/stride maximum boundary testing Fixed bugs in height/width validator function meaning some esixting avg_pool2d float tests need regening. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7aeab82d3bd3c49d02d54708f2c9d995cd3cf2df
2023-04-258K level: RESIZE scaling at 64 maximumJeremy Johnson
Add support for producing scaling 64 tests in RESIZE and make sure default tests are not generated with scaling greater than 64. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e99cde7e6bec6ea7854a139c6fc2ed151af5b1e
2023-04-118K level: rank 6 tests support using generator groupsJeremy Johnson
Split generation of conformance tests into groups to avoid a change in existing tests selection * A standard group is used for the normal/default conformance tests * An 8k level group covers the extension of testing for 8k level * Other arbitrary groups can be created and the group names will be used to tag the tests, so that in the future they could be filtered Group selection criteria is in a new dictionary * A default selection config is used for generation groups * But a different config can be used for a particular generation group Added parameter sparsity for pad of rank6 and above, and further parameter sparsity options to test_select to limit number of tests selected. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib9f1185c0ae6d0957df2116411c17e958157c6d3
2023-03-28Add CONST INT48 support to conformance.Jeremy Johnson
Remove ERRORIF tests for CONST as they do not create valid negative tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia96b184f34010a452d2c4848b0653514352c751e
2023-03-13Update framework test choices - SPLIT, RESIZEv0.60.2Jeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ifce52ef0e082e9e6289938baa6c73a6ebd9ded62
2023-03-01CONV2D/CONV3D/DEPTHWISE_CONV2D Test Regen for ofm_depth bugfixJames Ward
* Restricting size of tensors due to expanded values of ofm_depth Signed-off-by: James Ward <james.ward@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I582d8f0073c2d01244976f8e2e14783b506d8091
2023-02-22Create MI tests for Tensors: TRANSPOSE_CONV2DJames Ward
Change-Id: I4f967ce2dfb0cb44dc58307170fd9306a0ea1ed5 Signed-off-by: James Ward <james.ward@arm.com>
2023-02-22Bug fixes for max-batch-size/ofm-depthJames Ward
* Conformance regen required for tosa-bi: conv2d conv3d depthwise_conv2d transpose_conv2d * Include bug fix for testGen.makeShape() * Include json changes to limit size of tests which now have ofm_depth > 1 Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ic8221b9a742f5737df523997bee7260f4dfcdef3
2023-02-22Create MI tests for Tensor: FFT2D & RFFT2DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I44ad6513fad831ff50fcc5ab801b46a5bf25d0a8
2023-02-22Create MI tests for Tensor: CONV3DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I2c1e87ec586195f82e97bbc70e4e2bc18cd2cc88
2023-02-22Create MI tests for Tensors: DEPTHWISE_CONV2DJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I6c2087c94c2c024aabc2086d40cbca9c719c69dd
2023-02-14Select new TFLite SPLIT testsJeremy Johnson
Due to change in generator creating different test patterns. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I58224dc117c766b22fccd418f64004aea31905b5
2023-02-07Adjust ERROR_IF tests generated for tosa-mi profileJeremy Johnson
So they do not overwrite tosa-bi tests, ops affected: NEGATE, TILE, REDUCE_MIN, REDUCE_MAX, RESIZE, SCATTER, GATHER Update test-select exclude-patterns to work with ERRORIF tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Idb0d26512ce9a48f6d868663a630e343c8785899
2023-02-06Create MI tests for Tensors: AVG_POOL2D, FULLY_CONNECTED, MATMULJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I874c87745b929f9808a5e45490042420a57b22f7
2023-02-06Create MI tests for Tensor: CONV2DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Iefd8490d9eedc76d199d36943b87c99e4dd97612
2023-02-02Create MI tests for Reduction: REDUCE_SUM, REDUCE_PRODUCTJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I21b8f8139ce069d5fc6bbeabe292e3a44d99fe10
2023-01-31Create MI tests for Type Conversion: CASTJames Ward
* Add exclusion regex's to conformance generation Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I15bef7451efd5662065060242d35bd7fa3381487
2023-01-25Create MI tests for Comparison: EQUAL, GREATER, GREATER_EQUALJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I4d4834260698db75427590c0a8b80cc3c824ef3b
2023-01-25Create MI tests for EW Unary: LOG, RECIPROCAL, RSQRT, EXPJeremy Johnson
Also fix the ERROR_IF conformance test generator to apply correct operator filtering to create the tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5788932e3529ec9386900623b20c22c98e4201b7
2023-01-23Create MI tests for Data Layout: SLICEJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ief60ae8fa490ded21aa77726d738565be5f8d91c
2023-01-19Create MI tests for Activation: CLAMP; Data Layout: PADJames Ward
* Existing float attributes now serialized as bytes Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I415276706b9daf0893e3a59189f387f872ff07c2
2023-01-04Create MI tests for EW Binary: ADD, SUB, MULJeremy Johnson
Add option to control negative tests dim range to stop ERROR_IF test clashes between profiles. Change-Id: I1f806934480cc32b95e80613c66fecb1e6ff5426 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-01-04Create MI tests for EW Binary: POWJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8d478b7bd13559ea37e27f77b05ee5db56c8f9e3
generated by cgit v1.2.1 (git 2.23.0) at 2024-07-18 15:28:45 +0000