aboutsummaryrefslogtreecommitdiff
path: root/verif/conformance
AgeCommit message (Collapse)Author
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-11Fixes for conformance generator tags and test selectionJeremy Johnson
Incorrectly used "tag" in test.desc instead of "tags" Incorrectly selected all tests rather than just all positive tests depending on operator selection option. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib257bc57accf745beedfa4a3b2ece9539c0972c2
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-08-07Deprecation of parts of TOSA conformanceJeremy Johnson
* Add deprecation notice for conformance_tests repo * Add deprecation notice for framework tests in conformance Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92b17f4d6d993e47f30d7996a18fc13b934c4cdf
2023-07-26Add Verif Conformance READMEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If30b329905aad9e520e4a9a3ac1dffccd37bb0e9
2023-06-29Add support for ERF operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib42b867287b83a183a0d0fb1f1eb29974f58fae4
2023-05-17Add support for one dimension of size -1 in ReshapeOpJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I0ef7607f4266296a1204c5cccdb5be36f345b5ba
2023-05-168K level: rank 6 ARGMAX testsJeremy Johnson
Change-Id: I81097ae263a64ee8e35b8bd798598671ac089c3f Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-04-268K levels: Tensor op tests kernel/stride at 8192 maximumsJeremy Johnson
Operators updated: AVG_POOL2D, MAX_POOL2D, CONV2D, CONV3D, DEPTHWISE_CONV2D & TRANSPOSE_CONV2D tosa_verif_build_tests argument --level-8k-sizes used to allow kernel/stride maximum boundary testing Fixed bugs in height/width validator function meaning some esixting avg_pool2d float tests need regening. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7aeab82d3bd3c49d02d54708f2c9d995cd3cf2df
2023-04-258K level: RESIZE scaling at 64 maximumJeremy Johnson
Add support for producing scaling 64 tests in RESIZE and make sure default tests are not generated with scaling greater than 64. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e99cde7e6bec6ea7854a139c6fc2ed151af5b1e
2023-04-118K level: rank 6 tests support using generator groupsJeremy Johnson
Split generation of conformance tests into groups to avoid a change in existing tests selection * A standard group is used for the normal/default conformance tests * An 8k level group covers the extension of testing for 8k level * Other arbitrary groups can be created and the group names will be used to tag the tests, so that in the future they could be filtered Group selection criteria is in a new dictionary * A default selection config is used for generation groups * But a different config can be used for a particular generation group Added parameter sparsity for pad of rank6 and above, and further parameter sparsity options to test_select to limit number of tests selected. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib9f1185c0ae6d0957df2116411c17e958157c6d3
2023-03-28Add CONST INT48 support to conformance.Jeremy Johnson
Remove ERRORIF tests for CONST as they do not create valid negative tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia96b184f34010a452d2c4848b0653514352c751e
2023-03-13Update framework test choices - SPLIT, RESIZEv0.60.2Jeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ifce52ef0e082e9e6289938baa6c73a6ebd9ded62
2023-03-01CONV2D/CONV3D/DEPTHWISE_CONV2D Test Regen for ofm_depth bugfixJames Ward
* Restricting size of tensors due to expanded values of ofm_depth Signed-off-by: James Ward <james.ward@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I582d8f0073c2d01244976f8e2e14783b506d8091
2023-02-22Create MI tests for Tensors: TRANSPOSE_CONV2DJames Ward
Change-Id: I4f967ce2dfb0cb44dc58307170fd9306a0ea1ed5 Signed-off-by: James Ward <james.ward@arm.com>
2023-02-22Bug fixes for max-batch-size/ofm-depthJames Ward
* Conformance regen required for tosa-bi: conv2d conv3d depthwise_conv2d transpose_conv2d * Include bug fix for testGen.makeShape() * Include json changes to limit size of tests which now have ofm_depth > 1 Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ic8221b9a742f5737df523997bee7260f4dfcdef3
2023-02-22Create MI tests for Tensor: FFT2D & RFFT2DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I44ad6513fad831ff50fcc5ab801b46a5bf25d0a8
2023-02-22Create MI tests for Tensor: CONV3DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I2c1e87ec586195f82e97bbc70e4e2bc18cd2cc88
2023-02-22Create MI tests for Tensors: DEPTHWISE_CONV2DJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I6c2087c94c2c024aabc2086d40cbca9c719c69dd
2023-02-14Select new TFLite SPLIT testsJeremy Johnson
Due to change in generator creating different test patterns. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I58224dc117c766b22fccd418f64004aea31905b5
2023-02-07Adjust ERROR_IF tests generated for tosa-mi profileJeremy Johnson
So they do not overwrite tosa-bi tests, ops affected: NEGATE, TILE, REDUCE_MIN, REDUCE_MAX, RESIZE, SCATTER, GATHER Update test-select exclude-patterns to work with ERRORIF tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Idb0d26512ce9a48f6d868663a630e343c8785899
2023-02-06Create MI tests for Tensors: AVG_POOL2D, FULLY_CONNECTED, MATMULJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I874c87745b929f9808a5e45490042420a57b22f7
2023-02-06Create MI tests for Tensor: CONV2DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Iefd8490d9eedc76d199d36943b87c99e4dd97612
2023-02-02Create MI tests for Reduction: REDUCE_SUM, REDUCE_PRODUCTJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I21b8f8139ce069d5fc6bbeabe292e3a44d99fe10
2023-01-31Create MI tests for Type Conversion: CASTJames Ward
* Add exclusion regex's to conformance generation Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I15bef7451efd5662065060242d35bd7fa3381487
2023-01-25Create MI tests for Comparison: EQUAL, GREATER, GREATER_EQUALJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I4d4834260698db75427590c0a8b80cc3c824ef3b
2023-01-25Create MI tests for EW Unary: LOG, RECIPROCAL, RSQRT, EXPJeremy Johnson
Also fix the ERROR_IF conformance test generator to apply correct operator filtering to create the tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5788932e3529ec9386900623b20c22c98e4201b7
2023-01-23Create MI tests for Data Layout: SLICEJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ief60ae8fa490ded21aa77726d738565be5f8d91c
2023-01-19Create MI tests for Activation: CLAMP; Data Layout: PADJames Ward
* Existing float attributes now serialized as bytes Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I415276706b9daf0893e3a59189f387f872ff07c2
2023-01-04Create MI tests for EW Binary: ADD, SUB, MULJeremy Johnson
Add option to control negative tests dim range to stop ERROR_IF test clashes between profiles. Change-Id: I1f806934480cc32b95e80613c66fecb1e6ff5426 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-01-04Create MI tests for EW Binary: POWJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8d478b7bd13559ea37e27f77b05ee5db56c8f9e3
2022-12-15Create MI tests for Image: RESIZEJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I440590fa59433b9ddc1934188f2b34b377129c1f
2022-12-15Create MI tests for Activation: SIGMOID, TANHJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I75d6280ecb5219d50c91d14029e90ff3a2f6eb06
2022-12-14Create MI tests for EW Ternary: SELECTJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I1eb4fbadb9117e28f87a36ee828d65d896d834f0
2022-12-13Create MI tests for EW Unary: ABS, CEIL, FLOOR, NEGATEJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I2ee6d4f706958b6261fd41dd245a3d01d3f8cecf
2022-12-13Create MI tests for Reduction: REDUCE_MIN, REDUCE_MAXJames Ward
* Also include test_select improvements Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I306a8ed57975bc429d6cd5a8a24777b889593ddf
2022-12-09Create MI tests for EW Binary: MINIMUM, MAXIMUMJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I6824e30cece291fceaedd727132a6b211762baa9
2022-12-01Fix conformance desc.json contains correct profilesJeremy Johnson
Must seperate conformance generation by profile to stop desc.json being incorrectly overwritten by different profile information. Change-Id: Iead01c9fcc659e48ef1d350a860b03c2b0a05ffa Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2022-11-21Create MI tests for Data Nodes: CONST, IDENTITYJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I7280b3bf31aa377a6f6af7713627f34eb445e689
2022-11-21Create MI tests for Data Layout: CONCAT, PAD, RESHAPE, REVERSE, SLICE, TILE, ↵James Ward
TRANSPOSE Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I626d9ece0290c5104c9ac42d20be2f9db1e55fa7
2022-11-21Create MI tests for Scatter/Gather: SCATTER, GATHERJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I44c0843fa049546582ca1111795342d5035a329c
2022-11-21Create MI tests for Tensors: ARGMAX, MAX_POOL2DJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I908d88ddf7c52d98ce6d533396b5c439f15750f5
2022-11-19Initial set up of Main Inference conformance test genJeremy Johnson
tosa-verif-build-tests - option for setting FP values range - option for recursively finding tests - change from os.path to Path tosa_verif_result_check - option to supply FP tolerance - output difference and max tolerance on contents mismatch - change from os.path to Path MI conformance - contains examples of AVG_POOL2D and CONV2D tests Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8e1645cd8f10308604400ea53eef723ca163eed7
2022-10-26Add TILE bool tests & rename CAST/RESCALE testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If9639a94c8e7322d470a1ca97e9e95d821683629
2022-10-13Minor fixes & add FP16 support to refmodel testing and conformance genJeremy Johnson
Improve base inference conformance generation: * Change to target specific dtypes required for conformance tests. * Reduce dimension sizes of ERROR_IF tests. NOTE: Will impact tensor, clamp and all ERROR_IF tests. Add option to change seed on conformance generation for extra testing. Stop creation of convolution tests with negative output dimensions. Improve reporting on failing to allocate tensor due to above issue. Fix runner to correctly pass ref model debug flags. Update reference_model examples. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I992180dcfe265a7d50edfb151c9f38eeaef5c369
2022-08-30Update RESCALE testing to match no rank restrictionsEric Kunze
Matches changes to the specification loosening the rank requirements for the RESCALE operation. Includes removing wrong rank testing for RESCALE, and increasing conformance testing to rank 5 Signed-off-by: Eric Kunze <eric.kunze@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I4e15d4bd8f448e1ec81d9a8bbcd231c286b64a30
2022-08-11Update RESIZE operator in test generator for spec updatesJeremy Johnson
* Add common screen aspect ratios with borders to random pool of tests * Add up/downscaling to random pool Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Iee8e3f5ed6bd5c941816474df20a7fd433646d6b Signed-off-by: James Ward <james.ward@arm.com>
2022-07-14Add profile to all created conformance tests.Jeremy Johnson
Add supported profiles for each test in convert2conformance and tosa_verif_conformance_generator. Enable filtering of profile tests on running in tosa_verif_run_tests. Reorganize arguments in conformance_generator to have more important ones first. Change-Id: Ie6e5b68727adb3c39b04aa482dd6433788f7bcc9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2022-07-12Add conformance generator scriptsJeremy Johnson
Change-Id: I5cb16ea1f47ee454f03a1d5182827e3fd9f49128 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
generated by cgit v1.2.1 (git 2.23.0) at 2024-07-13 01:08:08 +0000