aboutsummaryrefslogtreecommitdiff
path: root/verif/generator
AgeCommit message (Collapse)Author
2023-10-25Improve verfiy and generate library validationJeremy Johnson
Improved libraries validation to catch unknown values. Improved verify output to match generate library. Refactored generate tests to use less code duplication. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9c38745fbc8e70f46c19ddae6c62ee248d33b5f1
2023-10-16Data generator library python interface addedJeremy Johnson
Added support for using generate library in tosa_verif_build_tests and tosa_verif_run_tests tosa tool scripts. Reduced scope of compliance test creation and verification to the supported type of FP32. Fix missing virtual destructor warning in generate_dot_product.h and add config file for generate library. Simple pytests included to check python interface. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6cdad9b00660d6ddc8bd07fdea813937fb48626a
2023-10-02Update verifier library data-type supportJeremy Johnson
Make compliance meta-data data-type required for all. Add data-type checking for all verifier modes. Add initial enum support for new ROUND compliance mode. Improve print out information from library. Use numpy ctypes.data_as to get f16 support compared to ctypes_lib. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie983ba4ea958a88556f30c09b3ebc19cd9ec96b7
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-09-01[reference model] Fix dim test generationv0.90a0Tai Ly
In tosa_test_gen.py, change output_shape for dim operators to [] instead of [1] Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I8e7901e02831e9a467bce17e55e9379cb5f1b739
2023-08-30Fix conformance regressions due to type and region changes.Jeremy Johnson
Compress flatbuffer JSON files for conformance to allow while_loop tests to fit in the 30MB file size limit. Do not include new SHAPE type in usable DTypes so that selection of ERROR_IF tests are not changed. Also enable strict json mode for flatbuffers. Change-Id: Id89a9963244c34769a4662f9951adde0aa3334d9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-07-31Update serialization library submoduleJeremy Johnson
* Update Flatbuffers version * Allows consts to be saved as inputs Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ibb9e38d381f8229bfca65b559e35c9ded889c384
2023-06-29Add support for ERF operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib42b867287b83a183a0d0fb1f1eb29974f58fae4
2023-06-15Add ERROR_IF to incorrect broadcast shapesJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I7460ad9eed3ed5c7cec6e855a0303753ed28eb1c
2023-05-17Add support for one dimension of size -1 in ReshapeOpJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I0ef7607f4266296a1204c5cccdb5be36f345b5ba
2023-05-10Refactor ref_model rank checking and add level check to argmaxJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Iad035b31d5e5e83040068e6311501490765bfff7
2023-04-268K levels: Tensor op tests kernel/stride at 8192 maximumsJeremy Johnson
Operators updated: AVG_POOL2D, MAX_POOL2D, CONV2D, CONV3D, DEPTHWISE_CONV2D & TRANSPOSE_CONV2D tosa_verif_build_tests argument --level-8k-sizes used to allow kernel/stride maximum boundary testing Fixed bugs in height/width validator function meaning some esixting avg_pool2d float tests need regening. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7aeab82d3bd3c49d02d54708f2c9d995cd3cf2df
2023-04-258K level: RESIZE scaling at 64 maximumJeremy Johnson
Add support for producing scaling 64 tests in RESIZE and make sure default tests are not generated with scaling greater than 64. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e99cde7e6bec6ea7854a139c6fc2ed151af5b1e
2023-04-118K level: rank 6 tests support using generator groupsJeremy Johnson
Split generation of conformance tests into groups to avoid a change in existing tests selection * A standard group is used for the normal/default conformance tests * An 8k level group covers the extension of testing for 8k level * Other arbitrary groups can be created and the group names will be used to tag the tests, so that in the future they could be filtered Group selection criteria is in a new dictionary * A default selection config is used for generation groups * But a different config can be used for a particular generation group Added parameter sparsity for pad of rank6 and above, and further parameter sparsity options to test_select to limit number of tests selected. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib9f1185c0ae6d0957df2116411c17e958157c6d3
2023-03-28Add CONST INT48 support to conformance.Jeremy Johnson
Remove ERRORIF tests for CONST as they do not create valid negative tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia96b184f34010a452d2c4848b0653514352c751e
2023-02-28Update rank limits for SLICE, TILE and TRANSPOSELuke Hutton
Updated to align with corresponding changes to the spec. In addition, some ERROR_IF tests have been updated to match the checks specified by the spec, including: PAD, SLICE, TILE, TRANSPOSE. Signed-off-by: Luke Hutton <luke.hutton@arm.com> Change-Id: Ie2c5f48e79a5610eb82739170e25057a63dac1d8
2023-02-22Bug fixes for max-batch-size/ofm-depthJames Ward
* Conformance regen required for tosa-bi: conv2d conv3d depthwise_conv2d transpose_conv2d * Include bug fix for testGen.makeShape() * Include json changes to limit size of tests which now have ofm_depth > 1 Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ic8221b9a742f5737df523997bee7260f4dfcdef3
2023-02-22Add tests for INT48 CONST supportLuke Hutton
* Add TOSA test coverage * Add some basic error checking Signed-off-by: Luke Hutton <luke.hutton@arm.com> Change-Id: Ib02df7319954a2dcb915e5f5a70699ca398fd483
2023-02-10Add FFT2d to the reference modelLuke Hutton
Includes: * FFT2d reference implementation * Basic TOSA tests Change-Id: Ie79fcb713542345d550ec013646810c1e890e388 Signed-off-by: Luke Hutton <luke.hutton@arm.com>
2023-02-02Remove accumulator attributes from all but AVG_POOL2DJames Ward
Signed-off-by: James Ward <james.ward@arm.com> Change-Id: If67f503a1848967bc1671646c3011d055b622c52
2023-01-31Create MI tests for Type Conversion: CASTJames Ward
* Add exclusion regex's to conformance generation Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I15bef7451efd5662065060242d35bd7fa3381487
2023-01-24Add RFFT2d to the reference modelLuke Hutton
Includes: * RFFT2d reference implementation * TFLite framework tests * Basic TOSA tests * Serialization submodule upgrade with support for FFT/RFFT Signed-off-by: Luke Hutton <luke.hutton@arm.com> Change-Id: I2a687e9cf87fb62a26160ea52439ba9830bea36e
2023-01-19Create MI tests for Activation: CLAMP; Data Layout: PADJames Ward
* Existing float attributes now serialized as bytes Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I415276706b9daf0893e3a59189f387f872ff07c2
2023-01-13Reference model update for control flow operators supportJerry Ge
Rationale for making this change: - In the original design, for control flow operators like WhileOp, child blocks couldn't read the tensor variables (global consts) in the root level block, this patch added the machanism for child blocks to access their parent level block's tensors. - This change also relies on another serialization change on adding another layer of abtraction called Region: - Serialization patch: [region] Add TosaSerializationRegion to serialization_lib - Updated the corresponding python version of the serialization code: TosaSerializerRegion to python version of serialization_lib - This change also relies on the TOSA MLIR Translator change: Add RegionBuilder to TOSA MLIR Translator - Added the WhileOp related test cases: While, LSTM, GRU, RNN - Other related fixes Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I13ae33628ad07e41d248e88652ce1328654694ab
2022-12-15Add extra control flow ERROR_IF testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7276dc686d8d18ba44663b73e35ceca2a1cbaadf
2022-11-21Create MI tests for Data Layout: CONCAT, PAD, RESHAPE, REVERSE, SLICE, TILE, ↵James Ward
TRANSPOSE Signed-off-by: James Ward <james.ward@arm.com> Change-Id: I626d9ece0290c5104c9ac42d20be2f9db1e55fa7
2022-11-19Initial set up of Main Inference conformance test genJeremy Johnson
tosa-verif-build-tests - option for setting FP values range - option for recursively finding tests - change from os.path to Path tosa_verif_result_check - option to supply FP tolerance - output difference and max tolerance on contents mismatch - change from os.path to Path MI conformance - contains examples of AVG_POOL2D and CONV2D tests Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8e1645cd8f10308604400ea53eef723ca163eed7
2022-11-09Add BF16 support to reference modelJames Ward
* Upgrade Eigen to 3.4.0 (for bfloat16 support) and add work- arounds for reduce.any() and reduce.all() bugs (introduced between 3.3.7 and 3.4.0) * Truncation to bfloat16 now performed in eval() methods Signed-off-by: James Ward <james.ward@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If5f5c988d76d3d30790acf3b97081726b89205fe
2022-10-26Add TILE bool tests & rename CAST/RESCALE testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If9639a94c8e7322d470a1ca97e9e95d821683629
2022-10-13Rename FLOAT type to FP32Jeremy Johnson
Update tensor operations naming to state input type as TxT in all cases. Effects CONV2D, CONV3D, DEPTHWISE_CONV2D, FULLY_CONNECTED, TRANSPOSE_CONV2D. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ic959acfcb3aa0a910b33b774a5a85fac08219205
2022-10-13Minor fixes & add FP16 support to refmodel testing and conformance genJeremy Johnson
Improve base inference conformance generation: * Change to target specific dtypes required for conformance tests. * Reduce dimension sizes of ERROR_IF tests. NOTE: Will impact tensor, clamp and all ERROR_IF tests. Add option to change seed on conformance generation for extra testing. Stop creation of convolution tests with negative output dimensions. Improve reporting on failing to allocate tensor due to above issue. Fix runner to correctly pass ref model debug flags. Update reference_model examples. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I992180dcfe265a7d50edfb151c9f38eeaef5c369
2022-10-11Reference model changes for fp16 supportJames Ward
Change-Id: I72f21fcfa153046274969d327313e3349981dbe6 Signed-off-by: James Ward <james.ward@arm.com>
2022-09-26Add CONST testing to Numpy refmodel testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id32f3b7c287c9e8b0fd42dd96922be5a255598e2
2022-09-21Add simple post commit reference model testing against NumpyJeremy Johnson
Only perform testing of this after build of ref model using postcommit pytest mark. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I771a18d2c9cd4051fecafad3e6079b44f2ed62fa
2022-08-30Update RESCALE testing to match no rank restrictionsEric Kunze
Matches changes to the specification loosening the rank requirements for the RESCALE operation. Includes removing wrong rank testing for RESCALE, and increasing conformance testing to rank 5 Signed-off-by: Eric Kunze <eric.kunze@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I4e15d4bd8f448e1ec81d9a8bbcd231c286b64a30
2022-08-26Align padding for transpose_conv2d to match specEric Kunze
Increasing out pad values now leads to increasing pad. Reference model changes, and test generator changes to match specification definition Change-Id: I4f3ebfbca5048354fb15bedc7ab640ff28ed853a Signed-off-by: Eric Kunze <eric.kunze@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2022-08-25Add PAD ERROR_IF test for output shapeJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I25a13540734fa30c0c21b46708dfabbec8c4d1e5
2022-08-24Enabled 16-bit TABLE REQUIRE statementJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib6e81814e022f33e45430e47ca99d6d9f9e0e101
2022-08-11Update RESIZE operator in test generator for spec updatesJeremy Johnson
* Add common screen aspect ratios with borders to random pool of tests * Add up/downscaling to random pool Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Iee8e3f5ed6bd5c941816474df20a7fd433646d6b Signed-off-by: James Ward <james.ward@arm.com>
2022-07-11Modify input limits for apply_scale_32Eric Kunze
Aligns with change in specification, as the limit to value being between (-1 << shift-2) and (1 << shift-2) was overly constraining. Signed-off-by: Eric Kunze <eric.kunze@arm.com> Change-Id: I72cc5743344d4036920c3df366e0a3930701a0cb
2022-06-15Remove quantization info from serialization attributesEric Kunze
Any needed information moves into the attributes for each operator. New serialization library version removes teh quantization information attributes from the schema Signed-off-by: Eric Kunze <eric.kunze@arm.com> Change-Id: Icf6165687ab1fd34a01f64c01b0b92b2820e72fa
2022-06-13Update transpose_conv2d to align with TOSA specTatWai Chong
Rename outpad to out_pad, and also fix the dilation in the generator. Change-Id: I4c1599871f0d0b41856e819d8c644a85ca6d8267 Signed-off-by: TatWai Chong <tatwai.chong@arm.com>
2022-05-31Remove RESHAPE -1 dimensions supportJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I098daf49c92da12c07143cdd23ac9bb58acebbb9
2022-05-26Add support for uint16_t to RESCALEJeremy Johnson
Update ref-model RESCALE op to support UINT16 conversions Add testing for RESCALE UINT16 and ERROR_IFs Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ic6e6e53de1f0b054bedb9e6ba3856e7475498aba
2022-05-04Fix for NEGATE using 32-bit accumulatorJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie5d119dc317303a0d2a71d018ac94ce6800ecbf5
2022-05-04Control oversized tests for TRANSPOSE_CONV2DJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I464871b412b3dfc9a84bd3d8fffe4c1de3072b69
2022-04-28Update tensor ops ERROR_IF criteriaJeremy Johnson
Update to ref model to check ERROR_IF criteria for pooling and convolution ops to match specification Update to tosa_verif_build_tests to produce valid test ranges and new ERROR_IF tests Plus update pooling ops big kernel to 9 (from 6) for better testing coverage and set dilation to 1 and add out_pad bottom & right for transpose_conv2d to match specification Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ic5759872d40ae8d3f3d07043d9a0f2fa0244d72e
2022-04-11Refactor verif/generator/tosa_test_gen.py into different filesJeremy Johnson
Move all error & validation into tosa_error_if.py Move all argument and tensor generation into tosa_arg_gen.py Move utility functions into tosa_utils.py Create new TosaTensorValuesGen class for specialising tensor value generation. Change-Id: Ib9ac65e2308b14471a567c6f11d775c76585bc5b Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
generated by cgit v1.2.1 (git 2.23.0) at 2024-07-08 06:49:55 +0000