aboutsummaryrefslogtreecommitdiff
path: root/verif/generator
AgeCommit message (Collapse)Author
2024-01-25Save Int8/UInt8 reference outputs to native dtypesJerry Ge
* Int8/UInt8 reference outputs were previously saved to INT32 * Save those in their native dtypes and updated other affected code Signed-off-by: Jerry Ge <jerry.ge@arm.com> Georgios Pinitas <georgios.pinitas@arm.com> Change-Id: Id65fc8773150d3c56bc2c72789a6a0d3c78cd363
2024-01-24Add conformance testing for shape operatorsWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ie80570146601c470a3be7c04a9d6e1016a7c547c
2024-01-23Update RESHAPE and TILE conformance testingWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iaf59472cb32e03a92bad87ae9dba1b7548f20268
2024-01-18[reference model] Add shape operatorsTai Ly
- fixed up reshape conformance tests to use shape input instead of attribute - fixed up tile conformance tests to use shape input instead of attribute - fixed output and output rank of dim op - allow rank 0 and rank 1 tensors for tosa.shape values (for shape = {}) - added initialization of rank 0 const_shape tensors (for shape = {}) - Update conformance tests to use new rescale attributes Signed-off-by: Tai Ly <tai.ly@arm.com> Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: I6cce0d2a9ab066fe20a2abf9d2cfde3eb3d8c18b
2024-01-18Main Compliance testing for TILEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7f7449c652844097fec6226bcc32ca765b40bc04
2024-01-17Main Compliance testing for CONCATJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If08c560eac171124641dac66d7de9d5053db72df
2024-01-17Main Compliance testing for SELECTJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7276f2db39e67314c950e972cc1a97b7796dcd18
2024-01-08Main Compliance: REDUCE_PRODUCT supportJeremy Johnson
Update and fix REDUCE_PRODUCT compliance verify lib support. Added compliance test generation with data range to not cause infs. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3b3004c6caa80d97e330a6393f435f5270b56e21
2024-01-08Main Conformance: Re-adjust TANH compliance checkJeremy Johnson
Add lower bound to ABS ERROR checks to allow for cancellation of small values in error bounds checking. Re-adjust the error bounds multiplier to match the specification. Fix up naming of verify library info structs. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e178c3d7d59fef9c3696178646b23ed2a3ffc61
2023-12-14Remove inferred dimension from RESHAPEJeremy Johnson
Test generation changed to only produce static reshape tests Reference model changed to produce ERROR_IF on inferred shapes Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92c92a40e7c0e457961bc654630040dff79a750b
2023-12-14Main Compliance testing for SCATTER and GATHERJeremy Johnson
Added indices shuffling and random INT32 support to generate lib with testing of these new random generator modes Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I058d8b092470228075e8fe69c2ededa639163003
2023-12-11Enforce no output rewrite REQUIRE in SCATTERJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3555e7216d403d436bf6e39d4b16bb000645c4bb
2023-12-04Path fixes for ref model and librariesJeremy Johnson
Support relative desc.json paths in refmodel of just "desc.json". Catch when generate and verify library paths are None and provide help in tosa_verif_check_result. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie52ac399ea002e5fcdcc1eec3d9df6153a778e88
2023-12-04Change TANH, SIGMOID to ABS_ERROR complianceJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icf04afc7fdae8f506ba4710aaa085d6ea53bb5bf
2023-12-04Main Compliance FP16 support - generate and verify.Jeremy Johnson
FP16 support for all existing operators for compliance: * DOT_PRODUCT * ULP * EXACT * ABS_ERROR Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I8d25448a793375b53880da3787d8f839767f02cf
2023-12-04Main Compliance testing support for RESHAPEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I750ef5d3c95daca10e7717437ed375a81b434582
2023-11-30Main Compliance testing support for CASTJeremy Johnson
Limit CAST input tensor to maximums of output type to avoid saturation and infinity. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I33350a4ce0ec828da7d2e7aa8cd3183a89a97431
2023-11-30Adjust random data ranges for Main Compliance to avoid FP inf and nanJeremy Johnson
POW - there are now 3 test sets to cover random ranges. Also added ROUND mode to data generator to force integer exponent values. LOG, EXP, RSQRT, REDUCE_SUM & FULLY_CONNECTED - have had their ranges reduced for each test. Fix generate library configuration defaults and checks. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie5d3bd78f690cc787a2ca4eb9b4bd6808bd9238c
2023-11-23Main Compliance testing support for COMPARISON opsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id6229cfaccad866b110630119eb045dbf6453bf5
2023-11-23Main Compliance testing support for LOG and ACTIVATIONsJeremy Johnson
Increase exp2 allowed range to account for denormals. Minor adjustments to verify to match spec updates for pseudo code. Set ranges of activation function inputs to match spec. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6fcf665932ac2c9080e284b865da8f7746801f59
2023-11-23Main Compliance testing support for AVG_POOL2DJeremy Johnson
Fix test set 4 to have more symmetrical data set for AVG_POOL2D. Increased size of AVG_POOL2D test shapes to meet MIN_DOT_PRODUCTS. Print out sign of bias magnitude in verify library to help debugging. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icf139cf5788cbc68f2c537567bdfb4cfe31274df
2023-11-16[reference_model] add local_bound attribute to test_genTai Ly
Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I9b87d1ad0f09648d07a6ecfc2075a174a9d04a3d
2023-11-16Fix FP16, BF16 data ranges for conformance testsJeremy Johnson
Enable use of data ranges for old data gen path as well as the new generate library path, so that FP16 and BF16 test data is produced within the correct ranges. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I749870a3112f8c3a75f4d16b8322c813fbf977cd
2023-11-16Main Compliance testing support for EXP & POWJeremy Johnson
Added new ABS_ERROR mode to verify lib and ref model. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ifb78290675833d3df7df91a4d6cef336b02b64a4
2023-11-16Main Compliance testing for simple UNARY opsJeremy Johnson
For RECIPROCAL, RSQRT, CEIL, FLOOR, ABS, NEGATE & IDENTITY. Improved ULP informational output. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I49644573b4c9a30b2b9d6c9624f2a1d46976a378
2023-11-10Main Compliance testing for FULLY_CONNECTEDJeremy Johnson
Updated shapes to meet MIN_DOT_PRODUCTS. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I82297917c009b3120306f8a9bb965209d109ef8d
2023-11-10Main Compliance testing support for ARGMAX, REDUCE_SUM/MAX/MINJeremy Johnson
Add extra tests for FP32 REDUCE_SUM that meet MIN_DOT_PRODUCTS. Plus improved dot product test generation skip information. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia8198a9500ddddfc86c5bb84230b9a4edf5ffd50
2023-11-10Main Compliance testing support for ADD, SUB, MINIMUM and MAXIMUMJeremy Johnson
Added main inference compliance mode data generation and verification. Improved error reporting in tosa_verif_build_tests. Fixed pseudo random invalid range check. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib538b5999444d67704ebc322be3c7becbe5ad206
2023-11-02Main compliance testing support for MULJeremy Johnson
Update verify ULP mode to allow fractions (e.g. 0.5). Update pseudo generator to accept ranges. Fix up pseudo random distribution based on ranges. Change-Id: I9168c5f7d37722678c0f1f9e906953c8cec367b1 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-11-02Compliance mode testing for CONV2DJeremy Johnson
Added CONV2D data generation. Updated verify dot product check to latest specification. Updated test generator and python datagenerator library to create const files during test generation. Add support for compliance test sets to conformance test_select. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5be3b761a1e3ef259c058e493877cd5a89d5778b
2023-10-26Compliance testing support for MAX_POOL2D & PADJeremy Johnson
Added Pseudo Random number generator in generate library. Enabled MAX_POOL2D, PAD FP32 tests to use new generator and compliance. Fixed verify library exact mode to expect reference data as FP64. Simplified tosa_verif_build_tests internal interfaces for new tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icc0ffa924cf38107c3a212efd452c47a650c9d98
2023-10-25Improve verfiy and generate library validationJeremy Johnson
Improved libraries validation to catch unknown values. Improved verify output to match generate library. Refactored generate tests to use less code duplication. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9c38745fbc8e70f46c19ddae6c62ee248d33b5f1
2023-10-16Data generator library python interface addedJeremy Johnson
Added support for using generate library in tosa_verif_build_tests and tosa_verif_run_tests tosa tool scripts. Reduced scope of compliance test creation and verification to the supported type of FP32. Fix missing virtual destructor warning in generate_dot_product.h and add config file for generate library. Simple pytests included to check python interface. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6cdad9b00660d6ddc8bd07fdea813937fb48626a
2023-10-02Update verifier library data-type supportJeremy Johnson
Make compliance meta-data data-type required for all. Add data-type checking for all verifier modes. Add initial enum support for new ROUND compliance mode. Improve print out information from library. Use numpy ctypes.data_as to get f16 support compared to ctypes_lib. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie983ba4ea958a88556f30c09b3ebc19cd9ec96b7
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-09-01[reference model] Fix dim test generationv0.90a0Tai Ly
In tosa_test_gen.py, change output_shape for dim operators to [] instead of [1] Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I8e7901e02831e9a467bce17e55e9379cb5f1b739
2023-08-30Fix conformance regressions due to type and region changes.Jeremy Johnson
Compress flatbuffer JSON files for conformance to allow while_loop tests to fit in the 30MB file size limit. Do not include new SHAPE type in usable DTypes so that selection of ERROR_IF tests are not changed. Also enable strict json mode for flatbuffers. Change-Id: Id89a9963244c34769a4662f9951adde0aa3334d9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-07-31Update serialization library submoduleJeremy Johnson
* Update Flatbuffers version * Allows consts to be saved as inputs Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ibb9e38d381f8229bfca65b559e35c9ded889c384
2023-06-29Add support for ERF operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib42b867287b83a183a0d0fb1f1eb29974f58fae4
2023-06-15Add ERROR_IF to incorrect broadcast shapesJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I7460ad9eed3ed5c7cec6e855a0303753ed28eb1c
2023-05-17Add support for one dimension of size -1 in ReshapeOpJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I0ef7607f4266296a1204c5cccdb5be36f345b5ba
2023-05-10Refactor ref_model rank checking and add level check to argmaxJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Iad035b31d5e5e83040068e6311501490765bfff7
2023-04-268K levels: Tensor op tests kernel/stride at 8192 maximumsJeremy Johnson
Operators updated: AVG_POOL2D, MAX_POOL2D, CONV2D, CONV3D, DEPTHWISE_CONV2D & TRANSPOSE_CONV2D tosa_verif_build_tests argument --level-8k-sizes used to allow kernel/stride maximum boundary testing Fixed bugs in height/width validator function meaning some esixting avg_pool2d float tests need regening. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7aeab82d3bd3c49d02d54708f2c9d995cd3cf2df
2023-04-258K level: RESIZE scaling at 64 maximumJeremy Johnson
Add support for producing scaling 64 tests in RESIZE and make sure default tests are not generated with scaling greater than 64. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e99cde7e6bec6ea7854a139c6fc2ed151af5b1e
2023-04-118K level: rank 6 tests support using generator groupsJeremy Johnson
Split generation of conformance tests into groups to avoid a change in existing tests selection * A standard group is used for the normal/default conformance tests * An 8k level group covers the extension of testing for 8k level * Other arbitrary groups can be created and the group names will be used to tag the tests, so that in the future they could be filtered Group selection criteria is in a new dictionary * A default selection config is used for generation groups * But a different config can be used for a particular generation group Added parameter sparsity for pad of rank6 and above, and further parameter sparsity options to test_select to limit number of tests selected. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib9f1185c0ae6d0957df2116411c17e958157c6d3
2023-03-28Add CONST INT48 support to conformance.Jeremy Johnson
Remove ERRORIF tests for CONST as they do not create valid negative tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia96b184f34010a452d2c4848b0653514352c751e
2023-02-28Update rank limits for SLICE, TILE and TRANSPOSELuke Hutton
Updated to align with corresponding changes to the spec. In addition, some ERROR_IF tests have been updated to match the checks specified by the spec, including: PAD, SLICE, TILE, TRANSPOSE. Signed-off-by: Luke Hutton <luke.hutton@arm.com> Change-Id: Ie2c5f48e79a5610eb82739170e25057a63dac1d8
2023-02-22Bug fixes for max-batch-size/ofm-depthJames Ward
* Conformance regen required for tosa-bi: conv2d conv3d depthwise_conv2d transpose_conv2d * Include bug fix for testGen.makeShape() * Include json changes to limit size of tests which now have ofm_depth > 1 Signed-off-by: James Ward <james.ward@arm.com> Change-Id: Ic8221b9a742f5737df523997bee7260f4dfcdef3
generated by cgit v1.2.1 (git 2.23.0) at 2024-07-01 04:37:08 +0000