aboutsummaryrefslogtreecommitdiff
path: root/verif/tosa_test_gen.py
AgeCommit message (Collapse)Author
2022-01-06Reorganize verif and create packagesJeremy Johnson
Split generator and runner scripts Add package setup Add py-dev-env.sh/.bash to allow editing source files during dev Update README.md with installation info Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I172fe426d99e2e9aeeacedc8b8f3b6a79c8bd39d
2022-01-05Fix crash with rank mismatch negative testsEric Kunze
When testing rank mismatch tests, could be accessing beyond the end a tensor's shape. Change-Id: I304463e50579d574d629abd13098017a26a1406b Signed-off-by: Eric Kunze <eric.kunze@arm.com>
2021-12-08Stop COND_IF neg tests creating bad arraysJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I267fa624241f60f1fa613e27e4fe46fc43768f8a
2021-11-24Do not generate tests that fail validation checksLes Bell
Change-Id: I33237ebfd946b9ec91352c2b0dc6298cc113cd77 Signed-off-by: Les Bell <les.bell@arm.com>
2021-11-18WrongRank negative test gen additions & fixesJeremy Johnson
REDUCE ops - test generation now limited to ranks 1-4 and so WrongRank tests created SCATTER/GATHER ops - enable WrongRank testing Change-Id: I0909573e0edabddbad8764e7542d654967a5c8b9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-11-18ERRORIF test generation fixesJeremy Johnson
TRANSPOSE - remove WrongRank testing as no limit in spec RESIZE - make floating point test naming same format as int - fix MaxDimExceeeded by not restricting dims and fix max dim check to match spec - catch stride less than/equal zero and regen PAD - remove zero point testing as not in spec Change-Id: I4eb0b78d7f577469d202866d95833e7d5fd2f2a8 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-11-18Convolutions ERROR_IF testsLes Bell
Signed-off-by: Les Bell <les.bell@arm.com> Change-Id: I68a13e1b337b1afc2ab5e0edcffda2b4b0cecdda
2021-11-11Add Broadcast DimensionMismatch errorsJeremy Johnson
Add RankMismatch and DimensionMismatch support for SELECT Update RankMismatch ops to also support DimensionMismatch Update POW op to have proper broadcast testing A few other broadcastable ops missing Rank/Dimension testing Change-Id: I6566f45a7a0db4f9f008456ea7a8e23d4192f4f9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-11-09Fix Transpose WrongRank test and add new test for ConcatMatthew Haddon
* Transpose WrongRank tests now use ranks 7, 8 * Concat ERROR_IF checks now test for inaccurate summation of output shape tensor dimension Change-Id: If32f43a4dbd872d0ef7625fa3d4969c863a11b8c Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Signed-off-by: Les Bell <les.bell@arm.com>
2021-11-09Add negative testing to cond_if, while_loopMatthew Haddon
Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Signed-off-by: Les Bell <les.bell@arm.com> Change-Id: Ie6c8c8653874f9eed6007a54a3ad526601a4a669
2021-11-09Add ERROR_IF checks to operators without specific ERROR_IFsMatthew Haddon
* Operators implemented: sigmoid, tanh, arthmetic_right_shift, mul, table, select, equal, greater, greater_equal, concat, reverse, tile, scatter, gather, case * Note that over the course of implementation some specific ERROR_IF checks have been added for some of the above operators Change-Id: I80595e6eb9a3e5efd1cc6fd7aa28bbc2dd614980 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Signed-off-by: Les Bell <les.bell@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-11-01RESIZE test generator updatesJeremy Johnson
Allow random shift values rather than just 11 Check for values that cause unpredictable behaviour Change-Id: I508c9d0982808ac03b8400d47ad9445d6afb1b79 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-10-28Changes for 0.23.0 releaseKevin Cheng
- update serialization_lib hash - PAD: 1. make padding as an attribute instead of tensor. 2. add pad_const_int (for non-float type) / pad_const_fp (for float type) - TRANSPOSE: make perm as an attribute instead of tensor - TABLE: make table as attribute instead of tensor - update examples/ tests Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Iddc446db4b356ba2f36ea4a79b7220b9cfc2aa4e
2021-10-25Limit tensor values for COND_IF/WHILE_LOOP tests to stop saturationJeremy Johnson
Change-Id: Idb36b1f1c0d78ec101c168865a9c8d03221b4c84 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-10-21Fix pool2d generation and check to match specificationJeremy Johnson
Change output size error_if check in ref model to match specification. Remove size check from test generation as output shape is always correctly calculated. Change-Id: I5be64f31e6448b47e80fc0a4af11bb312f366a26 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-10-18Add Negative tests for pad, reshape, slice, transposeMatthew Haddon
Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I659337aadfd0498bf88a95737f69c51efec797cc
2021-10-18Add negative testing support to fully_connected, matmul, argmaxMatthew Haddon
Change-Id: I75f2a4ab6790dcbdfaec064f42f601d8f44da70b Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-18Add negative testing support to RESCALEMatthew Haddon
* Negative tests for rescale op added Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I70aead1c6a67f159c7b7c9a05f7d5f0b92521584
2021-10-12COND_IF int8, int16 testsLes Bell
Change-Id: Id1c2d2a41bdfff6b3fe45f79bf99abd0b5beb704 Signed-off-by: Les Bell <les.bell@arm.com>
2021-10-12Add negative testing for avg_pool2d, max_pool2dMatthew Haddon
* Negative tests for ERROR_IFs given in spec * Constrict dimension size of latter ranks if rank is larger than 4 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: Iffea1874e876dba83c8a7c63049283bf7b3ba74b
2021-10-11Add negative testing support for reduce operatorsMatthew Haddon
* Negative testing support added for the following operators: reduce_all, reduce_any, reduce_max, reduce_min, reduce_product, reduce_sum Change-Id: Ife4f0307b681510e8910a23a39efc127c82ae120 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-11Fix rank and dtype filtering for ops like conv3d & fully_connectedJeremy Johnson
Change-Id: Ic2aebe40b5cce61d4576a64f4f48ff87b36475c2 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-10-07Add negative testing support for ew_unary operatorsMatthew Haddon
* Added negative testing support for the following operators: abs, bitwise_not, ceil, clz, exp, floor, log, logical_not, negate, reciprocal, rsqrt Change-Id: Icc6f146c6407502520330678420951749ba2a9ef Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-07Add negative testing support to (most) EW Binary OpsMatthew Haddon
* Negative testing support for the following operators: ADD, BITWISE_AND, BITWISE_OR, BITWISE_XOR, INTDIV, LOGICAL_AND, LOGICAL_LEFT_SHIFT, LOGICAL_RIGHT_SHIFT, LOGICAL_OR, LOGICAL_XOR, MAXIMUM, MINIMUM, POW, SUB Change-Id: I2271f00b0b619604e864e36e4a4f987f1b2a37d4 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-07Add ERROR_IF checks for mismatched batch/channelMatthew Haddon
Change-Id: I7c670c5f9b97a18a6f586b16f31bc9fc301f6bc3 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-07Separate positive and negative test generation and refactorMatthew Haddon
* Positive and negative tests are now produced entirely in isolation, if 'both' test types are chosen all positive tests are generated, then all the negative tests are generated, without combining the two * Moved tensor generation out of serialize test and into it's own function Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: Id8e9023d2c2966a0b14ae83d2d848a66cb125fcb
2021-10-07Fix shape generation with num-const-inputs-concat optionJeremy Johnson
Change-Id: I2a0aa63a4256629d12166638812dc5b854db2ddf Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-10-04Add shared ERROR_IF statementsMatthew Haddon
* Added WrongInputType and WrongOutputType checks * Delete and Add extra inputs/outputs when doing op graph checks * Add incorrect rank checks * Entire op dictionary is now passed to build functions * Error if validation now done in separate function * Duplicate tests are ignored Change-Id: Ie1117bdbc9bdeb42059123792ce1df4cc56db54e Signed-off-by: Matthew Haddon <matthew.haddon@arm.com>
2021-10-01Fix CONST test.Kevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Iea91076f3e0673ce5adfcca247d968416023fa58
2021-09-30Remove duplicate output tensor in cond_if_binary testKevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Ic0c4bd1a55c6782aeacb77163a3a093edb86d437
2021-09-28Remove ReluN op.Kevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Iaac727159a84de1f83de549c3a22704096f46bf9
2021-09-27Add ERROR_IF support for RESIZEMatthew Haddon
* TosaErrorValidator implemented to produce and test for ERROR_IF conditions * RESIZE specific ERROR_IF test support added * Set rank and type parameters before test generation loop to avoid multiple checks for valid parameters * Increase output dimensions if IFM/OFM ratio smaller than 1/16 Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I430e13383d99c2e25354f53d3703fb9be973f6d4
2021-09-24Extended tests for 2D/3D tensor functionsLes Bell
* larger values for stride/padding/dilation/kernel * sparse test generation, as there are too many variations Signed-off-by: Les Bell <les.bell@arm.com> Change-Id: If13ea17024d81262ab892e3111cbf5833e77a8c5
2021-09-16Generate more CONV3D testsLes Bell
* plus extra debug info for some exceptions Change-Id: I667ee9b8a1cba8c6bd47c31231f1805eba680ba5 Signed-off-by: Les Bell <les.bell@arm.com>
2021-09-16Implement Conv3D kernel.Kevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Ic16e918b1a2423ad563684e29ce70d9efdbf9c02
2021-09-15Rename attribute: Pool2d, Conv2d, TransposeConv2d -> Pool, Conv, TransposeConvKevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: I466dd1dcf5230e8e07df202ba88515e775e04a1e
2021-09-15Fix for ADD/SUB saturation in testsJeremy Johnson
Change-Id: I4f05b256f4f439f72e5ee8bce60e4e92b6aaa6e7 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2021-09-14Fix while_loop test.Kevin Cheng
- add output tensor to main block - fix bug where tensor cannot serialize list of const data Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: I40b96d634642e24e6fae6fa3ad6846f5911005f7
2021-09-08Remove invalid tests from test generatorMatthew Haddon
* Implemented InvalidValidator to remove existing invalid tests. * Removed invalid tests for resize, rescale, conv2d, depthwise_conv2d, transpose_conv2d, avg_pool2d, and max_pool2d (note default avg/max_pool never produced negative tests, but theoretically could). * Changed behaviour of computerMultiplierAndShift to produce the allowed range of shift values. Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I5e7b11030deb5322e2ca08fd4f4467fb02b7740d
2021-09-08Allow user to specify test type generatedMatthew Haddon
* The option --test-type allows the user to select 'positive', 'negative', or 'both' types of tests produced by the test generator. * Reset RNG when looping through negative test generation (generation not implemented) Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I1bfcb3170e7380be0f98b36b3d4abc4779a05abe
2021-09-07Fix batch size if target shape set for SCATTER operatorMatthew Haddon
* max batch size ignored if target shape is set * W size reduced if large input tensor used Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I13472ab768fa93a1d0b9e28964f56ec4a06dbdfd
2021-08-31Rename DIV operator to INTDIVMatthew Haddon
* In line with the TOSA spec the DIV operator has been renamed INTDIV Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: I1dc6e88220ef26b24487675600b6cd1e5bb8b0f7
2021-08-31Set TILE multiple to one if input dimension is largeMatthew Haddon
* Sets multiple of input dimension to one if the dimension of the input tensor is large to reduce ouput tensor size. Change-Id: I2ebcecf438282de032a33e0cf4b3847cbf94440d
2021-08-20Replace node level check ASSERT_MSG_NODE()/FATAL_ERROR_NODE() with REQUIRE() ↵Kevin Cheng
or ERROR_IF() - Adding return code enum class: {VALID, UNPREDICTABLE, ERROR} - Runtime errors (e.g. memory allocation failure) will abort immediately, or will return one of the three return codes Part of the codes are re-written to pass REQUIRE() to the top-level (e.g. apply_scale_32/16()) - Update setExpectedFailure() to setExpectedReturnCode() on test generation script - Update test regression script to interface with reference model change Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: Ia063c936bcb2a54d6e379a5bb6801aa72d1186f1
2021-08-20Make MATMUL output shape randomMatthew Haddon
* Currently when a target shape is specified, the W value of the input tensor is always equal to N, this is not the case when no target shape is defined. * A random value for W is generated every time. Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I8f8ecb32308cef4a1ece1871f76ebbd5f0cf881f
2021-08-19Produce Concat tests with multiple input tensorsMatthew Haddon
* Concat tests now contain between 2 and 5 input tensors concatenated together * Both input and const tensors are used as inputs to the operator * Option to add in const tensor inputs (this is slow), defaults to original behaviour Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I2a0cc622d31aceab8d24521668d0aae040ba73b1
2021-08-12Support int4 weights read. Added conv2d int8xint4 in test generation.Kevin Cheng
Signed-off-by: Kevin Cheng <kevin.cheng@arm.com> Change-Id: I61620f160c7dad6aac5fcc3da0a6e97f3bae5b40
2021-08-12test gen PAD agPad fix to cover all pad valuesLes Bell
* encode pad in the test name with leading zeros Change-Id: I7b1cc04014d390e707de099c5f78f93f5f344d69 Signed-off-by: Les Bell <les.bell@arm.com>
2021-08-12Add support for UINT8Matthew Haddon
* RESCALE can now produce tests with UINT8 as the input/output datatype. Signed-off-by: Matthew Haddon <matthew.haddon@arm.com> Change-Id: I0a5d7b3c6dd7c2501d14e5d558b1f18e5e967fa9
2021-08-12fully_connected test OC independent of input shapeLes Bell
Change-Id: Ib299b78abe0e5c2ef0aeff9d853e583078768e1c Signed-off-by: Les Bell <les.bell@arm.com>
generated by cgit v1.2.1 (git 2.23.0) at 2024-06-08 12:55:30 +0000