aboutsummaryrefslogtreecommitdiff
path: root/verif
AgeCommit message (Collapse)Author
2023-11-16Main Compliance testing for simple UNARY opsJeremy Johnson
For RECIPROCAL, RSQRT, CEIL, FLOOR, ABS, NEGATE & IDENTITY. Improved ULP informational output. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I49644573b4c9a30b2b9d6c9624f2a1d46976a378
2023-11-10Main Compliance testing for FULLY_CONNECTEDJeremy Johnson
Updated shapes to meet MIN_DOT_PRODUCTS. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I82297917c009b3120306f8a9bb965209d109ef8d
2023-11-10Main Compliance testing support for ARGMAX, REDUCE_SUM/MAX/MINJeremy Johnson
Add extra tests for FP32 REDUCE_SUM that meet MIN_DOT_PRODUCTS. Plus improved dot product test generation skip information. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ia8198a9500ddddfc86c5bb84230b9a4edf5ffd50
2023-11-10Main Compliance testing support for ADD, SUB, MINIMUM and MAXIMUMJeremy Johnson
Added main inference compliance mode data generation and verification. Improved error reporting in tosa_verif_build_tests. Fixed pseudo random invalid range check. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib538b5999444d67704ebc322be3c7becbe5ad206
2023-11-02Main compliance testing support for MULJeremy Johnson
Update verify ULP mode to allow fractions (e.g. 0.5). Update pseudo generator to accept ranges. Fix up pseudo random distribution based on ranges. Change-Id: I9168c5f7d37722678c0f1f9e906953c8cec367b1 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-11-02Compliance mode testing for CONV2DJeremy Johnson
Added CONV2D data generation. Updated verify dot product check to latest specification. Updated test generator and python datagenerator library to create const files during test generation. Add support for compliance test sets to conformance test_select. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I5be3b761a1e3ef259c058e493877cd5a89d5778b
2023-10-31Allow more integer types in result check.Jeremy Johnson
Added int8, int16, uint8 and uint16. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6b1174aa7369c34a613b322c4b05bf4fc586ee6e
2023-10-26Compliance testing support for MAX_POOL2D & PADJeremy Johnson
Added Pseudo Random number generator in generate library. Enabled MAX_POOL2D, PAD FP32 tests to use new generator and compliance. Fixed verify library exact mode to expect reference data as FP64. Simplified tosa_verif_build_tests internal interfaces for new tests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Icc0ffa924cf38107c3a212efd452c47a650c9d98
2023-10-25Improve verfiy and generate library validationJeremy Johnson
Improved libraries validation to catch unknown values. Improved verify output to match generate library. Refactored generate tests to use less code duplication. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9c38745fbc8e70f46c19ddae6c62ee248d33b5f1
2023-10-24Improve conformance backward version supportJeremy Johnson
Turn off ARGMAX rank 6 tests for v0.60.0 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I535b7887a9db099d39ad943e151f870ebc716791
2023-10-16Data generator library python interface addedJeremy Johnson
Added support for using generate library in tosa_verif_build_tests and tosa_verif_run_tests tosa tool scripts. Reduced scope of compliance test creation and verification to the supported type of FP32. Fix missing virtual destructor warning in generate_dot_product.h and add config file for generate library. Simple pytests included to check python interface. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I6cdad9b00660d6ddc8bd07fdea813937fb48626a
2023-10-12Conformance simple backward version and binary files supportJeremy Johnson
Add --test-version option to tosa_verif_conformance_generator to select the version for tests to output. Add --output-type to allow json, binary or both files to be created during conformance generation. Fix passing schema_path to test runner. Add explicit verify lib path arg to test runner. Change-Id: I5f1ad137d713fca408a98470ea77bddf8916c5f3 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-10-02Support for compliance checking testingJeremy Johnson
Updated to conformance generator to not generate tests with results for compliance tests. Updated test runner to run compliance mode version (precise & abs mode) of reference model to create test results to use against SUT results. Updated reference model to enable abs_mode on correct desc.json flags. Updated test checker to support compliance checking using verifier lib. Seperated color printing from test checker. Change-Id: I7e2fbfc6883916caa5d94d4ece122c48bf45f530 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-10-02Update verifier library data-type supportJeremy Johnson
Make compliance meta-data data-type required for all. Add data-type checking for all verifier modes. Add initial enum support for new ROUND compliance mode. Improve print out information from library. Use numpy ctypes.data_as to get f16 support compared to ctypes_lib. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie983ba4ea958a88556f30c09b3ebc19cd9ec96b7
2023-09-28Refactor path arguments to tosa-toolsJeremy Johnson
tosa_verif_conformance_generator - Move to using ref-model-path instead of ref-model-dir - Add schema-path and flatc-path - Add model_files module to locate default places for files convert2conformance - Remove default paths verifier - Switch to using exact path of verifier library tosa_verif_run_tests - Use conformance model_files to locate defaults Change-Id: Ieca3b164670e2a7dcb047743667cc4e8317daa97 Signed-off-by: Josef Malmström <josef.malmstrom@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-09-28Add 0-rank tensor support for concat in framework testWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iff77091e4a57f487431ffbf7ac1c89301a153c8b
2023-09-15[reference_model] Support StatefulOps and the tests for CallOnceOpTai Ly
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I03cb878736ccd7e1f5e1f780d7171949a19a9de2
2023-09-13Build compliance verifier library with a python interface.Jeremy Johnson
Build steps to create tosa_reference_verify_lib.so New python class to handle loading and calling verifier with pytests. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Id42c71c398ed7364742b4a039977412ef3081c4a
2023-09-13Restore coverage for conformance MATMULJeremy Johnson
Increase size of tensor dimensions to meet compliance size needs. Also improve warning output about this size requirement. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I9dc2b8dab05a1bac0083cdca9d99845c287d3ce9
2023-09-11Fixes for conformance generator tags and test selectionJeremy Johnson
Incorrectly used "tag" in test.desc instead of "tags" Incorrectly selected all tests rather than just all positive tests depending on operator selection option. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ib257bc57accf745beedfa4a3b2ece9539c0972c2
2023-09-07Initial lazy data-gen and compliance test build supportJeremy Johnson
Add initial support for compliance and lazy data-gen meta data added to desc.json for MATMUL. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I00c047814134a96d7c98d890e93b5884e25b8e64
2023-09-07Add desc.json schema validation with compliance/data_gen extensionsJeremy Johnson
Rename scripts utlities to tosa_* such as tosa_json2numpy Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ie8d584e2afb189fb74cf96b39590c7c27444ba14
2023-09-01[reference model] Fix dim test generationv0.90a0Tai Ly
In tosa_test_gen.py, change output_shape for dim operators to [] instead of [1] Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I8e7901e02831e9a467bce17e55e9379cb5f1b739
2023-08-30Fix conformance regressions due to type and region changes.Jeremy Johnson
Compress flatbuffer JSON files for conformance to allow while_loop tests to fit in the 30MB file size limit. Do not include new SHAPE type in usable DTypes so that selection of ERROR_IF tests are not changed. Also enable strict json mode for flatbuffers. Change-Id: Id89a9963244c34769a4662f9951adde0aa3334d9 Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-08-18Add DIM operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iea11ee5d3d98773e9c5e9b827593c05afb41ce3b
2023-08-16Remove `"` from the search pattern `"tosa.*"`TatWai Chong
The double quotation mark is not always appear in the assembly output of operations. Remove it to easier capture general printing pattern. Change-Id: I59301daa07abd905de7259f8ae527a74b1ce5a6c Signed-off-by: TatWai Chong <tatwai.chong@arm.com>
2023-08-08Re-write json2numpy testsJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I17227adfca203f353e51ae9297f7e01f8d7edbe9
2023-08-07Deprecation of parts of TOSA conformanceJeremy Johnson
* Add deprecation notice for conformance_tests repo * Add deprecation notice for framework tests in conformance Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I92b17f4d6d993e47f30d7996a18fc13b934c4cdf
2023-08-02Update Framework compiler scriptEric Kunze
Reuse the name deduced from the test run in results. Also move most string formatting to f strings. Move path handling to pathlib from os.path Signed-off-by: Eric Kunze <eric.kunze@arm.com> Change-Id: I44c7786d4b7af44306e218ff49608df35d4521a7
2023-07-31Update serialization library submoduleJeremy Johnson
* Update Flatbuffers version * Allows consts to be saved as inputs Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: Ibb9e38d381f8229bfca65b559e35c9ded889c384
2023-07-26Update framework tools to add name in jsonEric Kunze
Name of the test can now come from the json description rather than inferred from the directory. Old method remains to keep functionality with older tests Change-Id: I79f872103ae6a101b5cc799af91c933839f28dfb
2023-07-26Add Verif Conformance READMEJeremy Johnson
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: If30b329905aad9e520e4a9a3ac1dffccd37bb0e9
2023-07-25Update flatbuffer version and fix json2numpy testEric Kunze
Temporarily disable the json to numpy pytest Otherwise will load as the wrong dtype Also update the flatbuffer module version to match tosa_mlir_translator Change-Id: I40999f631adb1b230bc8a8b891e0955dd67371f0 Signed-off-by: Eric Kunze <eric.kunze@arm.com>
2023-07-11Fix size mismatch of placeholders for broadcast fuzzingWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib954db4f8f2fd5008e95d5651a6e84a8f5b7161d
2023-07-06Add reference model framework test for INT8 squared differenceWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: I6fba6907cef0616c18dc461dbb92d2aceb582f6c
2023-07-06Enable framework generator to save numpy files in native dtypesJerry Ge
With the following patch merged, the tosa serialization_lib is able to read u/int8, u/int16 in their native dtypes: https://review.mlplatform.org/c/tosa/serialization_lib/+/9871 Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I5e63fb70b055cc5ee33e7dc26237eef9bedeac30
2023-07-05Add a parameter to tensor generation function to disable fuzzingWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Iff31b33b818a181371904915d5477a169513aa2e
2023-06-29Add support for ERF operator to reference modelWon Jeon
Signed-off-by: Won Jeon <won.jeon@arm.com> Change-Id: Ib42b867287b83a183a0d0fb1f1eb29974f58fae4
2023-06-21Add tf broadcast-to testingTai Ly
This patch adds BoradcastTo Op to the tf tests. Did not add tflite testing because the tf.lite.TFLiteConverter converts tf.broadcast-to to tfl.mul by 1. Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: Icd372e619c318121c19eca87d5716bcd5fbbbb23
2023-06-15Fix framework test generator with custom_shapesJerry Ge
shape_list is global while the custom_shape is only applicable for a single operator, need to make a copy for the whole shape_list when dealing with custom_shapes Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I2c5b87cb6fcaa153b0ccfe5827bae814b9cfe2b9
2023-06-15Add ERROR_IF to incorrect broadcast shapesJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I7460ad9eed3ed5c7cec6e855a0303753ed28eb1c
2023-06-15Add accumtype to tf/tfl framework testsJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Ie45cc27433f5dbce3fadc90014dc5cc8e36a9950
2023-06-02Support custom_shapes for framework test generationJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Ia29c73cb5d0a7f91914e2a94ca52d06f375722e9
2023-05-17Add support for one dimension of size -1 in ReshapeOpJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I0ef7607f4266296a1204c5cccdb5be36f345b5ba
2023-05-168K level: rank 6 ARGMAX testsJeremy Johnson
Change-Id: I81097ae263a64ee8e35b8bd798598671ac089c3f Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
2023-05-10Refactor ref_model rank checking and add level check to argmaxJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: Iad035b31d5e5e83040068e6311501490765bfff7
2023-05-05[reference model] Add precise modeTai Ly
This adds --precise_mode=1 option to tosa_referece_model, which will cause reference model to convert all floating point tensors to FP64 tensors and compute all operators accordingly. Also adds optional -p arguments to test runners tosa_verif_run_tests.py and tosa_verif_framework_compiler_runner.py to run tests in precise mode Signed-off-by: Tai Ly <tai.ly@arm.com> Change-Id: I156055216ad61710096497a8fa1a653be2a602a3
2023-04-268K levels: Tensor op tests kernel/stride at 8192 maximumsJeremy Johnson
Operators updated: AVG_POOL2D, MAX_POOL2D, CONV2D, CONV3D, DEPTHWISE_CONV2D & TRANSPOSE_CONV2D tosa_verif_build_tests argument --level-8k-sizes used to allow kernel/stride maximum boundary testing Fixed bugs in height/width validator function meaning some esixting avg_pool2d float tests need regening. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I7aeab82d3bd3c49d02d54708f2c9d995cd3cf2df
2023-04-258K level: RESIZE scaling at 64 maximumJeremy Johnson
Add support for producing scaling 64 tests in RESIZE and make sure default tests are not generated with scaling greater than 64. Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com> Change-Id: I3e99cde7e6bec6ea7854a139c6fc2ed151af5b1e
2023-04-20Add level checking to TOSA Ref modelJerry Ge
Signed-off-by: Jerry Ge <jerry.ge@arm.com> Change-Id: I5689d7c6b902a319a68fa4628b59e0bcc23aeca4
generated by cgit v1.2.1 (git 2.23.0) at 2024-06-24 13:39:46 +0000