From 6dd341093507157aabbea00b90ca8902509cfd4f Mon Sep 17 00:00:00 2001 From: Eric Kunze Date: Sun, 25 Feb 2024 22:24:52 -0800 Subject: Modify TOSA profiles Create composable profiles and profile extensions. Define requirements for a TOSA implementation to fully implement at least one profile. Signed-off-by: Eric Kunze Change-Id: I02cfb0171b2d227727f530cb29108b479206b25b --- chapters/appendix_b.adoc | 12 + chapters/introduction.adoc | 37 +- chapters/operators.adoc | 4 +- tools/dictionary.dic | 1 + tools/genspec.py | 84 ++- tools/tosa.py | 49 +- tosa.xml | 1468 ++++++++++++++++++++++++++------------------ tosa.xsd | 76 ++- tosa_spec.adoc | 2 + 9 files changed, 1095 insertions(+), 638 deletions(-) create mode 100644 chapters/appendix_b.adoc diff --git a/chapters/appendix_b.adoc b/chapters/appendix_b.adoc new file mode 100644 index 0000000..64f9f3a --- /dev/null +++ b/chapters/appendix_b.adoc @@ -0,0 +1,12 @@ +// +// This confidential and proprietary software may be used only as +// authorised by a licensing agreement from ARM Limited +// (C) COPYRIGHT 2024 ARM Limited +// ALL RIGHTS RESERVED +// The entire notice above must be reproduced on all authorised +// copies and copies may only be made to the extent permitted +// by a licensing agreement from ARM Limited. + +== Appendix B - Profile operator tables + +include::{generated}/profile_ops.adoc[] diff --git a/chapters/introduction.adoc b/chapters/introduction.adoc index 0a94ef3..c34bf7b 100644 --- a/chapters/introduction.adoc +++ b/chapters/introduction.adoc @@ -89,27 +89,36 @@ The following principles govern the selection of operators within TOSA. === Profiles -TOSA supports three profiles that enable efficient implementation on different classes of device. -The Base Inference profile is intended for embedded integer/fixed-point designs performing inference only. -The Main Inference profile is intended for general inference functionality including integer and floating-point data types. -The Main Training profile adds training operators in addition to inference operators. -This version of the specification covers the Base Inference and Main Inference profiles. -Main Training profile is expected in a later version of the specification. -The following table summarizes the three profiles: +TOSA profiles enable efficient implementation on different classes of device. +Each profile is an independent set of operations and data type combinations. + +TOSA profile extensions define optional operation and data type combinations. + +Each operator's Supported Data Types table will define which profile or extension an operator and data type is in. +An operator / data type combination may be part of multiple profiles or extensions. +If so, each profile and extension will be listed in the Supported Data Types table. +In addition, a table listing all operations for each profile can be found in Appendix B. + +The following are required for compliant TOSA implementations: + +* A TOSA implementation must implement at least one profile. +* A TOSA implementation may choose to implement any extensions. +* If a TOSA implementation chooses to implement an extension, it must implement the complete extension. +* If a operator / data type combination requires multiple extensions, the combination is only required to be implemented if all extensions are implemented +** For example, a CAST from bf16 to fp8 is only required if both extensions are implemented. .Profiles -|=== -|Profile|Name|Integer Inference|Floating-point Inference|Training +include::{generated}/profiles.adoc[] -|Base Inference|TOSA-BI|Yes|No|No -|Main Inference|TOSA-MI|Yes|Yes|No -|Main Training|TOSA-MT|Yes|Yes|Yes -|=== +.Profile Extensions +include::{generated}/profile_extensions.adoc[] === Levels A TOSA level defines operator argument ranges that an implementation shall support. This is distinct from a profile that defines the operations and data-types supported. +One level must apply to all profiles and extensions supported by an implementation. + This version of the specification defines two TOSA levels: * No level : allows the full range of arguments specified by the operations according to the operation data types. @@ -487,6 +496,8 @@ To map indexes in an output tensor to that of an input tensor, see <`, abbreviated `T`, represents a tensor whose elements are of type `element_type` where `element_type` can be any of the data types supported in TOSA. -* `tensor_list_t` represents a list of tensors. When lists are homogeneous, i.e. contain tensors of the same type, their type is further qualified as follows: `tensor_list_t>`. +* `tensor_list_t` represents a list of tensors. When lists are homogeneous, containing tensors of the same type, their type is further qualified as follows: `tensor_list_t>`. * `tosa_graph_t` represents a TOSA graph (see <>). Arguments belong to one of three categories: Input, Output, or Attribute. The category to which an argument belongs further constrains its type: * An Input argument must be a tensor or a list of tensors used to provide the data read by the operation. * An Output argument must be a tensor or a list of tensors into which the data produced by the operation is written. -* An Attribute argument is constant, i.e. its value is known at compilation time. It may have any data type supported by TOSA. +* An Attribute argument is constant, its value is known at compilation time. It may have any data type supported by TOSA. [[operator-graphs]] === Operator Graphs diff --git a/tools/dictionary.dic b/tools/dictionary.dic index 53377a0..e5a70a2 100644 --- a/tools/dictionary.dic +++ b/tools/dictionary.dic @@ -4,6 +4,7 @@ adoc acc ARGMAX AsciiDoc +BFloat BILINEAR bilinearly bitwise diff --git a/tools/genspec.py b/tools/genspec.py index bdca32c..3865486 100755 --- a/tools/genspec.py +++ b/tools/genspec.py @@ -2,10 +2,25 @@ # Copyright (c) 2023-2024, ARM Limited. # SPDX-License-Identifier: Apache-2.0 import os +import re +from functools import cmp_to_key import tosa +def compare_profiles(a, b): + if a.profiles[0] == b.profiles[0]: + return 1 if a.mode > b.mode else -1 + if "EXT-" in a.profiles[0]: + if "EXT-" in b.profiles[0]: + return 1 if a.profiles[0] > b.profiles[0] else -1 + else: + return 1 + if "EXT-" in b.profiles[0]: + return -1 + return 1 if a.profiles[0] > b.profiles[0] else -1 + + class TOSASpecAsciidocGenerator: def __init__(self, spec): self.spec = spec @@ -73,12 +88,12 @@ class TOSASpecAsciidocGenerator: if op.typesupports: file.write("\n*Supported Data Types:*\n\n") file.write("|===\n") - header = "|Profile|Mode" + header = "|Profile/Extension|Mode" for ty in op.types: header += f"|{ty}" file.write(header) file.write("\n\n") - for tysup in op.typesupports: + for tysup in sorted(op.typesupports, key=cmp_to_key(compare_profiles)): profile = ", ".join(tysup.profiles) if tysup.profiles else "Any" entry = f"|{profile}|{tysup.mode}" for ty in op.types: @@ -108,6 +123,28 @@ class TOSASpecAsciidocGenerator: f.write(" draft") f.write("\n") + # Generate profile table + with open(os.path.join(outdir, "profiles.adoc"), "w") as f: + f.write("|===\n") + f.write("|Profile|Name|Description|Specification Status\n\n") + for profile in self.spec.profiles: + f.write( + f"|{profile.profile}|{profile.name}|" + f"{profile.description}|{profile.status}\n" + ) + f.write("|===\n") + + # Generate profile table + with open(os.path.join(outdir, "profile_extensions.adoc"), "w") as f: + f.write("|===\n") + f.write("|Name|Description|Specification Status\n\n") + for profile_extension in self.spec.profile_extensions: + f.write( + f"|{profile_extension.name}|{profile_extension.description}" + f"|{profile_extension.status}\n" + ) + f.write("|===\n") + # Generate level maximums table with open(os.path.join(outdir, "levels.adoc"), "w") as f: f.write("|===\n") @@ -137,6 +174,49 @@ class TOSASpecAsciidocGenerator: for enum in self.spec.enums: self.generate_enum(enum, f) + all_operators = [] + for group in self.spec.operatorgroups: + for op in group.operators: + all_operators.append(op) + + # Generate profile operator appendix + with open(os.path.join(outdir, "profile_ops.adoc"), "w") as f: + f.write("=== Profiles\n") + for profile in self.spec.profiles: + f.write(f"==== {profile.profile}\n") + f.write(f"{profile.description}\n\n") + f.write(f"Status: {profile.status}\n") + f.write("|===\n") + f.write("|Operator|mode\n\n") + for op in sorted(all_operators, key=lambda o: o.name): + if op.typesupports: + for tysup in op.typesupports: + if profile.name in tysup.profiles: + f.write(f"|{op.name}|{tysup.mode}\n") + f.write("|===\n") + + f.write("=== Profile Extensions\n") + for pext in self.spec.profile_extensions: + f.write(f"==== {pext.name} extension\n") + f.write(f"{pext.description}\n\n") + f.write(f"Status: {pext.status}\n") + f.write("|===\n") + f.write("|Operator|mode|note\n\n") + for op in sorted(all_operators, key=lambda o: o.name): + if op.typesupports: + for tysup in op.typesupports: + for profile in tysup.profiles: + if profile.find(pext.name) != -1: + note = "" + m = re.match(r"(.*) and (.*)", profile) + if m: + if m[1] == pext.name: + note = f"If {m[2]} is also supported" + else: + note = f"If {m[1]} is also supported" + f.write(f"|{op.name}|{tysup.mode}|{note}\n") + f.write("|===\n") + if __name__ == "__main__": import argparse diff --git a/tools/tosa.py b/tools/tosa.py index 803e478..e70b297 100644 --- a/tools/tosa.py +++ b/tools/tosa.py @@ -31,6 +31,23 @@ class TOSAOperatorArgumentCategory: self.profiles = profiles +class TOSAProfile: + def __init__(self, profile, name, description, status): + self.profile = profile + self.name = name + self.description = description + self.status = status + self.ops = [] + + +class TOSAProfileExtension: + def __init__(self, name, description, status): + self.name = name + self.description = description + self.status = status + self.ops = [] + + class TOSAEnum: def __init__(self, name, description, values): self.name = name @@ -95,6 +112,8 @@ class TOSASpec: def __init__(self, xmlpath): tree = ET.parse(xmlpath) self.xmlroot = tree.getroot() + self.profiles = [] + self.profile_extensions = [] self.levels = [] self.operatorgroups = [] self.enums = [] @@ -102,6 +121,12 @@ class TOSASpec: def __load_spec(self): self.__load_version() + for profile in self.xmlroot.findall("./profiles/profile"): + self.profiles.append(self.__load_profile(profile)) + for profile_ext in self.xmlroot.findall( + "./profile_extensions/profile_extension" + ): + self.profile_extensions.append(self.__load_profile_extension(profile_ext)) for level in self.xmlroot.findall("./levels/level"): self.levels.append(self.__load_level(level)) for group in self.xmlroot.findall("./operators/operatorgroup"): @@ -119,6 +144,19 @@ class TOSASpec: else: self.version_is_draft = False + def __load_profile(self, xml_profile): + profile = xml_profile.get("profile") + name = xml_profile.get("name") + description = xml_profile.get("description") + status = xml_profile.get("status") + return TOSAProfile(profile, name, description, status) + + def __load_profile_extension(self, ext): + name = ext.get("name") + description = ext.get("description") + status = ext.get("status") + return TOSAProfileExtension(name, description, status) + def __load_level(self, level): name = level.get("name") desc = level.text.strip() @@ -155,10 +193,17 @@ class TOSASpec: for tysup in op.findall("typesupport"): tsmode = tysup.get("mode") tsmap = {} - profiles = tysup.findall("profile") + profiles = tysup.findall("op_profile") tsprofiles = [] for p in profiles: - tsprofiles.append(p.get("name")) + tsp_name = p.get("name") + and_name = p.get("and_name") + if and_name is not None: + if and_name < tsp_name: + tsp_name = f"{and_name} and {tsp_name}" + else: + tsp_name = f"{tsp_name} and {and_name}" + tsprofiles.append(tsp_name) for ty in types: tsmap[ty] = tysup.get(ty) typesupports.append(TOSAOperatorDataTypeSupport(tsmode, tsmap, tsprofiles)) diff --git a/tosa.xml b/tosa.xml index 27b49b8..17c82f9 100644 --- a/tosa.xml +++ b/tosa.xml @@ -3,10 +3,19 @@ - Base Inference - Main Inference - Main Training + + + + + + + + + + + + No level Level 8K @@ -34,26 +43,26 @@ - - + + + + + + - - + - + - - + - - + - - + @@ -104,30 +113,29 @@ - - + + + + + + - - + - + - - + - - + - - + - - + @@ -192,31 +200,32 @@ - - - + + + + + + + + + - - + - + - - + - - + - - + - - + @@ -285,31 +294,32 @@ - - - + + + + + + + + + - - + - + - - + - - + - - + - - + @@ -374,31 +384,32 @@ - - - + + + + + + + + + - - + - + - - + - - + - - + - - + @@ -439,8 +450,7 @@ - - + @@ -477,24 +487,26 @@ - - - + + + + + + + + + - - + - - + - - + - - + @@ -525,30 +537,29 @@ - - + + + + + + - - + - + - - + - - + - - + - - + @@ -586,26 +597,26 @@ - - + + + + + + - - + - + - - + - - + - - + @@ -638,8 +649,7 @@ - - + @@ -700,31 +710,32 @@ - - - + + + + + + + + + - - + - + - - + - - + - - + - - + @@ -753,19 +764,20 @@ - - + + + + + + - - + - - + - - + @@ -785,16 +797,13 @@ - - + - - + - - + @@ -814,16 +823,13 @@ - - + - - + - - + @@ -843,16 +849,13 @@ - - + - - + - - + @@ -877,18 +880,18 @@ - + + + + - - + - - + - - + @@ -915,9 +918,15 @@ - - - + + + + + + + + + BITWISE_AND @@ -939,9 +948,15 @@ - - - + + + + + + + + + BITWISE_OR @@ -963,9 +978,15 @@ - - - + + + + + + + + + BITWISE_XOR @@ -987,9 +1008,15 @@ - - - + + + + + + + + + INTDIV @@ -1011,7 +1038,10 @@ - + + + + LOGICAL_AND @@ -1033,7 +1063,10 @@ - + + + + LOGICAL_LEFT_SHIFT @@ -1055,9 +1088,18 @@ - - - + + + + + + + + + + + + LOGICAL_RIGHT_SHIFT @@ -1079,9 +1121,18 @@ - - - + + + + + + + + + + + + LOGICAL_OR @@ -1103,7 +1154,10 @@ - + + + + LOGICAL_XOR @@ -1125,7 +1179,10 @@ - + + + + MAXIMUM @@ -1147,18 +1204,17 @@ - + + + - - + - - + - - + @@ -1181,18 +1237,17 @@ - + + + - - + - - + - - + @@ -1206,7 +1261,7 @@ Input tensor with the same rank as input1 - + Result right shift (i32_t data type only) @@ -1220,20 +1275,24 @@ - - - + + + + + + + + + + - - + - - + - - + @@ -1257,16 +1316,13 @@ - - + - - + - - + @@ -1289,18 +1345,18 @@ - + + + + - - + - - + - - + @@ -1310,7 +1366,7 @@ Input tensor - + Lookup table tensor @@ -1326,8 +1382,12 @@ - - + + + + + + @@ -1347,18 +1407,17 @@ - + + + - - + - - + - - + @@ -1377,9 +1436,15 @@ - - - + + + + + + + + + CEIL @@ -1398,16 +1463,13 @@ - - + - - + - - + @@ -1426,7 +1488,9 @@ - + + + COS @@ -1445,16 +1509,13 @@ - - + - - + - - + @@ -1474,16 +1535,13 @@ - - + - - + - - + @@ -1503,16 +1561,13 @@ - - + - - + - - + @@ -1532,16 +1587,13 @@ - - + - - + - - + @@ -1560,7 +1612,10 @@ - + + + + NEGATE @@ -1587,20 +1642,23 @@ - - - + + + + + + + + + - - + - - + - - + @@ -1620,16 +1678,13 @@ - - + - - + - - + @@ -1649,16 +1704,13 @@ - - + - - + - - + @@ -1687,21 +1739,27 @@ - - - - + + + + + + + + + + + + + - - + - - + - - + @@ -1721,16 +1779,13 @@ - - + - - + - - + @@ -1756,18 +1811,17 @@ - + + + - - + - - + - - + @@ -1791,18 +1845,17 @@ - + + + - - + - - + - - + @@ -1826,18 +1879,17 @@ - + + + - - + - - + - - + @@ -1861,7 +1913,10 @@ - + + + + REDUCE_ANY @@ -1882,7 +1937,10 @@ - + + + + REDUCE_MAX @@ -1903,20 +1961,23 @@ - - - + + + + + + + + + - - + - - + - - + @@ -1938,20 +1999,23 @@ - - - + + + + + + + + + - - + - - + - - + @@ -1974,16 +2038,13 @@ - - + - - + - - + @@ -2006,18 +2067,17 @@ - + + + - - + - - + - - + @@ -2042,28 +2102,33 @@ - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2091,28 +2156,33 @@ - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2134,28 +2204,33 @@ - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2178,28 +2253,33 @@ - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2222,28 +2302,33 @@ - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2271,28 +2356,33 @@ used. - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2315,28 +2405,33 @@ used. - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2359,28 +2454,33 @@ used. - - - - + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2404,27 +2504,29 @@ used. - - - + + + + + + + + + - - + - + - - + - - + - - + @@ -2450,27 +2552,29 @@ used. - - - + + + + + + + + + - - + - + - - + - - + - - + @@ -2510,21 +2614,26 @@ used. - - - - + + + + + + + + + + + + - - + - - + - - + @@ -2546,147 +2655,143 @@ used. - - - - - - + + + + + + + + + + + + + + + + + + - - + - - + - - + + + + + + + + + + - - - - - + - - + - - + + + + + + + + + + - - - - - + - - + - - + - - + - - + - - + - - + - + - - + - - + - - + - - + - + - + - + - - + - - + - - + - - + - + - - + - - + - - + - - + - - + - + - - + - - + @@ -2709,11 +2814,11 @@ used. Output tensor zero point.int8/uint8 can have zero point within their valid range. uint16 zero point must be either 0 or 32768. All other types must have zero point equal to 0. - + Scaling multiplier array - + Scaling shift array @@ -2742,18 +2847,42 @@ used. - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -2772,30 +2901,38 @@ used. - - - - - - + + + + + + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2813,30 +2950,38 @@ used. - - - - - - + + + + + + + + + + + + + + + + + + - - + - + - - + - - + - - + @@ -2862,6 +3007,13 @@ used. List of output tensors + + + + + + + @@ -2885,6 +3037,13 @@ used. List of output tensors + + + + + + + WHILE_LOOP @@ -2902,6 +3061,13 @@ used. List of output tensors + + + + + + + @@ -2926,6 +3092,18 @@ used. + + + + + + + + + + + + VARIABLE_WRITE @@ -2940,6 +3118,21 @@ used. + + + + + + + + + + + + + + + VARIABLE_READ @@ -2954,6 +3147,21 @@ used. + + + + + + + + + + + + + + + @@ -2970,6 +3178,14 @@ used. Output shape + + + + + + + + CONCAT_SHAPE @@ -2981,6 +3197,14 @@ used. Output shape + + + + + + + + CONST_SHAPE @@ -2992,6 +3216,14 @@ used. Output shape + + + + + + + + DIV_SHAPE @@ -3006,6 +3238,14 @@ used. Output shape + + + + + + + + MUL_SHAPE @@ -3020,6 +3260,14 @@ used. Output shape + + + + + + + + SUB_SHAPE @@ -3034,6 +3282,14 @@ used. Output shape + + + + + + + + diff --git a/tosa.xsd b/tosa.xsd index e0afbe2..b39a2f4 100644 --- a/tosa.xsd +++ b/tosa.xsd @@ -3,11 +3,18 @@ - + + + + + + + + @@ -66,6 +73,9 @@ + + + @@ -104,16 +114,6 @@ - - - - - - - - - - @@ -137,6 +137,23 @@ + + + + + + + + + + + + + + + + + @@ -145,6 +162,34 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -238,8 +283,8 @@ - - + + @@ -249,7 +294,11 @@ + + + + @@ -286,6 +335,7 @@ + diff --git a/tosa_spec.adoc b/tosa_spec.adoc index fb6b43e..7a07fbd 100644 --- a/tosa_spec.adoc +++ b/tosa_spec.adoc @@ -27,3 +27,5 @@ include::chapters/enumerations.adoc[] include::chapters/pseudocode.adoc[] include::chapters/appendix_a.adoc[] + +include::chapters/appendix_b.adoc[] -- cgit v1.2.1