1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
//
// This confidential and proprietary software may be used only as
// authorised by a licensing agreement from ARM Limited
// (C) COPYRIGHT 2020-2024 ARM Limited
// ALL RIGHTS RESERVED
// The entire notice above must be reproduced on all authorised
// copies and copies may only be made to the extent permitted
// by a licensing agreement from ARM Limited.
=== Activation Functions
==== CLAMP
Clamp to an arbitrary minimum and maximum value.
Maximum and minimum values are specified as values in the range of the input type.
No zero point subtraction is done to the values, thus to clamp to the zero point value, the zero point itself should be supplied as the minimum value.
include::{generated}/operators/CLAMP.adoc[]
[source,c++]
----
include::{pseudocode}/operators/CLAMP.tosac[lines=10..-1]
----
==== ERF
// erf(z) = \frac{2}{\sqrt{\pi}} \int_{0}^{z} e^{-t^2} dt
Error function:
.Calculation for the error function
image::erf.svg["Error function", align="center"]
For quantized integer data types, the TABLE operator should be used instead with
the following definition.
The ERF table has 513 entries each of 16-bit precision and covering the input range -4.0 to +4.0 in steps of 1/64.
[source,c++]
----
include::{pseudocode}/operators/tables/ERF.tosac[lines=10..-1]
----
include::{generated}/operators/ERF.adoc[]
==== SIGMOID
Applies the sigmoid logistic function to each element of the input tensor.
// sigmoid(x) = \frac{1}{1 + e^{-x}}
.Calculation for the sigmoid function
image::sigmoid.svg["Sigmoid definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit sigmoid.
This sigmoid table has 513 entries each of 16-bit precision and covering the input range -16.0 to +16.0 in steps of 1/16.
.Code for generating 16-bit sigmoid table
[source,c++]
----
include::{pseudocode}/operators/tables/SIGMOID.tosac[lines=10..-1]
----
include::{generated}/operators/SIGMOID.adoc[]
[source,c++]
----
include::{pseudocode}/operators/SIGMOID.tosac[lines=10..-1]
----
==== TANH
Parameterized hyperbolic tangent.
// tanh(x) = \frac{1 - e^{-2x}}{1 + e^{-2x}}
.Calculation for the sigmoid function
image::tanh.svg["Hyperbolic tangent definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit hyperbolic tangent.
This tanh_table has 513 entries each of 16-bit precision and covering the input range -8.0 to +8.0 in steps of 1/32.
.Calculation of an example 16-bit tanh table
[source,c++]
----
include::{pseudocode}/operators/tables/TANH.tosac[lines=10..-1]
----
include::{generated}/operators/TANH.adoc[]
[source,c++]
----
include::{pseudocode}/operators/TANH.tosac[lines=10..-1]
----
|
