1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
|
//
// This confidential and proprietary software may be used only as
// authorised by a licensing agreement from ARM Limited
// (C) COPYRIGHT 2020-2021 ARM Limited
// ALL RIGHTS RESERVED
// The entire notice above must be reproduced on all authorised
// copies and copies may only be made to the extent permitted
// by a licensing agreement from ARM Limited.
=== Activation Functions
==== CLAMP
Clamp to an arbitrary minimum and maximum value.
Maximum and minimum values are specified as values in the range of the input type.
No zero point subtraction is done to the values, thus to clamp to the zero point value, the zero point itself should be supplied as the minimum value.
include::{generated}/operators/CLAMP.adoc[]
[source,c++]
----
ERROR_IF(max_val < min_val);
for_each(index in shape) {
in_out_t value = tensor_read<in_out_t>(input, shape, index);
value = apply_clip<in_out_t>(value, min_val, max_val);
tensor_write<in_out_t>(output, shape, index, value);
}
----
==== ERF
// erf(z) = \frac{2}{\sqrt{\pi}} \int_{0}^{z} e^{-t^2} dt
Error function:
.Calculation for the error function
image::erf.svg["Error function", align="center"]
For quantized integer data types, the TABLE operator should be used instead with
the following definition.
The ERF table has 513 entries each of 16-bit precision and covering the input range -4.0 to +4.0 in steps of 1/64.
[source,c++]
----
int16_t erf_reference(int16_t x) { // input x range is -256 to + 256 inclusive
F64 v = (double)x / (double)64;
v = erf(v);
return round_to_nearest_int(32768.0 * v);
}
generate_lookup_table(&erf_table, &erf_reference);
----
include::{generated}/operators/ERF.adoc[]
==== SIGMOID
Applies the sigmoid logistic function to each element of the input tensor.
// sigmoid(x) = \frac{1}{1 + e^{-x}}
.Calculation for the sigmoid function
image::sigmoid.svg["Sigmoid definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit sigmoid.
This sigmoid table has 513 entries each of 16-bit precision and covering the input range -16.0 to +16.0 in steps of 1/16.
.Code for generating 16-bit sigmoid table
[source,c++]
----
int16_t sigmoid_reference(int16_t x) { // input x range is -256 to + 256 inclusive
fp64_t v = (fp64_t)x / (fp64_t)16;
v = 1.0/(1.0 + exp(-v));
return round_to_nearest_int(32768.0 * v);
}
generate_lookup_table(&sigmoid_table, &sigmoid_reference);
----
include::{generated}/operators/SIGMOID.adoc[]
[source,c++]
----
for_each(index in shape) {
in_out_t value1 = tensor_read<in_out_t>(input, shape, index);
value = sigmoid<in_out_t>(value1);
tensor_write<in_out_t>(output, shape, index, value);
}
----
==== TANH
Parameterized hyperbolic tangent.
// tanh(x) = \frac{1 - e^{-2x}}{1 + e^{-2x}}
.Calculation for the sigmoid function
image::tanh.svg["Hyperbolic tangent definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit hyperbolic tangent.
This tanh_table has 513 entries each of 16-bit precision and covering the input range -8.0 to +8.0 in steps of 1/32.
.Calculation of an example 16-bit tanh table
[source,c++]
----
int16_t tanh_reference(int16_t x) { // input x range is -256 to +256 inclusive
fp64_t v = (fp64_t)x/(fp64_t)32;
v = exp(-2.0*v);
v = (1.0-v)/(1.0+v);
return round_to_nearest_int(32768.0 * v);
}
generate_lookup_table(&tanh_table, &tanh_reference);
----
include::{generated}/operators/TANH.adoc[]
[source,c++]
----
for_each(index in shape) {
in_out_t value1 = tensor_read<in_out_t>(input, shape, index);
value = tanh<in_out_t>(value1);
tensor_write<in_out_t>(output, shape, index, value);
}
----
|
