1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
//
// This confidential and proprietary software may be used only as
// authorised by a licensing agreement from ARM Limited
// (C) COPYRIGHT 2020-2021 ARM Limited
// ALL RIGHTS RESERVED
// The entire notice above must be reproduced on all authorised
// copies and copies may only be made to the extent permitted
// by a licensing agreement from ARM Limited.
=== Activation Functions
==== CLAMP
Clamp to an arbitrary minimum and maximum value.
Maximum and minimum values are specified as values in the range of the input type.
No zero point subtraction is done to the values, thus to clamp to the zero point value, the zero point itself should be supplied as the minimum value.
include::{generated}/operators/CLAMP.adoc[]
[source,c++]
----
ERROR_IF(max_val < min_val);
for_each(index in shape) {
in_out_t value = tensor_read<in_out_t>(input, shape, index);
value = apply_clip<in_out_t>(value, min_val, max_val);
tensor_write<in_out_t>(output, shape, index, value);
}
----
==== SIGMOID
Applies the sigmoid logistic function to each element of the input tensor.
// sigmoid(x) = \frac{1}{1 + e^{-x}}
.Calculation for the sigmoid function
image::sigmoid.svg["Sigmoid definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit sigmoid.
This sigmoid table has 513 entries each of 16-bit precision and covering the input range -16.0 to +16.0 in steps of 1/16.
.Code for generating 16-bit sigmoid table
[source,c++]
----
int16_t sigmoid_reference(int16_t x) { // input x range is -256 to + 256 inclusive
fp64_t v = (fp64_t)x / (fp64_t)16;
v = 1.0/(1.0 + exp(-v));
return round_to_nearest_int(32768.0 * v);
}
generate_lookup_table(&sigmoid_table, &sigmoid_reference);
----
include::{generated}/operators/SIGMOID.adoc[]
[source,c++]
----
for_each(index in shape) {
in_out_t value1 = tensor_read<in_out_t>(input, shape, index);
value = sigmoid<in_out_t>(value1);
tensor_write<in_out_t>(output, shape, index, value);
}
----
==== TANH
Parameterized hyperbolic tangent.
// tanh(x) = \frac{1 - e^{-2x}}{1 + e^{-2x}}
.Calculation for the sigmoid function
image::tanh.svg["Hyperbolic tangent definition"]
For quantized integer data types, the TABLE operator should be used instead.
Each implementation may choose an appropriate TABLE given the scale and zero point of the input data.
Eight or sixteen bit precision tables may be used based on the input tensor to the sigmoid function.
Below we give an example table generation for 16-bit hyperbolic tangent.
This tanh_table has 513 entries each of 16-bit precision and covering the input range -8.0 to +8.0 in steps of 1/32.
.Calculation of an example 16-bit tanh table
[source,c++]
----
int16_t tanh_reference(int16_t x) { // input x range is -256 to +256 inclusive
fp64_t v = (fp64_t)x/(fp64_t)32;
v = exp(-2.0*v);
v = (1.0-v)/(1.0+v);
return round_to_nearest_int(32768.0 * v);
}
generate_lookup_table(&tanh_table, &tanh_reference);
----
include::{generated}/operators/TANH.adoc[]
[source,c++]
----
for_each(index in shape) {
in_out_t value1 = tensor_read<in_out_t>(input, shape, index);
value = tanh<in_out_t>(value1);
tensor_write<in_out_t>(output, shape, index, value);
}
----
|
