From b8522ba72949b3eca03410f931f2655ee8485fa5 Mon Sep 17 00:00:00 2001 From: Eric Kunze Date: Thu, 30 Jun 2022 14:46:07 -0700 Subject: Relax limit on input value for RESCALE The current value is too strict, and can lead to some cases to fail. By making the new value limit between (-1 << shift-1) and (1 << shift-1), we still restrict the input value a range that works for optimized implementations that left shift their values before multiplying. Signed-off-by: Eric Kunze Change-Id: Ib4f49732e361418884888d398087560da3ccd6e8 --- chapters/introduction.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/chapters/introduction.adoc b/chapters/introduction.adoc index 93206ca..be475d4 100644 --- a/chapters/introduction.adoc +++ b/chapters/introduction.adoc @@ -352,7 +352,7 @@ The values to achieve a scaling of 1.0 are shift=30, multiplier=1<<30 for apply_ int32_t apply_scale_32(int32_t value, int32_t multipler, uint6_t shift, bool_t double_round=false) { REQUIRE(multiplier >= 0); REQUIRE(2 <= shift && shift <= 62); - REQUIRE(value >= (-1<<(shift-2)) && value < (1<<(shift-2)); + REQUIRE(value >= (-1 << (shift - 1)) && value < (1 << (shift - 1)); int64_t round = 1 << (shift - 1); if (double_round) { if (shift > 31 && value >= 0) round += 1<<30; -- cgit v1.2.1