py/parsenum: fix rounding error when float ends in .0#5832
Conversation
dlech
force-pushed
the
float-bug
branch
2 times, most recently
from
c8e8da6 to
62a9539
Compare
It is common code style to write floating point numbers with trailing `.0` instead of just trailing `.`. When there is a trailing `.0`, the current parser will multiply the number by 10 and then divide it by 10 again later. For certain numbers, this can cause rounding. This adds a check for the special case of `.0` to avoid the extra work. Also it avoids a call to pow() and extra multiplication when the exp_val is 0. Even in cases where there is not a rounding problem, this should be more efficient, especially for soft float implementations.
|
How about checking all digits for 0 as you suggest in the issue? Then the behavior is correct without edge cases. |
|
I thought about it, but performance and code size also have to be taken into consideration. Since there are no instances in micropython and micropython-lib where x.00 is used, but if black is used to format code there will be many instances of x.0, this seems like a good compromise. Of course, I'm glad to change it if the consensus is different from my opinion. |
|
I would request that you check for any number of 0's. (Sometimes I use this criterion: am I going to spend a lot more time explaining this and are people going to spend a lot more time debugging something than the CPU time that would be saved?) In terms of performance, I think the check will be swamped by the rest of the conversion, and I don't think the code size increase would be significant. I think the whole conversion algorithm could use revisiting, but there are several other issues on that topic. |
|
superseded by #6024 |
ESP32S2 camera board draft
|
n.b. this remains an issue: .. because |
|
This was never merged. It was closed in favour of #6024, which is also not merged. |
5 participants
It is common code style to write floating point numbers with trailing
.0instead of just trailing.. When there is a trailing.0, the current parser will multiply the number by 10 and then divide it by 10 again later. For certain numbers, this can cause rounding.This adds a check for the special case of
.0to avoid the extra work. Even in cases where there is not a rounding problem, this should be more efficient, especially for soft float implementations.Fixes #5831