You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The discrepancy between line 4 and 5 is the bug. In one case the nominal value is rounded to 0.9993 and in another it is rounded to 9.992e-01. Based on the printout of the nominal value I would say case 4 has the right value.
This must be some bug that happens during the conversion to exponential format combine with floating point roundoff.
For what it's worth, sciform has what I consider to be the correct behavior on this example:
sciform avoids these sorts of floating point rounding issues by immediately casting all inputs to Decimal and then working with that representation. Some thought was generally put into Decimal representation strategy and rounding. See this example.
Resolution to this issue depends on the route we choose to go at #192. If we choose to use sciform as the formatting backend then this issue will be resolved by that change. Otherwise someone (likely me as the resident formatting expert) would need to dig into what is going wrong and resolve it. Might be easy, might be a rabbit hole!
The text was updated successfully, but these errors were encountered:
@jagerber48 My inclination is to do as little "fix formatting" here, with uncertainties, and rather a) move the existing formatting to a module where it can remain without much change or development, and b) work toward allowing other formatters, say with sciform.
Deferring to "registered user code" as pint would be OK, but I think sciform an optional dependency for "better formatting" would be preferable.
I came across this formatting bug "in the wild":
The discrepancy between line 4 and 5 is the bug. In one case the nominal value is rounded to
0.9993
and in another it is rounded to9.992e-01
. Based on the printout of the nominal value I would say case 4 has the right value.This must be some bug that happens during the conversion to exponential format combine with floating point roundoff.
For what it's worth, sciform has what I consider to be the correct behavior on this example:
sciform
avoids these sorts of floating point rounding issues by immediately casting all inputs toDecimal
and then working with that representation. Some thought was generally put intoDecimal
representation strategy and rounding. See this example.Resolution to this issue depends on the route we choose to go at #192. If we choose to use
sciform
as the formatting backend then this issue will be resolved by that change. Otherwise someone (likely me as the resident formatting expert) would need to dig into what is going wrong and resolve it. Might be easy, might be a rabbit hole!The text was updated successfully, but these errors were encountered: