Closed Bug 186563 Opened 22 years ago Closed 17 years ago

Number.toFixed: loss of precision

Categories

(Core :: JavaScript Engine, defect)

x86
Windows 2000
defect
Not set
critical

Tracking

()

VERIFIED INVALID

People

(Reporter: lapsap7+mz, Unassigned)

References

Details

(Keywords: testcase)

Attachments

(2 files, 2 obsolete files)

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201 In the following code, for 17 < f <= 20, toFixed loses precision: var a = 0.827; var b = 1.827; var f = 17; document.write( "a = " + a + " ==> a.toFixed(" + f + ") = " + a.toFixed(f) + "<br>" + "b = " + b + " ==> b.toFixed(" + f + ") = " + b.toFixed(f)) Reproducible: Always Steps to Reproduce:
(OUTPUT IN MOZILLA): a = 0.827 ==> a.toFixed(17) = 0.82699999999999996 b = 1.827 ==> b.toFixed(17) = 1.82699999999999996 This type of behavior is seen with many types of numerical operations in JavaScript, for example with multiplication: 5.1 * 10 51 5.1 * 100 509.99999999999994 The reason for this is the following: computers do arithmetic in base 2, not base 10. Certain numbers which have finite decimal expansions in base 10 have infinite decimal expansions in base 2. That, plus the fact that the computer must round off infinite expansions to finite ones, produces the results above. The algorithm for Number.prototype.toFixed involves arithmetic operations which are done in C (in the SpiderMonkey implementation of JavaScript). The C language adheres to the ANSI/IEEE standard 754 for floating-point arithmetic, which requires this behavior. That is why all the following bugs were marked invalid: bug 20140 "Rounding Error?" bug 42134 "JavaScript should be more lenient comparing (==) floats" bug 129300 "Error in javascript internal routine : parseFloat" bug 154103 "Arithmetic error in JavaScript Engine" bug 160173 "Error de punto flotante al sumar en JavaScript" Note that Netscape's web documentation is quite clear about loss of precision that may be encountered when using Number.prototype.toFixed : http://developer.netscape.com/docs/manuals/js/core/jsref15/number.html#1200964 contains an excellent explanation of some of the unexpected behavior. However, I will defer to Roger or Waldemar on the current bug, because IE6 seems to behave differently than we do in this case: OUTPUT IN MOZILLA: a = 0.827 ==> a.toFixed(17) = 0.82699999999999996 b = 1.827 ==> b.toFixed(17) = 1.82699999999999996 OUTPUT IN IE6: a = 0.827 ==> a.toFixed(17) = 0.82700000000000000 b = 1.827 ==> b.toFixed(17) = 1.82700000000000000 My questions for Roger or Waldemar are: 1. Has SpiderMonkey (also Rhino) violated the spec in any way here? 2. Even if not, would we want to emulate IE in cases like this?
Assignee: rogerl → khanson
Phil's analysis of the observed behavior is correct. The difference seen in IE and Mozilla is as follows. IE is storing 'a' as a string and Mozilla is storing 'a' as a value. The spec doesn't nail down the storage format. Thus when IE does a.toFixed it starts out with a exact string representation while Mozilla suffers the round trip conversions described by Phil.
Kenton: thanks! I will go ahead and mark this invalid, then. If anyone objects, please reopen -
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → INVALID
Marking Verified. seak_tengfong@yahoo.com: thank you for this report. This is an interesting issue that often comes up -
Status: RESOLVED → VERIFIED
Hi Phil, Glad you find this interesting. I hope reopening is appropriate. I am very reluctant to reopen this. // This should be 1.126, not 1.125. // Webkit, Spidermonkey, Futhark are all wrong. (1.1255).toFixed(3); for convenience: javascript:alert((1.1255).toFixed(3)) http://bclary.com/2004/11/07/#a-15.7.4.5 Look at steps 10 and 11. 10. Let n be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible. If there are two such n, pick the larger n. 11. If n =0, let m be the string "0". Otherwise, let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes). I've tried to correctly implement the algorithm in JS. ======================================================== /** * - Fix for rounding error in JScript. * - Fix for rounding in all other engines - step 10, adjust precision. * - Range for fractionDigits expanded to {-20...100} */ Number.prototype.toFixed = function(fractionDigits) { var f = parseInt(fractionDigits) || 0; if( f < -20 || f > 100 ) { throw new RangeError("Precision of " + f + " fractional digits is out of range"); } var x = Number(this); if( isNaN(x) ) { return "NaN"; } var s = ""; if(x <= 0) { s = "-"; x = -x; } if( x >= Math.pow(10, 21) ) { return s + x.toString(); } var m; // 10. Let n be an integer for which the exact mathematical value of // n ÷ 10^f - x is as close to zero as possible. // If there are two such n, pick the larger n. n = Math.round(x * Math.pow(10, f) ); if( n == 0 ) { m = "0"; } else { // let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes). m = n.toString(); } if( f == 0 ) { return s + m; } var k = m.length; if(k <= f) { var z = Math.pow(10, f+1-k).toString().substring(1); m = z + m; k = f+1; } if(f > 0) { var a = m.substring(0, k-f); var b = m.substring(k-f); m = a + "." + b; } return s + m; }; I had to improvise for step 10. The algorithm in the spec omits the detail. The spec doesn't nail down the storage format, as Kentop pointed out, but it does say that the larger number should be taken. I provided no code there to "pick the larger of the two," as the spec dictates. The function above produces the result for things like: (1.1255).toFixed(3); // 1.126 - correctly rounded. (0.827 ).toFixed(17); // No loss of precision. This behavior seem more practial and useful. ========================================================= Related bugs: Bug 397880 – Incorrect rounding by toFixed() for some values when last decimal digit is 5 (edit) Bug 226993 – Javascript toFixed rounds in unexpected manner (edit) Where can I find Mozilla's source code for Number.prototype.toFixed(fractionDigits)?
Status: VERIFIED → REOPENED
Resolution: INVALID → ---
Attached file Test case showing original vs patched versions (obsolete) (deleted) —
Test case showing original vs patched versions of Number.prototype.toFixed Allows for negative formatDigits, which was not properly supported in last post.
In the attachment, (.07).toFixed() == 0 can be ignored. This will always be true.
Attached file More Complete Test Case (obsolete) (deleted) —
Mozilla and Opera have rounding errors with (1.1255).toFixed(3) and (1.1255).toFixed(17), losing precision. Safari 2 got the wrong result with (1.1255).toFixed(3), but got the correct result with (1.1255).toFixed(17). Only Safari 3 got it right the first time. The RangeError it throws on a negative fractionDigits is perfectly valid. Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification.
Attachment #283513 - Attachment is obsolete: true
Attached file More Complete Test Case Description (deleted) —
Noticed that for .07, Safari 2 returns .1, but should instead return 0.1. Only Safari 3 gets the correct result.
Attachment #283857 - Attachment is obsolete: true
Instead of using Math.round, it would be more true to the spec to implement step 10 with the following: //---------------------------------------------------------------------------- // 10. Let n be an integer for which the exact mathematical value // of n ÷ 10^f - x is as close to zero as possible. var fTo10 = Math.pow(10, f); n = Math.floor(x * fTo10 ); // If there are two such n, pick the larger n. if (Math.abs(n / fTo10 - x) >= Math.abs((n + 1) / fTo10 - x)) { ++n; } //---------------------------------------------------------------------------- Math.round works correctly. http://bclary.com/2004/11/07/#a-15.8.2.15 So should Number.prototype.toFixed. It seems likely that the native code uses something like my first patch, but chooses error-prone rounding, and does not implement rounding as Math.round does. I can guess this to be the problem; where is the !#$@ing source code? I have found no problems with the first patch I wrote using javascript's Math.round. The second approach I have written should be translated to native code. I do, however, have a problem with the current implementation, which yeilds: (1.1255).toFixed(3) // 1.125 (.1255).toFixed(3) // 0.126
toPrecision is also broken in all browsers but IE. (I did not test Opera 9.5b/Futhark Engine, but it is probably broken there, too) javascript:alert(1.1255.toPrecision(4)) toExponential is broken in all browsers but IE and Safari 3/Webkit javascript:alert(1.1255.toExponential(20)) The JS Guide documentation for toPrecision is wrong. Found with Google Search: toPrecision documentation http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Predefined_Core_Objects:Number_Object "Returns a string representing the number to a specified precision in fixed-point notation." Incorrect and incomplete. The reference documentation is correct, but avoids discussion of the rounding errors/bugs in Mozilla: http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Global_Objects:Number ------------------------------------------------------------------------------ We need a meta bug with dependencies of the following: toPrecision, toFixed, toExponential For each of the above, there should be a bug rep't tests of boundary and error-prone values. * run the tests * write patches/get review/checkin et c.
Per the ECMAScript standard, the correct answer for 1.1255.toPrecision(4) is "1.125". This is what Mozilla produces. This is not a bug. The reason is that 1.1255 cannot be represented exactly as an IEEE double, so internally it is represented as 1.1254999999999999449329379785922355949878692626953125.
Status: REOPENED → RESOLVED
Closed: 22 years ago17 years ago
Resolution: --- → INVALID
In reply to #13: If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard.
We're doing that. ES4 will also have a decimal number type.
The code example, follows the ECMA 262 algorithm and gets a different result than the native code. A more desirable one, I might add. Does the example follow the algorithm incorrectly? If so, where. Where is the source code so we can have a look?
Status: RESOLVED → REOPENED
Keywords: testcase
Resolution: INVALID → ---
Assignee: khanson → general
Status: REOPENED → NEW
QA Contact: pschwartau → general
Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the very beginning -- it evaluates the token "1.1255" incorrectly. This token evaluates to 1.1254999999999999449329379785922355949878692626953125. This is required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls apart after this. You also wrote "Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification." How is this relevant here? If you're referring to the note at the end of 15.7.4.5, then please re-read that note. What it's stating is that instead of throwing RangeError, toFixed is allowed to, upon request, correctly produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5, we've established that the value given is 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can see all of its digits in the Mozilla implementation of toFixed.
Status: NEW → RESOLVED
Closed: 17 years ago17 years ago
Resolution: --- → INVALID
In the example the problem is in the emulation of step 10. The spec requires you to pick n to be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible. If there are two such n, pick the larger n. This is not implemented by n = Math.round(x * Math.pow(10, f)) because that expression has double (or triple) rounding. You round once in the * operator and again in Math.round; there may be a third rounding in Math.pow depending on the value of f. The spec requires exact arithmetic, not rounded intermediate results.
(In reply to comment #17) > Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the > very beginning -- it evaluates the token "1.1255" incorrectly. This token > evaluates to 1.1254999999999999449329379785922355949878692626953125. This is > required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls > apart after this. > > You also wrote "Aside from fixing the bugs in the other browsers, this patch > enhances toFixed with a greater precision. This is permitted by the > specification." How is this relevant here? If you're referring to the note at > the end of 15.7.4.5, then please re-read that note. What it's stating is that > instead of throwing RangeError, toFixed is allowed to, upon request, correctly > produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5, > we've established that the value given is > 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can > see all of its digits in the Mozilla implementation of toFixed. > Are you saying 1.1255.valueOf() should return 1.1254999999999999449329379785922355949878692626953125
Not only that, merely 1.1255 returns the exact mathematical value 1.1254999999999999449329379785922355949878692626953125. It happens to print as "1.1255" if you convert it back to a string, but it's actually less than the exact mathematical value 1.1255.
Garrett: now you may appreciate why decimal is coming in JS2/ES4 ;-). /be
Status: RESOLVED → VERIFIED
(In reply to comment #19) > (In reply to comment #17) > > Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the > > very beginning -- it evaluates the token "1.1255" incorrectly. This token > > evaluates to 1.1254999999999999449329379785922355949878692626953125. This is > > required by ECMA 262 sections 7.8.3 and 8.5.ntation of toFixed. > > [snip] > Are you saying 1.1255.valueOf() should return > 1.1254999999999999449329379785922355949878692626953125 No, that's not what waldemar wrote. The point is that implementations must scan the 1.1255 token using extra internal precision, necessary for correct rounding to the binary precision limits of IEEE double. Nothing in the spec requires or indeed allows visible results in the language to have greater than double precision. Which is why we have bugs on file about hardware such as modern Macs, which oddly use 80-bit precision in the x87 FPU but 64-bit in SSE (Mac OS X does not seem to allow you to override this strange FPU configuration; if anyone knows how, please mail me). /be
Makes sense. Looking fwd to ES4.
(In reply to comment #5) Thank you Garrett for this FIX ! I need a method to obtain a mathematically correct response. All this internal stuff on the specification is not a responsible response to a practical problem encountered by users. I agree with comment #14 "If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard".
here's another one: 565133671234567.89.toFixed(2) 565133671234567.88
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: