Closed
Bug 186563
Opened 22 years ago
Closed 17 years ago
Number.toFixed: loss of precision
Categories
(Core :: JavaScript Engine, defect)
Tracking
()
VERIFIED
INVALID
People
(Reporter: lapsap7+mz, Unassigned)
References
Details
(Keywords: testcase)
Attachments
(2 files, 2 obsolete files)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201
In the following code, for 17 < f <= 20, toFixed loses precision:
var a = 0.827;
var b = 1.827;
var f = 17;
document.write(
"a = " + a + " ==> a.toFixed(" + f + ") = " + a.toFixed(f) + "<br>" +
"b = " + b + " ==> b.toFixed(" + f + ") = " + b.toFixed(f))
Reproducible: Always
Steps to Reproduce:
Comment 1•22 years ago
|
||
(OUTPUT IN MOZILLA):
a = 0.827 ==> a.toFixed(17) = 0.82699999999999996
b = 1.827 ==> b.toFixed(17) = 1.82699999999999996
This type of behavior is seen with many types of numerical
operations in JavaScript, for example with multiplication:
5.1 * 10
51
5.1 * 100
509.99999999999994
The reason for this is the following: computers do arithmetic
in base 2, not base 10. Certain numbers which have finite decimal
expansions in base 10 have infinite decimal expansions in base 2.
That, plus the fact that the computer must round off infinite
expansions to finite ones, produces the results above.
The algorithm for Number.prototype.toFixed involves arithmetic
operations which are done in C (in the SpiderMonkey implementation
of JavaScript). The C language adheres to the ANSI/IEEE standard 754
for floating-point arithmetic, which requires this behavior.
That is why all the following bugs were marked invalid:
bug 20140 "Rounding Error?"
bug 42134 "JavaScript should be more lenient comparing (==) floats"
bug 129300 "Error in javascript internal routine : parseFloat"
bug 154103 "Arithmetic error in JavaScript Engine"
bug 160173 "Error de punto flotante al sumar en JavaScript"
Note that Netscape's web documentation is quite clear about loss of
precision that may be encountered when using Number.prototype.toFixed :
http://developer.netscape.com/docs/manuals/js/core/jsref15/number.html#1200964
contains an excellent explanation of some of the unexpected behavior.
However, I will defer to Roger or Waldemar on the current bug,
because IE6 seems to behave differently than we do in this case:
OUTPUT IN MOZILLA:
a = 0.827 ==> a.toFixed(17) = 0.82699999999999996
b = 1.827 ==> b.toFixed(17) = 1.82699999999999996
OUTPUT IN IE6:
a = 0.827 ==> a.toFixed(17) = 0.82700000000000000
b = 1.827 ==> b.toFixed(17) = 1.82700000000000000
My questions for Roger or Waldemar are:
1. Has SpiderMonkey (also Rhino) violated the spec in any way here?
2. Even if not, would we want to emulate IE in cases like this?
Assignee: rogerl → khanson
Comment 2•22 years ago
|
||
Phil's analysis of the observed behavior is correct. The difference seen in IE
and Mozilla is as follows. IE is storing 'a' as a string and Mozilla is storing
'a' as a value. The spec doesn't nail down the storage format. Thus when IE
does a.toFixed it starts out with a exact string representation while Mozilla
suffers the round trip conversions described by Phil.
Comment 3•22 years ago
|
||
Kenton: thanks! I will go ahead and mark this invalid, then.
If anyone objects, please reopen -
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → INVALID
Comment 4•22 years ago
|
||
Marking Verified.
seak_tengfong@yahoo.com: thank you for this report.
This is an interesting issue that often comes up -
Status: RESOLVED → VERIFIED
Comment 5•17 years ago
|
||
Hi Phil,
Glad you find this interesting. I hope reopening is appropriate. I am very reluctant to reopen this.
// This should be 1.126, not 1.125.
// Webkit, Spidermonkey, Futhark are all wrong.
(1.1255).toFixed(3);
for convenience:
javascript:alert((1.1255).toFixed(3))
http://bclary.com/2004/11/07/#a-15.7.4.5
Look at steps 10 and 11.
10. Let n be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible. If there are two such n, pick the larger n.
11. If n =0, let m be the string "0". Otherwise, let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes).
I've tried to correctly implement the algorithm in JS.
========================================================
/**
* - Fix for rounding error in JScript.
* - Fix for rounding in all other engines - step 10, adjust precision.
* - Range for fractionDigits expanded to {-20...100}
*/
Number.prototype.toFixed = function(fractionDigits) {
var f = parseInt(fractionDigits) || 0;
if( f < -20 || f > 100 ) {
throw new RangeError("Precision of " + f + " fractional digits is out of range");
}
var x = Number(this);
if( isNaN(x) ) {
return "NaN";
}
var s = "";
if(x <= 0) {
s = "-";
x = -x;
}
if( x >= Math.pow(10, 21) ) {
return s + x.toString();
}
var m;
// 10. Let n be an integer for which the exact mathematical value of
// n ÷ 10^f - x is as close to zero as possible.
// If there are two such n, pick the larger n.
n = Math.round(x * Math.pow(10, f) );
if( n == 0 ) {
m = "0";
}
else {
// let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes).
m = n.toString();
}
if( f == 0 ) {
return s + m;
}
var k = m.length;
if(k <= f) {
var z = Math.pow(10, f+1-k).toString().substring(1);
m = z + m;
k = f+1;
}
if(f > 0) {
var a = m.substring(0, k-f);
var b = m.substring(k-f);
m = a + "." + b;
}
return s + m;
};
I had to improvise for step 10. The algorithm in the spec omits the detail. The spec doesn't nail down the storage format, as Kentop pointed out, but it does say that the larger number should be taken. I provided no code there to "pick the larger of the two," as the spec dictates.
The function above produces the result for things like:
(1.1255).toFixed(3); // 1.126 - correctly rounded.
(0.827 ).toFixed(17); // No loss of precision.
This behavior seem more practial and useful.
=========================================================
Related bugs:
Bug 397880 – Incorrect rounding by toFixed() for some values when last decimal digit is 5 (edit)
Bug 226993 – Javascript toFixed rounds in unexpected manner (edit)
Where can I find Mozilla's source code for Number.prototype.toFixed(fractionDigits)?
Status: VERIFIED → REOPENED
Resolution: INVALID → ---
Comment 6•17 years ago
|
||
Test case showing original vs patched versions of
Number.prototype.toFixed
Allows for negative formatDigits, which was not properly supported in last post.
Comment 7•17 years ago
|
||
In the attachment,
(.07).toFixed() == 0
can be ignored. This will always be true.
Comment 8•17 years ago
|
||
Mozilla and Opera have rounding errors with (1.1255).toFixed(3) and (1.1255).toFixed(17), losing precision.
Safari 2 got the wrong result with (1.1255).toFixed(3), but got the correct result with (1.1255).toFixed(17).
Only Safari 3 got it right the first time. The RangeError it throws on a negative fractionDigits is perfectly valid.
Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification.
Attachment #283513 -
Attachment is obsolete: true
Comment 9•17 years ago
|
||
Noticed that for .07, Safari 2 returns .1, but should instead return 0.1.
Only Safari 3 gets the correct result.
Attachment #283857 -
Attachment is obsolete: true
Comment 10•17 years ago
|
||
Instead of using Math.round, it would be more true to the spec to implement step 10 with the following:
//----------------------------------------------------------------------------
// 10. Let n be an integer for which the exact mathematical value
// of n ÷ 10^f - x is as close to zero as possible.
var fTo10 = Math.pow(10, f);
n = Math.floor(x * fTo10 );
// If there are two such n, pick the larger n.
if (Math.abs(n / fTo10 - x) >= Math.abs((n + 1) / fTo10 - x)) {
++n;
}
//----------------------------------------------------------------------------
Math.round works correctly.
http://bclary.com/2004/11/07/#a-15.8.2.15
So should Number.prototype.toFixed.
It seems likely that the native code uses something like my first patch, but chooses error-prone rounding, and does not implement rounding as Math.round does. I can guess this to be the problem; where is the !#$@ing source code?
I have found no problems with the first patch I wrote using javascript's Math.round.
The second approach I have written should be translated to native code.
I do, however, have a problem with the current implementation, which yeilds:
(1.1255).toFixed(3) // 1.125
(.1255).toFixed(3) // 0.126
Comment 11•17 years ago
|
||
toPrecision is also broken in all browsers but IE. (I did not test Opera 9.5b/Futhark Engine, but it is probably broken there, too)
javascript:alert(1.1255.toPrecision(4))
toExponential is broken in all browsers but IE and Safari 3/Webkit
javascript:alert(1.1255.toExponential(20))
The JS Guide documentation for toPrecision is wrong. Found with Google Search: toPrecision documentation
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Predefined_Core_Objects:Number_Object
"Returns a string representing the number to a specified precision in fixed-point notation."
Incorrect and incomplete.
The reference documentation is correct, but avoids discussion of the rounding errors/bugs in Mozilla:
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Global_Objects:Number
------------------------------------------------------------------------------
We need a meta bug with dependencies of the following:
toPrecision, toFixed, toExponential
For each of the above, there should be a bug rep't tests of boundary and error-prone values.
* run the tests
* write patches/get review/checkin et c.
Comment 13•17 years ago
|
||
Per the ECMAScript standard, the correct answer for 1.1255.toPrecision(4) is "1.125". This is what Mozilla produces. This is not a bug. The reason is that 1.1255 cannot be represented exactly as an IEEE double, so internally it is represented as 1.1254999999999999449329379785922355949878692626953125.
Status: REOPENED → RESOLVED
Closed: 22 years ago → 17 years ago
Resolution: --- → INVALID
Reporter | ||
Comment 14•17 years ago
|
||
In reply to #13:
If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard.
Comment 15•17 years ago
|
||
We're doing that. ES4 will also have a decimal number type.
Comment 16•17 years ago
|
||
The code example, follows the ECMA 262 algorithm and gets a different result than the native code. A more desirable one, I might add.
Does the example follow the algorithm incorrectly? If so, where.
Where is the source code so we can have a look?
Updated•17 years ago
|
Assignee: khanson → general
Status: REOPENED → NEW
QA Contact: pschwartau → general
Comment 17•17 years ago
|
||
Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the very beginning -- it evaluates the token "1.1255" incorrectly. This token evaluates to 1.1254999999999999449329379785922355949878692626953125. This is required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls apart after this.
You also wrote "Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification." How is this relevant here? If you're referring to the note at the end of 15.7.4.5, then please re-read that note. What it's stating is that instead of throwing RangeError, toFixed is allowed to, upon request, correctly produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5, we've established that the value given is 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can see all of its digits in the Mozilla implementation of toFixed.
Status: NEW → RESOLVED
Closed: 17 years ago → 17 years ago
Resolution: --- → INVALID
Comment 18•17 years ago
|
||
In the example the problem is in the emulation of step 10. The spec requires you to pick n to be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible. If there are two such n, pick the larger n.
This is not implemented by n = Math.round(x * Math.pow(10, f)) because that expression has double (or triple) rounding. You round once in the * operator and again in Math.round; there may be a third rounding in Math.pow depending on the value of f. The spec requires exact arithmetic, not rounded intermediate results.
Comment 19•17 years ago
|
||
(In reply to comment #17)
> Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the
> very beginning -- it evaluates the token "1.1255" incorrectly. This token
> evaluates to 1.1254999999999999449329379785922355949878692626953125. This is
> required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls
> apart after this.
>
> You also wrote "Aside from fixing the bugs in the other browsers, this patch
> enhances toFixed with a greater precision. This is permitted by the
> specification." How is this relevant here? If you're referring to the note at
> the end of 15.7.4.5, then please re-read that note. What it's stating is that
> instead of throwing RangeError, toFixed is allowed to, upon request, correctly
> produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5,
> we've established that the value given is
> 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can
> see all of its digits in the Mozilla implementation of toFixed.
>
Are you saying 1.1255.valueOf() should return 1.1254999999999999449329379785922355949878692626953125
Comment 20•17 years ago
|
||
Not only that, merely 1.1255 returns the exact mathematical value 1.1254999999999999449329379785922355949878692626953125. It happens to print as "1.1255" if you convert it back to a string, but it's actually less than the exact mathematical value 1.1255.
Comment 21•17 years ago
|
||
Garrett: now you may appreciate why decimal is coming in JS2/ES4 ;-).
/be
Updated•17 years ago
|
Status: RESOLVED → VERIFIED
Comment 22•17 years ago
|
||
(In reply to comment #19)
> (In reply to comment #17)
> > Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the
> > very beginning -- it evaluates the token "1.1255" incorrectly. This token
> > evaluates to 1.1254999999999999449329379785922355949878692626953125. This is
> > required by ECMA 262 sections 7.8.3 and 8.5.ntation of toFixed.
> > [snip]
> Are you saying 1.1255.valueOf() should return
> 1.1254999999999999449329379785922355949878692626953125
No, that's not what waldemar wrote. The point is that implementations must scan the 1.1255 token using extra internal precision, necessary for correct rounding to the binary precision limits of IEEE double.
Nothing in the spec requires or indeed allows visible results in the language to have greater than double precision. Which is why we have bugs on file about hardware such as modern Macs, which oddly use 80-bit precision in the x87 FPU but 64-bit in SSE (Mac OS X does not seem to allow you to override this strange FPU configuration; if anyone knows how, please mail me).
/be
Comment 23•17 years ago
|
||
Makes sense. Looking fwd to ES4.
Comment 28•14 years ago
|
||
(In reply to comment #5)
Thank you Garrett for this FIX ! I need a method to obtain a mathematically correct response.
All this internal stuff on the specification is not a responsible response to a practical problem encountered by users.
I agree with comment #14 "If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard".
Comment 30•8 years ago
|
||
here's another one:
565133671234567.89.toFixed(2)
565133671234567.88
You need to log in
before you can comment on or make changes to this bug.
Description
•