Closed Bug 478796 Opened 16 years ago Closed 6 years ago

Number.toFixed() returning incorrect values for certain large numbers

Categories

(Tamarin Graveyard :: Virtual Machine, defect, P2)

x86
All
defect

Tracking

(Not tracked)

RESOLVED WONTFIX
Q1 12 - Brannan

People

(Reporter: jodyer, Unassigned)

References

Details

var x = 1000000000000000128; print(x); // 1000000000000000100 - strange but same as spidermonkey print(x.toFixed(0)); // 1000000000000000100 - sm returns: 1000000000000000128 print(x.toFixed(1)); // 1000000000000000130.0 - sm returns 1000000000000000128.0 Actual Results: see above Expected Results: same as spidermonkey Workaround: none tofixed works as expected for args 2-20 Transferred Comments: JIRA Migration Admin - Mon Jun 04 12:33:00 CDT 2007 Milestone ID = 1378 Milestone = FP 10 - Code Complete Build ID = 22846 Build = AVMPLUS_1_0_0_d686 Fix Build ID = null Fix Build = null JIRA Migration Admin - Mon Jun 04 12:33:00 CDT 2007 Move from BugDB issue number 188975 BugDB Migration - Mon Jun 04 12:33:00 CDT 2007 [cpeyer 10/20/06] Entered Bug. [cpeyer 10/20/06] Sending to QRB [cnuuja 10/20/06] This is probably not a compliance bug, but rather related to differences in how spidermonkey and (howler?) round numbers with more precision than can fit exactly into a double. Someone should see why our toFixed(1) output is different, but keep this in mind: First of all, this is a really huge number. Don't be decieved by the fact that its an integer. The number is just larger than can accurately fit in the 52 bits of full precision in a double (an excellent test case). It equivalent to the following in base 2: 110111100000101101101011001110100111011001000000000010000000 Only the first 52 digits fit exactly. 1101111000001011011010110011101001110110010000000000 10000000 that means we need to use some of the 11 bits of base 2 exponent in the double to fully represent this number. Note that there's a 1 just after that at the 53 bit, making the number an odd rather than an even number. Since the exponent is base 2, it can't represent odd numbers (it just multiplies the mantissa by a power of 2), so we're goint to have to round that 1 up or down. That's why the number displays different in Spidermonkey (and the same as Spidermonkey in Howler). Not sure why the toFixed representation is different, that may be a bug or just an o.k. (spec-wise) implementation difference. In any case, seems like a fp10 bug to me Mark DeMichele - Thu Sep 18 09:29:25 CDT 2008 This bug is also related to FP-557. Please see my last comment. It fails for any precision. If the first significant digit it > 5 and you are expecting to get a 0 as a result, you will get a 1 in the last precise spot. Here are some examples. This is VERY BAD. I CAN'T BELIEVE THIS HASN'T BEEN ADDRESSED YET. Numerically, it is absolutely wrong and since flash is no long used to make pretty animations, I think someone really needs to take this seriously and fix it. v=0.00005 v.toFixed(2)=0.01 v=0.00007 v.toFixed(2)=0.01 v=0.00009 v.toFixed(2)=0.01 v=5e-7 v.toFixed(2)=0.00 v=7e-7 v.toFixed(2)=0.01 v=9e-7 v.toFixed(2)=0.01 v=0.00005 v.toFixed(3)=0.001 v=0.00007 v.toFixed(3)=0.001 v=0.00009 v.toFixed(3)=0.001 v=5e-7 v.toFixed(3)=0.000 v=7e-7 v.toFixed(3)=0.001 v=9e-7 v.toFixed(3)=0.001 Trevor Baker - Tue Jan 27 16:41:36 CST 2009 send to internal review for prioritization This bug transferred from: http://bugs.adobe.com/jira/browse/ASC-2293
Blocks: AS3_Builtins
Priority: -- → P3
Target Milestone: --- → flash10.1
Flags: in-testsuite?
confirmed that tamarin produces results that do not match spidermonkey, sending to QRB, there are a handful of toFixed() bugs out there: 442293 513019 513039
Flags: flashplayer-triage+
Flags: flashplayer-qrb?
Priority: P3 → P4
Blocks: a2d-d2a
Assignee: nobody → lhansen
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
Reopening; the small-number cases are covered by bug #513039 but the large-number cases are unique to this bug.
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Summary: toFixed returning incorrect values → toFixed returning incorrect values for certain large numbers
(In reply to comment #0) > > First of all, this is a really huge number. Don't be decieved by the fact > that its an integer. The number is just larger than can accurately fit in > the 52 bits of full precision in a double (an excellent test case). It > equivalent to the following in base 2: > 110111100000101101101011001110100111011001000000000010000000 > > Only the first 52 digits fit exactly. > 1101111000001011011010110011101001110110010000000000 10000000 That analysis is incorrect, because the first '1' is represented implicitly. Thus the last '1' bit is actually representable, and our number parser parses the number correctly and the value in memory does represent that bit. (The hex representation of the number is 43abc16d674ec801.)
TC pushed: redux changeset: 2760:2e75d5d3010b
Summary: toFixed returning incorrect values for certain large numbers → Number.toFixed() returning incorrect values for certain large numbers
Any fix for this would likely have to be version-checked for backwards compatibility.
Flags: flashplayer-qrb? → flashplayer-qrb+
Status: REOPENED → ASSIGNED
Flags: flashplayer-needsversioning+
Whiteboard: Must revert
Whiteboard: Must revert
Assignee: lhansen → nobody
Priority: P4 → --
Target Milestone: flash10.1 → Future
Depends on: 535770
This is a mass change. Every comment has "assigned-to-new" in it. I didn't look through the bugs, so I'm sorry if I change a bug which shouldn't be changed. But I guess these bugs are just bugs that were once assigned and people forgot to change the Status back when unassigning.
Status: ASSIGNED → NEW
Priority: -- → P2
Target Milestone: Future → flash10.2
Assignee: nobody → lhansen
Status: NEW → ASSIGNED
Whiteboard: must–fix-candidate must–fix-candidate
Whiteboard: must–fix-candidate must–fix-candidate → must-fix-candidate must-fix-candidate
Target Milestone: Q3 11 - Serrano → Q1 12 - Brannan
Whiteboard: must-fix-candidate
Assignee: lhansen → nobody
Status: ASSIGNED → NEW
Status: NEW → RESOLVED
Closed: 15 years ago6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.