This was the set of failures I got with a near-pristine TM build yesterday. It could use categorization. So far I've been picking a test at a time to investigate and fix, so I don't know exactly how many issues this represents. I'm reasonably confident most of the Object.defineProperty failures are due to [].length not being redefinable through that method, so it'd probably be best for anyone looking to categorize these to start with the non-O.dP tests first.

We have some bugs, the tests have some bugs (to be reported shortly, need to open up a browser that has my test262 bugzilla password saved), about as you'd expect. I haven't fully investigated most of the tests that appear to be [].length redefinition, and arguments bizarreness oddities. I don't think there's much to gain investigating the former -- we know what they are in general, and the few that aren't, we can deal with after fixing the bulk of them, with much less effort. There's also a couple decodeURI tests that might be fixed by a reviewed patch to be landed soon for bug 660612, but given those tests' complexity I'm uninterested in investigating until I know that particular problem exists post-patch. Other than that everything's investigated, with bugs filed.

The split tests mentioned in attachment 543562 [details] are <https://bugs.ecmascript.org/show_bug.cgi?id=60>. The bad-this tests mentioned in attachment 543562 [details] are <https://bugs.ecmascript.org/show_bug.cgi?id=117>.

Some of the tests wrongly expect that U+0085 NEXT LINE is a space character. I filed <https://bugs.ecmascript.org/show_bug.cgi?id=118> to cover that.

S13.0_A5 checks that the engine doesn't implement destructuring parameters to function statements, which we do implement. I filed https://bugs.ecmascript.org/show_bug.cgi?id=133 on getting that bit of extension-testing removed from the test.

This updates waldo's list taking into account what we've fixed and what was fixed in test262 upstream. I've removed these from waldo's categories, since all the tests that rely on them no longer fail: bug 668024 (actually, this hasn't landed, but I tested with that patch included) bug 652771 bug 635017 bug 668934 bug 660612 Buggy global object in our DOM (for undefined, NaN, Infinity writing/deletion) Bug 331773 (isn't needed for test262; I guess bug 660612 fixed that.) There are the tests which no longer fail: 11.13.1-4-27-s 11.13.1-4-3-s 11.4.1-4.a-8-s 15.2.3.6-4-354-13 15.2.3.6-4-354-4 15.2.3.6-4-360-3 15.2.3.6-4-360-7 15.2.3.6-4-531-13 15.2.3.6-4-531-4 15.2.3.6-4-538-3 15.2.3.6-4-538-7 15.2.3.7-6-a-24 15.5.4.20-3-2 15.5.4.20-3-3 15.5.4.20-3-4 15.5.4.20-3-5 15.5.4.20-3-6 S15.1.2.2_A2_T10 S15.1.2.3_A2_T10 S15.1.3.1_A2.4_T1 S15.1.3.2_A2.4_T1 S15.10.6.2_A12 S15.10.6.2_A1_T16 S15.10.6.3_A1_T16 S15.4.4.12_A2_T1 S15.4.4.12_A2_T4 S15.4.4.12_A3_T2 S15.4.4.12_A3_T3 S15.5.4.10_A1_T4 S15.5.4.10_A1_T6 S15.5.4.10_A1_T7 S15.5.4.10_A1_T8 S15.5.4.10_A1_T9 S15.5.4.12_A1_T4 S9.3.1_A2 S9.3.1_A3_T1 S9.3.1_A3_T2

Nice :>

Test262 has fixed a lot of the problems with their tests, and we've fixed a few here, so the list is now a bit shorter.

There's a regression, Firefox Nightly (21) fails a lot tests that Fx stable (18) didn't, errors say, citing: "eval within global execution context".

Since Chrome fails only 15 tests now, Safari - 8, IE10 - 8, Opera - 10 (all of them in stable versions) and Firefox Nightly fails 209 tests, I think this bug needs the parity-webkit and parity-ie flags (I'd also say parity-opera but since they're switching to WebKit it has probably lost significance).

With non-writable array length implemented (note that bug's not fixed, the patch there isn't ready, it still has issues to work through), the failure count drops to 68. Looking at some of the tests that fail on the site, I'm not convinced all the test262 issues mentioned before have been resolved. But investigating too much further probably should wait til bug 858381 is fixed.

There are some new failures, that aren't present on Walden's attachment. But there are some failures that are gone! I have installed addons (I forgot to disable them, sorry). I hope they don't change the results.

All these Array.prototype.* ones are problems with the test suite. I reported them https://bugs.ecmascript.org/show_bug.cgi?id=1453.

A lot of new errors since last time. Ran with today's Nightly. Latest released Chrome has 11 errors. Nightly has 73. I also noticed that the tests ran a lot faster in Chrome than in Nightly.

By contrast 23.0.1 has 193 errors. So a good improvement compared to the latest release.

(In reply to Kai de Leeuw from comment #15) > I also noticed that the tests ran a lot faster in Chrome than in Nightly. I believe this is due to the test runner playing with iframes a lot rather than a raw JavaScript performance issue.

Tom's Hardware Guide's "Web Browser Grand Prix" (tracking bug 933949) includes test262.

Firefox 29 and Nightly 32 fail 42 test cases. Chrome 35 fails only 5 test cases.

Firefox 33 now failing 53 test cases: 10.4.3-1-104 Strict mode should not ToObject thisArg if not an object. Strict equality operator should succeed. Fail 10.4.3-1-106 Strict mode should not ToObject thisArg if not an object. Return type should be 'number'. Fail S12.1_A4_T2 Checking if execution of "y={x}" fails Fail S12.6.4_A15 Using block within "for-in" Expression Fail 15.11.4.4-8-2 Error.prototype.toString return empty string when 'name' is empty string and 'msg' is undefined Fail 15.2.3.4-1-4 Object.getOwnPropertyNames throws TypeError if 'O' is a boolean Fail 15.2.3.4-1-5 Object.getOwnPropertyNames throws TypeError if 'O' is a string Fail 15.2.3.4-1 Object.getOwnPropertyNames throws TypeError if type of first param is not Object Fail 15.2.3.6-4-292-1 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own property of 'O' which is also defined in [[ParameterMap]] of 'O', and 'desc' is data descriptor, test updating multiple attribute values of 'name' (10.6 [[DefineOwnProperty]] step 3 and 5.b) Fail 15.2.3.6-4-293-2 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Value]] attribute value of 'name' which is defined as unwritable and non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b) Fail 15.2.3.6-4-293-3 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is not thrown when updating the [[Value]] attribute value of 'name' which is defined as non-writable and configurable (10.6 [[DefineOwnProperty]] step 3 and step 5.b) Fail 15.2.3.6-4-294-1 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Writable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and 5b) Fail 15.2.3.6-4-295-1 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Enumerable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b) Fail 15.2.3.6-4-296-1 Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Configurable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b) Fail 15.2.3.6-4-354-13 Object.defineProperty will update [[Value]] attribute of indexed property successfully when [[Configurable]] attribute is true and [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step Note) Fail 15.2.3.6-4-354-4 Object.defineProperty will update [[Value]] attribute successfully when [[Configurable]] attribute is true and [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step Note) Fail 15.2.3.6-4-360-3 ES5 Attributes - Updating data property 'P' whose attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i) Fail 15.2.3.6-4-360-7 ES5 Attributes - Updating indexed data property 'P' whose attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i) Fail 15.2.3.6-4-531-13 Object.defineProperty will update [[Get]] and [[Set]] attributes of indexed accessor property 'P' successfully when [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step 11) Fail 15.2.3.6-4-531-17 ES5 Attributes - Updating an indexed accessor property 'P' using simple assignment is successful, 'O' is the global object (8.12.5 step 5.b) Fail 15.2.3.6-4-531-4 Object.defineProperty will update [[Get]] and [[Set]] attributes of named accessor property 'P' successfully when [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step 11) Fail 15.2.3.6-4-538-3 ES5 Attributes - Updating a named accessor property 'P' whose [[Configurable]] attribute is true to a data property is successful, 'O' is the global object Fail 15.2.3.6-4-538-7 ES5 Attributes - Updating an indexed accessor property 'P' whose [[Configurable]] attribute is true to a data property is successful, 'O' is the global object Fail 15.2.3.7-6-a-24 Object.defineProperties - 'O' is the global object which implements its own [[GetOwnProperty]] method to get 'P' (8.12.9 step 1 ) Fail 15.4.4.14-1-17 Array.prototype.indexOf applied to the global object Fail 15.4.4.14-2-15 Array.prototype.indexOf - 'length' is property of the global object Fail 15.4.4.14-9-b-i-23 Array.prototype.indexOf - This object is the global object Fail 15.4.4.15-1-17 Array.prototype.lastIndexOf applied to the global object Fail 15.4.4.15-2-15 Array.prototype.lastIndexOf - 'length' is property of the global object Fail 15.4.4.15-8-b-i-23 Array.prototype.lastIndexOf - This object is the global object Fail 15.4.4.16-2-15 Array.prototype.every - 'length' is property of the global object Fail 15.4.4.16-7-c-i-23 Array.prototype.every - This object is an global object which contains index property Fail 15.4.4.17-2-15 Array.prototype.some - 'length' is property of the global object Fail 15.4.4.17-7-c-i-23 Array.prototype.some - This object is an global object which contains index property Fail 15.4.4.18-2-15 Array.prototype.forEach - 'length' is property of the global object Fail 15.4.4.18-7-c-i-23 Array.prototype.forEach - This object is an global object which contains index property Fail 15.4.4.19-8-c-i-23 Array.prototype.map - This object is the global object which contains index property Fail 15.4.4.20-2-15 Array.prototype.filter - 'length' is property of the global object Fail 15.4.4.20-9-c-i-23 Array.prototype.filter - This object is the global object which contains index property Fail 15.4.4.21-2-15 Array.prototype.reduce - 'length' is property of the global object Fail 15.4.4.21-8-b-iii-1-23 Array.prototype.reduce - This object is the global object which contains index property Fail 15.4.4.21-9-c-i-23 Array.prototype.reduce - This object is the global object which contains index property Fail 15.4.4.22-2-15 Array.prototype.reduceRight - 'length' is property of the global object Fail 15.4.4.22-8-b-iii-1-23 Array.prototype.reduceRight - This object is the global object which contains index property Fail 15.4.4.22-9-c-i-23 Array.prototype.reduceRight - This object is an global object which contains index property Fail 15.4.4.4-5-c-i-1 Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i) Fail S15.4.4.4_A3_T1 [[Prototype]] of Array instance is Array.prototype, [[Prototype] of Array.prototype is Object.prototype Fail S15.9.3.1_A5_T1 2 arguments, (year, month) Fail S15.9.3.1_A5_T2 3 arguments, (year, month, date) Fail S15.9.3.1_A5_T3 4 arguments, (year, month, date, hours) Fail S15.9.3.1_A5_T4 5 arguments, (year, month, date, hours, minutes) Fail S15.9.3.1_A5_T5 6 arguments, (year, month, date, hours, minutes, seconds) Fail S15.9.3.1_A5_T6 7 arguments, (year, month, date, hours, minutes, seconds, ms) Fail Test suite version: ES5 | Test suite date: 2014-09-18

(In reply to Alexandre Folle de Menezes from comment #20) > Firefox 33 now failing 53 test cases: This triage is at a quick skim, going from memory of previous investigation in many cases, tempered with inspection of cases where memory of the error message wasn't helpful. So I may have gotten some of these wrong. Still, I'm pretty confident it's generally accurate. What it reveals is that test262 has a lot of tests that are broken, and a smallish handful that SpiderMonkey doesn't implement right, mostly in very edge-case behaviors. These failures look bogus in light of ES6 additions (note that test262-the-site may well not be up-to-date with ES6 changes, nor even up-to-date with test262 on Github): > S12.1_A4_T2 Checking if execution of "y={x}" fails Fail > S12.6.4_A15 Using block within "for-in" Expression Fail > 15.11.4.4-8-2 Error.prototype.toString return empty string when 'name' is > empty string and 'msg' is undefined Fail These look bogus in light of ES6 changes to ES5 semantics: > 15.2.3.4-1-4 Object.getOwnPropertyNames throws TypeError if 'O' is a boolean > Fail > 15.2.3.4-1-5 Object.getOwnPropertyNames throws TypeError if 'O' is a string > Fail > 15.2.3.4-1 Object.getOwnPropertyNames throws TypeError if type of first > param is not Object Fail > S15.4.4.4_A3_T1 [[Prototype]] of Array instance is Array.prototype, > [[Prototype] of Array.prototype is Object.prototype Fail These I think are bogus in a browser environment with the DOM's global object (which has interesting [[DefineOwnProperty]] and [[Set]] behaviors for "length" and indexed properties) -- they may fail, but they're not actual bugs: > 15.2.3.6-4-354-13 Object.defineProperty will update [[Value]] attribute of > indexed property successfully when [[Configurable]] attribute is true and > [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step > Note) Fail > 15.2.3.6-4-360-7 ES5 Attributes - Updating indexed data property 'P' whose > attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: > true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i) > Fail > 15.2.3.6-4-531-13 Object.defineProperty will update [[Get]] and [[Set]] > attributes of indexed accessor property 'P' successfully when > [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step > 11) Fail > 15.2.3.6-4-538-7 ES5 Attributes - Updating an indexed accessor property 'P' > whose [[Configurable]] attribute is true to a data property is successful, > 'O' is the global object Fail > 15.2.3.6-4-531-17 ES5 Attributes - Updating an indexed accessor property 'P' > using simple assignment is successful, 'O' is the global object (8.12.5 step > 5.b) Fail > 15.4.4.14-1-17 Array.prototype.indexOf applied to the global object Fail > 15.4.4.14-2-15 Array.prototype.indexOf - 'length' is property of the global > object Fail > 15.4.4.14-9-b-i-23 Array.prototype.indexOf - This object is the global > object Fail > 15.4.4.15-1-17 Array.prototype.lastIndexOf applied to the global object Fail > 15.4.4.15-2-15 Array.prototype.lastIndexOf - 'length' is property of the > global object Fail > 15.4.4.15-8-b-i-23 Array.prototype.lastIndexOf - This object is the global > object Fail > 15.4.4.16-2-15 Array.prototype.every - 'length' is property of the global > object Fail > 15.4.4.16-7-c-i-23 Array.prototype.every - This object is an global object > which contains index property Fail > 15.4.4.17-2-15 Array.prototype.some - 'length' is property of the global > object Fail > 15.4.4.17-7-c-i-23 Array.prototype.some - This object is an global object > which contains index property Fail > 15.4.4.18-2-15 Array.prototype.forEach - 'length' is property of the global > object Fail > 15.4.4.18-7-c-i-23 Array.prototype.forEach - This object is an global object > which contains index property Fail > 15.4.4.19-8-c-i-23 Array.prototype.map - This object is the global object > which contains index property Fail > 15.4.4.20-2-15 Array.prototype.filter - 'length' is property of the global > object Fail > 15.4.4.20-9-c-i-23 Array.prototype.filter - This object is the global object > which contains index property Fail > 15.4.4.21-2-15 Array.prototype.reduce - 'length' is property of the global > object Fail > 15.4.4.21-8-b-iii-1-23 Array.prototype.reduce - This object is the global > object which contains index property Fail > 15.4.4.21-9-c-i-23 Array.prototype.reduce - This object is the global object > which contains index property Fail > 15.4.4.22-2-15 Array.prototype.reduceRight - 'length' is property of the > global object Fail > 15.4.4.22-8-b-iii-1-23 Array.prototype.reduceRight - This object is the > global object which contains index property Fail > 15.4.4.22-9-c-i-23 Array.prototype.reduceRight - This object is an global > object which contains index property Fail > 15.4.4.4-5-c-i-1 Array.prototype.concat will concat an Array when index > property (read-only) exists in Array.prototype (Step 5.c.i) Fail I suspect DOM spec choices make these tests invalid, but I haven't deeply investigated them to be certain: > 15.2.3.6-4-354-4 Object.defineProperty will update [[Value]] attribute > successfully when [[Configurable]] attribute is true and [[Writable]] > attribute is false, 'O' is the global object (8.12.9 - step Note) Fail > 15.2.3.6-4-360-3 ES5 Attributes - Updating data property 'P' whose > attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: > true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i) > Fail > 15.2.3.6-4-531-4 Object.defineProperty will update [[Get]] and [[Set]] > attributes of named accessor property 'P' successfully when [[Configurable]] > attribute is true, 'O' is the global object (8.12.9 step 11) Fail > 15.2.3.6-4-538-3 ES5 Attributes - Updating a named accessor property 'P' > whose [[Configurable]] attribute is true to a data property is successful, > 'O' is the global object Fail > 15.2.3.7-6-a-24 Object.defineProperties - 'O' is the global object which > implements its own [[GetOwnProperty]] method to get 'P' (8.12.9 step 1 ) Fail These failures are valid: > 10.4.3-1-104 Strict mode should not ToObject thisArg if not an object. > Strict equality operator should succeed. Fail > 10.4.3-1-106 Strict mode should not ToObject thisArg if not an object. > Return type should be 'number'. Fail > 15.2.3.6-4-292-1 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own property of 'O' which is > also defined in [[ParameterMap]] of 'O', and 'desc' is data descriptor, test > updating multiple attribute values of 'name' (10.6 [[DefineOwnProperty]] > step 3 and 5.b) Fail > 15.2.3.6-4-293-2 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own data property of 'O' > which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown > when updating the [[Value]] attribute value of 'name' which is defined as > unwritable and non-configurable (10.6 [[DefineOwnProperty]] step 4 and step > 5b) Fail > 15.2.3.6-4-293-3 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own data property of 'O' > which is also defined in [[ParameterMap]] of 'O', test TypeError is not > thrown when updating the [[Value]] attribute value of 'name' which is > defined as non-writable and configurable (10.6 [[DefineOwnProperty]] step 3 > and step 5.b) Fail > 15.2.3.6-4-294-1 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own data property of 'O' > which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown > when updating the [[Writable]] attribute value of 'name' which is defined as > non-configurable (10.6 [[DefineOwnProperty]] step 4 and 5b) Fail > 15.2.3.6-4-295-1 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own data property of 'O' > which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown > when updating the [[Enumerable]] attribute value of 'name' which is defined > as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b) Fail > 15.2.3.6-4-296-1 Object.defineProperty - 'O' is an Arguments object of a > function that has formal parameters, 'name' is own data property of 'O' > which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown > when updating the [[Configurable]] attribute value of 'name' which is > defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b) > Fail Memory suggests this is valid, and easily fixt if someone investigates it: > S15.9.3.1_A5_T1 2 arguments, (year, month) Fail > S15.9.3.1_A5_T2 3 arguments, (year, month, date) Fail > S15.9.3.1_A5_T3 4 arguments, (year, month, date, hours) Fail > S15.9.3.1_A5_T4 5 arguments, (year, month, date, hours, minutes) Fail > S15.9.3.1_A5_T5 6 arguments, (year, month, date, hours, minutes, seconds) > Fail > S15.9.3.1_A5_T6 7 arguments, (year, month, date, hours, minutes, seconds, > ms) Fail

(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #21) > Memory suggests this is valid, and easily fixt if someone investigates it: > > > S15.9.3.1_A5_T1 2 arguments, (year, month) Fail > > S15.9.3.1_A5_T2 3 arguments, (year, month, date) Fail > > S15.9.3.1_A5_T3 4 arguments, (year, month, date, hours) Fail > > S15.9.3.1_A5_T4 5 arguments, (year, month, date, hours, minutes) Fail > > S15.9.3.1_A5_T5 6 arguments, (year, month, date, hours, minutes, seconds) > > Fail > > S15.9.3.1_A5_T6 7 arguments, (year, month, date, hours, minutes, seconds, > > ms) Fail I think the date tests are broken outside of PDT time zone: https://bugs.ecmascript.org/show_bug.cgi?id=293

https://github.com/tc39/test262/ is now really easy to run. It's somewhat annoying that we still don't run most the tests in our automation. - Ran 12134 tests - Passed 11606 tests (95.6%) - Failed 528 tests (4.4%)

s/2013-08-23/2015-03-08/

@evilpie what version are you running? I just ran Test262 on Firefox 36.0.1 on Windows and got Total tests ran: 11552 | Pass: 11449 | Fail: 103 | Failed to load: 0 Is there a newer version than whats on http://test262.ecmascript.org/? You ran more test than I did.

The website uses the "es5-tests" branch (https://github.com/tc39/test262/tree/es5-tests).

(In reply to Tom Schuster [:evilpie] from comment #23) > https://github.com/tc39/test262/ is now really easy to run. It's somewhat > annoying that we still don't run most the tests in our automation. Fodder for another SM job?

Found a new host: http://v8.github.io/test262/website/default.html