While running the Sunspider benchmark on B2G and Android on the 7x27 QRD device, here are the recorded scores (lower scores are better): V1.2 Android ---------------------------- 2974 2606 ---------------------------- The B2G numbers should at least match the Android numbers. Note: 1.2 Versions: Gaia version: c932c482c6944fa32724ce7af9e5423c4c2bcccd Gecko version: f98470e4c52dbcd3ee0236d7566df733b89f9fa9 This benchmark has been agreed upon as being supported.

These scores were obtained on Sunspider v1.0.1

On a later buid than the one reported above, the following score was recorded: 2927 Gaia version: a13c76f8d3c617ee57c302c103da04ed1a6298d1 Gecko version: 85ead3614a3de104ca8b52c63a5b9b35c68feaa5

This regression is explained by the backout out of Bug 903802. (In reply to Mandyam Vikram from comment #0) > Note: 1.2 Versions: > Gaia version: > c932c482c6944fa32724ce7af9e5423c4c2bcccd > Gecko version: > f98470e4c52dbcd3ee0236d7566df733b89f9fa9 http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=f98470e4c52dbcd3ee0236d7566df733b89f9fa9&st=commit&s=Bug+903802 (In reply to Mandyam Vikram from comment #2) > Gaia version: > a13c76f8d3c617ee57c302c103da04ed1a6298d1 > Gecko version: > 85ead3614a3de104ca8b52c63a5b9b35c68feaa5 http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=85ead3614a3de104ca8b52c63a5b9b35c68feaa5&st=commit&s=Bug+903802 Note that the second fix is included in 1.2: http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=refs%2Fheads%2Fv1.2&st=commit&s=Bug+903802 So the next build should not have the same problem. Mandyam: Can you check again to see if the problem is fixed as expected? (In reply to Mandyam Vikram from comment #0) > The B2G numbers should at least match the Android numbers. The Firefox for Android builds are tuned for high-end devices, which, as far as I know this is not our target market for B2G. So keep in mind that this is not a safe assumption on memory intensive benchmarks, even on identical devices. Note that Bug 898556 highlights this kind of issue where B2G results appears as being worse than Firefox for Android. We have plans to make the GC memory dependent of the available memory on the system[1], but this is not currently the case. [1] https://wiki.mozilla.org/JavaScript:Projects#Memory-dependent_GC_Configuration

Just a clarification that the Android scores are from running the benchmark on the stock Android browser on an Android build and not on the Firefox Android app. Understand your comment about tuning it for diff. devices. But, this is an issue we will face on 1.2 as the commercial devices already have 256Mb and 512 Mb configurations.

I tried it on a build I synced today. Just trying this on my own, I got scores of 3001, 2974 and 3330. Are you able to run this on a commercial device and check the scores? On a related topic, based on a question from Kannan on whether ION was enabled, please look at the config file that is being used below. Is ION supposed to be enabled here? https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla-b2g/gonk-misc/tree/default-gecko-config?h=mozilla/v1.2 (same as Moz 1.2 builds)

(In reply to Mandyam Vikram from comment #5) > I tried it on a build I synced today. Just trying this on my own, I got > scores of 3001, 2974 and 3330. Let me see if I can reproduce that on an Unagi. > On a related topic, based on a question from Kannan on whether ION was > enabled, please look at the config file that is being used below. Is ION > supposed to be enabled here? > > https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla-b2g/gonk-misc/tree/ > default-gecko-config?h=mozilla/v1.2 This is the recent configuration file, Ion should be enabled.

(In reply to Nicolas B. Pierron [:nbp] from comment #6) > (In reply to Mandyam Vikram from comment #5) > > I tried it on a build I synced today. Just trying this on my own, I got > > scores of 3001, 2974 and 3330. > > Let me see if I can reproduce that on an Unagi. I mnage to reproduce the bug, I still have some doubt about the validity of the measurement that I did because I see like a few 2ms slow down on benchmarks likes bits-in-bytes which are most likely to be cause by network latencies. Still I spotted that ss-format-* benchmarks are like 60ms slower. I will investigate more the current know regression (Bug 915167) which (fix?) might not have been backported.

Seems like https://bugzilla.mozilla.org/attachment.cgi?id=803048 is already in the build that was tested.

(In reply to Mandyam Vikram from comment #8) > Seems like https://bugzilla.mozilla.org/attachment.cgi?id=803048 is already > in the build that was tested. Yes, but the reason why I did not closed Bug 915167 is that the fix which is currently attached to it did not fix the issue, it only partially fixed it, and this is visible if you zoom on the latest regression of sunspider (AWFY Unagi). I presume that there might be another patch at the origin of the regression, but currently I have no detail since the harness was broken as explained in Bug 914671. So I will have to modify my harness to bisect with an extra patch applied on top to see from if I can pin-point a second regression patch. And hopefully see which patch fixed it later and might need to be backported too. These are all hypothesis, but I will first try to reproduce these results on a well managed network and not an Hotel network with unknown latencies (which might matter for sunspider & kraken).

Moved to 1.3 per triage decision

Unassinged my-self as I already keep track of regressions of FxOS on nightly, and the merge for 1.3 would take another version of Gecko where we are following the changes.

Sunspider score has regressed from the following versions: Gaia version: 5e0d0df6a762cf1e1812eeb735fba72e2539dc0c Gecko version: b318b86aeb90a517f183abecd6b45c99065eb473 Score : 2850 Gaia version: 313599c293f596519604070fa378ffc89e61e98f Gecko version: bd1bd2486c85e449244de79dd80e10040397d3cf Score: 2927

Testing on the following version shows that the regression in https://bugzilla.mozilla.org/show_bug.cgi?id=922295#c12 is not observed. Gaia version: 845801e0c74badf0b96657a864935ef1a983cf47 Gecko version: bbb4c0a8fa65cf1546a6cedc4c20fea16cd63ef2 We can move this to 1.3.

What is the goal of this bug? Tracking one regression? If yes, then we should close it and mark it as FIXED. If the goal is saying that we are bad without any specific detail which are actionable, then we should close this bug, and I will mark it as WORKSFORME. If you want to discuss any improvement plan, then we should have a real discussion on this topic involving the JavaScript Team to see how we can manage that but Bugzilla is not the best place to have these discussions. In any case, we should close these bugs because they are either non-actionable or completed. There is no point at keeping a vague bug around than causing confusion!

benchmarks goals are right now to match or better android scores on the same hardware all else equal.

(In reply to Sandip Kamat from comment #15) > benchmarks goals are right now to match or better android scores on the same > hardware all else equal. Who set this goal, is that part of the JavaScript team goals? I haven't seen any discussion about that?

Sandip: it sounds like there a few questions here. 1. Is this a benchmark goal of the FxOS team? Like Nicolas, I don't know of any specific JavaScript team goal or dedicated resources for SunSpider performance on FxOS. 2. Is this a meta bug for all FxOS SunSpider regressions for all time? I agree with Nicolas that such a meta bug would not be useful. The bug would never close and would combine multiple conversations about unrelated regressions. We should file separate bugs for specific performance regressions or slow test cases, so each problem can triaged, fixed, closed, and verified. 3. Comment 4 says this test is comparing the FxOS browser running on FxOS versus Android's stock WebKit browser running on Android. Is that a relevant comparison?

It would be a mistake to make strict score parity a high priority for the JS engine. Sunspider is very much a red-herring benchmark - more a collection of micro-benchmarks than a serious test of an engine's capabilities. The micro-benchmarks aren't even all that natural - forcing synthetic optimizations like the math cache to get competitive performance. Spending valuable resources getting sunspider to parity would draw them away from work that would apply directly to improving SpiderMonkey's general performance on mobile platforms.

Sorry, I was referring to general benchmark approach on FxOS in comment #15, and not specifically about JS. Hit send too early. I agree that we should take a different approach for JS rather than comparing with Android as it is not apples to apples. For now, we can can close this bug for v1.2. Vikram, pls let me know if you disagree. We do need to agree on what targets we should use for JS on v1.3. I am looking for your inputs here on how do we get to a definition (even if not super crisp) of what an "acceptable" JS performance is. What are better alternatives to SunSpider, and how do we arrive at the generic targets?(of course we can specify different targets for different HW/SW Specs as well)

(In reply to Sandip Kamat from comment #19) > We do need to agree on what targets we should use for JS on v1.3. I am > looking for your inputs here on how do we get to a definition (even if not > super crisp) of what an "acceptable" JS performance is. What are better > alternatives to SunSpider, and how do we arrive at the generic targets?(of > course we can specify different targets for different HW/SW Specs as well) Bugzilla is not the correct location for this discussion. Most likely, you will want to gather information from the JavaScript team and/or arrange a meeting.

(In reply to Sean Stangl [:sstangl] from comment #20) > (In reply to Sandip Kamat from comment #19) course we can specify different targets for different HW/SW Specs as well) > > Bugzilla is not the correct location for this discussion. Most likely, you > will want to gather information from the JavaScript team and/or arrange a > meeting. Agreed. Working with Kannan/Nicolas.

(In reply to Sandip Kamat from comment #19) > We do need to agree on what targets we should use for JS on v1.3. I am > looking for your inputs here on how do we get to a definition (even if not > super crisp) of what an "acceptable" JS performance is. What are better > alternatives to SunSpider, and how do we arrive at the generic targets?(of > course we can specify different targets for different HW/SW Specs as well) Concerning all performance benchmarks, including Octane (Bug 922291), I think the only thing we should look at is preventing unwanted regressions. Setting a goal in terms of benchmark score is stupid and leads to the implementation of things such as Bug 583939. If you want us to improve the benchmarks score, then the first question is to find what is not behaving correctly and if implementing such feature would improve the web or only this benchmark. Setting a goal in terms of desirable feature improves the engine overall and make the platform better for everybody (and not only benchmarks). Currently, our goal is to make the JavaScript engine ready for Haida, and none of these would show up on the usual JavaScript benchmarks. I do not think that it would be a good idea to have some meddling here. If you want to help us choose our goals, then tell us what is the most important for you, is that having a responsive phone, a fast start-up of applications, fast games, or fast benchmarks? My current understanding is that you want fast transitions and fast start-up, and we are currently working on both.

Mass-closing JS bugs for which the platform is Gonk (Firefox OS), since Firefox OS is gone. Feel free to re-open if still valid.