Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!

This adds reftests to aurora/beta as of today (aki should be doing a reconfig in moments). We'll touch base on mochi/other-tests timing/plan when we get closer to enabling those anywhere. Unknown yet on what reftests will fail here, but we can always hide the suites and/or disable tests. While I'm at it, I noticed we did not explicitly avoid esr10 as well, but I think that was already covered by not building there, but better-safe-than-sorry.

Comment on attachment 661024 [details] [diff] [review] [configs] do-it esr10 has 'lock_platforms' set, but its addition is harmless.

Comment on attachment 661024 [details] [diff] [review] [configs] do-it http://hg.mozilla.org/build/buildbot-configs/rev/6581ccd98aff

Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the timeline there?

armv6 reftests should be live on aurora+beta. (In reply to Alex Keybl [:akeybl] from comment #4) > Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the > timeline there? I think this bug will suffice if we keep it open.

(In reply to Aki Sasaki [:aki] from comment #5) > armv6 reftests should be live on aurora+beta. > > (In reply to Alex Keybl [:akeybl] from comment #4) > > Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the > > timeline there? > > I think this bug will suffice if we keep it open. Joel or Brad might be able to answer where it is being worked on and depend on them.

this bug was filed when we were running XUL android and didn't have the bandwidth to manage so many. The only thing holding us back from more tests is planning for capacity.

I see the reftests running: https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=armv6 https://tbpl.mozilla.org/?tree=Mozilla-Beta&jobname=armv6 It looks very pretty :)

(In reply to Joel Maher (:jmaher) from comment #7) > this bug was filed when we were running XUL android and didn't have the > bandwidth to manage so many. The only thing holding us back from more tests > is planning for capacity. Can we agree upon some cadence (daily, every x check-ins) for nightly/aurora and do this per-checkin for beta? We don't need to all the tests done all the time, and I'd like to try to get unblocked in time for Beta 16.

(In reply to Alex Keybl [:akeybl] from comment #0) > Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we > enable all tests on Aurora/Beta, at some reasonable cadence? Thanks! Alex: for "all tests", do you mean all unit tests, or unit tests + talos (perf numbers) ?

I want to make sure if we are running talos, that the results get published as a different os/platform/version combination so we don't have armv6 and armv7 builds running the same tests and reporting different numbers. We worked around this when we switched from xul to native by adding a flag to talos so it would report a different machine name (which is a 1:1 mapping of the platform/os/version).

(In reply to Aki Sasaki [:aki] from comment #10) > (In reply to Alex Keybl [:akeybl] from comment #0) > > Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we > > enable all tests on Aurora/Beta, at some reasonable cadence? Thanks! > > Alex: > for "all tests", do you mean all unit tests, or unit tests + talos (perf > numbers) ? I care most about unit tests for release, as user feedback on performance will be more important than talos numbers.

This will make android-armv6 run the exact same unit test suites as android (armv7).

Comment on attachment 661318 [details] [diff] [review] run all unit tests on armv6 For those following along at home, this will make these all run on all branches while we're at it, and turn on mochi-1..8 and crashtest, jsreftest, and xpc

Comment on attachment 661318 [details] [diff] [review] run all unit tests on armv6 http://hg.mozilla.org/build/buildbot-configs/rev/092fc3e36d78 This should go live on Monday's reconfig.

It looks like there may be a permaorange here: 26108 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug560780.html | got the mousedown events - got 1, expected 12 My push was the first that M1 was run on, so it is also possible that I somehow broke this test, though that seems unlikely. I can try backing it out if that will help.

You didn't. Hidden on inbound, aurora, beta and try.

Filed bug 792300, and hid them on every other trunk tree (except twigs, which always get the short end of the stick).

My understanding is that ARMv6 tests are running on ARMv7 (using the correct ARMv6 JS configuration). Resolved/fixed?

(In reply to Alex Keybl [:akeybl] from comment #19) > My understanding is that ARMv6 tests are running on ARMv7 (using the correct > ARMv6 JS configuration). Resolved/fixed? They are, but a handful of suites are still hidden in production due to test and code issues. Do you want this left open to track that, or would you rather a new bug to handle those "unhide" issues.

(In reply to Justin Wood (:Callek) from comment #20) > (In reply to Alex Keybl [:akeybl] from comment #19) > > My understanding is that ARMv6 tests are running on ARMv7 (using the correct > > ARMv6 JS configuration). Resolved/fixed? > > They are, but a handful of suites are still hidden in production due to test > and code issues. > > Do you want this left open to track that, or would you rather a new bug to > handle those "unhide" issues. I think we can resolve this bug, we'll track the three blockers for FF17.

I suspect that all of these armv6 test failures are due to the tool chain bug as explained in bug 790624 comment 17. You can confirm this by doing a test run with a custom mozconfig that has --enable-optimize that excludes -Os.

Fwiw, there's still a lot of random crashes in the armv6 test runs, and some on other Android too. It appears we get no crash stacks in the log file. Are these crashes being tracked somewhere?

(In reply to Mats Palmgren [:mats] from comment #23) > It appears we get no crash stacks in the log file. No stacks in log could be one of two issues: The first is Bug 795206 which is a releng bug, due to the controlling system the tegra jobs run from, and is on my plate to fix. The second could be a host OS (android) crash which we don't yet have a good (any?) way to automate getting a stack at this point.

But what you are most likely to mean is ye olde http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=732325&entireHistory=true&tree=trunk, which since the October explosion has mostly been eating a shutdown crash. I wouldn't be particularly surprised if it's the same shutdown crash that the reftest harness has had all along, which used to be eaten by just showing as a green run unless someone happened to look at the log.

I'm going to resolve this, as a fixed thing -- we have Armv6 tests on every branch we have Armv6 builds at the moment, we might want to eventually expand the amount of tests we run, and we do intend to switch to armv6 devices in the future, but I'm calling *this* bug done.