Bug 1440714 is working on running our jittest suite on Android. This has showed up a bunch of test failures: bug793385.js bug830943.js TypedObject/bug970285.js TypedObject/bug976530.js TypedObject/fuzz1.js TypedObject/fuzz10.js TypedObject/fuzz11.js TypedObject/fuzz4.js TypedObject/fuzz6.js TypedObject/fuzz8.js TypedObject/fuzz9.js arrays/reverse-frozen.js arrays/too-long-array-splice.js asm.js/bug1007512.js asm.js/bug1008636.js | Unknown (code 3, args "--no-asmjs") [0.3 s] asm.js/bug885976.js asm.js/testBug893519.js auto-regress/bug1183241.js auto-regress/bug1263857.js auto-regress/bug1317460.js https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebed2683e1533023a813ac50e6b332e809089ae5

[geckoview:klar:p2] because nbp prioritized this bug P2 and it is probably not a Klar+GeckoView blocker.

Sean, are these jittest failures different from ARM64 bug 1475648?

Chris, the only real bugs on hardware I was able to reproduce are in Bug 1500611; they were fixed in Bug 1500616 and pass on m-c. So as far as I know, all the tests pass now. This specific bug looks like it lists nearly every single test as failing. I'm not sure how that was determined, but it seems very unlikely. It's possible that it's another runner error, although likely different from Bug 1475648, since that patches a regression that was introduced about a week ago.

I'd suggest re-running the tests. Everything should pass, if not, it's likely a harness issue.

Bob, can we enable the jit tests for Moto G5 ARM32 devices? In bug 1475648 comment 26, you said the jit tests are currently only enabled on try for Pixel2 AArch64. Sean thinks the jit tests should pass on ARM32, so we should be able to run them on the Moto G5 devices, if we have enough capacity. According to my test matrix notes, the only tests running on Moto G5 are the Speedometer and Unity WebGL benchmarks. https://docs.google.com/spreadsheets/d/17hp38bKgyLYs-ydqM0kauOgwr13ytYxh3aii6RrIBAk/edit#gid=0

I'll set up a try run on the g5s and see how they do.

I set up the g5 and p2 to run the jittests for arm7 32bit on try. You can see the results at <https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=bclary%40mozilla.com&fromchange=bb0706ca41d0563339079e2786f5cff0c2de8c5a&tochange=ead5af56741a679f99d596cb926ff583a88f7660>. If what you want is arm7 32bit tests, I don't believe the g5s are the best choice but we can get arm7 32bit tests on the p2 where we have more devices which run the tests marginally faster while leaving the g5s for perf related testing. Let me know if the g5s are a requirement or if we can get the same results by testing arm7 32bit builds on p2s.

(In reply to Bob Clary [:bc:] from comment #7) > I set up the g5 and p2 to run the jittests for arm7 32bit on try. You can > see the results at > <https://treeherder.mozilla.org/#/ > jobs?repo=try&tier=1%2C2%2C3&author=bclary%40mozilla. > com&fromchange=bb0706ca41d0563339079e2786f5cff0c2de8c5a&tochange=ead5af56741a > 679f99d596cb926ff583a88f7660>. Thanks! Here are the test failures for each platform: Moto G5 (arm7): jit 1, 4,5, 9,10 Pixel 2 (arm7): jit 5, 9,10 Pixel 2 (aarch64): jit 1,2,4,5,6,7,8,9,10 So the arm7 failures on the Moto G5 were a superset of those on Pixel 2. Perhaps that's just a timing difference. Once the jit tests are all green on the Pixel 2, we can try another sanity check on the Moto G5. > If what you want is arm7 32bit tests, I don't believe the g5s are the best > choice but we can get arm7 32bit tests on the p2 where we have more devices > which run the tests marginally faster while leaving the g5s for perf related > testing. I see. I didn't realize we were running arm7 tests on the Pixel 2. We don't need to run arm7 tests on the Moto G5 if we can run both arm7 and aarch64 on the Pixel 2. In that case, Moto G5 devices would only be interesting for performance benchmarks like Speedometer and Unity WebGL. I'll note that in my test matrix spreadsheet.

Given that there are more ARMv7 failures on the G5 than on the Pixel 2, it seems like running on the G5 would be very valuable. The Jit4 failure looks like infra, while the Jit1 failure looks like something actionable. (I agree on the strategy of fixing the Pixel2 first and then re-testing; just saying that we should have the most stringent hardware for regression tests if we can. We know there are some wasm corner cases that are poorly implemented and /may/ show up on obscure hardware for example. Not that the G5 is obscure.)

We don't have enough g5s to spare and still reserve enough for perf testing that is coming. Assuming the jittests are run on 40% of pushes and we run them on built-projects it would take about 10 devices. If they run on a higher percentage of pushes it would take more devices. If someone oks 10-15 more g5s, it would be possible. We aren't necessarily limited to g5s. Any device which can be rooted could be used. We would just need the approval and budget.

P1 because fixing ARMv7 test failures is a prerequisite for ARM64. Test failures from "Android 7.0 MotoG5 opt": https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=ead5af56741a679f99d596cb926ff583a88f7660 Jit1 exception output: /data/local/tests/jit-tests/jit-tests/tests/TypedObject/fuzz4.js:6:3 Error: Assertion failed: got (void 0), expected ({0:0, 1:0, 2:0, 3:0, 4:0, 5:0, 6:0, 7:0, 8:0, 9:0}) exception output: /data/local/tests/jit-tests/jit-tests/tests/auto-regress/bug678086.js:15:5 Error: Assertion failed: got "a/undefined", expected "a/b" Jit5 TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/ctypes/conversion-finalizer.js | Segmentation fault (code 139, args "--ion-eager --ion-offthread-compile=off") [0.3 s] Jit9 TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/jaeger/bug670885.js | error: closed (code 1, args "--ion-eager --ion-offthread-compile=off") [0.1 s] Jit10 TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/self-test/assertRecoveredOnBailout-1.js | Segmentation fault (code 139, args "--no-baseline --no-ion") [0.4 s] TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/wasm/spec/float_memory.wast.js | #1 module successfully instantiated: PASS. (code 3, args "") [0.3 s] TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/wasm/spec/memory_redundancy.wast.js | #1 module successfully instantiated: PASS. (code 3, args "") [0.2 s]

I spun the wasm failures out into bug 1513231 and will try to repro locally.