Closed Bug 1498298 Opened 6 years ago Closed 6 years ago

Investigate running packet.net-based Android emulator unit tests against x86-64 build

Categories

(Testing :: General, enhancement, P3)

Product:
Testing
Component:
General
Version:
Version 3
Platform:
x86_64
Android
Type:
enhancement

Tracking

(firefox-esr60 wontfix, firefox65 wontfix, firefox66 wontfix, firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox-esr60 --- wontfix
firefox65 --- wontfix
firefox66 --- wontfix
firefox67 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

(Whiteboard: [geckoview:p2])

Attachments

(2 files, 1 obsolete file)

allow for android 7.0 x86 debug tests
(deleted), patch
gbrown
: review+
Details | Diff | Splinter Review
switch most tests to x86_64/opt and add x86_64/debug tests too
(deleted), patch
jmaher
: review+
snorp
: feedback+
cpeterson
: feedback+
Details | Diff | Splinter Review
If we switch our packet.net geckoview tests from the "Android 4.2 x86 opt" build to the "Android 5.0 x86-64 opt" build, how do the test results compare?
Assignee: nobody → gbrown
In the above try push, tests run against the "Android 5.0 x86-64 opt" build. Performance of all test suites seems very similar to normal packet.net x86. mochitest and geckoview-junit test results are the same as on normal packet.net x86. *** crashtest and jsreftests have persistent crashes not seen with x86 builds. *** :jchen -- Just making sure you are aware of these results; follow-up as you desire!
Flags: needinfo?(nchen)
No longer depends on: 1489264, 1490040, 1490962
Blocks: 1425322
Assignee: gbrown → nobody
Priority: P1 → P3
Blocks: Fennec-x86_64
OS: Unspecified → Android
Hardware: Unspecified → x86_64
Flags: needinfo?(nchen)
Updated, with proper e10s support (wpt excluded). Includes x86-64 opt and debug now: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3852a308e3b85f4fdc5c7e10d128a950a062e341
Blocks: 1500509
Depends on: 1512427
I'll land some task configuration changes now that will make this easier to switch over in the future (but won't make any actual changes to today's tasks).
Attachment #9034504 - Flags: review+
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/135c1ea8854e Prepare to run packet.net geckoview tests against x86_64 opt/debug builds; r=me,a=test-only
This should be all that's needed to switch to x86_64 and introduce x86_64/debug tests. Before landing, we need to green up tests (investigate crashes). https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=769eb3ffa0efccbd4be900784101827b54c5864f
https://hg.mozilla.org/mozilla-central/rev/135c1ea8854e
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox66: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 1518849

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

Assignee: nobody → gbrown

(In reply to Geoff Brown [:gbrown] from comment #10)

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

If I'm reading Treeherder correctly, are the results reversed? It looks like the crashtests (C) are green on opt but orange on debug and the jsreftests (J1-J8) are green on both opt and debug.

I filed bug 1524493 for the debug crashtest failure.

(In reply to Chris Peterson [:cpeterson] from comment #11)

(In reply to Geoff Brown [:gbrown] from comment #10)

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

If I'm reading Treeherder correctly, are the results reversed? It looks like the crashtests (C) are green on opt but orange on debug and the jsreftests (J1-J8) are green on both opt and debug.

Yes, that's correct. Thanks!

Depends on: 1524693

Let's do this:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=ef8fa87e327af92d0de25d902e15420aaf7434ef

  • stop running x86/opt mochitest-cl, mochitest-gpu, crashtest, jsreftest, geckoview-junit
  • start running x86_64/opt mochitest-cl, mochitest-gpu, crashtest, jsreftest, geckoview-junit
  • start running x86_64/debug mochitest-cl, mochitest-gpu, jsreftest
  • no /debug crashtests until at least bug 1524493 is resolved
  • no /debug geckoview-junit, and geckoview-junit runs tier 3, until at least bug 1521195 is resolved
  • web-platform tests remain on x86 for now; perhaps move them to x86_64 when they can run on geckoview
  • treeherder labels for x86_64 will be tidied up in bug 1524693
Attachment #9034505 - Attachment is obsolete: true
Attachment #9040860 - Flags: review?(jmaher)
Attachment #9040860 - Flags: feedback?(snorp)
Attachment #9040860 - Flags: feedback?(cpeterson)
Comment on attachment 9040860 [details] [diff] [review] switch most tests to x86_64/opt and add x86_64/debug tests too LGTM! Looking at your Treeherder results, I see: x86_64 opt: crashtests, jsreftests, mochitest-clipboard, and mochitest-gpu are all green. x86_64 debug: jsreftests, mochitest-clipboard, and mochitest-gpu are all green. Debug crashtests are disabled due to bug 1524493.
Attachment #9040860 - Flags: feedback?(cpeterson) → feedback+
Comment on attachment 9040860 [details] [diff] [review] switch most tests to x86_64/opt and add x86_64/debug tests too Review of attachment 9040860 [details] [diff] [review]: ----------------------------------------------------------------- ::: taskcluster/ci/test/test-sets.yml @@ +394,2 @@ > - web-platform-tests > - web-platform-tests-reftests So the only tests that will still be running on Android x86 will be the web-platform-tests?

(In reply to Chris Peterson [:cpeterso

So the only tests that will still be running on Android x86 will be the
web-platform-tests?

Correct!

Comment on attachment 9040860 [details] [diff] [review] switch most tests to x86_64/opt and add x86_64/debug tests too Review of attachment 9040860 [details] [diff] [review]: ----------------------------------------------------------------- ::: taskcluster/ci/test/reftest.yml @@ +69,3 @@ > chunks: > by-test-platform: > android-em-4.3-arm7-api-16/debug: 100 do we get any real value from jsreftests on android 4.3 arm? this seems like a very expensive cost and I would question the use of it. but unrelated to this bug! ::: taskcluster/ci/test/test-sets.yml @@ +389,3 @@ > # - mochitest-webgl1-core > # - reftest > # - test-verify why are these commented out? waiting on greening up? I was under the impression these would be running on opt/debug.
Attachment #9040860 - Flags: review?(jmaher) → review+

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #18)

do we get any real value from jsreftests on android 4.3 arm? this seems
like a very expensive cost and I would question the use of it. but
unrelated to this bug!

@ Sean, how important are the jsreftests compared to the jit-tests?

On our Google Pixel 2 test devices, we currently run the jit-tests (for both ARMv7 and ARM64 Gecko builds) but not jsreftests.

On our Android ARMv7 emulator, we currently run jsreftests but not jit-tests.

status-firefox65: --- → wontfix
status-firefox66: affected → wontfix
status-firefox67: --- → affected
status-firefox-esr60: --- → wontfix
Flags: needinfo?(sstangl)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #18)

do we get any real value from jsreftests on android 4.3 arm? this seems
like a very expensive cost and I would question the use of it. but
unrelated to this bug!

We're scheduled to consider turning off Android 4.3 jsreftests once this bug is resolved and we have seen a few days of x86_64 results; will pursue that in bug 1473368.

::: taskcluster/ci/test/test-sets.yml
@@ +389,3 @@

 # - mochitest-webgl1-core
 # - reftest
 # - test-verify

why are these commented out? waiting on greening up? I was under the
impression these would be running on opt/debug.

Yes, waiting on greening up. Most of that work is tracked in bug 1460411.

Attachment #9040860 - Flags: feedback?(snorp) → feedback+
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/908a14f72ae3 Switch most Android 7.0 x86 tests to x86_64 builds; add Android 7.0 x86_64/debug tests; r=jmaher

https://hg.mozilla.org/mozilla-central/rev/908a14f72ae3

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
status-firefox67: affected → fixed
Resolution: --- → FIXED
Target Milestone: mozilla66 → mozilla67
Flags: needinfo?(sstangl)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: