utils.py uses fix_stack_using_bpsyms.py. This can now be replaced with fix_stacks.py, for speed and code simplicity improvements.

Depends on D66363

This is similar to the same-named commit in bug 1619837.

For those few configurations that don't run xpcshell tests, we need a
fix-stacks dependency in the build job. This is because xpcshell self tests
run during the build job, but the existing xpcshell test dependency doesn't
cover that case.

gbrown: this bug is about changing the stack-fixing script used to process test output when breakpad symbols are present, as is the case for debug jobs on automation. The old script is tools/rb/fix_stack_using_bpsyms.py, the new script is tools/rb/fix_stacks.py. The patches work on Linux/Mac/Windows but there are test failures on Android, as this try push shows. These failures have "ImportError: No module named fix_stacks" in testing/mozbase/mozrunner/mozrunner/utils.py, because the newish fix_stacks.py script isn't present.

glandium helped me investigate this, and my understanding is that utils.py is using utilityPath as the path to find fix_stacks.py. utilityPath is specified on a command line (somehow) to the path of a 71.0a1 archive in tooltool(!). That archive unsurprisingly doesn't have fix_stacks.py. It would be better if utilityPath was somehow set to point to the bin/ directory of the extracted contents of target.common.tests.tar.gz, which end up in /builds/worker/workspace/build/tests.

Do you have any idea of how to achieve this? Thanks.

https://hg.mozilla.org/mozilla-central/rev/4a5f14b37c79

The executables in/bin from an android build's target.common.tests.tar.gz are appropriate to the android device, often arm binaries. But the android test harnesses usually need to use those programs on the host -- they need the ones from a Linux build -- so they download that "host-utils" archive and use that instead. host-utils are updated from a Linux build as needed...and we haven't needed to for awhile: currently based on 71.0a1. Of course this Android/Linux build dance is not necessary for scripts and python, but some of that goes along for the ride: It usually doesn't hurt to use the Linux build for such things.

I think the easiest way forward is to update host-utils, and make sure they have your fix-stacks changes. Would that be okay?

(In reply to Geoff Brown [:gbrown] from comment #6)

I think the easiest way forward is to update host-utils, and make sure they have your fix-stacks changes. Would that be okay?

That makes sense, we'll need the Linux fix-stacks executable on the host as well as the script. Is that something you can do or at least explain how to do?

It looks like the Linux fix-stacks executable is already being fetched for all android test tasks, and I can see that happening in test logs: That part should be okay already. We'll get fix_stacks.py added in bug 1623134.

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293528417&repo=mozilla-central&lineNumber=34

[fetches 2020-03-17T16:39:13.851Z] Extracting /builds/worker/fetches/fix-stacks.tar.xz to /builds/worker/fetches

Thank you, gbrown!

Backed out changeset 3b38351e9979 (bug 1622701) for multiple crashtest failures.

Push that started the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Csuperseded%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&fromchange=f753bf2c8d70cd31970e42dad254c54b17705da7&searchStr=android%2C7.0%2Cx86-64%2Cquantumrender%2Cdebug%2Creftests%2Ctest-android-em-7.0-x86_64-qr%2Fdebug-geckoview-crashtest-e10s%2Cr%28c%29&selectedJob=293988201

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293988201&repo=autoland&lineNumber=1334

Backout: https://hg.mozilla.org/integration/autoland/rev/9cd87640f1d39ac7334ab0114437d7fc9b88923d

This breaks the Linux x64 debug builds on Google Cloud Platform: https://treeherder.mozilla.org/logviewer.html#?job_id=296759520&repo=autoland

check> tracebacks = ['Traceback (most recent call last):\n File "/builds/worker/checkouts/gecko/testing/xpcshell/runxpcshelltests.py", li... fix-stacks; please run ./mach bootstrap')\nException: cannot find fix-stacks; please run ./mach bootstrap\n']

That's a tier 3 job, right? (I've never even heard of that job kind before now.)

I've seen similar failures before and it's a simple fix, requiring the addition of a single line to a taskcluster config file. It's late here now, I'll do it tomorrow morning.

https://hg.mozilla.org/mozilla-central/rev/fc7f46174390

https://hg.mozilla.org/mozilla-central/rev/2aa65fbb4aee