Treeherder Bug Filer Reporter
	Description • 6 years ago
treeherder

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=164106529&repo=mozilla-central

https://queue.taskcluster.net/v1/task/ZoYyzIuwQGusDDlh-JGgtA/runs/0/artifacts/public/logs/live_backing.log

[task 2018-02-24T10:59:07.306Z] PATH="/builds/worker/workspace/sixgill/usr/bin:${PATH}" SOURCE='/builds/worker/checkouts/gecko' ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' LD_LIBRARY_PATH='/builds/worker/workspace/obj-haz-shell/dist/bin' XDB='/builds/worker/workspace/sixgill/usr/bin/xdb.so' /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeCallgraph.js typeInfo.txt > callgraph.txt
[task 2018-02-24T10:59:26.352Z] Finished loading data structures
[task 2018-02-24T11:11:59.451Z] Running gcFunctions to generate ('gcFunctions.txt', 'gcFunctions.lst', 'gcEdges.txt', 'suppressedFunctions.lst')
[task 2018-02-24T11:11:59.451Z] PATH="/builds/worker/workspace/sixgill/usr/bin:${PATH}" SOURCE='/builds/worker/checkouts/gecko' ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' LD_LIBRARY_PATH='/builds/worker/workspace/obj-haz-shell/dist/bin' XDB='/builds/worker/workspace/sixgill/usr/bin/xdb.so' /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js callgraph.txt gcFunctions.txt gcFunctions.lst gcEdges.txt suppressedFunctions.lst
[task 2018-02-24T11:12:50.486Z] Traceback (most recent call last):
[task 2018-02-24T11:12:50.486Z]   File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 323, in <module>
[task 2018-02-24T11:12:50.486Z]     run_job(step, data)
[task 2018-02-24T11:12:50.486Z]   File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 199, in run_job
[task 2018-02-24T11:12:50.486Z]     subprocess.check_call(command, env=env(config))
[task 2018-02-24T11:12:50.486Z]   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
[task 2018-02-24T11:12:50.486Z]     raise CalledProcessError(retcode, cmd)
[task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' returned non-zero exit status -11
[task 2018-02-24T11:12:50.489Z] + onexit
[task 2018-02-24T11:12:50.489Z] + grab_artifacts /builds/worker/workspace/analysis /builds/worker/artifacts
[task 2018-02-24T11:12:50.489Z] + local analysis_dir
[task 2018-02-24T11:12:50.489Z] + analysis_dir=/builds/worker/workspace/analysis
[task 2018-02-24T11:12:50.489Z] + local artifacts
[task 2018-02-24T11:12:50.489Z] + artifacts=/builds/worker/artifacts
[task 2018-02-24T11:12:50.489Z] + cd /builds/worker/workspace/analysis
[task 2018-02-24T11:12:50.490Z] + ls -lah
[task 2018-02-24T11:12:50.491Z] total 3.7G

	Noemi Erli[:noemi_erli]
	Comment 3 • 6 years ago

There are 31 failures in the past 7 days, all of them on Linux X64 debug.
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=164373725&lineNumber=39277

Relevant part of the log: 
[task 2018-02-26T17:30:45.521Z] Traceback (most recent call last):
[task 2018-02-26T17:30:45.521Z]   File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 323, in <module>
[task 2018-02-26T17:30:45.521Z]     run_job(step, data)
[task 2018-02-26T17:30:45.521Z]   File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 199, in run_job
[task 2018-02-26T17:30:45.521Z]     subprocess.check_call(command, env=env(config))
[task 2018-02-26T17:30:45.521Z]   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
[task 2018-02-26T17:30:45.521Z]     raise CalledProcessError(retcode, cmd)
[task 2018-02-26T17:30:45.521Z] subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' returned non-zero exit status -11
[task 2018-02-26T17:30:45.524Z] + onexit
[task 2018-02-26T17:30:45.524Z] + grab_artifacts /builds/worker/workspace/analysis /builds/worker/artifacts
[task 2018-02-26T17:30:45.524Z] + local analysis_dir
[task 2018-02-26T17:30:45.524Z] + analysis_dir=/builds/worker/workspace/analysis
[task 2018-02-26T17:30:45.524Z] + local artifacts
[task 2018-02-26T17:30:45.524Z] + artifacts=/builds/worker/artifacts
[task 2018-02-26T17:30:45.525Z] + cd /builds/worker/workspace/analysis
[task 2018-02-26T17:30:45.525Z] + ls -lah
[task 2018-02-26T17:30:45.526Z] total 3.7G

This bug blocks 903519 which has some recent activity

	(not currently active) Ted Mielczarek
	Comment 4 • 6 years ago

(In reply to Treeherder Bug Filer from comment #0)
> [task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command
> '['/builds/worker/workspace/obj-haz-shell/dist/bin/js',
> '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/
> computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp',
> 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']'
> returned non-zero exit status -11

This is a SIGSEGV. sfink: could these hazard builds use the same Breakpad integration bits you added for other JS builds?

	Steve Fink [:sfink] [:s:]
	Comment 5 • 6 years ago

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> (In reply to Treeherder Bug Filer from comment #0)
> > [task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command
> > '['/builds/worker/workspace/obj-haz-shell/dist/bin/js',
> > '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/
> > computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp',
> > 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']'
> > returned non-zero exit status -11
> 
> This is a SIGSEGV. sfink: could these hazard builds use the same Breakpad
> integration bits you added for other JS builds?

Yes. I started working on that last night. I should file the bug though... bug 1441946. I actually hadn't realized how frequent JS crashes during the analysis are; it looks like it has been generating quite a few intermittents for a while.

The first thing I've run into is that the binary I built and am successfully using for the SM jobs is incompatible with the hazard environment -- minidump_stackwalk wants a newer libstdc++ than the hazard builds have. Which makes no sense to me, since  they both get gcc from the toolchain task, and they're using libstdc++ from that gcc?!

But this is motivating me to finally make a toolchain task for breakpad. I'm scrambling at the moment to see if there's some way I might be able to squeeze nursery strings in before the soft freeze, which is looking highly unlikely, but once that deadline whooshes by I'll get back to it.

	(not currently active) Ted Mielczarek
	Comment 6 • 6 years ago

(In reply to Steve Fink [:sfink] [:s:] from comment #5)
> The first thing I've run into is that the binary I built and am successfully
> using for the SM jobs is incompatible with the hazard environment --
> minidump_stackwalk wants a newer libstdc++ than the hazard builds have.
> Which makes no sense to me, since  they both get gcc from the toolchain
> task, and they're using libstdc++ from that gcc?!

Oh, the minidump_stackwalk binaries in tooltool are not built with the toolchain gcc because I have never gotten around to fixing bug 1391408. That might be a lot easier nowadays because we have cross-compiled Mac and mingw builds in CI, so I could just reuse most of those toolchains for building minidump_stackwalk binaries for Mac/Windows.

The minidump_stackwalk binaries were built with this script (which builds binaries for all of linux32, linux64, mac, and win32):
https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/misc/minidump_stackwalk.sh

in this Docker image:
https://hub.docker.com/r/luser/breakpad-builder/

built from the Dockerfile + scripts here:
https://github.com/luser/breakpad-taskcluster

Humorously, I did write a script to do the actual build itself in Taskcluster, but it predates toolchain tasks.

	Firefox Bug Husbandry Bot
	Comment 8 • 6 years ago

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup