Filed by: tomcat [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=37763720&repo=mozilla-inbound https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-pgo/1476700355/mozilla-inbound_ubuntu64_hw_test-chromez-e10s-pgo-bm104-tests1-linux-build1379.txt.gz

Frequency increase on Feb 24 may be related to bug 1339594.

marking as fixed as this is greatly reduced.

given the fact that these are almost all win8 pgo/opt, we should look into this. tresize is where we crash- the simple thing is to not run tresize on win8 :) any thoughts on why we get an unknown top frame? do we need to do something different for stackwalk? here is an example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=82538140&lineNumber=1666 11:29:42 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/eG2s1dbXQx-PgU5F3jPmeQ/artifacts/public/build/firefox-55.0a1.en-US.win64.crashreporter-symbols.zip 11:29:46 INFO - mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld~1.t-w\appdata\local\temp\tmpkw8zcu\profile\minidumps\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp c:\users\cltbld~1.t-w\appdata\local\temp\tmpbfkner 11:29:46 INFO - mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp 11:29:47 INFO - PROCESS-CRASH | tresize | application crashed [unknown top frame] 11:29:47 INFO - Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\tmpkw8zcu\profile\minidumps\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp 11:29:47 INFO - stderr from minidump_stackwalk: 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4359: INFO: Minidump opened minidump c:\users\cltbld~1.t-w\appdata\local\temp\tmpkw8zcu\profile\minidumps\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4479: INFO: Minidump not byte-swapping minidump 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 15 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 7 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 7 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 1197932545 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 6 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 1197932546 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 4 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 5 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4956: INFO: GetStream: type 3 not present 11:29:47 INFO - 2017-03-08 11:29:46: minidump_processor.cc:152: ERROR: Minidump c:\users\cltbld~1.t-w\appdata\local\temp\tmpkw8zcu\profile\minidumps\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp has no thread list 11:29:47 INFO - 2017-03-08 11:29:46: stackwalk.cc:139: ERROR: MinidumpProcessor::Process failed 11:29:47 INFO - 2017-03-08 11:29:46: minidump.cc:4331: INFO: Minidump closing minidump 11:29:47 INFO - minidump_stackwalk exited with return code 1 11:29:47 INFO - TEST-UNEXPECTED-ERROR | tresize | Found crashes after test run, terminating test :ted, would you have ideas why we have no top frame?

These are all timeouts: 08:19:51 INFO - TEST-INFO | started process 2756 (C:\slave\test\build\application\firefox\firefox -profile c:\users\cltbld~1.t-w\appdata\local\temp\tmp2o5cos\profile http://localhost:49280/startup_test/tresize/addon/content/tresize-test.html) 08:22:21 INFO - Timeout waiting for test completion; killing browser... 08:22:21 INFO - Terminating psutil.Process(pid=2756, name=u'firefox.exe') There is a good crash report sometimes, as in https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=81698927&lineNumber=1695, but not often.

this looks to be the same stack as mentioned in bug 1342685. If this is the case (which makes sense if this is a timeout), then we have issues creating a new window (although this could be the process creation).

Also for :ted's consideration, there is another Talos, Windows crash report in https://bugzilla.mozilla.org/show_bug.cgi?id=1346707#c0 with breakpad in the stack.

so it appears that :blassey fixed the win7 issues!! Now we have 100+ failures/week of win8 crashes in talos, will ask for more help on Monday if we don't have any action here.

> 11:29:47 INFO - 2017-03-08 11:29:46: minidump_processor.cc:152: ERROR: Minidump c:\users\cltbld~1.t-w\appdata\local\temp\tmpkw8zcu\profile\minidumps\05fc0085-4ed2-4af6-93d6-eb03f1953fbd.dmp has no thread list That's the smoking gun--the dump is missing all the useful information we'd need to actually produce a stack. This can happen sometimes when we're writing a minidump for the chrome process but the process is in a really bad state (heap corruption or certain kinds of OOM).

So then what do we do to figure this out? We have almost 200 instances/week of this between all the different talos bugs. Do we need to fix the minidump tool?

(In reply to Joel Maher ( :jmaher) from comment #14) > So then what do we do to figure this out? We have almost 200 instances/week > of this between all the different talos bugs. Do we need to fix the > minidump tool? The issue here is that Firefox is crashing in a way that it can't generate a valid minidump. The minidump writing code we use on Windows is just MinidumpWriteDump from Microsoft's dbghelp.dll. I don't have any great solutions for you.

Most of these are still Windows 8 with no crash report, as before. I found some crash reports in bug 1345735 and am waiting on a needinfo there. There are also now a few Linux crashes reported here...those look different, so I'll spin off another bug for Linux.

following up here, we are looking closer at bug 1345735