Filed by: nerli [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=154070870&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/G1Lk9HwpQs-EOo_raWmewA/runs/0/artifacts/public/logs/live_backing.log https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/G1Lk9HwpQs-EOo_raWmewA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Earliest failure I found: https://treeherder.mozilla.org/logviewer.html#?job_id=153976899&repo=autoland JS bugs bug 1423225 and bug 1427870 landed before that.

PROCESS-CRASH | automation.py | application crashed [@ js::NativeObject::getReservedSlot] Crash dump filename: /tmp/tmpAsvmHO.mozrunner/minidumps/338ec24a-e6ca-b3ec-3688-c06fda57860c.dmp Operating system: Linux 0.0.0 Linux 4.4.0-98-generic #121~14.04.1-Ubuntu SMP Wed Oct 11 11:54:55 UTC 2017 x86_64 CPU: amd64 family 6 model 62 stepping 4 2 CPUs GPU: UNKNOWN Crash reason: SIGSEGV Crash address: 0x0 Process uptime: not available Thread 0 (crashed) 0 libxul.so!js::NativeObject::getReservedSlot [Shape.h:36187d67ca82 : 122 + 0x18] 1 libxul.so!js::SavedStacks::adoptAsyncStack [SavedStacks.cpp:36187d67ca82 : 172 + 0x5] 2 libxul.so!js::SavedStacks::insertFrames [SavedStacks.cpp:36187d67ca82 : 1403 + 0x10] 3 libxul.so!js::SavedStacks::saveCurrentStack [SavedStacks.cpp:36187d67ca82 : 1175 + 0x14] 4 libxul.so!JS::CaptureCurrentStack [jsapi.cpp:36187d67ca82 : 7685 + 0x5] 5 libxul.so!PromiseDebugInfo::create [Promise.cpp:36187d67ca82 : 191 + 0x5] 6 libxul.so!CreatePromiseObjectInternal [Promise.cpp:36187d67ca82 : 1504 + 0x5] 7 libxul.so!CreatePromiseObjectWithoutResolutionFunctions [Promise.cpp:36187d67ca82 : 856 + 0x5] 8 libxul.so!NewPromiseCapability [Promise.cpp:36187d67ca82 : 920 + 0xc] 9 libxul.so!js::OriginalPromiseThen [Promise.cpp:36187d67ca82 : 2482 + 0x5] 10 libxul.so!js::Promise_then [Promise.cpp:36187d67ca82 : 3062 + 0x5] 11 0x1685232cf011 12 0x7f7658e471a8

Taking this since it fails frequently. Either (1) we really have a proxy here (probably a wrapper or a dead object proxy) or (2) it's memory corruption and this assert fails because we have a garbage Class. I'm not super familiar with the SavedStacks code but I can do some Try pushes to see which one it is, then we can go from there.

Most of the time we fail in SavedFrame code, but in some cases we failed this assertion under EnvironmentObject and CTypes code. Try instrumentation suggests the Class itself is getting corrupted - I'm now doing another Try push to confirm this.

Nominating for tracking because this is a frequent assertion failure that may indicate memory corruption.

Retriggers on autoland are strongly pointing at bug 1424866 as the culprit. https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=debug%20h16&group_state=expanded&fromchange=71b4e1ee7d995ba671c95417abae0cc4bb9bde6e&tochange=b2de74e80df2c095c346bb058ecd4f439248c56f

But of course, that makes no #)*!#$ sense, so I dunno...

And on inbound, the failures definitely appear to have started on the autoland merge that brought those commits over: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=f78a83244fbebe8a469ae3512fce7f638cab7e1f

It looks like this disappeared again? :/ I did some more Try pushes but it's very random and seems build specific.

It's possible this could be caused by a moving GC if we fail to trace something. I'll kick off a try build with GC zeal turned on and see if this catches anything.

this went away on Jan 6th, I don't see any comments as to why this is fixed- it would be nice if we could associate this with a changeset/bug.

Bah, it looks like this is back on inbound. Could be this change I landed: https://hg.mozilla.org/integration/mozilla-inbound/rev/4ef2425934f236201df33293a34c2400d5c5f19d This is very weird; we should try to get it in rr so we can see what memory is corrupted where...

So it looks like this went away again with glandium's toolchain changes here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=7c92680e18dbc44de8b2bb5efbb23c686e0fc504 The commit message mentions massive changes to the binaries, so it makes some sense I guess. I'll see if I can run tests locally with one of the buggy builds from Treeherder...

OK I'm running headless mochitests now on Linux with one of the affected builds. I'll see if that fails -- if it does I can try again with rr.

I did some retriggers on inbound but no failures at all: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4ef2425934f236201df33293a34c2400d5c5f19d&filter-searchStr=linux64%20debug&group_state=expanded Is it possible the test machines were updated or something? I would have expected at least one failure with all these retriggers...

The only recent thing going on that I know of is AWS rolling out their Spectre/Meltdown mitigations.

Mike, are we actually using the Debian-based instances yet or just testing them?

No, and even if we were, I ensured all the way down the road that we weren't ending up with uncontrolled changes. The one Jan mentions in comment 19 is an effective no-op, and the commit message mention large diffs, but they are because things are shifted, not because code actually changes.

Is this still a problem for 59 beta?

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #24) > Is this still a problem for 59 beta? No because apparently this disappeared again. It's unsettling that this can show up (and disappear) on random commits :/ At least so far it only affected Linux64 Debug, if that's any consolation...

OK, I'll stop tracking this for 59, for now.

Unassigning as this disappeared again - we never figured out what caused this.