https://treeherder.mozilla.org/logviewer.html#?job_id=192736884&repo=mozilla-inbound&lineNumber=8118 https://queue.taskcluster.net/v1/task/MXqUI0qqS2qLy8CVeCFEcA/runs/0/artifacts/public/logs/live_backing.log [task 2018-08-08T10:11:01.939Z] 10:11:01 INFO - STDOUT: tests/web-platform/tests/webdriver/tests/actions/special_keys.py::test_webdriver_special_key_sends_keydown[RETURN-expected1] [task 2018-08-08T10:11:01.941Z] 10:11:01 INFO - PID 1043 | 1533723061937 webdriver::server DEBUG -> POST /session/b74d802b-959e-4cb3-a16b-11bd3d88a8e7/window/rect {"width": 800, "height": 600} [task 2018-08-08T10:11:01.941Z] 10:11:01 INFO - PID 1043 | 1533723061939 Marionette TRACE 0 -> [0,1491,"WebDriver:SetWindowRect",{"height":600,"width":800,"x":null,"y":null}] [task 2018-08-08T10:11:01.949Z] 10:11:01 INFO - PID 1043 | 1533723061943 Marionette TRACE 0 <- [1,1491,null,{"x":0,"y":0,"width":800,"height":600,"state":"normal"}] [task 2018-08-08T10:11:01.950Z] 10:11:01 INFO - PID 1043 | 1533723061944 webdriver::server DEBUG <- 200 OK {"value": {"height":600,"width":800,"x":0,"y":0}} [task 2018-08-08T10:11:01.950Z] 10:11:01 INFO - PID 1043 | 1533723061945 webdriver::server DEBUG -> POST /session/b74d802b-959e-4cb3-a16b-11bd3d88a8e7/url {"url": "http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html"} [task 2018-08-08T10:11:01.951Z] 10:11:01 INFO - PID 1043 | 1533723061946 Marionette TRACE 0 -> [0,1492,"WebDriver:Navigate",{"url":"http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html"}] [task 2018-08-08T10:11:01.967Z] 10:11:01 INFO - PID 1043 | 1533723061960 Marionette DEBUG [4294967297] Received DOM event beforeunload for http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html [task 2018-08-08T10:11:01.983Z] 10:11:01 INFO - PID 1043 | 1533723061975 Marionette DEBUG [4294967297] Received DOM event pagehide for http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html [task 2018-08-08T10:11:01.984Z] 10:11:01 INFO - PID 1043 | 1533723061979 Marionette DEBUG [4294967297] Received DOM event unload for http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html [task 2018-08-08T10:11:02.025Z] 10:11:02 INFO - PID 1043 | 1533723062018 Marionette DEBUG [4294967297] Received DOM event DOMContentLoaded for http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html [task 2018-08-08T10:11:02.042Z] 10:11:02 INFO - PID 1043 | 1533723062029 Marionette DEBUG [4294967297] Received DOM event pageshow for http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html [task 2018-08-08T10:11:02.043Z] 10:11:02 INFO - PID 1043 | 1533723062035 Marionette TRACE 0 <- [1,1492,null,{"value":null}] [task 2018-08-08T10:11:02.047Z] 10:11:02 INFO - PID 1043 | 1533723062040 webdriver::server DEBUG <- 200 OK {"value": null} [task 2018-08-08T10:11:02.048Z] 10:11:02 INFO - PID 1043 | 1533723062042 webdriver::server DEBUG -> POST /session/b74d802b-959e-4cb3-a16b-11bd3d88a8e7/element {"using": "css selector", "value": "#keys"} [task 2018-08-08T10:11:02.069Z] 10:11:02 INFO - PID 1043 | 1533723062061 Marionette TRACE 0 -> [0,1493,"WebDriver:FindElement",{"using":"css selector","value":"#keys"}] [task 2018-08-08T10:11:02.085Z] 10:11:02 INFO - PID 1043 | *** Compartment mismatch 0x6080000108a0 vs. 0x608000081120 [task 2018-08-08T10:11:02.086Z] 10:11:02 INFO - PID 1043 | AddressSanitizer:DEADLYSIGNAL [task 2018-08-08T10:11:02.086Z] 10:11:02 INFO - PID 1043 | ================================================================= [task 2018-08-08T10:11:02.087Z] 10:11:02 ERROR - PID 1043 | ==1149==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f48789befa8 bp 0x7fff2b3a4e10 sp 0x7fff2b3a4b20 T0) [task 2018-08-08T10:11:02.088Z] 10:11:02 INFO - PID 1043 | ==1149==The signal is caused by a WRITE memory access. [task 2018-08-08T10:11:02.088Z] 10:11:02 INFO - PID 1043 | ==1149==Hint: address points to the zero page. [task 2018-08-08T10:11:02.709Z] 10:11:02 INFO - PID 1043 | #0 0x7f48789befa7 in fail /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:39:9 [task 2018-08-08T10:11:02.709Z] 10:11:02 INFO - PID 1043 | #1 0x7f48789befa7 in check /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:54 [task 2018-08-08T10:11:02.710Z] 10:11:02 INFO - PID 1043 | #2 0x7f48789befa7 in check /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:66 [task 2018-08-08T10:11:02.710Z] 10:11:02 INFO - PID 1043 | #3 0x7f48789befa7 in check<js::NativeObject *> /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:77 [task 2018-08-08T10:11:02.711Z] 10:11:02 INFO - PID 1043 | #4 0x7f48789befa7 in assertSameCompartment<JS::Handle<js::NativeObject *>, JS::Handle<js::NativeObject *> > /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:219 [task 2018-08-08T10:11:02.712Z] 10:11:02 INFO - PID 1043 | #5 0x7f48789befa7 in InitializePropertiesFromCompatibleNativeObject /builds/worker/workspace/build/src/js/src/vm/JSObject.cpp:1378 ...... [task 2018-08-08T10:11:02.868Z] 10:11:02 INFO - PID 1043 | #83 0x7f486cceba5c in MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:298 [task 2018-08-08T10:11:02.868Z] 10:11:02 INFO - PID 1043 | #84 0x7f48777c739a in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:768:34 [task 2018-08-08T10:11:02.869Z] 10:11:02 INFO - PID 1043 | #85 0x4f22e4 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:50:30 [task 2018-08-08T10:11:02.870Z] 10:11:02 INFO - PID 1043 | #86 0x4f22e4 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:287 [task 2018-08-08T10:11:02.967Z] 10:11:02 INFO - PID 1043 | #87 0x7f488b5f882f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291 [task 2018-08-08T10:11:02.967Z] 10:11:02 INFO - PID 1043 | #88 0x421708 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x421708) [task 2018-08-08T10:11:02.968Z] 10:11:02 INFO - PID 1043 | [task 2018-08-08T10:11:02.969Z] 10:11:02 INFO - PID 1043 | AddressSanitizer can not provide additional info. [task 2018-08-08T10:11:02.969Z] 10:11:02 INFO - PID 1043 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/js/src/vm/JSContext-inl.h:39:9 in fail [task 2018-08-08T10:11:02.970Z] 10:11:02 INFO - PID 1043 | ==1149==ABORTING [task 2018-08-08T10:11:03.014Z] 10:11:03 INFO - PID 1043 | [task 2018-08-08T10:11:03.014Z] 10:11:03 INFO - PID 1043 | ###!!! [Parent][MessageChannel] Error: (msgtype=0x17007C,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv [task 2018-08-08T10:11:03.015Z] 10:11:03 INFO - PID 1043 | [task 2018-08-08T10:11:03.030Z] 10:11:03 INFO - PID 1043 | [task 2018-08-08T10:11:03.031Z] 10:11:03 INFO - PID 1043 | ###!!! [Parent][MessageChannel] Error: (msgtype=0x17007C,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv [task 2018-08-08T10:11:03.032Z] 10:11:03 INFO - PID 1043 | [task 2018-08-08T10:11:03.093Z] 10:11:03 INFO - PID 1043 | 1533723063090 Marionette DEBUG [17] Frame script loaded [task 2018-08-08T10:11:03.095Z] 10:11:03 INFO - PID 1043 | 1533723063091 Marionette DEBUG [17] Frame script registered [task 2018-08-08T10:11:03.177Z] 10:11:03 INFO - PID 1043 | A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down [task 2018-08-08T10:11:03.356Z] 10:11:03 INFO - PID 1043 | 1533723063345 Marionette DEBUG Received DOM event unload for [object XULDocument] [task 2018-08-08T10:11:03.376Z] 10:11:03 INFO - PID 1043 | 1533723063370 Marionette DEBUG Received observer notification message-manager-disconnect [task 2018-08-08T10:11:03.392Z] 10:11:03 INFO - PID 1043 | 1533723063384 Marionette TRACE 0 <- [1,1493,null,{"value":null}] [task 2018-08-08T10:11:03.409Z] 10:11:03 INFO - PID 1043 | 1533723063397 webdriver::server DEBUG <- 500 Internal Server Error {"value":{"error":"unknown error","message":"Failed to convert data to an object","stacktrace":""}} [task 2018-08-08T10:11:03.418Z] 10:11:03 INFO - PID 1043 | 1533723063413 Marionette DEBUG Closed connection 0 [task 2018-08-08T10:11:03.556Z] 10:11:03 INFO - STDOUT: ERROR [task 2018-08-08T10:11:03.571Z] 10:11:03 INFO - PID 1043 | 1533723063554 webdriver::server DEBUG -> DELETE /session/b74d802b-959e-4cb3-a16b-11bd3d88a8e7/actions [task 2018-08-08T10:11:03.572Z] 10:11:03 INFO - PID 1043 | 1533723063555 webdriver::server DEBUG Deleting session [task 2018-08-08T10:11:03.991Z] 10:11:03 INFO - PID 1043 | 1533723063987 Marionette DEBUG Received observer notification xpcom-will-shutdown [task 2018-08-08T10:11:03.993Z] 10:11:03 INFO - PID 1043 | 1533723063988 Marionette DEBUG Resetting recommended pref security.turn_off_all_security_so_that_viruses_can_take_over_this_computer [task 2018-08-08T10:11:03.993Z] 10:11:03 INFO - PID 1043 | 1533723063988 Marionette DEBUG Resetting recommended pref apz.content_response_timeout [task 2018-08-08T10:11:03.994Z] 10:11:03 INFO - PID 1043 | 1533723063988 Marionette DEBUG Resetting recommended pref browser.download.panel.shown [task 2018-08-08T10:11:03.994Z] 10:11:03 INFO - PID 1043 | 1533723063989 Marionette DEBUG Resetting recommended pref browser.pagethumbnails.capturing_disabled [task 2018-08-08T10:11:03.995Z] 10:11:03 INFO - PID 1043 | 1533723063989 Marionette DEBUG Resetting recommended pref browser.search.update [task 2018-08-08T10:11:03.996Z] 10:11:03 INFO - PID 1043 | 1533723063989 Marionette DEBUG Resetting recommended pref toolkit.cosmeticAnimations.enabled [task 2018-08-08T10:11:03.997Z] 10:11:03 INFO - PID 1043 | 1533723063991 Marionette DEBUG Resetting recommended pref browser.tabs.disableBackgroundZombification [task 2018-08-08T10:11:03.999Z] 10:11:03 INFO - PID 1043 | 1533723063991 Marionette DEBUG Resetting recommended pref browser.tabs.warnOnCloseOtherTabs

INFO - PID 1043 | *** Compartment mismatch 0x6080000108a0 vs. 0x608000081120 That sounds bad.

This is a Linux64 ASan build. We're failing the assertSameCompartment(cx, src, dst) in InitializePropertiesFromCompatibleNativeObject. "0x6080000108a0 vs. 0x608000081120" suggests it's probably a valid JS::Compartment* pointer and not a GC or memory corruption issue. HTMLDocument_Binding::Wrap is calling InitializePropertiesFromCompatibleNativeObject with: * dst = aReflector. This object was allocated right before the call so it's probably not this one. * src = unforgeableHolder. unforgeableHolder we pulled out of a reserved slot on canonicalProto and canonicalProto is the return value of GetProtoObjectHandle. That involves caches so maybe there's an issue with that?

Looking at this job on inbound/autoland, this orange showed up just once. There's also a recent and frequent GC assertion failure, bug 1482029, that affects this Wd test suite (debug builds).

So the relevant call is like so: if (!JS_InitializePropertiesFromCompatibleNativeObject(aCx, expando, unforgeableHolder)) { "expando" comes from DOMProxyHandler::EnsureExpandoObject(aCx, aReflector)) which can do one of two things: 1) Return an expando from the ExpandoAndGeneration on the document. 2) Create an object in the current Realm of aCx via JS_NewObjectWithGivenProto. That Realm is the Realm of "global", which we enter in Wrap(). "unforgeableHolder" comes from a reserved slot on canonicalProto, which comes from GetProtoObjectHandle(aCx) after entering the Realm of "global". The canonicalProto returned here will definitely be same-Realm as "global" in this case. OK, so could we have a stale "expando"? We got here via NativeInterface2JSObjectAndThrowIfFailed calling nsHTMLDocument::WrapNode, which means that GetWrapper() on the HTMLDocument returned null. And HTMLDocument_Binding::DOMProxyHandler::finalize clears out the expando in the ExpandoAndGeneration when the thing GetWrapper() returns is finalized. However, it's possible that we're in the state where EdgeNeedsSweepUnbarriered() returns true for the reflector (the thing whose proxy handler is HTMLDocument_Binding::DOMProxyHandler) but that object has not been finalized yet. In that case, we can land in NativeInterface2JSObjectAndThrowIfFailed, call GetWrapper(), get back null, go ahead and call HTMLDocument_Binding::Wrap, determine the desired global for the document and see that for some reason it has changed (maybe because it's disconnected from the window and we recently changed the behavior of that case in bug 451172?) and go ahead and create a new reflector in the new global. But we still have a stale expando hanging around in ExpandoAndGeneration, so we hit the above assert. Jon, does that all sound plausible?

And if it does, is there some way we could try reproducing this more reliably so we can test a proposed fix?

Bug 1482510 looks similar.

(In reply to Boris Zbarsky [:bzbarsky, bz on IRC] (vacation Aug 16-27) from comment #4) > OK, so could we have a stale "expando"? As I understand it we preserve the wrapper when we create the expando object, and the trace hook for ShadowingDOMProxyHandler traces the exando, so in theory neither of these objects should be getting finalized. I'll try adding some assertions around this and see if I can figure something out.

> As I understand it we preserve the wrapper when we create the expando object This is true (sorry for the lag; I needed to carefully double-check whether we in fact do end up doing it in some special cases). However, when we detect that the C++ object should be cycle-collected, we stop preserving the wrapper (so it can actually get collected). In theory at this point no one should be touching the objects anymore (because they are cc-unlinked), but if someone does... and evidence is that people do. In the stack linked above in comment 0, the access is coming via a weak reference (xpcJSWeakReference::Get in frame #25), so it's a nasty situation where we thought the object could go away and started tearing it down, but then someone revived it. I think what we should do in practical terms is clear out the expando when we get asked to wrap a thing that would have an ExpandoAndGeneration.

That said, HTMLDocument_Binding::Wrap has: MOZ_ASSERT(aObject->mExpandoAndGeneration.expando.isUndefined()); which I would expect to be triggering in this case. Certainly in a debug build like bug 1482029 has... So I'm not quite sure what's up there. I am also a bit confused by the stack in bug 1482029 comment 27. The stack there has: 22:09:48 INFO - 0 xul.dll!js::WrappedPtrOperations<JS::Value,JS::Heap<JS::Value> >::isObject() [Value.h:420590ac01b2353c6105228b5046418c4f3a2215 : 1303 + 0x22] 22:09:48 INFO - 1 xul.dll!mozilla::dom::DOMProxyHandler::EnsureExpandoObject(JSContext *,JS::Handle<JSObject *>) [DOMJSProxyHandler.cpp:420590ac01b2353c6105228b5046418c4f3a2215 : 124 + 0x8] 22:09:48 INFO - 2 xul.dll!mozilla::dom::HTMLDocument_Binding::Wrap(JSContext *,nsHTMLDocument *,nsWrapperCache *,JS::Handle<JSObject *>,JS::MutableHandle<JSObject *>) [HTMLDocumentBinding.cpp: : 2140 + 0xb] but for the HTMLDocument case the Value at line 124 of DOMJSProxyHandler.cpp should be a PrivateValue. It's really not clear to me where the isObject() on a Heap<JS::Value> is coming from there!

> which I would expect to be triggering in this case. Ah. That was just added in bug 1483487 a week ago, and these stacks largely predate it. > It's really not clear to me where the isObject() on a Heap<JS::Value> is coming from there! Still not sure about that part. That said, are the asserts from bug 1483487 showing up now?

I did just look, and the most recent failures in bug 1483702 are all in opt builds. I wonder whether it's worth making some of the bug 1483487 asserts diagnostic asserts....

As suggested in comment 8, clear ExpandoAndGeneration::expando before use.

Comment on attachment 9004938 [details] [diff] [review] bug1481844-clear-expando r=me

Comment on attachment 9004938 [details] [diff] [review] bug1481844-clear-expando [Security approval request comment] How easily could an exploit be constructed based on the patch? Very difficult. Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem? No. Which older supported branches are affected by this flaw? Possibly everything back to FF23. If not all supported branches, which bug introduced the flaw? Bug 820846. Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be? Should be trivial. How likely is this patch to cause regressions; how much testing does it need? Unlikely. This just clears a field that was already asserted to be empty.

(In reply to Jan de Mooij [:jandem] from comment #3) > Looking at this job on inbound/autoland, this orange showed up just once. > There's also a recent and frequent GC assertion failure, bug 1482029, that > affects this Wd test suite (debug builds). Could the other bug be the same one as this one? Bug 1482029 comment 19 shows assertion frames which are similar to those as mentioned here.

(In reply to Henrik Skupin (:whimboo) from comment #15) > Could the other bug be the same one as this one? I think so. Let's see what happens when this patch lands.

Comment on attachment 9004938 [details] [diff] [review] bug1481844-clear-expando sec-approval+ for trunk.

Hi Jon. Can this bug be closed now? All of the Marionette tests working and we no longer see the assertions / crashes.

(In reply to Henrik Skupin (:whimboo) from comment #21) Great. Let's close this.

I still see it marked as tracking esr60 63+. But as it looks like we missed to land the patch for that release. Can you please check the status on that?

I don't know exactly what these flags mean. I'm guessing we should still fix this on esr60 though.

Comment on attachment 9004938 [details] [diff] [review] bug1481844-clear-expando [Approval Request Comment] If this is not a sec:{high,crit} bug, please state case for ESR consideration: sec-high bug. User impact if declined: Possible crash / security vulnerability. Fix Landed on Version: 63. Risk to taking this patch (and alternatives if risky): Low. String or UUID changes made by this patch: None.

(In reply to Jon Coppeard (:jonco) from comment #24) > I don't know exactly what these flags mean. I'm guessing we should still > fix this on esr60 though. Oh, wait. We had the ESR60.2 release yesterday. The ESR60.3 (63+) will be together with the 63 release. So lets get back the tracking flag which you have removed.

Yep.

Comment on attachment 9004938 [details] [diff] [review] bug1481844-clear-expando Fixes a sec-high, approved for ESR 60.3.

Needs a rebased patch for ESR60 (or the issues in bug 1483487 to be sorted out). Backed out: https://hg.mozilla.org/releases/mozilla-esr60/rev/976fc330c260cc583009bc9d3f0d86dae4b2a4f1

Patch for esr60 including only the necessary parts of the patch in bug 1483487.

Thanks, Jon! https://hg.mozilla.org/releases/mozilla-esr60/rev/09939be135d8

Comment on attachment 9017884 [details] [diff] [review] bug1481844-esr60 [ESR Uplift Approval Request] If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fix for sec-high bug. User impact if declined: Possible crash / security vulnerability. Fix Landed on Version: 63 Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): I think this is low risk, although I'm still not clear why we couldn't get the assertions from bug 1483487 to stick. The patch itself is pretty simple, just clearing the mExpandoAndGeneration field before using it rather than assuming it's empty. String or UUID changes made by this patch: None.

Comment on attachment 9017884 [details] [diff] [review] bug1481844-esr60 Moving over the approval to the rebased patch.