STR: 1) Start up Nightly in a new profile, and make sure browser.tabs.remote.autostart is set to false (this is the default, so this should be the case) 2) Go to File > New e10s window ER: A new browser window should open with a working remote tab. AR: A new browser window opens, but the initial tab in it immediately hits the crashed state. I attached lldb, and got the following stack: * thread #1: tid = 0x887f07, 0x000000010006b31d libmozalloc.dylib`mozalloc_abort(msg=0x00007fff5fbfd418) + 93 at mozalloc_abort.cpp:37, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0) * frame #0: 0x000000010006b31d libmozalloc.dylib`mozalloc_abort(msg=0x00007fff5fbfd418) + 93 at mozalloc_abort.cpp:37 frame #1: 0x000000010066cc25 XUL`Abort(aMsg=0x00007fff5fbfd418) + 21 at nsDebugImpl.cpp:469 frame #2: 0x000000010066c680 XUL`NS_DebugBreak(aSeverity=3, aStr=0x000000010710dc1e, aExpr=0x0000000000000000, aFile=0x000000010710d615, aLine=1634) + 1232 at nsDebugImpl.cpp:426 frame #3: 0x0000000102e7c248 XUL`mozilla::dom::ContentChild::ProcessingError(this=0x000000011122a830, what=MsgProcessingError) + 232 at ContentChild.cpp:1634 frame #4: 0x0000000100fa6452 XUL`mozilla::dom::PContentChild::OnProcessingError(this=0x000000011122a830, aCode=MsgProcessingError) + 34 at PContentChild.cpp:5607 frame #5: 0x0000000100dbb5f5 XUL`mozilla::ipc::MessageChannel::MaybeHandleError(this=0x000000011122a890, code=MsgProcessingError, aMsg=0x00007fff5fbfdbf0, channelName=0x0000000106f7c13d) + 421 at MessageChannel.cpp:1599 frame #6: 0x0000000100dbb442 XUL`mozilla::ipc::MessageChannel::DispatchAsyncMessage(this=0x000000011122a890, aMsg=0x00007fff5fbfdbf0) + 290 at MessageChannel.cpp:1233 frame #7: 0x0000000100dba6ef XUL`mozilla::ipc::MessageChannel::DispatchMessage(this=0x000000011122a890, aMsg=0x00007fff5fbfdbf0) + 207 at MessageChannel.cpp:1115 frame #8: 0x0000000100db6799 XUL`mozilla::ipc::MessageChannel::OnMaybeDequeueOne(this=0x000000011122a890) + 505 at MessageChannel.cpp:1098 frame #9: 0x0000000100dd3463 XUL`void DispatchToMethod<mozilla::ipc::MessageChannel, bool (obj=0x000000011122a890, method=a0 65 db 00 01 00 00 00, arg=0x000000011121bb70)()>(mozilla::ipc::MessageChannel*, bool (mozilla::ipc::MessageChannel::*)(), Tuple0 const&) + 131 at tuple.h:383 frame #10: 0x0000000100dd335e XUL`RunnableMethod<mozilla::ipc::MessageChannel, bool (this=0x000000011121bb40)(), Tuple0>::Run() + 78 at task.h:307 frame #11: 0x0000000100dd5ee8 XUL`mozilla::ipc::MessageChannel::RefCountedTask::Run(this=0x0000000111226190) + 40 at MessageChannel.h:411 frame #12: 0x0000000100dd5eb4 XUL`mozilla::ipc::MessageChannel::DequeueTask::Run(this=0x0000000111206d20) + 36 at MessageChannel.h:428 frame #13: 0x0000000100d529c0 XUL`MessageLoop::RunTask(this=0x00007fff5fbfe180, task=0x0000000111206d20) + 96 at message_loop.cc:358 frame #14: 0x0000000100d52f3f XUL`MessageLoop::DeferOrRunPendingTask(this=0x00007fff5fbfe180, pending_task=0x00007fff5fbfddc8) + 79 at message_loop.cc:366 frame #15: 0x0000000100d53174 XUL`MessageLoop::DoWork(this=0x00007fff5fbfe180) + 292 at message_loop.cc:444 frame #16: 0x0000000100dbe7e3 XUL`mozilla::ipc::MessagePumpForChildProcess::Run(this=0x00000001112211f0, aDelegate=0x00007fff5fbfe180) + 627 at MessagePump.cpp:297 frame #17: 0x0000000100d528a5 XUL`MessageLoop::RunInternal(this=0x00007fff5fbfe180) + 117 at message_loop.cc:230 frame #18: 0x0000000100d527b5 XUL`MessageLoop::RunHandler(this=0x00007fff5fbfe180) + 21 at message_loop.cc:223 frame #19: 0x0000000100d5275d XUL`MessageLoop::Run(this=0x00007fff5fbfe180) + 45 at message_loop.cc:197 frame #20: 0x00000001047c6a52 XUL`XRE_RunAppShell + 306 at nsEmbedFunctions.cpp:708 frame #21: 0x0000000100dbe634 XUL`mozilla::ipc::MessagePumpForChildProcess::Run(this=0x00000001112211f0, aDelegate=0x00007fff5fbfe180) + 196 at MessagePump.cpp:272 frame #22: 0x0000000100d528a5 XUL`MessageLoop::RunInternal(this=0x00007fff5fbfe180) + 117 at message_loop.cc:230 frame #23: 0x0000000100d527b5 XUL`MessageLoop::RunHandler(this=0x00007fff5fbfe180) + 21 at message_loop.cc:223 frame #24: 0x0000000100d5275d XUL`MessageLoop::Run(this=0x00007fff5fbfe180) + 45 at message_loop.cc:197 frame #25: 0x00000001047c62b6 XUL`XRE_InitChildProcess(aArgc=3, aArgv=0x00007fff5fbff730) + 2678 at nsEmbedFunctions.cpp:550 frame #26: 0x0000000100000d19 plugin-container`content_process_main(argc=6, argv=0x00007fff5fbff730) + 185 at plugin-container.cpp:158 frame #27: 0x0000000100000de2 plugin-container`main(argc=7, argv=0x00007fff5fbff730) + 34 at MozillaRuntimeMain.cpp:11

It looks like the message we're attempting to process at the time of the error is one with name "PBrowser::Msg_Show".

Digging in, it looks like we're failing out in InitTabChildGlobal - specifically, when calling InitTabChildGlobalInternal, at this point: http://hg.mozilla.org/mozilla-central/file/14665b1de5ee/content/base/src/nsFrameMessageManager.cpp#l1598

Here's what's failing: http://hg.mozilla.org/mozilla-central/file/14665b1de5ee/js/xpconnect/src/XPCWrappedNative.cpp#l191 XPCWrappedNativeProto *proto = XPCWrappedNativeProto::GetNewOrUsed(scope, nativeHelper.GetClassInfo(), &sciProto, /* callPostCreatePrototype = */ false); if (!proto) return NS_ERROR_FAILURE; // <-- returning this

So I'm really down in the guts of XPConnect here, and I really don't know what's happening, but this is what I see: We're entering XPCNativeSet::GetNewOrUsed, passing some nsIClassInfo about ContentFrameMessageManager. We reach this point of XPCNativeSet::GetNewOrUsed, which appears to be some kind of fallback... we "give up", and get the set for nsISupports: http://hg.mozilla.org/mozilla-central/file/14665b1de5ee/js/xpconnect/src/XPCWrappedNativeInfo.cpp#l519 Which causes us to re-enter XPCNativeSet::GetNewOrUsed. I think what happens then is we hit this code: ClassInfo2NativeSetMap* map = rt->GetClassInfo2NativeSetMap(); if (!map) return nullptr; And we don't find a map, so we return nullptr.

Felipe says he's starting a bisect.

This breaks background thumbnailing :/

FTR, seems to affect custom builds only. Bad enough for development but nothing that breaks official builds.

This bisecting has been strange, but I think I got it down to: $ hg bisect --bad Due to skipped revisions, the first bad revision could be any of: changeset: 207867:3ff20d4c7f39 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:50:52 2014 -0700 summary: Mac v2 signing - Bug 1047584 - Enable DMG v2 signing. r=bhearsum changeset: 207868:fbc1ceea4d6f user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:50:56 2014 -0700 summary: Mac v2 signing - Bug 1047584 - Modify the .app file structure to allow for OSX v2 signing. r=bsmedberg changeset: 207869:f877c9b66cec user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:00 2014 -0700 summary: Mac v2 signing - Bug 1048687 - Read dependentlibs.list from Contents/Resources due to the new .app bundle structure. r=bsmedberg changeset: 207870:11d48c719ec9 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:04 2014 -0700 summary: Mac v2 signing - Bug 1050944 - Get Firefox to launch and run on OSX with the new .app bundle structure, made necessary by Apple's v2 signatures. r=smichaud, r=ted, sr=bsmedberg changeset: 207871:0ce9e74b0abe user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:08 2014 -0700 summary: Mac v2 signing - Bug 1046924 - Move updates directory out of the .app bundle. r=rstrong changeset: 207872:b503560736fa user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:13 2014 -0700 summary: Mac v2 signing - Bug 1055774 - Update lookup for crashreporter.ini for the new v2 bundle structure on OSX. r=bsmedberg changeset: 207873:cbd3f8e4bf49 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:17 2014 -0700 summary: Mac v2 signing - Bug 1059504 - Avoid plugin-container from crashing due to the new v2 bundle structure on OSX. r=bsmedberg changeset: 207874:24e1e95d7af2 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:21 2014 -0700 summary: Mac v2 signing - Bug 1053838 - Fix GTests to run with the new v2 bundle structure. r=bsmedberg changeset: 207875:427a562e16d4 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:25 2014 -0700 summary: Mac v2 signing - Bug 1060652 - Make mochitest harness work with the new bundle structure for v2 signatures on OSX. r=jmaher changeset: 207876:33000c22f91f user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:29 2014 -0700 summary: Mac v2 signing - Bug 1060562 - Update xpcshell-tests for the new v2 bundle structure on OSX. r=jmaher changeset: 207877:5af85a6db219 user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:39 2014 -0700 summary: Mac v2 signing - Bug 1064952 - Update Cpp unit test harness for the new v2 bundle structure on OSX. r=jmaher changeset: 207878:09e4c6245f6d user: Stephen Pohl <spohl.mozilla.bugs@gmail.com> date: Mon Sep 29 11:51:43 2014 -0700 summary: Mac v2 signing - Bug 1064910 - Update Webapp Runtime to work with the new v2 bundle structure of Firefox on OSX. r=myk

To double check, we should see if 20c7e70e1b1aworks and c5ba012a6944 presents the bug

(In reply to :Felipe Gomes from comment #9) > To double check, we should see if 20c7e70e1b1a works and c5ba012a6944 > presents the bug mccr8 confirmed this, so the bug started happening with these csets. I'm not sure if it's worth bisecting inside them because they are all part of the same work, but if someone thinks it will be useful, I can do it. I had tried but some csets didn't compile or presented other problems due to being incomplete, so I then skipped them during bisect. > c5ba012a6944 Robert Strong — Mac v2 signing - Bug 1074044 - Force add instead of patch the removed-files file. r=bhearsum > c5bb9c221c3e Robert Strong — Mac v2 signing - Bug 1074000 - Update xpcshell-tests for new OSX v2 bundle structure to run on comm-central. r=jmaher > be53b555dda6 Robert Strong — Mac v2 signing - Bug 1072722 - With older clients the new maintenance service checks the updated directory's updater.exe when verifying the updater.exe for replace requests. r=spohl > 28ae112eba8d Robert Strong — Mac v2 signing - Bug 1072716 - Update B2G removed-files.in for mac v2 signing. r=mwu > 4f0a10f05200 Robert Strong — Mac v2 signing - Bug 1072554 - Remove update directories when running xpcshell tests. r=spohl > 4932dbbc1cc8 Robert Strong — Mac v2 signing - Bug 1071465 - old precomplete file is not removed on update. r=spohl > 818b73513ce2 Robert Strong — Mac v2 signing - Bug 1071134 - Fix command line help for updater. r=spohl > 2cc605ebaad8 Robert Strong — Mac v2 signing - Bug 1070863 - The preprocessed channel-prefs.js file needs to be the same for each build. r=ted > 5d6da7c4a822 Robert Strong — Mac v2 signing - Bug 1070149 - Add explicit file removals for channel-prefs.js, update-settings.ini, and precomplete files. r=spohl > 972e4d9da1b5 Robert Strong — Mac v2 signing - Temporary workaround for Bug 1070148 - Copy maintenance service binary into its install directory when the installed binary is different. r=bbondy > 1599a302abfb Robert Strong — Mac v2 signing - Bug 1070661 - Error setting access/modification time on application bundle. r=spohl > 67261da9cb7d Robert Strong — Mac v2 signing - Bug 1068439 - Move the distribution directory from Content/MacOS to Contents/Resources on app update due to v2 signing requirements. r=spohl > b1d8f769952a Robert Strong — Mac v2 signing - Bug 1058182 - Fix app update xpcshell tests due to the Mac v2 bundle structure. r=bbondy > 71611abb9569 Robert Strong — Mac v2 signing - Bug 1064523 - Create staging directory outside of the Mac bundle. r=bbondy > 432ef1ab72b0 Robert Strong — Mac v2 signing - Bug 1059567 - Packaging changes for the move of removed-files file from Contents/MacOS to Contents/Resources. r=bbondy, r=nthomas > d3025e55dfc3 Robert Strong — Mac v2 signing - Bug 1059467 - Move precomplete file from the root of the Mac bundle to Contents/Resources. r=bbondy, r=nthomas > c6e905b1e209 Steven Michaud — Mac v2 signing - Bug 1047738 - Make distribution code look for the distribution directory under Contents/Resources due to v2 signing requirements. r=bsmedberg > 47c2f60b1174 Stephen Pohl — Mac v2 signing - Bug 1066123 - Ensure b2g desktop OSX builds still work after the v2 signature changes to Firefox.app bundles. r=ted > 09e4c6245f6d Stephen Pohl — Mac v2 signing - Bug 1064910 - Update Webapp Runtime to work with the new v2 bundle structure of Firefox on OSX. r=myk > 5af85a6db219 Stephen Pohl — Mac v2 signing - Bug 1064952 - Update Cpp unit test harness for the new v2 bundle structure on OSX. r=jmaher > 33000c22f91f Stephen Pohl — Mac v2 signing - Bug 1060562 - Update xpcshell-tests for the new v2 bundle structure on OSX. r=jmaher > 427a562e16d4 Stephen Pohl — Mac v2 signing - Bug 1060652 - Make mochitest harness work with the new bundle structure for v2 signatures on OSX. r=jmaher > 24e1e95d7af2 Stephen Pohl — Mac v2 signing - Bug 1053838 - Fix GTests to run with the new v2 bundle structure. r=bsmedberg > cbd3f8e4bf49 Stephen Pohl — Mac v2 signing - Bug 1059504 - Avoid plugin-container from crashing due to the new v2 bundle structure on OSX. r=bsmedberg > b503560736fa Stephen Pohl — Mac v2 signing - Bug 1055774 - Update lookup for crashreporter.ini for the new v2 bundle structure on OSX. r=bsmedberg > 0ce9e74b0abe Stephen Pohl — Mac v2 signing - Bug 1046924 - Move updates directory out of the .app bundle. r=rstrong > 11d48c719ec9 Stephen Pohl — Mac v2 signing - Bug 1050944 - Get Firefox to launch and run on OSX with the new .app bundle structure, made necessary by Apple's v2 signatures. r=smichaud, r=ted, sr=bsmedberg > f877c9b66cec Stephen Pohl — Mac v2 signing - Bug 1048687 - Read dependentlibs.list from Contents/Resources due to the new .app bundle structure. r=bsmedberg > fbc1ceea4d6f Stephen Pohl — Mac v2 signing - Bug 1047584 - Modify the .app file structure to allow for OSX v2 signing. r=bsmedberg > 3ff20d4c7f39 Stephen Pohl — Mac v2 signing - Bug 1047584 - Enable DMG v2 signing. r=bhearsum

Looking into it.

I'd appreciate it if someone could confirm what I'm seeing, namely that locally packaged builds don't exhibit this issue (created via 'make package' in the obj-dir, then extracted from the .dmg). It looks to me like only builds that were built via ./mach build run into this issue. Just want to make sure that I'm not missing anything.

I won't have a chance to confirm this still today, but I believe it's highly likely that that's right, because that would explain well why we run into this bug when building locally, but not on the regular Nightly builds

Okay, once the chrome.manifest file and the components directory is copied from Contents/Resources to Contents/MacOS we don't crash anymore. The packaged build doesn't crash because the chrome.manifest and the components directory appear to be compiled into the omni.ja file and we seem to be reading this in successfully from Contents/Resources. I'll figure out where we read in the chrome.manifest and components directory tomorrow. It should now look in Contents/Resources instead of Contents/MacOS.

Found the issue. The GreD for the plugin-container is set incorrectly. Working on a fix.

Forgot to add curly brackets after the |if (NS_FAILED(rv))|.

r=me in either case

With this patch applied, I no longer see the crash. Thanks spohl!!

Thanks, bsmedberg and mconley! Landed on inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/cabd17a5837a

Also landed on oak: https://hg.mozilla.org/projects/oak/rev/f1f1bdaccadf

Landed on aurora in the Mac V2 signing combined patch in bug 1047584