1399866 - Intermittent GECKO(1136) | Assertion failure: js::CheckGrayMarkingState(mJSRuntime), at z:/build/build/src/xpcom/base/CycleCollectedJSRuntime.cpp:1207

Reporter

Description

•

7 years ago

treeherder

Filed by: archaeopteryx [at] coole-files.de https://treeherder.mozilla.org/logviewer.html#?job_id=131009814&repo=autoland https://queue.taskcluster.net/v1/task/Ql1gEelxQD2EP70evR_W-A/runs/0/artifacts/public/logs/live_backing.log ... 11:46:15 INFO - GECKO(1136) | from Object 000001DD62354880 fun_environment edge 11:46:15 INFO - GECKO(1136) | from ObjectGroup 000001DD63E19DF0 group_proto edge 11:46:15 INFO - GECKO(1136) | from Object 000001DD63483640 group edge 11:46:15 INFO - GECKO(1136) | from Object 000001DD685F2520 object slot edge 11:46:15 INFO - GECKO(1136) | from Object 000001DD6888D8C0 object slot edge 11:46:15 INFO - GECKO(1136) | from ObjectGroup 000001DD5E351070 group_proto edge 11:46:15 INFO - GECKO(1136) | from Object 000001DD69394CA0 group edge 11:46:15 INFO - GECKO(1136) | from root Preserved wrapper 11:46:15 INFO - GECKO(1136) | Assertion failure: js::CheckGrayMarkingState(mJSRuntime), at z:/build/build/src/xpcom/base/CycleCollectedJSRuntime.cpp:1207 11:46:29 INFO - GECKO(1136) | #01: nsCycleCollector::BeginCollection(ccType,nsICycleCollectorListener *) [xpcom/base/nsCycleCollector.cpp:3909] 11:46:29 INFO - GECKO(1136) | #02: nsCycleCollector::Collect(ccType,js::SliceBudget &,nsICycleCollectorListener *,bool) [xpcom/base/nsCycleCollector.cpp:3761] 11:46:29 INFO - GECKO(1136) | #03: nsCycleCollector_collectSlice(js::SliceBudget &,bool) [xpcom/base/nsCycleCollector.cpp:4315] 11:46:29 INFO - GECKO(1136) | #04: nsJSContext::RunCycleCollectorSlice(mozilla::TimeStamp) [dom/base/nsJSEnvironment.cpp:1542] 11:46:29 INFO - GECKO(1136) | #05: CCRunnerFired [dom/base/nsJSEnvironment.cpp:1936] 11:46:29 INFO - GECKO(1136) | #06: std::_Func_impl<bool (*)(mozilla::TimeStamp),std::allocator<int>,bool,mozilla::TimeStamp>::_Do_call(mozilla::TimeStamp &&) [vs2015u3/VC/include/functional:214] 11:46:29 INFO - GECKO(1136) | #07: mozilla::IdleTaskRunner::Run() [xpcom/threads/IdleTaskRunner.cpp:62] 11:46:29 INFO - GECKO(1136) | #08: mozilla::TimedOut [xpcom/threads/IdleTaskRunner.cpp:85] 11:46:29 INFO - GECKO(1136) | #09: nsTimerImpl::Fire(int) [xpcom/threads/nsTimerImpl.cpp:515] 11:46:29 INFO - GECKO(1136) | #10: nsTimerEvent::Run() [xpcom/threads/TimerThread.cpp:289] 11:46:29 INFO - GECKO(1136) | #11: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:1040] 11:46:29 INFO - GECKO(1136) | #12: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/threads/nsThreadUtils.cpp:521] 11:46:29 INFO - GECKO(1136) | #13: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:97] 11:46:29 INFO - GECKO(1136) | #14: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:320] 11:46:29 INFO - GECKO(1136) | #15: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:300] 11:46:29 INFO - GECKO(1136) | #16: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:160] 11:46:29 INFO - GECKO(1136) | #17: nsAppShell::Run() [widget/windows/nsAppShell.cpp:232] 11:46:29 INFO - GECKO(1136) | #18: nsAppStartup::Run() [toolkit/components/startup/nsAppStartup.cpp:288] 11:46:29 INFO - GECKO(1136) | #19: XREMain::XRE_mainRun() [toolkit/xre/nsAppRunner.cpp:4697] 11:46:29 INFO - GECKO(1136) | #20: XREMain::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [toolkit/xre/nsAppRunner.cpp:4861] 11:46:29 INFO - GECKO(1136) | #21: XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [toolkit/xre/nsAppRunner.cpp:4956] 11:46:29 INFO - GECKO(1136) | #22: do_main [browser/app/nsBrowserApp.cpp:237] 11:46:29 INFO - GECKO(1136) | #23: NS_internal_main(int,char * *,char * *) [browser/app/nsBrowserApp.cpp:311] 11:46:29 INFO - GECKO(1136) | #24: wmain [toolkit/xre/nsWindowsWMain.cpp:118] 11:46:29 INFO - GECKO(1136) | #25: __scrt_common_main_seh [f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:253] 11:46:29 INFO - GECKO(1136) | #26: KERNEL32.DLL + 0x12774 11:46:29 INFO - GECKO(1136) | #27: ntdll.dll + 0x70d61 11:46:29 INFO - GECKO(1136) | Hit MOZ_CRASH(Aborting on channel error.) at z:/build/build/src/ipc/glue/MessageChannel.cpp:2544 11:46:29 INFO - GECKO(1136) | ### ERROR: CreateThread: Access is denied.

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 2

•

7 years ago

Attached patch bug1399866-proxy-target-assertions (deleted) — Details — Splinter Review

This is a gray marking issue and the logs indicate that it's for a ProxyObject's private slot. Here's a patch to add assertions when setting this slot. Not a fix.

Assignee: nobody → jcoppeard

Attachment #8909886 - Flags: review?(sphink)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 3

•

7 years ago

Attached patch bug1399866-gray-marking-info (deleted) — Details — Splinter Review

Here's a patch to output object class names when a gray marking problem is detected, which should make this easier to track down. Not a fix.

Attachment #8909889 - Flags: review?(sphink)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Updated

•

7 years ago

Keywords: leave-open

Steve Fink [:sfink] [:s:]

Updated

•

7 years ago

Attachment #8909886 - Flags: review?(sphink) → review+

Steve Fink [:sfink] [:s:]

Updated

•

7 years ago

Attachment #8909889 - Flags: review?(sphink) → review+

Pulsebot

Comment 4

•

7 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/fbdfaa4bae2d Add gray marking assertions when setting proxy target r=sfink https://hg.mozilla.org/integration/mozilla-inbound/rev/a079a0a2971d Dump more info on gray marking failure r=sfink

Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)

Comment 5

•

7 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/fbdfaa4bae2d https://hg.mozilla.org/mozilla-central/rev/a079a0a2971d

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Updated

•

7 years ago

Depends on: 1401804

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 6

•

7 years ago

Recent crashes are like: [task 2017-09-25T18:26:11.827Z] 18:26:11 INFO - GECKO(1096) | Found black to gray edge to Object XULElement 0xe53dffa0 [task 2017-09-25T18:26:11.831Z] 18:26:11 INFO - GECKO(1096) | from Object Proxy 0xd1869c70 proxy target edge [task 2017-09-25T18:26:11.835Z] 18:26:11 INFO - GECKO(1096) | from Object Object 0xe3133a60 object slot edge [task 2017-09-25T18:26:11.843Z] 18:26:11 INFO - GECKO(1096) | from Object Proxy 0xe53ded30 proxy target edge [task 2017-09-25T18:26:11.845Z] 18:26:11 INFO - GECKO(1096) | from Object XULElement 0xe53dffa0 object slot edge [task 2017-09-25T18:26:11.847Z] 18:26:11 INFO - GECKO(1096) | from root Preserved wrapper

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 12

•

7 years ago

Attached patch bug1399866-more-gray-marking-info (deleted) — Details — Splinter Review

I still can't figure out what's going on here. Here's a patch to dump even more information when we detect things have gone wrong in debug builds.

Attachment #8924506 - Flags: review?(pbone)

Paul Bone [:pbone]

Comment 13

•

7 years ago

Comment on attachment 8924506 [details] [diff] [review] bug1399866-more-gray-marking-info Review of attachment 8924506 [details] [diff] [review]: ----------------------------------------------------------------- This looks good to me. r+

Attachment #8924506 - Flags: review?(pbone) → review+

Pulsebot

Comment 14

•

7 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/203ea4c2406f Dump even more info on gray marking failure r=pbone

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 15

•

7 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/203ea4c2406f

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 26

•

7 years ago

Attached patch bug1399866-proxy-target-assert (deleted) — Details — Splinter Review

We're seeing a black proxy object with a gray target. Add an assert that we never create a proxy with a gray target. Currently we only assert we never create a black to gray edge, but a newly created proxy will not be black outside of incremental GC.

Attachment #8937160 - Flags: review?(sphink)

Steve Fink [:sfink] [:s:]

Updated

•

7 years ago

Attachment #8937160 - Flags: review?(sphink) → review+

Comment hidden (Intermittent Failures Robot)

Pulsebot

Comment 28

•

7 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/dd94a26765f3 Assert proxy private is not gray on creation r=sfink

Raul Gurzau (:RaulG)

Comment 29

•

7 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/dd94a26765f3

Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]

Updated

•

7 years ago

See Also: → https://bugzilla.mozilla.org/show_bug.cgi?id=1425902

Comment hidden (Intermittent Failures Robot)

Tiberius Oros[:tiberius_oros]

Comment 42

•

7 years ago

There have been a total of 32 failures in the last week, according to Orange Factor. All the failures occur on debug. As per platform, below you can see the no. of occurrences for each platform: -linux64-stylo-disabled: 10 -windows10-64-ccov: 9 -Linux x64: 8 -Windows 7: 2 -linux32-stylo-disabled: 1 -OS X 10.10: 1 -windows10-64: 1 Here is a recent log file and a snippet with the failure: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=159260208&lineNumber=4081 [task 2018-01-30T05:10:11.082Z] 05:10:11 INFO - GECKO(2040) | "engines": null (shape 7f25e57cb318 enumerate slot 34) [task 2018-01-30T05:10:11.196Z] 05:10:11 INFO - GECKO(2040) | MEMORY STAT | vsize 1669MB | residentFast 189MB | heapAllocated 34MB [task 2018-01-30T05:10:11.563Z] 05:10:11 INFO - GECKO(2040) | Assertion failure: js::CheckGrayMarkingState(mJSRuntime), at /builds/worker/workspace/build/src/xpcom/base/CycleCollectedJSRuntime.cpp:1231 It seems that all dependencies have been resolved. :jonco , would you please take a look at this?

Flags: needinfo?(jcoppeard)

Whiteboard: [stockwell needswork]

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 43

•

7 years ago

(In reply to Tiberius Oros[:tiberius_oros] from comment #42) I'm currently working on resolving a talos regression (bug 1433128). I'll take a look at this next.

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 46

•

7 years ago

Attached patch bug1399866-gray-asserts (deleted) — Details — Splinter Review

I found somewhere we don't have gray marking asserts: the public/Proxy.h header proves some functions that are used by gecko to write proxy slots. This patch adds assertions to ensure we don't create black to gray edges.

Flags: needinfo?(jcoppeard)

Attachment #8951343 - Flags: review?(sphink)

Steve Fink [:sfink] [:s:]

Updated

•

7 years ago

Attachment #8951343 - Flags: review?(sphink) → review+

Pulsebot

Comment 47

•

7 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/42eb9bb1033e Add gray marking asserts to proxy write functions r=sfink

Raul Gurzau (:RaulG)

Comment 48

•

7 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/42eb9bb1033e

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 50

•

7 years ago

I filed bug 1440739 to improve our gray marking asserts further.

Comment hidden (Intermittent Failures Robot)

Jason Orendorff [:jorendorff]

Comment 60

•

6 years ago

This is a good candidate for rr.

Flags: needinfo?(overholt)

Andrew Overholt [:overholt]

Updated

•

6 years ago

Flags: needinfo?(overholt)

Comment hidden (Intermittent Failures Robot)

Cristina Coroiu [:ccoroiu]

Comment 69

•

6 years ago

Recent failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=225652005&repo=autoland&lineNumber=20772

06:53:59 INFO - GECKO(1076) | Assertion failure: js::CheckGrayMarkingState(mJSRuntime), at /builds/worker/workspace/build/src/xpcom/base/CycleCollectedJSRuntime.cpp:1119
21680 06:55:49 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::CycleCollectedJSRuntime::CheckGrayBits() const]
24917 06:55:50 INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab missing output line for total leaks!
33236 07:04:13 ERROR - Return code: 1
33237 07:04:13 ERROR - Got 1 unexpected crashes
33282 [taskcluster:error] exit status 1

Comment hidden (Intermittent Failures Robot)

Razvan Maries

Comment 76

•

5 years ago

There are 20 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-09-29&endday=2019-10-06&tree=trunk&bug=1399866

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=269846152&repo=mozilla-central&lineNumber=15196

Comment hidden (Intermittent Failures Robot)

Alexandru Michis [:malexandru]

Comment 79

•

5 years ago

Jon, can you redirect this to someone who can look into it?

Flags: needinfo?(jcoppeard)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 80

•

5 years ago

The recent failures all look like this:

Found black to gray edge to gray JS Object Window 0x2b51ece5f060 (compartment 0x7f6b3097c8e0)
  from black JS ObjectGroup 0x298433092040 group_global edge
  from gray JS Object Proxy 0x2984330d8040 (compartment 0x7f6b3097c8e0) group edge
  from gray JS Object LexicalEnvironment 0x2b51ece61040 (compartment 0x7f6b3097c8e0) object slot edge
  from gray JS Object Function 0x2b51ece67470 (compartment 0x7f6b3097c8e0) fun_environment edge
  ...

So it's complaining that we have a black object group whose global is gray, which should not be possible.

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 81

•

5 years ago

Attached file Bug 1399866 - Add assertions that we don't create black to gray edges from ObjectGroups to global objects r=sfink (deleted) — Details

Not a fix. This patch adds some assertions that we don't ever set the realm of an ObjectGroup such that it points to a gray global object. We have these asserts already for other types of edge but this one is indirect because we store a pointer to the Realm, and get the global from that.

Pulsebot

Comment 82

•

5 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/67b51087d673 Add assertions that we don't create black to gray edges from ObjectGroups to global objects r=sfink

Narcis Beleuzu [:NarcisB]

Comment 83

•

5 years ago

Backed out for assertion failures on GC.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/e3dc5cfd4d368d281ff8938f5a232f0bf92fe6b6
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271361424&repo=autoland&lineNumber=21640

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 84

•

5 years ago

(In reply to Narcis Beleuzu [:NarcisB] from comment #83)
Great news, this may have found the problem.

Flags: needinfo?(jcoppeard)

Phabricator Automation

Updated

•

5 years ago

Attachment #9101300 - Attachment description: Bug 1399866 - Add assertions that we don't create black to gray edges from ObjectGroups to global objects r?sfink → Bug 1399866 - Add assertions that we don't create black to gray edges from ObjectGroups to global objects r=sfink

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 85

•

5 years ago

Attached file Bug 1399866 - Trigger global's read barrier in js::NewProxyObject r?sfink (deleted) — Details

When remapping wrappers the embedding can call this to create a new proxy object while in a realm whose global is gray. This breaks the JS API invariants and can cause black to gray GC edges to be creaated. Fix this by triggering the global's read barrier here.

Pulsebot

Comment 86

•

5 years ago

Pushed by jcoppeard@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/cec21bd20658 Trigger global's read barrier in js::NewProxyObject r=sfink https://hg.mozilla.org/integration/autoland/rev/0f2ab1f1dc33 Add assertions that we don't create black to gray edges from ObjectGroups to global objects r=sfink

Stefan Hindli [:stefan_hindli]

Comment 87

•

5 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/cec21bd20658
https://hg.mozilla.org/mozilla-central/rev/0f2ab1f1dc33

Comment hidden (Intermittent Failures Robot)

Jon Coppeard (:jonco) (PTO until 14th September)

Assignee

Comment 89

•

5 years ago

Closing as there are no failures since the last patches landed. If this starts happening again please file a new bug.

Status: NEW → RESOLVED

Closed: 5 years ago

Keywords: leave-open

Resolution: --- → FIXED

Ryan VanderMeulen [:RyanVM]

Updated

•

5 years ago

status-firefox67: --- → unaffected

status-firefox68: --- → unaffected

status-firefox69: --- → unaffected

status-firefox70: --- → unaffected

status-firefox71: --- → fixed

status-firefox-esr60: --- → unaffected

status-firefox-esr68: --- → unaffected

Target Milestone: --- → mozilla71

Andrew McCreight [:mccr8]

Updated

•

1 year ago

See Also: → https://bugzilla.mozilla.org/show_bug.cgi?id=1842584

bug1399866-proxy-target-assertions 7 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), patch	sfink : review+	Details \| Diff \| Splinter Review
bug1399866-gray-marking-info 7 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), patch	sfink : review+	Details \| Diff \| Splinter Review
bug1399866-more-gray-marking-info 7 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), patch	pbone : review+	Details \| Diff \| Splinter Review
bug1399866-proxy-target-assert 7 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), patch	sfink : review+	Details \| Diff \| Splinter Review
bug1399866-gray-asserts 7 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), patch	sfink : review+	Details \| Diff \| Splinter Review
Bug 1399866 - Add assertions that we don't create black to gray edges from ObjectGroups to global objects r=sfink 5 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), text/x-phabricator-request		Details
Bug 1399866 - Trigger global's read barrier in js::NewProxyObject r?sfink 5 years ago Jon Coppeard (:jonco) (PTO until 14th September) (deleted), text/x-phabricator-request		Details