This is particularly bad on the webrender builds (test-linux1804-64-qr/debug-mochitest-media-e10s-1, test-linux1804-64-qr/debug-mochitest-media-fis-e10s-1, test-linux1804-64-qr/debug-mochitest-media-swr-e10s-1, and test-linux1804-64-qr/debug-mochitest-media-fis-gli-e10s-1). This also shows up a little on linux x64 debug without webrender. Linux x64 webrender debug asan is in horrible shape due to leaks, but it is unclear whether that's the same problem.

Current state of affairs on m-c:

https://treeherder.mozilla.org/jobs?repo=try&revision=8e30e0a9c14f2ced362f0f5a7510a82a842717cf

Seems to be fine at this revision: https://treeherder.mozilla.org/jobs?repo=try&revision=df92ad4c79470a6afdd1af02ab206cf1cb911d24

And seems to be broken at this revision: https://treeherder.mozilla.org/jobs?repo=try&revision=6327505ba363c430743654b27f6fe8c474c94f37

That indicates that this started when bug 1693677 landed.

Any ideas here? Is this 50 second timer causing some references to live longer than they should? Or maybe it is just making things live long enough to expose some pre-existing shutdown bug?

(In reply to Byron Campen [:bwc] from comment #4)

Seems to be fine at this revision: https://treeherder.mozilla.org/jobs?repo=try&revision=df92ad4c79470a6afdd1af02ab206cf1cb911d24

I don't think so. I'm seeing 2 LEAKING THE WORLD failures there.

"Microphone:"... Paul, any thoughts?

[task 2021-06-07T18:55:32.316Z] 18:55:32 INFO - GECKO(2797) | [Parent 2797, Main Thread] ###!!! ASSERTION: 2 dynamic atom(s) with non-zero refcount: tree-input,Microphone: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #7)

Created attachment 9225750 [details]
snapshot1.png

(In reply to Byron Campen [:bwc] from comment #4)

Seems to be fine at this revision: https://treeherder.mozilla.org/jobs?repo=try&revision=df92ad4c79470a6afdd1af02ab206cf1cb911d24

I don't think so. I'm seeing 2 LEAKING THE WORLD failures there.

So those 2 came after 20 more retriggers. 40 vs 40 yields 6 failures vs 2. So, maybe it was not introduced, but it seems to have gotten worse?

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #8)

"Microphone:"... Paul, any thoughts?

[task 2021-06-07T18:55:32.316Z] 18:55:32 INFO - GECKO(2797) | [Parent 2797, Main Thread] ###!!! ASSERTION: 2 dynamic atom(s) with non-zero refcount: tree-input,Microphone: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443

I also see these in the logging:

Camera,ac-separator-text,scrolldir: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443
EW: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443
Screen,ac-separator,place,arrowpanel-color: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443

Also this:

device-share,popup-notification-dropmarker,chrome://global/skin/splitter.css,chrome://global/skin/close-icon.css,error-icon: 'nonZeroRefcountAtomsCount == 0', file /builds/worker/checkouts/gecko/xpcom/ds/nsAtomTable.cpp:443

I see a ton of stuff from here showing up in that logging:

https://searchfox.org/mozilla-central/rev/da5d08750e504f3710f7ea051327d9c311c39902/browser/base/content/webrtcIndicator.xhtml

When did we first promote linux-qr to tier 1? I did a try push from an m-c from last October, and it seems completely broken:

https://treeherder.mozilla.org/jobs?repo=try&revision=1e85ca2d8f01bb6b41ba33aa154ae35340f38b04

I'm not sure, this line implies the tier comes from here.

I guess you can dig around in the blame of those two places to try and piece it together.

Ok, even a non-webrender push does the same thing that happened in comment 12:

https://treeherder.mozilla.org/jobs?repo=try&revision=86ac0aac3c7ecd2468cb009a42a2acf03d7271d8

Looks like this is due to an expired cert in the tree from that long ago.

I'm not having any luck finding the specific test that is triggering this. It may require running the entire suite.

I spoke to bwc about this over Matrix. One of the things here that doesn't pass the sniff test is that we never got around to shipping the new indicator on Linux. It's an incomplete implementation. I actually think we might want to turn it off on Linux in Nightly until we can complete the implementation (which is blocked on bug 1668358).

We currently have it on by default on Linux in Nightly only. Maybe we should stop doing that. https://searchfox.org/mozilla-central/rev/b52cf6bbe214bd9d93ed9333d0403f7d556ad7c8/browser/app/profile/firefox.js#1869-1880

That might end up kicking the leak-can down the road a bit, but might also be the most expedient way forward?

Seems to be much rarer on spi runs, but still a non-zero rate.

I'm seeing a few anomalies that I will have to investigate further:

1. xpcom-shutdown notifications coming before xpcom-will-shutdown notifications. This seems like a bug to me.
2. xpcom-shutdown notifications coming before STS shutdown observers are notified. This also is a bug, and may share a cause with #1.
3. Shutdown occurring despite there being media::ShutdownTickets active. This is probably also a bug.

So, this seems like a necko-related leak (note the list of leaked objects here), and the fact that it happens on spi jobs and still contains lots of content-related objects, and the fact that it does not appear that anything webrtc-related is leaking, it does not appear that this is strictly a webrtc problem. At least not related to our network code, anyway.

So, tracing this further back, nsLayoutStatics is being leaked, which is resulting in the necko leaks. This is (in part) due to leaks of things like nsGlobalWindowInner/nsGlobalWindowOuter.

Any ideas on debugging this? I cannot repro locally, it only seems to happen on try.

One thing to keep in mind is that the "leaking the world" warning is just that, a warning. It is the "TEST-UNEXPECTED-FAIL | leakcheck | default 286868 bytes leaked " stuff that is the real error. Also, the leaked atoms are just a symptom that shows up when you are leaking a bunch of stuff. The "default" means that this is in the main process. The ASan errors are probably the same leak, showing up in a way that is harder to understand.

Looking at a few logs, it appears that it is leaking one nsGlobalWindowInner. Once you are leaking an inner window, you are going to leak all sorts of stuff. So the first goal is to figure out why the window is alive.

There are directions on how to get CC logs on try here, starting in the section "To get cycle collector logs on Try server". In addition to setting MOZ_CC_LOG_DIRECTORY and MOZ_CC_LOG_SHUTDOWN as it describes, you'll also want to set MOZ_CC_LOG_THREAD=main and MOZ_CC_LOG_PROCESS=main . The logs should show up under the "Artifacts" tab. You'll have to retrigger the job until you get a failure, as you'd expect.

The log name will contain the pid, so you need the pid of the leaking process. That'll be at the start of the table before all of the errors, and it'll look like "BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 3158". Then you'll want the last CC log for that process where there's a nsGlobalWindowInner in it, then you run find_roots.py from my heapgraph repo on that log with nsGlobalWindowInner and it'll tell you why any remaining windows are alive. I can help walk you through that part of it once you get the logs on a failing run.

If I am reading those right, this is webrtcIndicator.xhtml. I should note that I tried testing with the legacy indicator, but we still leaked the world. I will get cc logs for that case next, but I expect that they will tell a similar story.

Why would webrtcIndicator.xhtml be leaked?

Hm. Either it's not getting closed, or even after it's closed, something is keeping the window alive.

This is the section of code that manages the lifetime of the webrtcIndicator.xhtml window: https://searchfox.org/mozilla-central/rev/294f10eff7398d6b05beac6aa256d86ac3cb7113/browser/modules/webrtcUI.jsm#821-847

Do we know if the webrtcIndicator.xhtml window ever runs its onUnload method here?: https://searchfox.org/mozilla-central/rev/294f10eff7398d6b05beac6aa256d86ac3cb7113/browser/base/content/webrtcIndicator.js#446-456

Perhaps we could add some logging there to find out. If not, then the window is not being closed. If so, then the window is being closed, but something is still holding onto it (presumably, that gIndicatorWindow variable somehow).

The window (which is for webrtcIndicator.xhtml like you said) is being held alive by a style sheet. The output means that the style sheet has a refcount of 3, but there are only two references known to the cycle collector, a document and a CharacterData object. I'm not sure what the other reference might be. If that doesn't spur any ideas, the next step would be to use DMD heap scan mode in combination with the CC logs. I can write up some steps on how to do that. I don't remember exactly how to get DMD going on try, so it'll take a little bit of research.

To use DMD heap scan mode, you'll have add some additional environment variables to the CC ones that you did before. The basic idea is that you correlate the CC log and the DMD log for a particular leaking run, and that lets you find missing references to the object holding things alive.

• MOZ_DMD_SHUTDOWN_LOG has to be set to something like os.environ["MOZ_UPLOAD_DIR"] + "/"
  (unfortunately the DMD path is a bit wonky and it needs to end in /, and I think MOZ_UPLOAD_DIR does not end in /)
• MOZ_DMD_LOG_PROCESS should be set to "default", so that it only logs the main process. (Unfortunately the main process is specified in a different way than the CC logs.)
• DMD should be set to "--mode=scan" to enable DMD in heap scan mode.

The resulting DMD log will be unsymbolized. Symbolication can be done with dmd.py, but I'm not sure exactly how it will work with a build done on try, given that it involves paths. Probably it won't work. I can try taking a look at the unsymbolized stacks and hope that there's enough information to be useful.

(In reply to Byron Campen [:bwc] from comment #32)

If I am reading those right, this is webrtcIndicator.xhtml. I should note that I tried testing with the legacy indicator, but we still leaked the world. I will get cc logs for that case next, but I expect that they will tell a similar story.

Why would webrtcIndicator.xhtml be leaked?

Yeah, legacy indicator behaves similarly:

0x7f1d02564500 [StyleSheet]
--[mRelevantGlobal]--> 0x7f1d0a3f3800 [nsGlobalWindowInner # 24 inner chrome://browser/content/webrtcLegacyIndicator.xhtml]

Root 0x7f1d02564500 is a ref counted object with 1 unknown edge(s).
known edges:
   0x7f1d02c3e000 [Document normal (xhtml) chrome://browser/content/webrtcLegacyIndicator.xhtml] --[mStyleSheets[i]]--> 0x7f1d02564500
   0x7f1d15603450 [CharacterData (len=72)] --[mStyleSheet]--> 0x7f1d02564500

(In reply to Andrew McCreight [:mccr8] from comment #35)

• DMD should be set to "--mode=scan" to enable DMD in heap scan mode.

How does one do this on a try push?

"DMD" is an environment variable, so you set it in the same way as MOZ_CC_LOG_DIRECTORY etc.

Looks like there's a successful capture of a DMD log with a leak in that try run. I'll see if I can figure anything out.

The unsymbolicated output is unfortunately not useful, and I'm not sure if there's some way I can use the crash reporter symbols from the build here on OSX to symbolicate them.

I think I hacked up a way to do symbolication locally on OSX by downloading symbols. I'll look at it more tomorrow.

As seen above, the window is being held alive by a style sheet.

Digging through the DMD log, there is a chain of ownership going from the webrtcIndicator.xhtml Document to the StyleSheet as follows:

0x7fede4f31000 nsHTMLDocument mPreloadService -->
0x7fede10bcbe0 PreloadService nsRefPtrHashtable<PreloadHashKey, PreloaderBase> mPreloads -->
0x7fede6af4940 SheetLoadData RefPtr<StyleSheet> mSheet -->
0x7fede0ecd710 StyleSheet
(SheetLoadData is a subclass of PreloaderBase.)

PreloadService and SheetLoadData are not cycle collected, so these edges are all invisible to the cycle collector.

The document is cycle collected, so we can go back to the CC logs to figure out why it is alive:

0x7fedf3694348 [PrototypeDocumentContentSink]
--[mDocument]--> 0x7fede4f31000 [Document normal (xhtml) chrome://browser/content/webrtcIndicator.xhtml]

Root 0x7fedf3694348 is a ref counted object with 1 unknown edge(s).
known edges:
0x7fedea8e5f20 [Localization] --[mContentSink]--> 0x7fedf3694348
0x7fede64ac1a0 [PrototypeDocumentParser] --[mOriginalSink]--> 0x7fedf3694348

Then I went back to DMD again to find the missing reference to the content sink. That looks like it is being held alive by 0x7fede6af4940, which is the SheetLoadData from above. Maybe the reference is nsCOMPtr<nsICSSLoaderObserver> mObserver.

So, to summarize, the cycle here causing the leak is:
SheetLoadData --> PrototypeDocumentContentSink --> Document --> PreloadService --> SheetLoadData

It looks like there's some code in Document::Destroy() which is supposed to clear the PreloadService, but I guess that's not happening here? Olli, you reviewed bug 1618289, which added the mPreloadService field to Document. Do you have any ideas about what might be going wrong? Should we cycle collect PreloadService and SheetLoadData?

It is odd that ::Destroy isn't called. I think that is the main issue here.

about comment 21.
"xpcom-shutdown notifications coming before xpcom-will-shutdown notifications. This seems like a bug to me."
My guess is that someone is violating xpcom-will-shutdown contract and does some complicated stuff there.

mconley's question in comment 33 wasn't answered, does the unload ever run?

(In reply to Olli Pettay [:smaug] from comment #44)

It is odd that ::Destroy isn't called. I think that is the main issue here.

about comment 21.
"xpcom-shutdown notifications coming before xpcom-will-shutdown notifications. This seems like a bug to me."
My guess is that someone is violating xpcom-will-shutdown contract and does some complicated stuff there.

mconley's question in comment 33 wasn't answered, does the unload ever run?

It appears that onUnload is being called, and we are nulling out every gIndicatorWindow we've created before reaching the "LEAKING THE WORLD" logging. Is there anything else that I should check?

(In reply to Byron Campen [:bwc] from comment #45)

(In reply to Olli Pettay [:smaug] from comment #44)

It is odd that ::Destroy isn't called. I think that is the main issue here.

about comment 21.
"xpcom-shutdown notifications coming before xpcom-will-shutdown notifications. This seems like a bug to me."
My guess is that someone is violating xpcom-will-shutdown contract and does some complicated stuff there.

mconley's question in comment 33 wasn't answered, does the unload ever run?

It appears that onUnload is being called, and we are nulling out every gIndicatorWindow we've created before reaching the "LEAKING THE WORLD" logging. Is there anything else that I should check?

I took a count of the number of WebRTCIndicators we init, and we have the same number of onUnload calls (usually around 165 for the entire suite). Interestingly, it appears that one or two of the WebRTCIndicators never get an onLoad call. Not sure if that is an indication of a problem.

So, it looks like Document::Destroy is running, but then something is inserting a preload (calling PreloadService::RegisterPreload) afterward:

https://treeherder.mozilla.org/logviewer?job_id=343785039&repo=try&lineNumber=19302

https://hg.mozilla.org/try/rev/ba11355a3a0e098064496e3697ef671c1a79b3bb#l1.17

It seems plausible that this could leave a cycle. Thoughts?

FWIW, adding a check like this:

https://hg.mozilla.org/try/rev/49023d6fdb7b774fe5617dbc1ab40202f2dd527f#l1.17

Leads to clean try runs:

https://treeherder.mozilla.org/jobs?repo=try&revision=02c484b47360d2cff0731d89b4eeefea5f767d8e

Probably not the "real" fix, of course.

So, based on my naive reading of the stack in comment 47, this seems to be where stuff starts to take a left turn from teardown code to code that wasn't intended to be run in a Document that has been destroyed:

https://searchfox.org/mozilla-central/rev/fc95c6ad297d9d257f05599d01741503f3f57326/parser/prototype/PrototypeDocumentParser.cpp#144

Any ideas on the proper fix here?

FWIW, other things in Document::Destroy() do something similar to what your patch in comment 48 does.

While investigating bug 1703346, I found that the place where that bug was triggered (which resulted in a ysod which later broke our screensharing test) was dom/media/webrtc/tests/mochitests/test_1488832.html:

https://treeherder.mozilla.org/logviewer?job_id=343885964&repo=try&lineNumber=8803

This is the same test case that tripped over the assertion I added in comment 47:

https://treeherder.mozilla.org/logviewer?job_id=343785039&repo=try&lineNumber=19302

It seems that these two bugs have a common cause, related to gUM shutdown races.

Any ideas on where we might fix cases where the listeners for OnStopRequest are called after the document in question has started tearing down? I think this is likely the cause of bug 1703346 (and bug 1639821) also.

(In reply to Byron Campen [:bwc] from comment #52)

Any ideas on where we might fix cases where the listeners for OnStopRequest are called after the document in question has started tearing down? I think this is likely the cause of bug 1703346 (and bug 1639821) also.

IIRC necko guarantees that OnStartRequest and OnStopRequest are always called for every channel, so it seems expected to me that we'd be calling OnStopRequest during teardown when the channel is cancelled due to the docshell/document going away.

(In reply to Nika Layzell [:nika] (ni? for response) from comment #53)

(In reply to Byron Campen [:bwc] from comment #52)

Any ideas on where we might fix cases where the listeners for OnStopRequest are called after the document in question has started tearing down? I think this is likely the cause of bug 1703346 (and bug 1639821) also.

IIRC necko guarantees that OnStartRequest and OnStopRequest are always called for every channel, so it seems expected to me that we'd be calling OnStopRequest during teardown when the channel is cancelled due to the docshell/document going away.

So, adding a check to Document::BeginLoad like so seems to solve the leaks, without introducing new problems.

https://hg.mozilla.org/try/rev/d5784fc5c6c298e2e0386f57b94b9bf6894eb8cf

Does this seem like a reasonable approach? Or is there some deeper change we ought to be making (eg; unhooking PrototypeDocumentContentSink from the OnStopRequest callback when this teardown code is run)? Should we be adding !mIsGoingAway checks to other places in Document?

I wonder what happens if we MOZ_ASSERT(!mIsGoingAway) in other places...

We now have six mochitests that have been disabled due to this bug, which is essentially caused by the same thing as bug 1703346. Both this bug and bug 1703346 are stalled until necko and dom can come to an agreement on how to proceed there.

I am going to try to re-enable these tests, and disable test_1488832.html, since that's what is triggering this mess (see comment 51).

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f218be1fb37b10c2b710c5fb27b23bdc181895ef

Re-disabled a test (test_setSinkId_default_addTrack.html) that had broken due to a different bug (bug 1731076) sometime after it was disabled back in April for leaking.

https://treeherder.mozilla.org/jobs?repo=try&revision=0e293ba9deaef95c019969ef27a011d270e3aeba

Seeing some TabCapturer::OnFrame crashes, not sure if that's new for linux, but here's a baseline push:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f8e936d6521537888e6c422c402fc536abc1884

Re-enables the other tests that were innocent victims. In one case, one of
the 'innocent victims' has totally broken since it was disabled.

https://hg.mozilla.org/mozilla-central/rev/3cf151a4b9a8

Clearing the needinfo. Most of this code was ported from very old code, so I'm not sure of a proper fix.

Changing to S3 - this issue is observed in webrtc tests, not user impacts.

This leak probably does affect users, though I don't know how severe it is. It is also causing a big mess of intermittent failures in WebRTC tests, which makes understanding new failures more difficult, so I think we should fix this at some point.

(In reply to Andrew McCreight [:mccr8] from comment #66)

This leak probably does affect users, though I don't know how severe it is. It is also causing a big mess of intermittent failures in WebRTC tests, which makes understanding new failures more difficult, so I think we should fix this at some point.

My previous reckon and read from bug 1703346 was that it seems there are questions about the test writing. However, I just tried to review other see-also bugs like bug 1639821.

Also bug 1749861#c5 mentioned We weren't sure if that is just the test that Byron was fixing, or does it happen in the wild for our users and this bug is to confirm that it happens in the wild to users :) But as we see frequent user reports recently, bumping the severity/priority for now for better tracking; and I plan to take this to WebRTC team to see how we should plan forward.

"regression" keyword was added by the release mgmt bot when we added "regressed by: 1693677". However, the keyword wasn't cleared automatically when we removed the "regressed by: 1693677" later. So, I'm simply removing the keyword to reflect the situation. If I misunderstood anything, feel free to correct this, thank you.

I caught this leak on try on linux debug nonopt with shutdown CC and DMD logs: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=AnVABknDSfq3pBueZ8ddAA.0&revision=cedf6c7ba65c55c3501cb71294b73ee80a5b6013

I'll look for clues.

There is one window alive at the last CC, with webrtcIndicator.xhtml loaded. The outer and the inner window have the same root. This is find_roots.py output for the outer window since it has a bit more info (mOuterWindow):

0x7fb2a5062710 [StyleSheet]
    --[mRelevantGlobal]--> 0x7fb29cd2c400 [nsGlobalWindowInner # 32 inner chrome://browser/content/webrtcIndicator.xhtml]
    --[mOuterWindow]--> 0x7fb29cd5a560 [nsGlobalWindowOuter # 31 outer ]

    Root 0x7fb2a5062710 is a ref counted object with 1 unknown edge(s).
    known edges:
        0x7fb2ac0c4400 [Document normal (xhtml) chrome://browser/content/webrtcIndicator.xhtml]  --[mStyleSheets[i]]--> 0x7fb2a5062710
        0x7fb2a5062500 [CharacterData (len=65)]  --[mStyleSheet]--> 0x7fb2a5062710

1 unknown edge, hopefully we can find it. No luck on getting stacks in DMD yet but will update again with more info if I'm successful.

block_analyzer.py gives the following for 0x7fb2a5062710:

0x7fb2a04781c0 size = 32 bytes at byte offset 16                                                                                                                                                             nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long) [xpcom/ds/nsTArray-inl.h:223]                        nsTArray_Impl<RefPtr<mozilla::StyleSheet>, nsTArrayInfallibleAllocator>::InsertElementAtInternal<nsTArrayInfallibleAllocator, mozilla::StyleSheet*>(unsigned long, mozilla::StyleSheet*&&) [xpcom/ds/nsTArray.h:2613]                                                                                                                                                                                               
  mozilla::dom::Document::InsertSheetAt(unsigned long, mozilla::StyleSheet&) [dom/base/Document.cpp:7266]                             
  mozilla::css::Loader::InsertSheetInTree(mozilla::StyleSheet&) [layout/style/Loader.cpp:0]                                                                                                                
  mozilla::SharedStyleSheetCache::LoadCompletedInternal(mozilla::SharedStyleSheetCache*, mozilla::css::SheetLoadData&, nsTArray<RefPtr<mozilla::css::SheetLoadData> >&) [layout/style/SharedStyleSheetCache
.cpp:0]                                                                                                                                                                                                      mozilla::SharedStyleSheetCache::LoadCompleted(mozilla::SharedStyleSheetCache*, mozilla::css::SheetLoadData&, nsresult) [layout/style/SharedStyleSheetCache.cpp:66]
  mozilla::MozPromise<bool, bool, true>::ThenValue<mozilla::css::Loader::ParseSheet(nsTSubstring<char> const&, mozilla::css::SheetLoadData&, mozilla::css::Loader::AllowAsyncParse)::$_7, mozilla::css::Loa
der::ParseSheet(nsTSubstring<char> const&, mozilla::css::SheetLoadData&, mozilla::css::Loader::AllowAsyncParse)::$_8>::DoResolveOrRejectInternal(mozilla::MozPromise<bool, bool, true>::ResolveOrRejectValu
e&) [xpcom/threads/MozPromise.h:861]                                                                                                                                                                       
  mozilla::MozPromise<bool, bool, true>::ThenValueBase::ResolveOrRejectRunnable::Run() [xpcom/threads/MozPromise.h:490]
  mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:540]                                                                                                                                      
  mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:852]
  mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:0]
  mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:462]                                                                                                               
  mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_2>::Run() [xpcom/threads/nsThreadUtils.h:547]
  nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1229]                                                                                                                                
  NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:477]                                                                                                                              
  mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:86]                                                                                                                 MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:0]                                                                                                                                     
  MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:357]     
  nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:150]                                                                                                                                                    
  nsAppStartup::Run() [toolkit/components/startup/nsAppStartup.cpp:296]

0x7fb29c3cf230 size = 264 bytes at byte offset 120 
  mozilla::StyleSheet::StyleSheet(mozilla::css::SheetParsingMode, mozilla::CORSMode, mozilla::dom::SRIMetadata const&) [layout/style/StyleSheet.cpp:43]  
  mozilla::css::Loader::CreateSheet(nsIURI*, nsIContent*, nsIPrincipal*, mozilla::css::SheetParsingMode, mozilla::CORSMode, mozilla::Encoding const*, nsTSubstring<char16_t> const&, bool, mozilla::css::St
ylePreloadKind) [layout/style/Loader.cpp:1031]                                                                                                                                                             
  mozilla::css::Loader::CreateSheet(mozilla::dom::LinkStyle::SheetInfo const&, mozilla::css::SheetParsingMode, bool, mozilla::css::StylePreloadKind) [layout/style/Loader.h:512]
  mozilla::css::Loader::LoadStyleLink(mozilla::dom::LinkStyle::SheetInfo const&, nsICSSLoaderObserver*) [layout/style/Loader.cpp:1889]
  mozilla::dom::LinkStyle::DoUpdateStyleSheet(mozilla::dom::Document*, mozilla::dom::ShadowRoot*, nsICSSLoaderObserver*, mozilla::dom::LinkStyle::ForceUpdate) [dom/base/LinkStyle.cpp:316]
  mozilla::dom::LinkStyle::UpdateStyleSheet(nsICSSLoaderObserver*) [dom/base/LinkStyle.cpp:190]
  mozilla::dom::PrototypeDocumentContentSink::InsertXMLStylesheetPI(nsXULPrototypePI const*, nsINode*, nsINode*, mozilla::dom::XMLStylesheetProcessingInstruction*) [dom/prototype/PrototypeDocumentContent
Sink.cpp:403]                                                                                                                                                                                              
  mozilla::dom::PrototypeDocumentContentSink::CreateAndInsertPI(nsXULPrototypePI const*, nsINode*, nsINode*) [dom/prototype/PrototypeDocumentContentSink.cpp:370]
  mozilla::dom::PrototypeDocumentContentSink::PrepareToWalk() [dom/prototype/PrototypeDocumentContentSink.cpp:319]
  mozilla::dom::PrototypeDocumentContentSink::OnPrototypeLoadDone(nsXULPrototypeDocument*) [dom/prototype/PrototypeDocumentContentSink.cpp:275]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]                                                                                                              mozilla::net::ParentChannelListener::OnStopRequest(nsIRequest*, nsresult) [netwerk/protocol/http/ParentChannelListener.cpp:96]                                 
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  nsJARChannel::OnStopRequest(nsIRequest*, nsresult) [modules/libjar/nsJARChannel.cpp:1270]                                                                                                                
  {virtual override thunk({offset(-24)}, nsJARChannel::OnStopRequest(nsIRequest*, nsresult))} [modules/libjar/nsJARChannel.cpp:0]                           
  nsInputStreamPump::OnStateStop() [netwerk/base/nsInputStreamPump.cpp:696]                 
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) [netwerk/base/nsInputStreamPump.cpp:424]                            
  {virtual override thunk({offset(-24)}, nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*))} [netwerk/base/nsInputStreamPump.cpp:0]
  NS_NewCancelableRunnableFunction<CallbackHolder::CallbackHolder(nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}>(char const*, CallbackHolder::CallbackHolder(
nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}&&)::FuncCancelableRunnable::Run() [xpcom/threads/nsThreadUtils.h:668]
  mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:540]

0x7fb29d09fbd0 size = 272 bytes at byte offset 192 
  mozilla::MakeRefPtr<mozilla::css::SheetLoadData, mozilla::css::Loader*, nsTString<char16_t> const&, nsCOMPtr<nsIURI> const&, RefPtr<mozilla::StyleSheet>&, bool const&, nsIContent* const&, mozilla::dom:
:LinkStyle::IsAlternate&, mozilla::dom::LinkStyle::MediaMatched&, mozilla::css::StylePreloadKind, nsICSSLoaderObserver*&, nsIPrincipal*&, nsCOMPtr<nsIReferrerInfo> const&>(mozilla::css::Loader*, nsTStrin
g<char16_t> const&, nsCOMPtr<nsIURI> const&, RefPtr<mozilla::StyleSheet>&, bool const&, nsIContent* const&, mozilla::dom::LinkStyle::IsAlternate&, mozilla::dom::LinkStyle::MediaMatched&, mozilla::css::St
ylePreloadKind, nsICSSLoaderObserver*&, nsIPrincipal*&, nsCOMPtr<nsIReferrerInfo> const&) [mfbt/RefPtr.h:603]
  mozilla::css::Loader::LoadStyleLink(mozilla::dom::LinkStyle::SheetInfo const&, nsICSSLoaderObserver*) [layout/style/Loader.cpp:1905]
  mozilla::dom::LinkStyle::DoUpdateStyleSheet(mozilla::dom::Document*, mozilla::dom::ShadowRoot*, nsICSSLoaderObserver*, mozilla::dom::LinkStyle::ForceUpdate) [dom/base/LinkStyle.cpp:316]
  mozilla::dom::LinkStyle::UpdateStyleSheet(nsICSSLoaderObserver*) [dom/base/LinkStyle.cpp:190]
  mozilla::dom::PrototypeDocumentContentSink::InsertXMLStylesheetPI(nsXULPrototypePI const*, nsINode*, nsINode*, mozilla::dom::XMLStylesheetProcessingInstruction*) [dom/prototype/PrototypeDocumentContent
Sink.cpp:403]
  mozilla::dom::PrototypeDocumentContentSink::CreateAndInsertPI(nsXULPrototypePI const*, nsINode*, nsINode*) [dom/prototype/PrototypeDocumentContentSink.cpp:370]
  mozilla::dom::PrototypeDocumentContentSink::PrepareToWalk() [dom/prototype/PrototypeDocumentContentSink.cpp:319]
  mozilla::dom::PrototypeDocumentContentSink::OnPrototypeLoadDone(nsXULPrototypeDocument*) [dom/prototype/PrototypeDocumentContentSink.cpp:275]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  mozilla::net::ParentChannelListener::OnStopRequest(nsIRequest*, nsresult) [netwerk/protocol/http/ParentChannelListener.cpp:96]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  nsJARChannel::OnStopRequest(nsIRequest*, nsresult) [modules/libjar/nsJARChannel.cpp:1270]
  {virtual override thunk({offset(-24)}, nsJARChannel::OnStopRequest(nsIRequest*, nsresult))} [modules/libjar/nsJARChannel.cpp:0]
  nsInputStreamPump::OnStateStop() [netwerk/base/nsInputStreamPump.cpp:696]
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) [netwerk/base/nsInputStreamPump.cpp:424]
  {virtual override thunk({offset(-24)}, nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*))} [netwerk/base/nsInputStreamPump.cpp:0]
  NS_NewCancelableRunnableFunction<CallbackHolder::CallbackHolder(nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}>(char const*, CallbackHolder::CallbackHolder(
nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}&&)::FuncCancelableRunnable::Run() [xpcom/threads/nsThreadUtils.h:668]
  mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:540]
  mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:852]
  mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:0]

0x7fb2a5062500 size = 176 bytes at byte offset 136
  NS_NewXMLProcessingInstruction(nsNodeInfoManager*, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&) [dom/xml/ProcessingInstruction.cpp:29]
  mozilla::dom::PrototypeDocumentContentSink::CreateAndInsertPI(nsXULPrototypePI const*, nsINode*, nsINode*) [dom/prototype/PrototypeDocumentContentSink.cpp:362]
  mozilla::dom::PrototypeDocumentContentSink::PrepareToWalk() [dom/prototype/PrototypeDocumentContentSink.cpp:319]
  mozilla::dom::PrototypeDocumentContentSink::OnPrototypeLoadDone(nsXULPrototypeDocument*) [dom/prototype/PrototypeDocumentContentSink.cpp:275]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  mozilla::net::ParentChannelListener::OnStopRequest(nsIRequest*, nsresult) [netwerk/protocol/http/ParentChannelListener.cpp:96]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  nsJARChannel::OnStopRequest(nsIRequest*, nsresult) [modules/libjar/nsJARChannel.cpp:1270]
  {virtual override thunk({offset(-24)}, nsJARChannel::OnStopRequest(nsIRequest*, nsresult))} [modules/libjar/nsJARChannel.cpp:0]
  nsInputStreamPump::OnStateStop() [netwerk/base/nsInputStreamPump.cpp:696]
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) [netwerk/base/nsInputStreamPump.cpp:424]
  {virtual override thunk({offset(-24)}, nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*))} [netwerk/base/nsInputStreamPump.cpp:0]
  NS_NewCancelableRunnableFunction<CallbackHolder::CallbackHolder(nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}>(char const*, CallbackHolder::CallbackHolder(nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}&&)::FuncCancelableRunnable::Run() [xpcom/threads/nsThreadUtils.h:668]
  mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:540]
  mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:852]
  mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:0]
  mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:462]
  mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_2>::Run() [xpcom/threads/nsThreadUtils.h:547]
  nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1229]
  NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:477]

There are 4 blocks.

• 1 is a strong ref in DocumentOrShadowRoot::mStyleSheets. It is a known edge.
• 2 appears to be this weak ref. Can be disregarded.
• 3 appears to be SheetLoadData::mSheet from here. There are options on where it ended up, but we can use block_analyzer to look into it, so more on this later.
• 4 is the strong ref LinkStyle::mStyleSheet which is a base class of this XMLStylesheetProcessingInstruction which further up the hierarcy also is a CharacterData. It is the other known edge.

So SheetLoadData is leaking the world here, let's look at where it's kept alive:

0x7fb29d35f3a0 size = 352 bytes at byte offset 344                         
  PLDHashTable::MakeEntryHandle(void const*, std::nothrow_t const&) [xpcom/ds/PLDHashTable.cpp:631]  
  PLDHashTable::MakeEntryHandle(void const*) [xpcom/ds/PLDHashTable.cpp:676]
  mozilla::PreloadService::RegisterPreload(mozilla::PreloadHashKey const&, mozilla::PreloaderBase*) [uriloader/preload/PreloadService.cpp:25]
  mozilla::PreloaderBase::NotifyOpen(mozilla::PreloadHashKey const&, mozilla::dom::Document*, bool) [uriloader/preload/PreloaderBase.cpp:133]
  mozilla::css::Loader::LoadSheet(mozilla::css::SheetLoadData&, mozilla::css::Loader::SheetState, unsigned long, mozilla::css::Loader::PendingLoad) [layout/style/Loader.cpp:0]
  mozilla::css::Loader::LoadStyleLink(mozilla::dom::LinkStyle::SheetInfo const&, nsICSSLoaderObserver*) [layout/style/Loader.cpp:1939]
  mozilla::dom::LinkStyle::DoUpdateStyleSheet(mozilla::dom::Document*, mozilla::dom::ShadowRoot*, nsICSSLoaderObserver*, mozilla::dom::LinkStyle::ForceUpdate) [dom/base/LinkStyle.cpp:316]
  mozilla::dom::LinkStyle::UpdateStyleSheet(nsICSSLoaderObserver*) [dom/base/LinkStyle.cpp:190]
  mozilla::dom::PrototypeDocumentContentSink::InsertXMLStylesheetPI(nsXULPrototypePI const*, nsINode*, nsINode*, mozilla::dom::XMLStylesheetProcessingInstruction*) [dom/prototype/PrototypeDocumentContent
Sink.cpp:403]
  mozilla::dom::PrototypeDocumentContentSink::CreateAndInsertPI(nsXULPrototypePI const*, nsINode*, nsINode*) [dom/prototype/PrototypeDocumentContentSink.cpp:370]
  mozilla::dom::PrototypeDocumentContentSink::PrepareToWalk() [dom/prototype/PrototypeDocumentContentSink.cpp:319]
  mozilla::dom::PrototypeDocumentContentSink::OnPrototypeLoadDone(nsXULPrototypeDocument*) [dom/prototype/PrototypeDocumentContentSink.cpp:275]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  mozilla::net::ParentChannelListener::OnStopRequest(nsIRequest*, nsresult) [netwerk/protocol/http/ParentChannelListener.cpp:96]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]
  nsJARChannel::OnStopRequest(nsIRequest*, nsresult) [modules/libjar/nsJARChannel.cpp:1270]
  {virtual override thunk({offset(-24)}, nsJARChannel::OnStopRequest(nsIRequest*, nsresult))} [modules/libjar/nsJARChannel.cpp:0]
  nsInputStreamPump::OnStateStop() [netwerk/base/nsInputStreamPump.cpp:696]
  nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) [netwerk/base/nsInputStreamPump.cpp:424]
  {virtual override thunk({offset(-24)}, nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*))} [netwerk/base/nsInputStreamPump.cpp:0]
  NS_NewCancelableRunnableFunction<CallbackHolder::CallbackHolder(nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}>(char const*, CallbackHolder::CallbackHolder(
nsIAsyncInputStream*, nsIInputStreamCallback*, unsigned int, nsIEventTarget*)::{lambda()#1}&&)::FuncCancelableRunnable::Run() [xpcom/threads/nsThreadUtils.h:668]

0x7fb29d9a1ca0 size = 32 bytes at byte offset 8                                                                                                                                                            
  mozilla::SupportsWeakPtr::SelfReferencingWeakReference() const [mfbt/WeakPtr.h:227]                                                                                                                      
  mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::detail::WeakPtrDestructorBehavior)0>::operator=(mozilla::css::SheetLoadData const*) [mfbt/WeakPtr.h:0]
  nsTHashtable<nsBaseHashtableET<mozilla::SheetLoadDataHashKey, mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::detail::WeakPtrDestructorBehavior)0> > >::EntryHandle::InsertInternal<mozilla::css:
:SheetLoadData*>(mozilla::css::SheetLoadData*&&) [xpcom/ds/nsTHashtable.h:0]
  nsBaseHashtable<mozilla::SheetLoadDataHashKey, mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::detail::WeakPtrDestructorBehavior)0>, mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::deta
il::WeakPtrDestructorBehavior)0>, nsDefaultConverter<mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::detail::WeakPtrDestructorBehavior)0>, mozilla::WeakPtr<mozilla::css::SheetLoadData, (mozilla::
detail::WeakPtrDestructorBehavior)0> > >::EntryHandle::InsertOrUpdate<mozilla::css::SheetLoadData*>(mozilla::css::SheetLoadData*&&) [xpcom/ds/nsBaseHashtable.h:784]
  _ZN12PLDHashTable15WithEntryHandleIZN12nsTHashtableI17nsBaseHashtableETIN7mozilla20SheetLoadDataHashKeyENS3_7WeakPtrINS3_3css13SheetLoadDataELNS3_6detail25WeakPtrDestructorBehaviorE0EEEEE15WithEntryHan
dleIZN15nsBaseHashtableIS4_SA_SA_18nsDefaultConverterISA_SA_EE15WithEntryHandleIZNSH_14InsertOrUpdateIPS7_EERSA_RKS4_OT_EUlSO_E_EENSt13invoke_resultISO_JONSH_11EntryHandleEEE4typeESN_SP_EUlSO_E_EENSR_ISO
_JONSC_11EntryHandleEEE4typeESN_SP_EUlSO_E_EENSR_ISO_JONS_11EntryHandleEEE4typeEPKvSP_ [xpcom/ds/PLDHashTable.h:603]
  mozilla::SharedSubResourceCache<mozilla::SharedStyleSheetCacheTraits, mozilla::SharedStyleSheetCache>::LoadStarted(mozilla::SheetLoadDataHashKey const&, mozilla::css::SheetLoadData&) [layout/style/Shar
edSubResourceCache.h:480]                                                                                                                                                                                  
  mozilla::css::Loader::LoadSheet(mozilla::css::SheetLoadData&, mozilla::css::Loader::SheetState, unsigned long, mozilla::css::Loader::PendingLoad) [layout/style/Loader.cpp:1555]
  mozilla::css::Loader::LoadStyleLink(mozilla::dom::LinkStyle::SheetInfo const&, nsICSSLoaderObserver*) [layout/style/Loader.cpp:1939]
  mozilla::dom::LinkStyle::DoUpdateStyleSheet(mozilla::dom::Document*, mozilla::dom::ShadowRoot*, nsICSSLoaderObserver*, mozilla::dom::LinkStyle::ForceUpdate) [dom/base/LinkStyle.cpp:316]
  mozilla::dom::LinkStyle::UpdateStyleSheet(nsICSSLoaderObserver*) [dom/base/LinkStyle.cpp:190]                                                                                                            
  mozilla::dom::PrototypeDocumentContentSink::InsertXMLStylesheetPI(nsXULPrototypePI const*, nsINode*, nsINode*, mozilla::dom::XMLStylesheetProcessingInstruction*) [dom/prototype/PrototypeDocumentContent
Sink.cpp:403]
  mozilla::dom::PrototypeDocumentContentSink::CreateAndInsertPI(nsXULPrototypePI const*, nsINode*, nsINode*) [dom/prototype/PrototypeDocumentContentSink.cpp:370]
  mozilla::dom::PrototypeDocumentContentSink::PrepareToWalk() [dom/prototype/PrototypeDocumentContentSink.cpp:319]
  mozilla::dom::PrototypeDocumentContentSink::OnPrototypeLoadDone(nsXULPrototypeDocument*) [dom/prototype/PrototypeDocumentContentSink.cpp:275]                 
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]                                                                                                            
  mozilla::net::ParentChannelListener::OnStopRequest(nsIRequest*, nsresult) [netwerk/protocol/http/ParentChannelListener.cpp:96]
  nsDocumentOpenInfo::OnStopRequest(nsIRequest*, nsresult) [uriloader/base/nsURILoader.cpp:217]                                                                                                            
  nsJARChannel::OnStopRequest(nsIRequest*, nsresult) [modules/libjar/nsJARChannel.cpp:1270]                                                                                                                
  {virtual override thunk({offset(-24)}, nsJARChannel::OnStopRequest(nsIRequest*, nsresult))} [modules/libjar/nsJARChannel.cpp:0]                                   
  nsInputStreamPump::OnStateStop() [netwerk/base/nsInputStreamPump.cpp:696]

There are two blocks for this SheetLoadData, where we can see without looking closer that the second is a WeakPtr. The first is PreloadService. The same ref cycle identified in comment 43.

pre-patch: https://treeherder.mozilla.org/jobs?repo=try&revision=16e2cf06a4ab8a1a80de69551a255edb5ef9d956
post-patch: https://treeherder.mozilla.org/jobs?repo=try&revision=c44acb2dc83ef17c61d913ad8f1b141ada8a4cbb

related to comment 47, it sounds like we need to figure out who is trying to preload after Destroy.