This bug was filed from the Socorro interface and is report bp-7f7be854-b5bf-43be-8d87-f94b02160611. ============================================================= Crashing Thread (0) Frame Module Signature Source 0 mozglue.dll mozalloc_abort(char const* const) memory/mozalloc/mozalloc_abort.cpp:33 1 mozglue.dll mozalloc_handle_oom(unsigned int) memory/mozalloc/mozalloc_oom.cpp:46 2 mozglue.dll moz_xmalloc memory/mozalloc/mozalloc.cpp:85 3 xul.dll std::_Allocate(unsigned int, unsigned int, bool) vs2015u2/VC/include/xmemory0:69 4 xul.dll std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) vs2015u2/VC/include/sstream:151 5 msvcp140.dll std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, __int64) f:\dd\vctools\crt\crtw32\stdhpp\streambuf:411 6 msvcp140.dll std::basic_streambuf<char, std::char_traits<char> >::sputn(char const*, __int64) f:\dd\vctools\crt\crtw32\stdhpp\streambuf:208 7 msvcp140.dll std::basic_ostream<unsigned short, std::char_traits<unsigned short> >::write(unsigned short const*, __int64) f:\dd\vctools\crt\crtw32\stdhpp\ostream:564 8 xul.dll mozilla::gfx::RecordedSourceSurfaceCreation::RecordToStream(std::basic_ostream<char, std::char_traits<char> >&) gfx/2d/RecordedEvent.cpp:1274 9 xul.dll mozilla::gfx::DrawEventRecorderPrivate::RecordEvent(mozilla::gfx::RecordedEvent const&) gfx/2d/DrawEventRecorder.cpp:33 10 xul.dll mozilla::gfx::StoreSourceSurface gfx/2d/DrawTargetRecording.cpp:52 11 xul.dll mozilla::gfx::EnsureSurfaceStored gfx/2d/DrawTargetRecording.cpp:67 12 xul.dll mozilla::gfx::DrawTargetRecording::EnsurePatternDependenciesStored(mozilla::gfx::Pattern const&) gfx/2d/DrawTargetRecording.cpp:724 13 xul.dll mozilla::gfx::DrawTargetRecording::FillRect(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::Pattern const&, mozilla::gfx::DrawOptions const&) gfx/2d/DrawTargetRecording.cpp:321 14 xul.dll gfxWindowsNativeDrawing::PaintToContext() gfx/thebes/gfxWindowsNativeDrawing.cpp:286 15 xul.dll nsNativeThemeWin::DrawWidgetBackground(nsRenderingContext*, nsIFrame*, unsigned char, nsRect const&, nsRect const&) widget/windows/nsNativeThemeWin.cpp:1993 16 xul.dll nsDisplayThemedBackground::PaintInternal(nsDisplayListBuilder*, nsRenderingContext*, nsRect const&, nsRect*) layout/base/nsDisplayList.cpp:3178 17 xul.dll nsDisplayThemedBackground::Paint(nsDisplayListBuilder*, nsRenderingContext*) layout/base/nsDisplayList.cpp:3162 18 xul.dll mozilla::FrameLayerBuilder::PaintItems(nsTArray<mozilla::FrameLayerBuilder::ClippedDisplayItem>&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, gfxContext*, nsRenderingContext*, nsDisplayListBuilder*, nsPresContext*, mozilla::gfx::IntPointTyped<mozilla::gfx::UnknownUnits> const&, float, float, int) layout/base/FrameLayerBuilder.cpp:5715 19 xul.dll mozilla::FrameLayerBuilder::DrawPaintedLayer(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*) layout/base/FrameLayerBuilder.cpp:5890 20 xul.dll mozilla::layers::BasicPaintedLayer::PaintThebes(gfxContext*, mozilla::layers::Layer*, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*) gfx/layers/basic/BasicPaintedLayer.cpp:91 21 xul.dll mozilla::layers::BasicLayerManager::PaintSelfOrChildren(mozilla::layers::PaintLayerContext&, gfxContext*) gfx/layers/basic/BasicLayerManager.cpp:730 22 xul.dll nsDisplayWrapList::GetBounds(nsDisplayListBuilder*, bool*) layout/base/nsDisplayList.cpp:4060 23 xul.dll mozilla::layers::BasicLayerManager::PaintLayer(gfxContext*, mozilla::layers::Layer*, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*) gfx/layers/basic/BasicLayerManager.cpp:910 24 xul.dll mozilla::layers::BasicLayerManager::PaintSelfOrChildren(mozilla::layers::PaintLayerContext&, gfxContext*) gfx/layers/basic/BasicLayerManager.cpp:751 25 xul.dll nsDisplayWrapList::GetBounds(nsDisplayListBuilder*, bool*) layout/base/nsDisplayList.cpp:4060 26 xul.dll mozilla::layers::BasicLayerManager::PaintLayer(gfxContext*, mozilla::layers::Layer*, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*) gfx/layers/basic/BasicLayerManager.cpp:910 27 xul.dll mozilla::layers::BasicLayerManager::PaintSelfOrChildren(mozilla::layers::PaintLayerContext&, gfxContext*) gfx/layers/basic/BasicLayerManager.cpp:751 28 xul.dll nsDisplayWrapList::GetBounds(nsDisplayListBuilder*, bool*) layout/base/nsDisplayList.cpp:4060 29 xul.dll mozilla::layers::BasicLayerManager::PaintLayer(gfxContext*, mozilla::layers::Layer*, void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*) gfx/layers/basic/BasicLayerManager.cpp:910 30 xul.dll mozilla::layers::BasicLayerManager::EndTransactionInternal(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) gfx/layers/basic/BasicLayerManager.cpp:639 31 xul.dll nsDisplayList::PaintRoot(nsDisplayListBuilder*, nsRenderingContext*, unsigned int) layout/base/nsDisplayList.cpp:1889 32 xul.dll nsLayoutUtils::PaintFrame(nsRenderingContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) layout/base/nsLayoutUtils.cpp:3585 33 xul.dll nsSimplePageSequenceFrame::PrintNextPage() layout/generic/nsSimplePageSequenceFrame.cpp:772 34 xul.dll nsPrintEngine::PrintPage(nsPrintObject*, bool&) layout/printing/nsPrintEngine.cpp:2733 35 xul.dll nsPagePrintTimer::Run() layout/printing/nsPagePrintTimer.cpp:89 36 xul.dll nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:1073 37 xul.dll mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:98 38 xul.dll mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:315 39 xul.dll MessageLoop::RunHandler() ipc/chromium/src/base/message_loop.cc:228 40 xul.dll MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:208 41 xul.dll nsBaseAppShell::Run() widget/nsBaseAppShell.cpp:156 42 xul.dll nsAppShell::Run() widget/windows/nsAppShell.cpp:262 43 xul.dll XRE_RunAppShell toolkit/xre/nsEmbedFunctions.cpp:809 44 xul.dll mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:283 45 xul.dll MessageLoop::RunHandler() ipc/chromium/src/base/message_loop.cc:228 46 xul.dll MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:208 47 xul.dll XRE_InitChildProcess toolkit/xre/nsEmbedFunctions.cpp:644 48 plugin-container.exe content_process_main(int, char** const) ipc/contentproc/plugin-container.cpp:231 49 plugin-container.exe wmain toolkit/xre/nsWindowsWMain.cpp:123 50 plugin-container.exe __scrt_common_main_seh f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:255 51 kernel32.dll BaseThreadInitThunk 52 ntdll.dll __RtlUserThreadStart 53 ntdll.dll _RtlUserThreadStart

3 crashes in this signature in the last week, all on Windows 7.

Crash on Windows 10. ID: b5417d4e-1221-4df4-b407-845e32160723 Print Preview -> Print -> Crash

this signature is regressing in volume in firefox 50 builds (though there have been single occurrences in 48.0a1 & 49.0a1 as well). it's happening on all versions of windows and is accounting for about 0.1% of all crashes on 50.0b.

Mike, do you have any quick off-the-cuff guesses as to what's up here?

[Tracking Requested - why for this release]: #8 content crash for 50 at the moment

Tracked for 50 in case we have a good and safe fix ready in time for 50.1.0

Someone told me this is probably because we do print preview (maybe printing, too?) differently with e10s, shuffling data to the parent process.

Yeah, the crash appears to be happening within the code that takes the draw instructions and attempts to serialize them up to the parent. Any idea what's happening here, bobowen?

(In reply to Mike Conley (:mconley) from comment #8) > Yeah, the crash appears to be happening within the code that takes the draw > instructions and attempts to serialize them up to the parent. > > Any idea what's happening here, bobowen? Yes as people are saying, looks like it's running out of memory while it's trying to record the print draw instructions. All the ones I just looked at were recording SourceSurfaces, which I guess are some of the bigger things that get stored. I suppose all we could do is to try and use something other that a StringBuffer that is fallible and just fail the print somehow. Is it likely that these people are going to OOM for other reasons though? The recording then gets copied to shared memory shipped to the parent and then copied to a string buffer again, so we could improve the memory management here. (It's done page by page, so that helps.) Although I'm not sure how easy it would be to have an istream in the parent backed by shared memory. In the child using a shared memory backed ostream would be trickier still I guess, as we don't know up front how big the page recording will be. Anyway, we could still run out of memory.

Hmm, looks like it's trying to allocate really large amounts.

I think part of the problem here is it looks like it is doing some sort of exponential allocation each time it has to extend.

Tracking 53+ for this crash since it seems to be in the top 10.

32-bit builds of Firefox are very prone to memory fragmentation, and due to how our memory allocation strategy works, any infallible allocation of more than 2Mb of contiguous memory is guaranteed to eventually crash the browser. (Because the jemalloc we're using divides the pool into chunks and treats anything >2M as special, and won't necessarily try to free/coalesce >2M chunks, IIIRC). You can talk to the MemShrink people for further details, by you're going to have to find some way to split up your memory requests. (Making this fallible, detecting that and then failing to print doesn't sound user friendly)

The simplest fix here seems to be falling back to files to record the data. All of the different sandboxes now have a writeable temp directory we can use. I've got a patch. We might want to look into reducing recording size and creating something that does a better job of allocating/using memory, so that we can go back to shared memory in the future. However that will probably be quite tricky to get right and not be as suitable for uplift.

MozReview-Commit-ID: IXssquDuZYv

(In reply to Gian-Carlo Pascutto [:gcp] from comment #13) > 32-bit builds of Firefox are very prone to memory fragmentation, and due to > how our memory allocation strategy works, any infallible allocation of more > than 2Mb of contiguous memory is guaranteed to eventually crash the browser. > (Because the jemalloc we're using divides the pool into chunks and treats > anything >2M as special, and won't necessarily try to free/coalesce >2M > chunks, IIIRC). > > You can talk to the MemShrink people for further details, by you're going to > have to find some way to split up your memory requests. (Making this > fallible, detecting that and then failing to print doesn't sound user > friendly) But graphics is always going to need to be able to allocate memory chunks bigger than 2MB... a simple 512x512 image is already 2MB when decoded... So as a general concept.. that doesn't work, even if we do have a fix here.

Comment on attachment 8812724 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Review of attachment 8812724 [details] [diff] [review]: ----------------------------------------------------------------- Sounds good to me, I doubt we can realistically guarantee this never goes over 2MB (just think of an SVG with complex paths in them).

(In reply to Bas Schouten (:bas.schouten) from comment #16) > But graphics is always going to need to be able to allocate memory chunks > bigger than 2MB... a simple 512x512 image is already 2MB when decoded... So > as a general concept.. that doesn't work, even if we do have a fix here. You can split things up in chunks, or make them fallible. If you need infallible allocations of contiguous buffers of more than 2MB then you're eventually going to have a bad time, though, until we get to 64 bits. This isn't really new, AFAIU it's been this way since we switched to jemalloc, and that's even from before my time.

In order to use NS_APP_CONTENT_PROCESS_TEMP_DIR for the temporary print recording I needed to add in the following in xpcom/io/nsAppDirectoryServiceDefs.h, when not already defined: #define NS_APP_CONTENT_PROCESS_TEMP_DIR "TmpD" I couldn't specify NS_OS_TEMP_DIR without also including nsDirectoryServiceDefs.h in that header file. froydnj - just wanted to make sure you were OK with this. Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3692b8c3451ef515cf1700dce695bb855f576b95 MozReview-Commit-ID: KRT0TDYXdKU

Comment on attachment 8812844 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Review of attachment 8812844 [details] [diff] [review]: ----------------------------------------------------------------- )Sorry, I forgot to hit "submit" on this yesterday.) I guess this is OK...at least the nsAppDirectoryServiceDefs.h bit. How concerned are we about serializing things to the filesystem, and then assuming those things have not been modified? I guess we think this is not that much different than somebody futzing with the IPC channel?

Pushed by bobowencode@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/a4b0052954d2 Use temporary files instead of shared memory to store the page recordings when printing via parent. r=bas, r=froydnj

Please request Aurora/Beta approval on this when you think this has baked for long enough on Nightly.

Track 51- as the volume of crash is low but still happy to take the fix in Beta51.

Comment on attachment 8812844 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Not sure if this crash is considered to be serious enough for 50.1 Approval Request Comment [Feature/regressing bug #]: Bug 1156742. [User impact if declined]: The OOM content process crashes will continue to occur on certain web pages/machines. [Describe test coverage new/current, TreeHerder]: There is no current automated testing for printing, so it would be good to get some manual coverage for this change. [Risks and why]: Medium - the change is not that complicated, but saying medium because of the lack of automated testing. [String/UUID change made/needed]: None.

Comment on attachment 8812844 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Fix a crash related to printing. Beta51+ and Aurora52+. Should be in 51 beta 4.

needs rebase for aurora grafting 376478:a4b0052954d2 "Bug 1279699: Use temporary files instead of shared memory to store the page recordings when printing via parent. r=bas, r=froydnj" merging layout/printing/ipc/RemotePrintJobChild.cpp merging layout/printing/ipc/RemotePrintJobChild.h merging layout/printing/ipc/RemotePrintJobParent.cpp merging layout/printing/ipc/RemotePrintJobParent.h warning: conflicts while merging layout/printing/ipc/RemotePrintJobParent.cpp! (edit, then use 'hg resolve --mark') warning: conflicts while merging layout/printing/ipc/RemotePrintJobParent.h! (edit, then use 'hg resolve --mark') abort: unresolved conflicts, can't continue (use 'hg resolve' and 'hg graft --continue')

Sorry Tomcat, I'd forgotten about the whole IPCResult change. This patch applies cleanly to Aurora and Beta.

Comment on attachment 8812844 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Looks like this might be might have caused the crash now tracked in bug 1320863 to reappear. It started again on 53.0a1 after this landed and has just appeared in 52.0a2, which had the uplift the day before. It's possible that memory issues, that were causing this crash then appear in fonts failing to be created in the parent. It's a nightly/aurora only crash. In beta and release it will fail to print that piece of text and count in Telemetry I believe.

Comment on attachment 8812844 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Removing the release uplift request for the moment as this looks like it might have caused or uncovered bugs later on (bugs 1320863 and 1321522).

Backed out at Bob's request. https://hg.mozilla.org/integration/mozilla-inbound/rev/a18b118a0bfcea69db595ee0cb8e8201f392de22

https://hg.mozilla.org/releases/mozilla-aurora/rev/c6e3c316f1c0 https://hg.mozilla.org/releases/mozilla-beta/rev/c0a36f11f8de

Thanks again for backing this out Ryan. So, as I was looking into the problems this caused I realised that I'd forgotten that we can hit [1] before the page has finished printing in the parent. Unfortunately, despite its name nsPrintEngine::PrePrintPage does start printing the next page and because I'm using the same file name for each page ... bad things happen. This should be pretty easy to fix. For completeness the backout has been merged to mozilla-central here: https://hg.mozilla.org/mozilla-central/rev/a18b118a0bfc [1] http://searchfox.org/mozilla-central/rev/0c055ccbcf96846044fc9a71421bd9b7978686f7/layout/printing/nsPagePrintTimer.cpp#152

This just has changes to nsDeviceContextSpecProxy and DrawEventRecorderFile to create a new temporary file per page. I was already passing the file name up to the parent each time in the old patch, so no changes needed there. jimm - flagging you as I think Bas might not be available, in the hope that you might have time to pick this up, thanks. MozReview-Commit-ID: 7hrg9VlTWfd

Comment on attachment 8816445 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Review of attachment 8816445 [details] [diff] [review]: ----------------------------------------------------------------- I can't test this, but the changes in nsDeviceContextSpecProxy.cpp look safe. R= and maybe Bas can look this over post landing for any additional issues. ::: widget/nsDeviceContextSpecProxy.cpp @@ +157,5 @@ > + > + char uuidChars[NSID_LENGTH]; > + uuid.ToProvidedString(uuidChars); > + mRecordingFileName.AssignASCII(uuidChars); > + nit - ws @@ +213,5 @@ > + nsresult rv = CreateUniqueTempPath(recordingPath); > + if (NS_FAILED(rv)) { > + return rv; > + } > + nit - ws

Pushed by bobowencode@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/d614a623d681 Use temporary files instead of shared memory to store the page recordings when printing via parent. r=bas, r=froydnj, r=jimm

New version of the uplift patch.

Comment on attachment 8816445 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Approval Request Comment [Feature/regressing bug #]: Bug 1156742. [User impact if declined]: The OOM content process crashes will continue to occur on certain web pages/machines. [Describe test coverage new/current, TreeHerder]: There is no current automated testing for printing, so it would be good to get some manual coverage for this change. [Risks and why]: Medium - the change is not that complicated, but saying medium because of the lack of automated testing. [String/UUID change made/needed]: None.

Comment on attachment 8816445 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent let's try again to fix this printing issue in aurora52

Comment on attachment 8816445 [details] [diff] [review] Use temporary files instead of shared memory to store the page recordings when printing via parent Fix a crash. Beta51+. Should be in 51 beta 7.

I tried to reproduce this crash on Windows 10 32-bit/64-bit and Windows 7 64bit using old Nightly 2016-07-16 - 32bit build, but without success. I loaded different heavy loaded websites and used print preview and print but it would not crash. baffclan, I don't know if you did reproduced this crash initially, if yes can you please check if this is fixed in latest Beta 51 build?

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #47) > baffclan, I don't know if you did reproduced this crash initially, if yes > can you please check if this is fixed in latest Beta 51 build? Sorry, I do not use Beta channel. I use Nightly channel.

Oh, I see. You can have more than one Firefox on your PC without affecting one or another, installed or unzipped. Can you please: 1. Download the zip file for Firefox beta that I was talking about (https://archive.mozilla.org/pub/firefox/candidates/51.0b10-candidates/build1/win32/en-US/) 2. Extract somewhere on your drive 3. Start it 4. Try to see if the problem still happens 5. Close it and delete it. While you are there can you also take a look in Nightly as well? This will help us close this bug if the issue does not reproduce for you anymore. Thanks and sorry for keep asking you this, but it really helps us!

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #49) I tested 51.0b10.win64 with New Profile. (firefox.exe -no-remote -p NewProfile) Cannot reproduce this problem. PS. I just used 51.0 for a few minutes, but it crashed :-( [@ IPCError-browser | ShutDownKill ]

(In reply to baffclan from comment #50) > I just used 51.0 for a few minutes, but it crashed :-( > [@ IPCError-browser | ShutDownKill ] Sorry, This was not 51.0, it was Nightly's crash. m(_._)m

(In reply to baffclan from comment #50) > Created attachment 8822191 [details] > 51.0b10 about:support > > (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #49) > I tested 51.0b10.win64 with New Profile. > (firefox.exe -no-remote -p NewProfile) > > Cannot reproduce this problem. Many Thanks! Marking the status flag as fixed for 51. > PS. > I just used 51.0 for a few minutes, but it crashed :-( > [@ IPCError-browser | ShutDownKill ] Yeah, that's bug 1279293 you are seeing... On another note, happy new year! \o/