Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Description • 8 years ago

Amazon have reported a 10% playback failure rate spike in Firefox >= 49 32bit. They're seeing a MEDIA_DECODE_ERROR being fired at the media element.

Shift72 have reported the same thing, and can repro in house. I've made builds with extra logging to try to pinpoint the problem, and we appear to be failing when calling AllocUnsafeShmem in BufferTextureData::CreateInternal().

Logging failures of CreateFileMapping() in SharedMemory::Create() didn't get hit for some reason. Not sure why.

This is a recent regression, which just spiked in 49. So I'd say it coincides with the e10s rollout.

	Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Comment 1 • 8 years ago

[Tracking Requested - why for this release]:

Having a 10% failure rate on Amazon Prime Video in Firefox 49 is not something we want to ship to our release users.

	Mike Conley (:mconley) (:⚙️)
	Comment 2 • 8 years ago

Is this bad enough to warrant clamping down on the e10s rollout on release?

	Mike Conley (:mconley) (:⚙️)
	Comment 3 • 8 years ago

(Note that blassey is currently on PTO - not sure who decisions of that nature fall to while he's away)

	Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Comment 4 • 8 years ago

We have observed this in the beta 49 population only. Both these populations had/have almost the same users with e10s enabled; both have about 20% of the total population enabled, but 49 has 2% more as there are a few more users with addons that are in the e10s whitelist.

So given that this occurs in 48 and 49 and there wasn't a significant change in the e10s-ness between these populations, I think we can concluded that this is not an issue with e10s.

We can reproduce this issue if we have the WebConsole open. about:memory reports that there are strings leaking in the web console. We don't think this can be the cause of the 10% Amazon failure; we don't believe that 10% of Amazon Prime Video session have the WebConsole open!

So we think that we're failing to allocate frames as a consequence of some other leak. Possibly, the heap size is growing so much that allocation of shmems is failing. This could be either the shmems used to shuffle video frames between the gmp and content process, or between the content and chrome process.

	Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Comment 5 • 8 years ago

(In reply to Chris Pearce (:cpearce) from comment #4)
> So given that this occurs in 48 and 49 and there wasn't a significant change
> in the e10s-ness between these populations, I think we can concluded that
> this is not an issue with e10s.

I meant to say: given that this doesn't occur in 48 but does in 49 and there wasn't a significant change in the e10s settings between these populations, I think we can conclude that this is not an issue with e10s.

	Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)
	Comment 9 • 8 years ago
mozreview-review

Comment on attachment 8783819 [details]
Bug 1296453: [MSE] P1. Fix eviction.

https://reviewboard.mozilla.org/r/73490/#review71304

	Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)
	Comment 10 • 8 years ago
mozreview-review

Comment on attachment 8783820 [details]
Bug 1296453: [MSE] P2. Clean up SourceBufferResource.

https://reviewboard.mozilla.org/r/73492/#review71310

r+

Just wondering why the offset is signed (and then you need to make it unsigned in a few places). But it's everywhere, so probably too much work touching it right now.

	Pulsebot
	Comment 11 • 8 years ago

Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f87d2ddadaf
[MSE] P1. Fix eviction. r=gerald
https://hg.mozilla.org/integration/autoland/rev/13294e0b2e81
[MSE] P2. Clean up SourceBufferResource. r=gerald

	Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Comment 12 • 8 years ago

Summary of debugging; we were able to observe two leaks using about:memory, by watching the heap-unclassified rise over time while watching Amazon Prime. One leak occurs if the WebConsole is open; Firefox is leaking strings. The other (fixed by jya here) was a regression from https://hg.mozilla.org/integration/mozilla-inbound/rev/57a920361b81 .

Our theory is that the leaks cause the heap to expand. Once the heap expands, it doesn't contract. As it grows it reserves virtual address space. If it grows enough, it'll overlap the address space used to allocate shmems, and we'll fail to allocate shmems used to shuffle frames between the GMP <-> content <-> main processes. This results in a failure to allocate a video frame, which percolates up into an error event fired at the video element.

	Ryan VanderMeulen [:RyanVM]
	Comment 13 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/6f87d2ddadaf
https://hg.mozilla.org/mozilla-central/rev/13294e0b2e81

	Chris Pearce [:cpearce (Not reading bugmail)] Reporter
	Comment 14 • 8 years ago

I've filed Bug 1297525 for the Web Console leak.

	Jean-Yves Avenard [:jya] Assignee
	Comment 15 • 8 years ago

Comment on attachment 8783819 [details]
Bug 1296453: [MSE] P1. Fix eviction.

Approval Request Comment
[Feature/regressing bug #]:1261900
[User impact if declined]: Memory leak, frequent OOM particularly on 32 bits platforms.
[Describe test coverage new/current, TreeHerder]: In central, manually checked, confirmed fixed with 3rd party
[Risks and why]: None, we used to evict everything anyway for the past year.
[String/UUID change made/needed]: None

	Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)
	Comment 16 • 8 years ago

Comment on attachment 8783819 [details]
Bug 1296453: [MSE] P1. Fix eviction.

Fix for memory leak causing crashes with media player. 
Looks low-risk, let's uplift to beta. This should make it into tomorrow's beta 7 build.

	Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)
	Comment 17 • 8 years ago

jya- I assume both patches should uplift here.

	Jean-Yves Avenard [:jya] Assignee
	Comment 18 • 8 years ago

first one is enough, but both are okay too :)

	Ryan VanderMeulen [:RyanVM]
	Comment 19 • 8 years ago
bugherder uplift

https://hg.mozilla.org/releases/mozilla-aurora/rev/fc33239f919a
https://hg.mozilla.org/releases/mozilla-aurora/rev/a453051db7bf

	Ryan VanderMeulen [:RyanVM]
	Comment 20 • 8 years ago
bugherder uplift

https://hg.mozilla.org/releases/mozilla-beta/rev/cbe685bc5bc4
https://hg.mozilla.org/releases/mozilla-beta/rev/4c552e756236