When loading https://www.wikipedia.org/ we're seeing that Fenix is slower to first paint and most visual metrics than Fennec68.

It looks like there's a long reflow in Fenix, where we spent time preparing the fonts and chatting between the content process and the parent process to initialize a font family,

Fenix, 743ms reflow:
https://share.firefox.dev/2Uomn4J

Fennec68, 447ms reflow
https://share.firefox.dev/3lpMPqA

I thought that perhaps this occurred only the first site visited, but in this scenario I preloaded another site before capturing the profile and I still see 260ms in gfxPlatformFontList::InitializeFamily
https://share.firefox.dev/3eSUDid

:jfkthame, do you see anything here?

Here's a profile with IPC messages track visible:
https://share.firefox.dev/2Ip2Hvs

In this case the browser had loaded https://www.mozilla.org/en-US/, waited ~10 seconds, navigated to about:blank, and then loaded wikipedia.org

I imagine that Wikipedia's home page with at least 10 languages must be part of the reason why this particular site is problematic.

Right, the wikipedia home page will cause a whole bunch of fonts to be needed -- not just for the 10 or so languages around the globe, but also for the much longer list in the search-field language dropdown. In particular, the profile indicates that we're encountering some text that isn't supported by any of the common default fonts we have listed, so we end up searching all available fonts to try and find one that supports it.

The good news is that this shouldn't happen on subsequent visits, as the font data is now loaded; but it hurts the first time. (And preloading a different site will not necessarily make much difference, as that other site may only cause a handful of the fonts to be loaded.)

Is the performance pretty comparable to fennec after that first visit to wikipedia.org (within the same browser session, but e.g. in a new tab)?

We might be able to persist a bunch of this data across browser sessions, which would mean that it'd only be an issue on first run; subsequent sessions would reuse the cached data.

(In reply to Jonathan Kew (:jfkthame) from comment #4)

We might be able to persist a bunch of this data across browser sessions, which would mean that it'd only be an issue on first run; subsequent sessions would reuse the cached data.

I've hoped for a long time that would could do this at some point. I would like to strongly endorse this idea.

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #3)

I imagine that Wikipedia's home page with at least 10 languages must be part of the reason why this particular site is problematic.

One "missing glyph" is enough to send us down this path. So the language dropdown makes it more likely to encounter a glyph that we don't have, but in general, this is a performance cliff where the cost is either all or nothing, and not in proportion to e.g. the number of used languages.

A lot of the overhead here seems to be in bouncing the IPC messages back and forth. It would probably be an improvement to have just one IPC message that does all the work, for example an InitializeFamiliesUntilGlyphFound(glyph) message.

The good news is that this shouldn't happen on subsequent visits, as the font data is now loaded; but it hurts the first time. (And preloading a different site will not necessarily make much difference, as that other site may only cause a handful of the fonts to be loaded.)

That is true - on revisits to the site in the same session I don't see the font family being initialized.
This is a much faster load in general because the resources are all in cache.
e.g.
https://share.firefox.dev/36tPr0s

Is the performance pretty comparable to fennec after that first visit to wikipedia.org (within the same browser session, but e.g. in a new tab)?

Fennec is still faster than Fenix for this site on subsequent visits to site within the same session.
However it looks to me like it's network timing that's slowing down Fenix and not fonts (backend time is ~74ms in Fennec, and ~160-200ms in Fenix)
I haven't look at these in depth, but I'll make a bug for Fennec/Fenix warm load performance if this occurs on other sites.

Fenix
https://share.firefox.dev/2UrnMri

Fennec
https://share.firefox.dev/38D6z6J

We might be able to persist a bunch of this data across browser sessions, which would mean that it'd only be an issue on first run; subsequent sessions would reuse the cached data.

I thought that we were cacheing platform font data between sessions, as per Bug 1554183, but my knowledge here is pretty limited.

The severity field is not set for this bug.
:boris, could you have a look please?

For more information, please visit auto_nag documentation.

(In reply to Markus Stange [:mstange] from comment #6)

A lot of the overhead here seems to be in bouncing the IPC messages back and forth. It would probably be an improvement to have just one IPC message that does all the work, for example an InitializeFamiliesUntilGlyphFound(glyph) message.

I've wondered about taking that approach. It should reduce the IPC overhead, but has the drawback that it could result in severely janking the parent process the first time a content process encounters an unknown character that forces an exhaustive search.

Anyhow, I put together a tentative patch to see how this would behave; there's a try build at https://treeherder.mozilla.org/jobs?repo=try&revision=cb77be495dd16c3e523afac3b6f5d2dc0bbe13aa. Could you try this and see how it compares? (My suspicion is that it won't actually be much better, because the IPC overhead is quite small compared to all the filesystem access that's going on.)

Thanks for providing that patch, :jfkthame.

The above plot shows some early results where I compared it (in Fenix) to a baseline (two commits back,
https://hg.mozilla.org/try/rev/3636cdf0b4874329e6daf9fea3455a6979a6f18d).

I included multiple runs of both the baseline and the patch (fenix_globalFontSearch), each time doing 10 cold page loads for the site.

From this initial test, it looks like a ~25% improvement to median visual metrics for this particular site.
Easily statistically significant at 90% confidence.

https://docs.google.com/spreadsheets/d/1FjhbYxNH37FZCeJiBcz2_pH7KmXvkiclV2ppQGz3jes/edit#gid=1414745497

Some profiles:

Baseline
https://share.firefox.dev/36o9Sgz

Global Font Search
https://share.firefox.dev/36o7Htd

At least in this case I don't see jank in the parent process, but the jank in the content process is reduced.

This looks really promising.

I'll compare this over a broader set of sites.

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #11)

Global Font Search
https://share.firefox.dev/36o7Htd

At least in this case I don't see jank in the parent process, but the jank in the content process is reduced.

Unless I'm misunderstanding, this profile does show a significant (237ms) jank in the parent, doesn't it?

Maybe this would be an acceptable trade-off, particularly on mobile where the user may be less likely to be trying to interact in other ways with the parent at the same time as the content process is doing its initial reflow, but it's not a "free" win.

(In reply to Jonathan Kew (:jfkthame) from comment #12)

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #11)

Global Font Search
https://share.firefox.dev/36o7Htd

At least in this case I don't see jank in the parent process, but the jank in the content process is reduced.

Unless I'm misunderstanding, this profile does show a significant (237ms) jank in the parent, doesn't it?

Yes, sorry, I missed that somehow.
And all profiles with the globalFontSearch show a similar jank in the parent.

Something we will have to consider.

I'd say this is an acceptable trade-off on Android for sure, because the Android app UI will still be responsive because it doesn't live on the parent process main thread. I'm less sure about whether it's an acceptable trade-off on desktop. My gut says yes.

I wonder if we could have both good things at the same time: A single synchronous IPC message, and chunked processing in the parent process. Maybe we could spin a nested event loop in the parent process, inside RecvTheSyncMessage? Or we could teach the IPC system to allow using MozPromise in the receiver of a sync message while still making the message look synchronous to the sender.

It would be best if this didn't need to be a synchronous message in the first place. How feasible would it be to make all this asynchronous? It would be nice if we could treat the unknown "fallback" fonts like webfonts which haven't loaded yet.
So it would work like this: If a character is not found, draw a placeholder box in its place, and kick off an asynchronous search for that character in all available fonts. Once found, refresh the fonts for the affected parts of the page.

Oh, if we can move the parent process side of this to a background thread, then there would be another alternative: We could create a direct IPC connection between the content process main thread and the parent process font background thread.

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #13)

And all profiles with the globalFontSearch show a similar jank in the parent.

Something we will have to consider.

I have an idea that might reduce this somewhat (specifically on Android), by reducing our use of FreeType in there; I think accessing the font data via harfbuzz APIs may be cheaper. I'm putting a patch together at the moment to see if it helps.

(In reply to Markus Stange [:mstange] from comment #14)

I'd say this is an acceptable trade-off on Android for sure, because the Android app UI will still be responsive because it doesn't live on the parent process main thread.

Yeah, that makes sense.

I'm less sure about whether it's an acceptable trade-off on desktop. My gut says yes.

Much less sure here, as the parent-process jank can be quite noticeable -- and desktop users are more likely to have huge font collections, so that even on a fast machine the impact can be quite severe. IMO one slow reflow (in an otherwise-responsive browser) is less painful than janking the entire browser UI; to the user, it's pretty much equivalent to a momentary network slowdown, for example. (Though we'd still prefer to eliminate it, of course!)

I wonder if we could have both good things at the same time: A single synchronous IPC message, and chunked processing in the parent process. Maybe we could spin a nested event loop in the parent process, inside RecvTheSyncMessage? Or we could teach the IPC system to allow using MozPromise in the receiver of a sync message while still making the message look synchronous to the sender.

I wondered about things like this, but I suspect that if we start chunking the processing in the parent, spinning the event loop, etc, we'll end up with something that's not much better than the current "ipc-chatty" model where the processing is "chunked" by virtue of the content process querying each font family separately. This lets the parent remain responsive, but it makes the overall time longer.

It would be best if this didn't need to be a synchronous message in the first place.

Yes!

How feasible would it be to make all this asynchronous? It would be nice if we could treat the unknown "fallback" fonts like webfonts which haven't loaded yet.
So it would work like this: If a character is not found, draw a placeholder box in its place, and kick off an asynchronous search for that character in all available fonts. Once found, refresh the fonts for the affected parts of the page.

This is an interesting possibility that may be worth exploring, though the trade-off here will be the risk of a "FOUT"-like visual glitch after font-search completes, or content shifting after initial rendering, which there's currently some pressure to try and minimize.

Andrew, there's a try push at https://treeherder.mozilla.org/jobs?repo=try&revision=bb205e69625f95be1e4bac1b56ac4f43e9d9905f with an additional patch (on top of the earlier GlobalFontSearch patch) that may help a bit more here.

I don't know how much difference this will make (maybe it'll be lost in the noise), but if you could try the same profiling with this patch added, it'd be interesting to see what happens.

I tested the initial GlobalFontSearch patch on a larger set of sites.
These are live sites so there's a fair deal of noise, but I only see wikipedia being improved w/ 90% confidence.
https://docs.google.com/spreadsheets/d/1FjhbYxNH37FZCeJiBcz2_pH7KmXvkiclV2ppQGz3jes/edit#gid=1867115751

(In reply to Markus Stange [:mstange] from comment #14)

I'd say this is an acceptable trade-off on Android for sure, because the Android app UI will still be responsive because it doesn't live on the parent process main thread. I'm less sure about whether it's an acceptable trade-off on desktop. My gut says yes.

I'm concerned that introducing a long jank on the parent process main thread (~240ms on a reasonably fast Pixel 3) will block network calls that are running during pageload.

(In reply to Markus Stange [:mstange] from comment #15)

Oh, if we can move the parent process side of this to a background thread, then there would be another alternative: We could create a direct IPC connection between the content process main thread and the parent process font background thread.

Intuitively, this seems ideal.

(In reply to Jonathan Kew (:jfkthame) from comment #17)

Andrew, there's a try push at https://treeherder.mozilla.org/jobs?repo=try&revision=bb205e69625f95be1e4bac1b56ac4f43e9d9905f with an additional patch (on top of the earlier GlobalFontSearch patch) that may help a bit more here.

I don't know how much difference this will make (maybe it'll be lost in the noise), but if you could try the same profiling with this patch added, it'd be interesting to see what happens.

Will do.

(In reply to Jonathan Kew (:jfkthame) from comment #17)

Andrew, there's a try push at https://treeherder.mozilla.org/jobs?repo=try&revision=bb205e69625f95be1e4bac1b56ac4f43e9d9905f with an additional patch (on top of the earlier GlobalFontSearch patch) that may help a bit more here.

I don't know how much difference this will make (maybe it'll be lost in the noise), but if you could try the same profiling with this patch added, it'd be interesting to see what happens.

The HarfBuzz api's look to be much faster on Android.

On wikipedia they improve loadtime and vismets ~40-50%, Pixel 3
https://docs.google.com/spreadsheets/d/1FjhbYxNH37FZCeJiBcz2_pH7KmXvkiclV2ppQGz3jes/edit#gid=296070648

In addition, I don't see any parent main thread jank in the profiles (I looked at 4 in total)
https://share.firefox.dev/37tEBb8
https://share.firefox.dev/2HYlMEt

That's awesome -- better than I would have dared to hope, even.

Given that, I think the first thing to do here (well, let's spin it off to a dependent bug) is to go ahead and take some form of the harfbuzz-apis patch, as that will be a significant win regardless of what else we do with the IPC messages etc. I'll check it over again and try to get it up for review tomorrow.

(If you're up for a bit more profiling, you could also try the harfbuzz-apis patch without the GlobalFontSearch one -- they should be independent -- and see how that compares.)

(In reply to Jonathan Kew (:jfkthame) from comment #16)

I'm less sure about whether it's an acceptable trade-off on desktop. My gut says yes.

Much less sure here, as the parent-process jank can be quite noticeable -- and desktop users are more likely to have huge font collections, so that even on a fast machine the impact can be quite severe. IMO one slow reflow (in an otherwise-responsive browser) is less painful than janking the entire browser UI; to the user, it's pretty much equivalent to a momentary network slowdown, for example. (Though we'd still prefer to eliminate it, of course!)

I've thought about this again and I agree with you now.
Let's keep the current behavior on desktop.

I wonder if we could have both good things at the same time: A single synchronous IPC message, and chunked processing in the parent process. Maybe we could spin a nested event loop in the parent process, inside RecvTheSyncMessage? Or we could teach the IPC system to allow using MozPromise in the receiver of a sync message while still making the message look synchronous to the sender.

I wondered about things like this, but I suspect that if we start chunking the processing in the parent, spinning the event loop, etc, we'll end up with something that's not much better than the current "ipc-chatty" model where the processing is "chunked" by virtue of the content process querying each font family separately.

Maybe not much better, but I expect it to be at least a little bit better. But maybe not enough to justify the code complexity.

How feasible would it be to make all this asynchronous? It would be nice if we could treat the unknown "fallback" fonts like webfonts which haven't loaded yet.
So it would work like this: If a character is not found, draw a placeholder box in its place, and kick off an asynchronous search for that character in all available fonts. Once found, refresh the fonts for the affected parts of the page.

This is an interesting possibility that may be worth exploring, though the trade-off here will be the risk of a "FOUT"-like visual glitch after font-search completes, or content shifting after initial rendering, which there's currently some pressure to try and minimize.

Sure, that trade-off exists. However, the visual glitch only appears if all of the following are true:

• The glyph is eventually found in one of the installed fonts.
• The glyph is used in text that is currently visible on the screen.
• The affected page is loaded shortly after Firefox startup, when the background enumeration of fonts hasn't completed yet.

In the egregiously-slow cases that I've seen, the glyphs in question are usually used in a language dropdown that's not visible. And in the slowest cases, the glyph is not found in any font. But my experience is very English language centric, so I'm not sure how much it applies to the general public.

(In reply to Jonathan Kew (:jfkthame) from comment #22)

That's awesome -- better than I would have dared to hope, even.

Given that, I think the first thing to do here (well, let's spin it off to a dependent bug) is to go ahead and take some form of the harfbuzz-apis patch, as that will be a significant win regardless of what else we do with the IPC messages etc. I'll check it over again and try to get it up for review tomorrow.

(If you're up for a bit more profiling, you could also try the harfbuzz-apis patch without the GlobalFontSearch one -- they should be independent -- and see how that compares.)

I ran a test with just the harfbuzzapis - fenix_harbuzz_only it improves vismets on wikipedia by ~20-25%.
By itself it looks to be just a bit faster than the initial globalFontSearch change.

fenix_baseline - unmodified
fenix_globalFontSearch - all in one global font search
fenix_global_harfbuzz - all in one global font search + harfbuzz api
fenix_harfbuzz_only - just the harfbuzz api seach

These results also show (at 90% confidence) improvements to speedIndex on https://accounts.google.com using just the harfbuzz APIs.
https://docs.google.com/spreadsheets/d/1FjhbYxNH37FZCeJiBcz2_pH7KmXvkiclV2ppQGz3jes/edit#gid=1092288141

This looks like a great improvement to Fenix.
I'll continue looking at this particular change in Bug 1680234 - I'm expecting bigger improvements on slower phones and there must be many more affected sites.

If anyone is aware of other popular sites that could result in this performance cliff, let me know.

But it does look like there are still gains possible from moving work of the parent process main thread.

I've put together a patch that makes global font fallback non-blocking; instead, we send an async message asking the main process to load the data required for fallback searches, it does that using idle-time runnables, and when finished, notifies the content process(es) to re-render.

This means that we may temporarily render hexboxes for characters that aren't found via font prefs or "common" font fallback, but as soon as the cmap-loading tasks are finished, we'll refresh the rendering with the proper fallbacks.

In local testing, this dramatically improves the initial rendering of wikipedia.org, for example: an initial reflow of 1400ms is reduced to 250ms. To see the effect on fallback, load wikipedia.org and expand the "Read wikipedia in your language" section so that the big list of language names is displayed; then quit the browser, re-launch, load wikipedia.org and immediately scroll down to see the language names. On macOS, I initially see hexboxes for a few of them, such as "ᨅᨔ ᨕᨙᨁᨗ / Basa Ugi" and "ދިވެހިބަސް", but then after a second or so they correct themselves. (If I'm a bit slow about scrolling down, I don't get to see the hexboxes at all.)

Try build with the patch: https://treeherder.mozilla.org/jobs?repo=try&revision=507ed73ee330cfbcce6d5ec7a24fb46d90db439e

Testing/profiling on Android would be much appreciated; I expect/hope this should make a significant difference.

Here's a patch that makes the cmap-loading required by global fallback happen asynchronously in the parent process (with "idle" priority, so it shouldn't jank things there), and then refresh content rendering when it's finished.

With my local build on macOS, this takes about 1.5 seconds off the time to render wikipedia.org when launching the browser and directly loading that page.

Profile with the async loading preffed-off (i.e. current trunk behavior), launched with MOZ_PROFILER_STARTUP=1 and loading https://www.wikipedia.org/ from the command line: https://share.firefox.dev/37BGl2g

We can see that the content appears at about the 5-second mark, after a hugely-long (1.8s) reflow in the content process, which was mostly waiting for the parent to read character maps for each font it wanted to look at.

With async loading preffed-on (gfx.font_rendering.fallback.async=true): https://share.firefox.dev/36RgRid

Here, the content appears at around 3.5s; the content-process reflow was 242ms instead of 1.8s. The font-loading work in the parent process starts during this time, but we don't wait for it to finish; it continues past the 4.5s mark. When it does finally complete, it triggers an extra reflow in the content process, which is when any temporary hexbox-fallback rendering would get fixed, though in this case there isn't anything visible unless I have scrolled the page so as to see the more "obscure" language names.

Note that this is normally a once-per-session issue; after the first global-fallback has been triggered, all character maps are present in the shared font list, so future fallback (in any process) will happen without requiring any of this font access.

Testing/profiling on Android would be much appreciated; I expect/hope this should make a significant difference.

I'm on it -- very curious myself.

I built the change from https://phabricator.services.mozilla.com/D98904 into a Fenix build w/ PGO'ed gecko.

Comparing gfx.font_rendering.fallback.async:false to gfx.font_rendering.fallback.async:true so far I don't see a difference.

I did verify that the preference is being set appropriately while the test is running.

I'm grabbing profiles to see if anything stands out.

gfx.font_rendering.fallback.async:false
https://share.firefox.dev/2Iv8Je0

gfx.font_rendering.fallback.async:true
https://share.firefox.dev/3gzuplT

In both cases I'm seeing two reflows of 140-160ms and the cmap-loading in the parent process during the pageload.

Maybe the EventQueuePriority::Idle is running right away because we have an idle slice available during pageload?

Something's not right there, in terms of the pref getting set/respected during the test. With the pref set to false (https://share.firefox.dev/2Iv8Je0), there shouldn't be any sign of LoadCmapsRunnable in the main process. Which suggests that somehow the pref isn't being successfully disabled, or there's something broken in the code that checks it. I'll look around a bit...

On the bright side, if I'm reading things correctly, these runs are on a par with the earlier fenix_global_harfbuzz tests in comment 24, or in other words a significant improvement over current trunk code. I'm just puzzled why the pref doesn't seem to be working.

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #30)

gfx.font_rendering.fallback.async:false
https://share.firefox.dev/2Iv8Je0

gfx.font_rendering.fallback.async:true
https://share.firefox.dev/3gzuplT

In both cases I'm seeing two reflows of 140-160ms and the cmap-loading in the parent process during the pageload.

It'd be interesting to confirm whether the content is rendered and the page usable after the first of those two reflows. (According to the behavior I see on desktop, I think it ought to be.)

(In reply to Jonathan Kew (:jfkthame) from comment #32)

Something's not right there, in terms of the pref getting set/respected during the test. With the pref set to false (https://share.firefox.dev/2Iv8Je0), there shouldn't be any sign of LoadCmapsRunnable in the main process. Which suggests that somehow the pref isn't being successfully disabled, or there's something broken in the code that checks it. I'll look around a bit...

On the bright side, if I'm reading things correctly, these runs are on a par with the earlier fenix_global_harfbuzz tests in comment 24, or in other words a significant improvement over current trunk code. I'm just puzzled why the pref doesn't seem to be working.

I'll compare the try push from Comment 25 with its parent commit.
https://treeherder.mozilla.org/jobs?repo=try&revision=507ed73ee330cfbcce6d5ec7a24fb46d90db439e

I haven't done a broader comparison yet, but this shows your try push having an impact (comparing to the previous commit in m-c).

Baseline:
https://share.firefox.dev/2JDL23P

Async cmaps
https://share.firefox.dev/2VT5G29

(In reply to Jonathan Kew (:jfkthame) from comment #33)

In both cases I'm seeing two reflows of 140-160ms and the cmap-loading in the parent process during the pageload.

It'd be interesting to confirm whether the content is rendered and the page usable after the first of those two reflows. (According to the behavior I see on desktop, I think it ought to be.)

When capturing screenshots in the profiles it looks like the page is only rendered after the second reflow:
https://share.firefox.dev/3gtbZTq

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #36)

When capturing screenshots in the profiles it looks like the page is only rendered after the second reflow:
https://share.firefox.dev/3gtbZTq

I take that back, it's more like it's rendered during the 2nd reflow.

I'm assuming that ContentParent::RecvInitializeFamily will no longer be called when the async path is enabled. If that's incorrect, please remove the whiteboard bhr annotation again. ContentParent::RecvInitializeFamily currently shows up in two stacks on http://queze.net/bhr/test/ .

(In reply to Markus Stange [:mstange] from comment #38)

I'm assuming that ContentParent::RecvInitializeFamily will no longer be called when the async path is enabled. If that's incorrect, please remove the whiteboard bhr annotation again. ContentParent::RecvInitializeFamily currently shows up in two stacks on http://queze.net/bhr/test/ .

It will still be called in some cases: I'd expect it to be called for individual font families that are explicitly referenced from CSS or browser prefs, because these will be initialized (synchronously) when first used. The async path will only be triggered in the case where we need to resort to global font fallback, which not all sites ever hit. (If that occurs, once it is complete there won't be any need for individual RecvInitializeFamily calls.)

So this isn't designed to eliminate all use of RecvInitializeFamily, but it should avoid the case where we call RecvInitializeFamily for dozens or hundreds of fonts in rapid succession.

As such, I'd guess it's much less likely to show up in BHR once we get this landed. (Currently looking into a failure on tryserver, so it's not ready to push quite yet.)

The async cmap change shows some significant improvements including a 5% speedIndex on stackoverflow as well as more improvements to wikipedia and others.

https://docs.google.com/spreadsheets/d/1FjhbYxNH37FZCeJiBcz2_pH7KmXvkiclV2ppQGz3jes/edit#gid=1301882518
** improvements highlighted in green are not necessarily statistically significant - see the 90% confidence interval**

We have a high degree of noise here, partially from live sites, so I'll run this test again with recorded sites to get a better idea.

The async cmap looks great.
I only picked up statistically significant changes on wikipedia.org, but they are quite big -
On recorded sites (for reproducibility) speedIndex improvements of ~33%.
On live sites, speedIndex improvements between 10% and 20%.

Nicely done!

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:jfkthame, could you have a look please?
For more information, please visit auto_nag documentation.

I'm looking into a few intermittent test issues that could be related to this, but hope to get the patch landed soon.

https://hg.mozilla.org/mozilla-central/rev/4a6818e6c81a

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=XbZdwZjjQg-Kqjw6JxEarQ.0&revision=68458324746edf2e36d6f51977933f96a993fbc6&searchStr=mochitest&failure_classification_id=2

Log: https://treeherder.mozilla.org/logviewer?job_id=325527888&repo=autoland&lineNumber=10990

Backout: https://hg.mozilla.org/integration/autoland/rev/4465e82f70bf0157249e8d5b3aaa5960442a5f9a

This also seems to have caused bug 1235836

https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=AIOb7nceS0eGjKBQI3zpog.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&tochange=4a6818e6c81a2c0c9221b8e0306d3c47d8cc666e&fromchange=48f46a7eada94d51d8bd508b5122642865ddef3d&test_paths=browser%2Fcomponents%2Fsessionstore%2Ftest

and bug 1684362
Retriggers: https://treeherder.mozilla.org/jobs?repo=autoland&test_paths=docshell%2Ftest%2Fmochitest&searchStr=android%2Cwebrender&tochange=4a6818e6c81a2c0c9221b8e0306d3c47d8cc666e&fromchange=48f46a7eada94d51d8bd508b5122642865ddef3d&group_state=expanded&selectedTaskRun=GuoYrMEIQFCilTn0taZyfA.0

Backed out changeset 52eb1fa54b88 (Bug 1676966) for causing mochitest failures in AsyncCompositionManager.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/89bfa55afeb071f6f4d4df3dd05f4be4c5ca459a

Push with failures:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&tochange=89bfa55afeb071f6f4d4df3dd05f4be4c5ca459a&fromchange=10487bde26d342ad1cfc02a67c33698243c09329&selectedTaskRun=e8ty5VXkQ4eqf4YyzZ7AUQ.0
Retriggers:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&fromchange=7461667f04f186a682e76060383497373b86f14d&searchStr=windows%2C10%2Cx64%2Cdebug%2Cmochitests%2Cwithout%2Ce10s%2Ctest-windows10-64%2Fdebug-mochitest-chrome-1proc%2Cc2&selectedTaskRun=HKwT24jwQZ2BwW0Tuyk-tg.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=325557393&repo=autoland&lineNumber=5538

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=345d6be71db15bda1400aa8cc2978e99948f6135&searchStr=without%2Ce10s&selectedTaskRun=KWrC-plWRaWjUQaKPhagdA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=325584308&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/6b775ea9c239da24a6daefd8ecdc8b525e13a933

https://hg.mozilla.org/mozilla-central/rev/02d875d4f7d7