In bug 1353461 we are in the process of switching reftest to "run-by-manifest" mode. This means that between every manifest of tests we will restart Firefox. In general this leads to improved stability, though in a few cases existing tests that happened to depend on side-effects of previous tests can start failing. This is one of those cases. Looks like all four localized-family-names-00{1-4}.html tests are permafailing on OSX, both opt and debug. Here is a push demonstrating this: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d04e70fecd06900a5c7634b42d35dd681ad197fc Example log: https://treeherder.mozilla.org/logviewer.html#?job_id=163438739&repo=try&lineNumber=1474 It looks like there are a few (very infrequent) intermittents already filed against these tests. So I suspect that the run-by-manifest change has just made the issue more apparent. We'll need to either fix this prior to landing run-by-manifest, or disable these tests on OSX.

Jonathan, can you please take a look?

I believe the issue here is that these tests depend for correct font selection on font data (localized names) that we load shortly after startup (the FontInfoLoader that runs in the background after a short delay). So if we try to run the tests in a newly-launched browser, they tend to fail; if they run later in the test job, after the browser has had time for the font-loader task to complete, they pass. Switching to run-by-manifest will tend to make this a much more reliable failure, as we'll always be starting this manifest with a freshly-launched browser, so it won't have had time to load the font info yet. So what to do here? Not sure, but I'm toying with a few ideas....

This should make these tests much more reliable, even if they run shortly after the browser has started.

Just FTR, it turns out that the font-loading that's most relevant here is not actually what the FontInfoLoader handles (cmap data and font fullnames and psnames, on platforms where these aren't available from system APIs), but rather the localized family names that are loaded by gfxPlatformFontList::InitOtherFamilyNames. Normally, this is done lazily (and then after the task finishes, we force a reflow in order to pick up any font-family requests that now match the newly-added names); but that means the initial rendering of a testcase loaded shortly after startup may miss fonts that are requested via localized names. So we can fix this by running these tests with a pref that causes InitOtherFamilyNames to run "eagerly" (potentially janking the main thread the first time a font-family name is used that isn't in the canonical name list, but that's OK in the context of reftests) instead of posting a background task. There isn't an existing pref for this, but it seems reasonable to hook it up to the gfx.font_loader.delay setting: if a zero delay is requested for other font-info loading, then we'll also load localized names eagerly. With this, together with the run-by-manifest patches from bug 1353461, none of these font-matching tests are failing on tryserver any longer: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f3360b5dbf0c32695309d3df829f69565a4b9baa. (The oranges that remain here are unrelated.)

Pushed by jkew@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/9e189d5ae829 For font-matching reftests that test localized names, force the font-name loader to run eagerly so that we aren't dependent on how long the browser has been running. r=jrmuizel