After doing some detailed analysis of the telemetry data we're receiving about cache corruption, I realised that we are always detecting distributions as having corrupt caches. This is because of two reasons:

• Distribution engines are loaded separately from the legacy config, so we don't know how many engines would be loaded when we check the cache. As a result, the distribution is likely to have more engines than we expect, and therefore we think the cache is corrupt.
• Distribution engines are not being marked as built-in. This will also potentially confuse the detection, as we're comparing built-in engines only (bug 1623597)

There's a couple of adverse effects as a result of this:

• We'll be rebuilding the search engine cache every start-up, which may affect start-up performance (though I expect the effect to be small).
• If the user removes an engine that is added or altered by the distribution, this setting will not be preserved.

I'm going to handle the cache detection issue in this bug. The built-in issue I'll handle in bug 1623597, since I think that needs a little bit of its own focus to ensure there's no separate issue with that.

When building the search cache, we currently have no idea if engines have been added or modified by a distribution. Therefore we need to skip the corruption check for distributions for now.

One distribution engines are incoporated into the modern configuration, we should no longer have this issue, since we'll know what should and shouldn't be there.

https://hg.mozilla.org/mozilla-central/rev/18602acc4527

For QA:

STR:

Note: please make sure this is an up-to-date build, or one where updates are turned off, otherwise you may hit bug 1623597 when reproducing this.

1. Set up a build to have a distribution.ini and a search engine file in distributions/searchplugins/common/ (I can supply if necessary).
2. Start up Firefox.
3. Restart Firefox (to ensure any region changes have been picked up).
4. Open about:telemetry, navigate to the scalars section and if this bug is hit, browser.searchinit.engines_cache_corrupted will be true.
5. Remove the distribution engine via the search engine preferences.
6. Restart Firefox.
7. If the bug is still present, then the search engine will re-appear.

Comment on attachment 9135172 [details] [diff] [review]
Beta Patch

Beta/Release Uplift Approval Request

• User impact if declined: If using a distribution build, there are several effects:

1. The search engine cache will be incorrectly detected as corrupt, which affects our reporting.
2. The cache will be unnecessarily rebuilt on every startup, potentially slowing startup though I expect the effect to be very small.
3. If the user removes an engine that has been added by the distribution, then it will appear after restart.

• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: See comment 4
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This stops our cache corruption detection happening for distributions which has the effect of reverting distributions to how they were before bug 1578807.
• String changes made/needed: None

(In reply to Mark Banner (:standard8) from comment #4)

For QA:

STR:

Note: please make sure this is an up-to-date build, or one where updates are turned off, otherwise you may hit bug 1623597 when reproducing this.

1. Set up a build to have a distribution.ini and a search engine file in distributions/searchplugins/common/ (I can supply if necessary).
2. Start up Firefox.
3. Restart Firefox (to ensure any region changes have been picked up).
4. Open about:telemetry, navigate to the scalars section and if this bug is hit, browser.searchinit.engines_cache_corrupted will be true.
5. Remove the distribution engine via the search engine preferences.
6. Restart Firefox.
7. If the bug is still present, then the search engine will re-appear.

Mark, Can you please provide the build needed at step 1? Or some steps on how to provide it? Thanks.

https://hg.mozilla.org/releases/mozilla-beta/rev/87734aa6dfeb

For setting up a build:

1. Download a nightly/beta whatever build is under test and unpack or install it.
2. Download the distribution.zip
3. On Windows/Linux extract the zip into the top level directory of that build, so that there is a distribution folder at the same level as firefox.exe (windows) or firefox (linux)
4. On Mac, extract the zip into Firefox.app/Contents/Resources so that the distribution folder is within the Resources folder.
5. Start Firefox, go to about:config, and you should be able to see distribution.id set to test.

(In reply to Bodea Daniel [:danibodea] from comment #8)

(In reply to Mark Banner (:standard8) from comment #4)

For QA:

STR:

Note: please make sure this is an up-to-date build, or one where updates are turned off, otherwise you may hit bug 1623597 when reproducing this.

1. Set up a build to have a distribution.ini and a search engine file in distributions/searchplugins/common/ (I can supply if necessary).
2. Start up Firefox.
3. Restart Firefox (to ensure any region changes have been picked up).
4. Open about:telemetry, navigate to the scalars section and if this bug is hit, browser.searchinit.engines_cache_corrupted will be true.
5. Remove the distribution engine via the search engine preferences.
6. Restart Firefox.
7. If the bug is still present, then the search engine will re-appear.

Mark, Can you please provide the build needed at step 1? Or some steps on how to provide it? Thanks.

Thank you Mark. I have succeeded in setting up the build.
I still do need some more information on how to perform point 5 and how to verify point 7. Help?

(In reply to Bodea Daniel [:danibodea] from comment #11)

Thank you Mark. I have succeeded in setting up the build.
I still do need some more information on how to perform point 5 and how to verify point 7. Help?

5. Remove the distribution engine via the search engine preferences.

Preferences -> Search -> One Click Search Engines section, click the distribution engine (Amazon.co.uk) and click remove.

7. If the bug is still present, then the search engine will re-appear.

Check the "One Click Search Engines" section again.

I've just realised that despite my intention, you might need an en-GB build with browser.search.region set to GB to reproduce this properly. Try without first, if it doesn't work, then try with.

This bug was reproduced in Nightly v76.0a1 from 2020-03-21 and verified in Nightly v76.0a1 from 2020-03-29 and Beta v75.0b10.