probably not a high priority, but we are working towards getting tests running in CI that have a more complete profile that is updated daily. Many tests are written assuming a fresh/blank profile.

browser/components/extensions/test/xpcshell/
test_ext_bookmarks.js
test_ext_topSites.js
test_ext_urlbar.js
toolkit/mozapps/extensions/test/xpcshell/
test_XPIStates.js
test_shutdown_early.js
test_sideloads_after_rebuild.js
test_system_upgrades.js
test_systemaddomstartupprefs.js
test_update_isPrivileged.js
toolkit/components/extensions/test/xpcshell/
test_ExtensionStorageSync_migration_kinto.js
test_ext_schemas_manifest_permissions.js
test_ext_schemas_privileged.js
test_ext_startup_cache_telemetry.js
test_extension_permissions_migration.js

The Bugbug bot thinks this bug should belong to the 'Core::Gecko Profiler' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Sorry Bugbug, I think the "profiles" in the description refer to Firefox user profiles, not Gecko Profiler profiles.
Moving bug back to the WebExternsions product.

What many of these tests have in common is that they involve updated extension schemas.
I suspect that the schema updated are not correctly processed because part of the schemas / registrations are stored in StartupCache.

In bug 1783828, I have shown that the consequence of this is that any change to an extension API schema is going to result in a test failure, and that the failure may disappear once the conditioned profile has been updated in 1~2 days. E.g. because of:

• Schemas.loadCachedSchemas - https://searchfox.org/mozilla-central/rev/c0bed29d643393af6ebe77aa31455f283f169202/toolkit/components/extensions/Schemas.jsm#3685-3695
• apiManager.getModuleJSONURLs, called by lazyInit - https://searchfox.org/mozilla-central/rev/c0bed29d643393af6ebe77aa31455f283f169202/toolkit/components/extensions/ExtensionParent.jsm#166-181
  • e.g. test_ext_isPrivileged.js modifies the registration, which would be skipped if the StartupCache is read.
  • the general issue affects tests from bug 1757168.

This is a condprof test-only issue. In Firefox, the StartupCache is wiped whenever a new version is released. That logic is triggered from https://searchfox.org/mozilla-central/rev/c0bed29d643393af6ebe77aa31455f283f169202/toolkit/xre/nsAppRunner.cpp#5178-5181, e.g. wiped when the app version info in the profile directory's compatibility.ini does not match the application version. The caller of this is ultimately XRE_main, which is called by Firefox on desktop and Android, but not by xpcshell (XRE_XPCShellMain).

To test whether the startupcache may be affecting test results, run the condprof tests with MOZ_PURGE_CACHES=1 environment variable to forcibly clear the caches.