Whether you are pushing actual mozilla-beta or a simulation of mozilla-central-as-beta with "MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-1}", talos fails by timing out on startup after a pair of "Add-on talos-powers@mozilla.org is not correctly signed" "Add-on pageloader@mozilla.org is not correctly signed". I traditionally dealt with that in simulation pushes by just going back to not requiring signed addons, but after I got burned by that in the 53 merge, causing a couple of days of tree closure by not detecting that some webext tests were going to fail, I'm now dealing with it by just not testing talos, because it's too much of a pain in the ass to do one "-u all -t none" push with signing enforced and a second "-u none -t all" push with it not enforced, then another pair with EARLY_BETA_OR_EARLIER unset, then two more for devedition builds whenever they start actually working if we wind up running talos on them.

:aswan, we currently run talos automation on mozilla beta, so there are no requirements there. Do we need to resign our addons with a different cert or tool?

we do not run talos on try with signed addons for easier development, you can hack this on try by removing this condition: https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/ffsetup.py#99 I am happy to add options in talos and hook them up to try, lets make sure we sort out the workflow first and all the requirements.

I'm not sure what the question is/was here. But this is a good time to point out that extensions that are not webextensions will not be loadable into release or beta firefox beginning in 57. I think we'll be carving out an exception for tests running in automation, ie those where this test is true: http://searchfox.org/mozilla-central/rev/3dc6ceb42746ab40f1441e1e659ffb8f62ae78e3/js/xpconnect/src/xpcpublic.h#612-619 If that's true for talos then we should be fine for 57. If not, we should come up with a plan very soon.

yes, we use that nasty preference in talos and I believe all other automation.

It looks like security.turn_off_all_security_so_that_viruses_can_take_over_this_computer (!!) is set to true in talos, but the MOZ_DISABLE_NONLOCAL_CONNECTIONS environment variable is not set. I'm a little confused since it gets set here: http://searchfox.org/mozilla-central/rev/1a0d9545b9805f50a70de703a3c04fc0d22e3839/testing/talos/talos/ffsetup.py#72 But I jumped into a browser instance started from Talos and ran this in the Browser Console and it returned false: Cc["@mozilla.org/process/environment;1"].getService(Ci.nsIEnvironment).exists("MOZ_DISABLE_NONLOCAL_CONNECTIONS") Running a local talos test prints out the environment that mach creates for run_tests.py but not the environment that run_tests.py creates for firefox so I'm not sure where this is going wrong.

odd, it could be that the MOZ_DISABLE_NONLOCAL_CONNECTIONS is only for the warmup run we do to initialize the profile. We need to investigate this and fix it if this isn't set for all tests (except for Q1 as we use mitmproxy and it handles dns for us) I assume we also need the additional extensions.legacy.enabled=true. We have this bug on our short list of Talos/Perf work in June. Ionut, can you try adding: setup.env['MOZ_DISABLE_NONLOCAL_CONNECTIONS'] = '1' specifically when we are not using mitmproxy (an else condition): https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/ttest.py#110 Also in config.py add extensions.legacy.enabled=true. Then run all talos tests on all platforms.

I did a try push with extensions.legacy.enabled=True, and things are looking good so far: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dd6df8f0c05a229c94e863837ceb67fa6f54032 In addition we already have the MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 set recently in bug 1370328.

:aswan, if we land extensions.legacy.enabled=True and MOZ_DISABLE_NONLOCAL_CONNECTIONS=1, then will we be able to ignore signing on mozilla-beta for web extensions?

(In reply to Joel Maher ( :jmaher) from comment #9) > :aswan, if we land extensions.legacy.enabled=True and > MOZ_DISABLE_NONLOCAL_CONNECTIONS=1, then will we be able to ignore signing > on mozilla-beta for web extensions? Yes, though note you will also need to set xpinstall.signatures.required to false. The complete list of what I think is needed is: 1. The pref security.turn_off_all_security_so_that_viruses_can_take_over_this_computer is set to true 2. The environment variable MOZ_DISABLE_NONLOCAL_CONNECTIONS is set 3. The pref xpinstall.signatures.required is set to false 4. The pref extensions.legacy.enabled is set to true To be clear, #1, #2, and #3 are needed to load an unsigned webextension #1, #2, and #4 are needed to load the talos extension(s) in 57

great, so #1, #2 are already done, and this patch does #4. for #3, we specifically do that for the tests that have web extensions, and for all try runs. so I think we will be good once this patch lands! Thanks :aswan.

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5eecee293a5c Add extensions.legacy.enabled=True to Talos. r=rwood

on mozilla-beta the failures are still happening, despite having all 4 items in place- :aswan, are there other areas where we can work on to fix this, or should I go ahead and sign the webextension?

I found an error, the profile warmup run didn't add the environment variables

Comment on attachment 8877241 [details] [diff] [review] use the env variables for the profile setup Review of attachment 8877241 [details] [diff] [review]: ----------------------------------------------------------------- Ahh, I missed that also

it seems that when using the environment I get marionette failed to connect errors, and it seems to be only when settings: MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 here is a try push highlighting this: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c9291870e61b37a62a3a2204ac93bce564045be0&filter-tier=1&filter-tier=2&filter-tier=3 here is a try push with everything else setup in the environment except for MOZ_DISABLE_NONLOCAL_CONNECTIONS: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d511491c704b71a1b6afd9e9d0f0b63362fa3d3&filter-tier=1&filter-tier=2&filter-tier=3 possibly Marionette requires other preferences to work properly with MOZ_DISABLE_NONLOCAL_CONNECTIONS? I have cc'd a bunch of people that know much more about Marionette than I do and I will start by asking :whimboo. :whimboo, would you know more about using MOZ_DISABLE_NONLOCAL_CONNECTIONS and marionette?

This is not supported yet. Please see bug 1272255.

Btw the NoSuchWindowException is thrown because you are trying to close a tab which doesn't exist anymore. As long as I don't know the code you are running through Marionette it's hard to tell something.

Just to clarify, Joel was asking about connecting to a marionette server with MOZ_DISABLE_NONLOCAL_CONNECTIONS, not about marionette based tests. To answer comment 19, I don't know if there are any bugs around that.. but I can say both mochitest and reftest use marionette with NONLOCAL. I don't know if they're doing anything extra that talos isn't, but they seem to work.

I guess the next step is to use the mozrunner test_environment to see if we get marionette working, possibly there are other environment variables we could use.

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1556492464a3 ensure we use environment variables during the profile warmup run. r=rwood

I don't believe this is fixed that patch is setting MOZ_DISABLE_NONLOCAL_CONNECTIONS to 0 but we need it to be 1 (once the referenced bug is addressed)

still more work todo on nonlocal_connections, I see a regression, I think there are improvements as well, still need more data. == Change summary for alert #7358 (as of June 16 2017 14:37 UTC) == Regressions: 8% tpaint windows7-32 pgo e10s 205.25 -> 221.32 3% tpaint windows7-32 opt e10s 272.78 -> 280.55 For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=7358

I talked to Joel and it seems like Talos is only using Marionette to install web extensions (and only in two tests). So I think the plan to get around bug 1272255 is simply to stop using marionette and instead use mozprofile to install the webextensions (once bug 1377510 is fixed). That change should be a lot less scary than trying to fix the marionette issue.

we removed marionette and now use mozprofile