When addon-signing lands starting in beta 41, releng will be creating two flavors of each build: one that requires addons (that will be shipped to end users) and one which doesn't, which can be used by addon authors to develop and test addons. Addons are used in much of our test automation, so we're going to have to figure out how to get test addons (which are in the tree, and which are subject to some amount of churn) signed for test automation; see bug 1132995. We're also going to have to decide which tests to run on each type of build. Until bug 1132995 is resolved (which is likely to be tricky), there are a significant number of tests which will only work on the unsigned-addons-allowed build. As bug 1132995 and any related harness changes are addressed, I envision migrating testing to the signed-addons-required build, leaving only a small subset of tests running on the unsigned-addons-allowed build, to make sure we don't break it.

Dave, do you have any preference or comments here?

When this was first discussed in February we discussed splitting the add-ons manager tests which constitute the bulk of the extension related tests into their own suite so we could run them only on the unbranded builds (bug 1136892). That would probably get us to running the rest of the tests on the normal builds fastest.

Splitting them into a subsuite seems pretty doable, like we have with the devtools tests.

(In reply to Dave Townsend [:mossop] (PTO till July 21st) from comment #2) > When this was first discussed in February we discussed splitting the add-ons > manager tests which constitute the bulk of the extension related tests into > their own suite so we could run them only on the unbranded builds (bug > 1136892). That would probably get us to running the rest of the tests on the > normal builds fastest. Great, we'll do that. It doesn't help a lot with getting the rest of the tests running on normal builds, since many of those other tests rely on addons incidentally, and so we'll have to implement a solution for test addon signing one way or the other. But, it does give us a small set of tests we can run on unbranded builds.

Thanks jgriffin. Who from the ateam side of things will be working on this so we can coordinate with them?

I'm working on bug 1136892 and will handle general coordination. At this point, it isn't clear what additional work will be needed, but you can use me as a contact point and I'll loop in other people as necessary.

This is in progress in a group of other bugs; see the meta bug at bug 1233200.