One of our long term goals is to move more testing to wpt instead of mochitests. This is great because it means we can improve compatibility with other browsers that run the same tests. Right now, however, we face the challenge that wpt does not get us coverage across all our shipped platforms. We've made great strides here and now have opt/debug and e10s/non-e10s. It would be great to have wpt tests running on b2g in automation, however, since b2g is dramatically different in how it handles apps and security. I don't think we have any other platforms that exercise app ids in our principal code or the IPC security infrastructure. We are writing a lot of wpt tests for ServiceWorker now, so this effects SW in particular. For example, see bug 1204596 comment 6. I'm tentatively marking this as ServiceWorkers-v2 blocking, but realize it may not be practical to actually block the release on this. Its more of a wish. :-)

Jonathon, James, what are the odds this could happen in Q1 of 2016? We'd like to make a push for service workers on b2g, but many of our tests are in wpt. It would help a lot of if we could run them on b2g. (Note, we expect many of them to fail there at the moment.)

Ben, we have quite a bit on our plate already for Q1. Would it be ok if we just did service worker tests from WPT? Limiting the scope gives us a better chance of being able to do this next quarter (but no promises). I am meeting with jgriffin later today and we can discuss. Will update the bug then. N-i to remind myself.

(In reply to David Burns :automatedtester from comment #2) > Ben, we have quite a bit on our plate already for Q1. Would it be ok if we > just did service worker tests from WPT? > > Limiting the scope gives us a better chance of being able to do this next > quarter (but no promises). I am meeting with jgriffin later today and we can > discuss. Will update the bug then. N-i to remind myself. For my purposes this would be fine. It feels like the hard part is infrastructure stuff, though. I would personally be happy if we got a small subset of wpt tests running on b2g with most disabled. Then we can enable our tests as we fix service workers for b2g.

So there are a few of components to this work: 1) Teaching wptrunner to launch the B2G emulators 2) Running tests on the emulators once they are launched 3) Getting the tests to give stable results 1) should in theory not be that hard since the framework for creating new targets already exists and we previously had this working on B2G devices. But my expectations around B2G are that things will take longer than anticipated for some reason. 2) is something that was working on devices in the 1.4 era. I don't know to what extent the test app we used then is still usable in modern times. 3) has historically been problematic on every new platform we've targeted, and is the part where limiting the scope to fewer tests is worthwhile so that, for example, we aren't held up on serviceworker tests trying to get navigation tests to pass. So I think that the idea of limiting the scope can help to get this done, but there are many other factors that could contrive to make this task either relatively straightforward or unexpectedly difficult.

Ben, I have spoken to jgriffin and we will organise a meeting with you next week so we can discuss this more in depth and come up with a plan for getting what you need.