This will help track anything left.

The documenation has been refactored to take into account: * mozharness moving into the Gecko tree * the work to get the symbols stack https://wiki.mozilla.org/Auto-tools/Projects/Marionette_update_tests

Status summary ############## I assume more issues will be filed as these tests became the go to solution for release drivers. Current issues: --------------- Our main issues at the moment are: 1) bug 1176358 - Linux 64-bit jobs are crashing due to the lack of a font on the Linux _build_ machines 2) bug 1188402 - Windows build machines are failing to checkout In order to fix #1 there are various options: A) Install the font on the Linux build machines through: - A.1 - Puppet - A.2 - Using mock B) Run the jobs on the _test_ machines (Ubuntu vs CentOS) instead of the _build_ machines since they're capable of running Firefox unit tests - This would require re-testing everything To fix #2, it is a matter of investigating and having a fix for mozharness. This is the second time we regressed here. I would not be surprised at all if more issues arised as this has been the norm. After these last two issues are resolved we should enable this project for mozilla-release and esr releases. Improve testing test harness changes: ------------------------------------- Right now, we have to land changes on various trees and wait for the next beta to come along and tag the repos (which can be days) so we can see the changes being executed on Release Engineering infrastructure. Unfortunately, this cycle is too slow to make significant contributions and we easily loose context from moving to other things in the meantime. There's a chance that we could hack something together from the tree where we can push to try and impersonate build jobs as Firefox UI test jobs. This is worth exploring as it would make investigate any issues easily. Production ready: ----------------- In order for this whole project to be bullet proof and easier for release drivers: * bug 1159527 - Run Firefox UI updates tests on mozilla-central and mozilla-aurora ** This will allow catching issues earlier in the release cycle * bug 1173976 - Reduce testing matrix of Firefox UI updates tests ** We currently test all locales for all versions since Gecko 38 ** This makes getting results slower and slower after each new release * bug 1182797 - The Windows Firefox UI updates were running for many hours ** mozharness or buildbot should have Long term: ---------- Currently we report a status summary of failures at the end of the logs and this is sent over email [1] Once build promotions are running on Treeherder it will be easier to get to these logs, however, we believe that a better visualization solution is required. [1] https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/release-automation-notifications

This bug is getting a real blocker for us now given that we will not be able to run any Mozmill tests once Firefox 42.0 hits beta! All details see bug 1195975. We really have to find someone to get those remaining depending bugs fixed if Armen doesn't have any time for it. David or Jonathan, can you please give an advice or find a solution? Thanks.

FTR: Gecko 42 moves to beta on the week of September 22, 2015

I'm just going to change this bug to critical instead of blocker because it is not a bug that indicates an immediate problem with releng services being down. And in my understanding, this work is being asked of ateam. (The bug is creating noise on our #buildduty channel due to blocker status)

Given that bug 1195975 is fixed, is this still critical?

swapping needinfo to me

We got the Mozmill addons signed and working in recent nightly builds and beta, but the signature has been given to us only due to the critical situation. And it's only for a limited time, until we have our Firefox UI tests running properly. Reason is that with those signed add-ons you will be able to control the browser from the outside, which is totally not wanted. Means we can be happy about this current signature. That said we have a bit more time but we should still try to get it done in the near future.

Armen, who is driving getting this completed? We are running the Marionette tests (which report to treeherder) now for builds from all channels via the CI machines and the old process of manually triggering them, I'm really eager to get this finally moved to being run automatically for beta/release/esr builds.

catlee, at the time I picked this up it was because releng had no bandwidth. Now that there are more hires and release promotion's big hill has been crossed over; is there someone on releng that could make this part of the release process?

There's still some work to do in release promotion; perhaps we can look at this late Q2 or in Q3.

We never finished the project and already landed code will be removed. If we still need such a thing we would implement it via jobs in TC nowadays.