Once we have it running for the release process we should look into adding to mozilla-aurora to catch issues as soon as possible.

whimboo is looking into porting his Firefox UI updates to use mozharness. That will benefit this bug. We can use this bug for scheduling it on Releng infra once the porting works on his side.

Rail, you should be good to go by integrating the update tests into the funsize partial generation jobs.

(In reply to Henrik Skupin (:whimboo) from comment #2) > Rail, you should be good to go by integrating the update tests into the > funsize partial generation jobs. Can you elaborate on this? Do you have any example on how to use them? Can you file a bug with some details under https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=General%20Automation&blocked=1149142

Alright, here some details... Also I don't see why a new bug needs to be filed given that this one is already in the right component. So I will just add the blocking dependency. The update tests can be triggered by the `update.py` script as located under the directory as listed below. Please don't obey `update_release.py` which is for releases only. http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/firefox_ui_tests/ Here you can find an example of how to use the script: scripts/firefox_ui_tests/update.py --cfg firefox_ui_tests/qa_jenkins.py --firefox-ui-branch mozilla-central --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/09/2015-09-22-03-02-04-mozilla-central/firefox-44.0a1.en-US.linux-x86_64.tar.bz2 --update-target-buildid %to_buildid% --update-target-version %version% [--update-channel nightly] * Basically we feed the script with a specific config which contains some settings for our machines. You should have a look at. I assume you might need the same for taskcluster so we could create a duplicated config file for you in the same folder. * The branch of the tests have to be specified so that the script operates on the correct versions. It will be mozilla-central or mozilla-aurora depending if you use a Nightly or Aurora build. * A bit more complicated is the installer URL. It will be the build as given by %from_buildid%. In our case we use mozdownload to fetch the URL given that it is not part of the via funsize created manifest. The appropriate code you can find here: https://github.com/mozilla/mozmill-ci/blob/master/pulse.py#L132. But maybe you have a better way of doing it. * --update-target-buildid is optional but if specified it will check if the updated build has the correct build id. I would suggest that you make use of it. * --update-target-version is optional but if specified it will check if the updated build has the correct build id. I would suggest that you make use of it. * --update-channel is only necessary if you have to test on a different channel. Given that we want to run those tests before we have them on the nightly channel you might wanna use it with `nightlytest` or whatever you have in mind. The script returns with an exit code != 0 in case of a failure. Detailed results of the run will be located in the log files under build/upload/logs. Let me know if you need more information in how to run those tests.

Thank you for the explanations! Sounds like a plan. :) These tests should detect issues with already published updates. I wonder if we can also check them before publishing. Maybe publish to a staging update server, run the tests against the staging server, then publish the updates for reals. And maybe test again. :) But this is definitely not a blocker to use the current approach.

That's correct. We can file a follow-up bug for test runs which gets executed before the publishing step. Getting the current tests run via Releng will help a lot.

We never finished the project and already landed code will be removed. If we still need such a thing we would implement it via jobs in TC nowadays.