Due to the changes around signing of binaries with SHA-2 instead of SHA-1, we did set up Windows XP users to always get a "watershed" build from downloads instead of the very recent build (which would fail to install at least on XP SP2), and people need to update from there to the recent build (updating to new builds and running them works fine, only the installer fails). We need to ensure this workflow is not broken as long as we support Windows XP, so we need to set up some testing for this. Requirements: 1) Needs to be fully automated. 2) Needs to run for every new build we release on the affected channels (release, beta, esr - aurora/DevEd if possible, nightly is optional). 3) At least the en-US locale needs to be tested, representative locales would be nice to have. 4) Needs to test that updates from the watershed build to the new build work. 5) Would be nice to test if the "bouncer" download.m.o URL actually serves the expected watershed build. For the moment, our usual update tests for release, beta, and I think esr still cover this case at least to some extent, but when the watershed builds fall out of the normal testing regimen, it will not be. Until then (probably end of this quarter), we will need to have a solution to this bug up and running.

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #0) > Requirements: 0) Implicit, but best to state it here, this needs to run on WinXP.

Sylvetre, Do you have someone that can do this? We are happy to mentor but currently don't have the bandwidth to create new test.

To give more details what would be necessary to do: * If needed new Windows XP SP2 VMs would have to be created and configured (If not we could re-use the 4 Windows XP SP3 VMs) * We could reuse our update tests but only for a single step update! We still don't cover multiple update steps. * Serving updates from those old builds will mean complete MAR files. Given that we do no longer know when those are getting created (not via the build message and not via the funsize messages because those don't support full mar files yet), we would have to investigate with RelEng about a possible solution, and have to also add special code to our pulse listener to always trigger update tests for those old builds on WinXP. * We would somehow have to mark those test results on Treeherder. This would require additional work to be done.

(In reply to Henrik Skupin (:whimboo) from comment #3) > To give more details what would be necessary to do: > > * If needed new Windows XP SP2 VMs would have to be created and configured > (If not we could re-use the 4 Windows XP SP3 VMs) We don't need SP2 VMs for this, we can just have this run on the SP3 ones - ALL of WinXP gets the same builds. > * We could reuse our update tests but only for a single step update! We > still don't cover multiple update steps. There is no multipstep update in those requirements! > * Serving updates from those old builds will mean complete MAR files. Given > that we do no longer know when those are getting created (not via the build > message and not via the funsize messages because those don't support full > mar files yet), we would have to investigate with RelEng about a possible > solution, and have to also add special code to our pulse listener to always > trigger update tests for those old builds on WinXP. Not sure what you mean with that, the test doesn't need to care which MAR files the update gets. > * We would somehow have to mark those test results on Treeherder. This would > require additional work to be done. I never expected this to go without someone needing to do some work.

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4) > > * Serving updates from those old builds will mean complete MAR files. Given > > that we do no longer know when those are getting created (not via the build > > message and not via the funsize messages because those don't support full > > mar files yet), we would have to investigate with RelEng about a possible > > solution, and have to also add special code to our pulse listener to always > > trigger update tests for those old builds on WinXP. > > Not sure what you mean with that, the test doesn't need to care which MAR > files the update gets. The fixed (old) Firefox build you want to test updates from, will not change. So after 4 days it will only get full updates offered. As I said we do not get any notification when such an update has been made available. And therefor cannot really test it. So I think we need bug 1195365 fixed first in some way. > > * We would somehow have to mark those test results on Treeherder. This would > > require additional work to be done. > > I never expected this to go without someone needing to do some work. And my plate is already pretty full this quarter. Usually our team also expects a heads-up with details at least during the former quarter so we can figure out if it's doable for us. This didn't really happen here, so we should let the managers decide that. The above information I only put in here, so you can see which kind of work would be necessary to do, and where we depend on.

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #0) > 1) Needs to be fully automated. > 2) Needs to run for every new build we release on the affected channels > (release, beta, esr - aurora/DevEd if possible, nightly is optional). release, beta, and esr you can cut out of this list. We cannot run updates for releases fully automated in our system. So if you want that we would need bug 1182796 fixed.

David, I think that Softvision can help with that! Florin, do you know who could help with that?

(In reply to Sylvestre Ledru [:sylvestre] from comment #7) > David, I think that Softvision can help with that! Florin, do you know who > could help with that? Sorry guys, we currently only do manual testing - we lost our automation team about a year ago :). So as long as the request here is to create an automated test we cannot help with that.

(In reply to Henrik Skupin (:whimboo) from comment #5) > The fixed (old) Firefox build you want to test updates from, will not > change. So after 4 days it will only get full updates offered. As I said we > do not get any notification when such an update has been made available. And > therefor cannot really test it. So I think we need bug 1195365 fixed first > in some way. Don't we get messages when a new build is available on the update channels? At that point all updates including the full MARs will be available. > And my plate is already pretty full this quarter. Usually our team also > expects a heads-up with details at least during the former quarter so we can > figure out if it's doable for us. This didn't really happen here, so we > should let the managers decide that. The above information I only put in > here, so you can see which kind of work would be necessary to do, and where > we depend on. Well, I only got the information that we have that watershed in mid December and the information that we need to run tests around the holidays (which is when I first emailed you), so sorry I couldn't give you any prior warning as I didn't know about it myself. The whole story about this SHA-1/SHA-2 binary signing stuff was very rushed and coming at us at the last minute. (In reply to Florin Mezei, QA (:FlorinMezei) from comment #8) > So as long as the request here is to create an automated test we cannot help > with that. Machine hours are much cheaper than human hours and I'm pretty sure that setting up those tests will take a fraction of the time it would take us over the next years to do manual testing on this, so yes, I strongly believe this should be done with automation. I don't care if it runs on which part of our infrastructure this runs on, I mainly care about ESR/release/beta there, aurora/DevEd is optional and nightly a nice-to-have. But we will need to have this running in some form.

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #9) > Don't we get messages when a new build is available on the update channels? > At that point all updates including the full MARs will be available. We get but your assumption is no longer valid. Since funsize is producing updates they will appear after the build has been made available. We do not know after which interval. And as said, we do not get notifications yet for complete patches, which we would need here. > I don't care if it runs on which part of our infrastructure this runs on, I > mainly care about ESR/release/beta there, aurora/DevEd is optional and > nightly a nice-to-have. But we will need to have this running in some form. If you really only care about releases and betas then you should definitely talk to RelEng. There is nothing we can do here in mozmill-ci. I feel this is a very good time to finally get our update tests setup on the RelEng systems!

(In reply to Henrik Skupin (:whimboo) from comment #10) > If you really only care about releases and betas then you should definitely > talk to RelEng. There is nothing we can do here in mozmill-ci. I feel this > is a very good time to finally get our update tests setup on the RelEng > systems! Fine, I don't care where they run, just that they run. Maybe getting this up as a first step is even better than the normal set of update tests.

May I ask how this decision plays into the process? https://blog.mozilla.org/security/2016/01/06/man-in-the-middle-interfering-with-increased-security/

This has no direct connection at all. Signing of Windows installers is a completely different matter from what HTTPS connections we support on our side.

Not needed anymore, and Mozmill-CI will be shutdown on Monday next week. So this is not needed anymore.