ATM we submit complete updates to balrog and let funsize generate partials. Funsize also copies the completes to a separate bucket and publishes their new location to balrog. We should let funsize publish the initial complete MAR to avoid moments when we have only complete MARs published. Funsize uses balrog to get information about both from and to MAR files. With this scenario we should use something other than balrog for the "to" MAR information. This could be either completeMarUrl buildbot property or indexed artifacts. Probably using the latter is preferable, even thought the routes are not stable yet.

Given that there is no information in the pulse message anymore which builds will get the complete updates served, we can no longer test the update process with Firefox UI Tests for builds older than 4 days. So this is a blocker for our testing matrix.

Hmmm, there is no real "from" build ID for complete updates. It supposed to fit all builds, except the ones with partials. I don't think you can use the "from" build IDs that we build partials for - they will get partial updates. In other words you need to have a build ID, which is old enough to get a complete MAR and be downloadable at the same time. Is this assumption correct?

Yes, this is totally correct. Best would be last partial - 1 day. I don't think that we should put more focus on older builds.

FYI we currently do not test any of the full update mar files which will be used for builds older then 4 days. But we indirectly check them with the fallback update tests, which gets a full mar after failing a former download.

We are actually not blocked on testing the complete mar files. We already do that as side-effect of the partial fallback update tests which run updates going back 4 days. In those cases the complete update gets downloaded via the old software update dialog and not the about window, but I don't think that this is that important here, given that the functionality of the about window is tested with the partial update downloads.

Looks like you have been working on this bug ;)

Comment on attachment 8799388 [details] As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. https://reviewboard.mozilla.org/r/84574/#review83198 Based on https://treeherder.mozilla.org/#/jobs?repo=try&revision=d09a20ea63102e51a5c39fc779ff61229d9fca46&selectedJob=28895159 this shouldn't be an issue.

Comment on attachment 8799388 [details] As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. https://reviewboard.mozilla.org/r/84576/#review83200

Pushed by Callek@gmail.com: https://hg.mozilla.org/integration/autoland/rev/be9b0021a857 As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. r=Callek

Marking leave-open because the pushed code doesn't fully fix this bug. (though bugs are cheap, so we can modify this bug and open a new one if :sfraser deems it appropriate)

I'm happy to use this one as the central bug and fire off new sub-tasks for other components of the fix

Comment on attachment 8799388 [details] As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. Approval Request Comment [Feature/regressing bug #]: 1195365 [User impact if declined]: n/a [Describe test coverage new/current, TreeHerder]: [Risks and why]: We want to improve the publishing of complete and partial MARs, and that requires an extra field for l10n repacks. Risk low as tested on mozilla-central, change is not destructive. [String/UUID change made/needed]:

(In reply to Simon Fraser [:sfraser] from comment #16) > Comment on attachment 8799388 [details] > As part of the work on bug 1195365 we need funsize to known about the MAR > URLs for the latest build without that information coming from balrog. > > Approval Request Comment > [Feature/regressing bug #]: 1195365 > [User impact if declined]: n/a > [Describe test coverage new/current, TreeHerder]: > [Risks and why]: We want to improve the publishing of complete and partial > MARs, and that requires an extra field for l10n repacks. Risk low as tested > on mozilla-central, change is not destructive. > [String/UUID change made/needed]: I'll make the call that this is releng-infra-only of a change and doesn't need a relman approval. Sheriffs can you help Simon land this on aurora please.

Comment on attachment 8799388 [details] As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. Per comment #17, take it in 51 aurora.

Comment on attachment 8802912 [details] Bump version number for bug 1195365, fix previous version pinning https://reviewboard.mozilla.org/r/87178/#review86236

Comment on attachment 8802912 [details] Bump version number for bug 1195365, fix previous version pinning https://reviewboard.mozilla.org/r/87178/#review86294

We'll need to confirm where we should turn off the uploads to balrog, and publishing of releases. Funsize is only interested in Windows/Mac/Linux builders, and only does nightlies. Since it's not getting the latest release information from Balrog any more, we can turn off the balrog updates earlier in the workflow. desktop_l10n.py is an obvious candidate, and removing https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#182 mobile_l10n.py is not, since funsize doesn't look at those builders. en-US is presumably https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050 and making sure that self.submit_balrog_updates() isn't called. Before I submit a patch, can we confirm that those are: a) The right two files (desktop_l10n.py and buildbase.py) b) The only two files we'll need to change

(In reply to Simon Fraser [:sfraser] from comment #24) > We'll need to confirm where we should turn off the uploads to balrog, and > publishing of releases. > > Funsize is only interested in Windows/Mac/Linux builders, and only does > nightlies. Since it's not getting the latest release information from Balrog > any more, we can turn off the balrog updates earlier in the workflow. > > desktop_l10n.py is an obvious candidate, and removing > https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#182 That should be one way, but better may be to remove balrog_servers from the config... https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/balrog/production.py#2 Note that funsize is not used for Thunderbird of course, so we want to make sure we don't break that. > mobile_l10n.py is not, since funsize doesn't look at those builders. > > en-US is presumably > https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050 and making sure that > self.submit_balrog_updates() isn't called. As you note, we don't want to get rid of this whole call, if nothing else because we need https://dxr.mozilla.org/mozilla-central/rev/944cb0fd05526894fcd90fbe7d1e625ee53cd73d/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050,2063-2069 for the taskcluster nightly stuff.. > Before I submit a patch, can we confirm that those are: > a) The right two files (desktop_l10n.py and buildbase.py) They are indeed right for at least one approach :-) > b) The only two files we'll need to change Depending on the exact goal/approach that sounds accurate. If you go with the "config files will modify behavior" instead you can use some of the stuff in https://dxr.mozilla.org/mozilla-central/search?q=path%3Amozharness+path%3Asingle_locale&redirect=true :-)

Comment on attachment 8806756 [details] bug 1195365 more rework to add site directories instead of plain path modifications r? https://reviewboard.mozilla.org/r/90088/#review89708

ping - is this bug still a release affecting issue?

(In reply to Jordan Lund (:jlund) from comment #28) > ping - is this bug still a release affecting issue? unping - sorry, there was some confusion, this was linked to a failed step in balrog submission. I believe that part of this has been resolved

Fixed as a result of Bug 1342392

Removing leave-open keyword from resolved bugs, per :sylvestre.