This is to track potential issues for the staging releases that I'm currently running as part of the 2019-01-29 upcoming mergeduty.

There's two staging releases that I attempted to run:

the future 66.0b1

Still has issues in Ship-ip with setting up partials. Seems like even though I trim the entries to just one, it still includes the default list which is auto-completed by Ship-it UI. There could be a bug in Ship-it side. I'll try again with Devedeition as initially I attempted a Firefox.

The treeherder tracking job is here while action task failing is here

the future 65.0RC

This one worked slightly smoother. Except two things:

• bouncer submission job is failing. More here but the corrupt entries that it's been complaining about is because the bouncer submission job submits

  "paths_per_bouncer_platform": {
    "win": "/firefox/releases/65.0/win32/:lang/Firefox%20Setup%2065.0.msi",
    "win64": "/firefox/releases/65.0/win64/:lang/Firefox%20Setup%2065.0.msi"
  }
  

while bouncer returns

/firefox/releases/65.0/win32/:lang/Firefox%20Installer.msi
/firefox/releases/65.0/win64/:lang/Firefox%20Installer.msi

There seems to be an inconsistency here with respect to the naming. I remember Callek pushing a patch to fix this a while ago so my first guess is that the naming was done correctly for beta but potentially not done for RC? Doubtful though since we have a single kind in taskcluster for all but I'll investigate further.

• UV is failing completely in the graph. Not sure if this is expected.

The treeherder tracking job is here with graph being here.

@tom: I haven't touched UV for a long time. Are these expected to fail for staging releases or could it be because of the partials?

The UV failing is https://hg.mozilla.org/mozilla-central/rev/4192b6560bf7 (which only affects staging and wasn't uplifted to beta). If you look at the errors, they are https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/common/updatererrors.h#38 (at least the one I checked was).

It looks like the bouncer entries already exists (from a previous staging run?) and have the old values.

(In reply to Tom Prince [:tomprince] from comment #3)

The UV failing is https://hg.mozilla.org/mozilla-central/rev/4192b6560bf7 (which only affects staging and wasn't uplifted to beta). If you look at the errors, they are https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/common/updatererrors.h#38 (at least the one I checked was).

That makes sense, I'll ignore this then.

It looks like the bouncer entries already exists (from a previous staging run?) and have the old values.

Your scenario makes much more sense. I wiped off the locations and the product, I'll be starting a new release. I expect it to be green this time.

Thanks for your help!

Latest set of staging releases are now:

• 66.0b1 with e5e5176e7fb8a9e69eb058efb8a24d00f0d63a22 on try and treeherder job
• 65.0 with 4ee312fd2454ee1f708cd0e020adb33a7c2e12d1 on try and treeherder job

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)

Your scenario makes much more sense. I wiped off the locations and the product, I'll be starting a new release. I expect it to be green this time.

Follow-up 65.0 try staging release's bouncer submission job worked smoothly after having a clean state on bouncer side.

I may do another set tomorrow morning Friday just to make sure things are fine and ready for Monday.
Or maybe just a 66.0b1 which was slightly troublous on windows-side earlier today.

Calling this done. I think we're in good shape for Monday.

I tried to create another 66.0b1 today but failed from Ship-it side when it attempted to "promote_firefox". That's because of bug 1496890 and its https://hg.mozilla.org/integration/autoland/rev/aee9f213f3c7 which changed the .taskcluster structure.

Rail is attempting a fix on Ship-it side (enrich the jsone context there to encompass the newly added clientId) so that when central -> beta on Monday, we won't be hitting this issue on production Ship-it too.

Good thing we have these try staging releases, thanks again Tom for the work in making them available.

Rail fixed the bug on ship-it side, we should have a good state on Monday for upcoming production 66.0b1