See https://firefox-ci-tc.services.mozilla.com/tasks/d5nmjR0BQkO5NQ-P2rNRzg
and log: https://firefoxci.taskcluster-artifacts.net/d5nmjR0BQkO5NQ-P2rNRzg/1/public/logs/live_backing.log

Priming gnome-3-34-extension
Sorry, an error occurred in Snapcraft:
A tool snapcraft depends on could not be found: 'snap'.
Ensure the tool is installed and available, and try again.

This is blocking releases.

I know what happened and a fix is currently under test on try[0].

The root cause is that snapcraft started to require the snap binary about a month ago[1]. 2 weeks ago, bug 1617369 landed on autoland. That bug is definitely unrelated to snap. That said, it changed this line[2], which slightly changed the docker image called image_builder. If image_builder changes, then all other docker images get rebuilt[3]. That's exactly what happened[4]. So 2 weeks ago, the snap docker image was rebuilt and downloaded the latest version of snapcraft which included last month's changes. It wasn't clear what version was pulled from the logs. I'm improving that.

We hadn't seen this in the past 2 weeks because the hash of image_builder on mozilla-beta wasn't changed until merge day. This is why the bug appeared today. We could have seen this issue if bug 1451694 was addressed.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c9c7004c088b4436388b860f3f94168e9466acd1
[1] https://github.com/snapcore/snapcraft/pull/3173/files#diff-84615e6204c0bf79064d03915b64c656R196
[2] https://hg.mozilla.org/integration/autoland/rev/b234115ff57eaebae0ebccff0586971db9b3edca#l4.13
[3] https://searchfox.org/mozilla-beta/rev/757c9db9ef294f402d223faebef380a31464bc46/taskcluster/taskgraph/transforms/docker_image.py#221
[4] https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=b234115ff57eaebae0ebccff0586971db9b3edca&searchStr=docker

It became mandatory in https://github.com/snapcore/snapcraft/pull/3173/files#diff-84615e6204c0bf79064d03915b64c656R196

Depends on D85150

Comment on attachment 9166557 [details]
Bug 1655669 - part 2: Install snap binary because it's now required by snapcraft r=KenVanDine,sfraser

Beta/Release Uplift Approval Request

• User impact if declined: Users won't be able to install the latest Firefox Snap
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Packaging issue that was tested on try at https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=BJ017FzoRPqa0H9LB17sgg.0&revision=4572db87de988b3323981b1e932e7bf6ad167195.
• String changes made/needed: None

Comment on attachment 9166557 [details]
Bug 1655669 - part 2: Install snap binary because it's now required by snapcraft r=KenVanDine,sfraser

approved for 80.0b1

https://hg.mozilla.org/releases/mozilla-beta/rev/a2bf14c2a751
https://hg.mozilla.org/releases/mozilla-beta/rev/854acc505256

https://hg.mozilla.org/mozilla-central/rev/0e143f95ec6d
https://hg.mozilla.org/mozilla-central/rev/2424b1a371a6

https://hg.mozilla.org/releases/mozilla-esr78/rev/a4d5d83dd05b
https://hg.mozilla.org/releases/mozilla-esr78/rev/cb919a80aa24