https://firefox-ci-tc.services.mozilla.com/tasks/ZJLcv00LQPiXDZEFYd6I1Q
https://firefoxci.taskcluster-artifacts.net/ZJLcv00LQPiXDZEFYd6I1Q/1/public/logs/live_backing.log

Reading package lists... Done

+ snapcraft
/snap/bin/snapcraft: 3: exec: /snap/snapcraft/current/usr/bin/python3: not found
[taskcluster 2022-08-02 22:13:29.378Z] === Task Finished ===
[taskcluster 2022-08-02 22:13:29.447Z] Unsuccessful task run with exit code: 127 completed in 141.528 seconds

release-snap-repackage-firefox was running fine on beta on July 31.

Maybe related to the image rebuild, https://firefox-ci-tc.services.mozilla.com/tasks/DTppit47RNmcBdHe72QGLw ?

gbrown@mozpad2:~/src$ ./mach taskcluster-load-image --task-id DTppit47RNmcBdHe72QGLw
Downloading from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DTppit47RNmcBdHe72QGLw/artifacts/public/image.tar.zst
e722d396f503: Loading layer [==================================================>]  65.52MB/65.52MB
2280e40b0eab: Loading layer [==================================================>]  3.518GB/3.518GB
Loaded image: firefox-snap:latest
Found docker image: docker.io/library/firefox-snap:latest
Try: docker run -ti --rm docker.io/library/firefox-snap:latest bash
gbrown@mozpad2:~/src$ docker run -ti --rm docker.io/library/firefox-snap:latest bash
root@e774abc43394:/scripts# python3
Python 3.6.9 (default, Jun 29 2022, 11:45:57) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
root@e774abc43394:/scripts# snapcraft
/snap/bin/snapcraft: 3: exec: /snap/snapcraft/current/usr/bin/python3: not found
root@e774abc43394:/scripts# ls -l /snap/snapcraft/current/usr/bin/py*
-rwxr-xr-x 1 root root    7816 Jul  8 11:26 /snap/snapcraft/current/usr/bin/py3clean
-rwxr-xr-x 1 root root   12123 Jul  8 11:26 /snap/snapcraft/current/usr/bin/py3compile
lrwxrwxrwx 1 root root      31 Mar 13  2020 /snap/snapcraft/current/usr/bin/py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx 1 root root       9 Mar 13  2020 /snap/snapcraft/current/usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5494544 Jul  8 11:28 /snap/snapcraft/current/usr/bin/python3.8
lrwxrwxrwx 1 root root      10 Mar 13  2020 /snap/snapcraft/current/usr/bin/pyvenv -> pyvenv-3.8
root@e774abc43394:/scripts# /snap/snapcraft/current/usr/bin/python3  
bash: /snap/snapcraft/current/usr/bin/python3: No such file or directory
root@e774abc43394:/scripts# /snap/snapcraft/current/usr/bin/python3.8
bash: /snap/snapcraft/current/usr/bin/python3.8: No such file or directory
root@e774abc43394:/scripts# file /snap/snapcraft/current/usr/bin/python3.8
/snap/snapcraft/current/usr/bin/python3.8: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /snap/core20/current/lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=1f3df9df2b5e575fdee41890fe17f6de614f93f6, stripped

:jlorenzo - I'm not sure this is making sense! Do you have an idea of what's gone wrong?

Thank you very much for the ping and for all the pointers 👍 TIL ./mach taskcluster-load-image 💯

First lead followed: core20

I ran the image image on my machine and interestingly, I have an additional log line:

root@8d5161d86921:/scripts# /snap/snapcraft/current/usr/bin/python3
qemu-x86_64: Could not open '/snap/core20/current/lib64/ld-linux-x86-64.so.2': No such file or directory

Is core20 the actual core issue?

I don't know for sure whether that's because I'm running an M1 Macbook or if that's really tied to the issue we see. I'd lean towards the latter because of a few additional things:

1. /snap/core20/[...] doesn't exist in the image but /snap/core18/[...] does!
2. Per the TC index[1], the firefox-snap image has been built 3 times for the past year. The last 2 being in the last week and the first one being exactly 1 year.
3. When comparing the logs between last year and this week, I see one particular difference around core20. Let's deep dive into them!

snapcraft: I Know What You Did Last Summer[2]

We install snapcraft by using its snap package because back in bug 1591751, we needed the latest version of snap. Luckily, we log that version of snap is installed thanks to bug 1655669. Based on this: here's what snapcraft we installed this week[3]:

Running: [/bin/sh -c bash download_and_install_snap.sh 'snapcraft'] 
[..]
++ curl --header 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable'
+ SNAP_METADATA='{
    [...]
    "base": "core20",
    "last_updated": "2022-07-08T11:29:16.074763+00:00",
    "revision": 7923,
    "version": "7.0.11",
}'
[...]

And here's what we got last year[4]:

Running: [/bin/sh -c bash download_and_install_snap.sh 'snapcraft'] 
[...]
++ curl --header 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable'
+ SNAP_METADATA='{
    "base": "core18",
    "last_updated": "2021-06-18T22:03:42.534110+00:00",
    "revision": 6596,
    "version": "4.8.3",
}'
[...]

Thus, something is wrong with the version of core.

Potential fixes

The immediate fix is likely to bump this line[5] to get this unblocked. A longer-term fix could be to stop hardcoding the version of core and reading it from the metadata at https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable. If we do so, we will likely have to perform the same change for all snap-flavored dependencies of snapcraft. I also see we're building our snap on Ubuntu bionic (aka 18.04 LTS) which gets no additional free support in April. We may want to bump the snap to jammy, the latest LTS.

[1] https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.cache.level-3.docker-images.v2.firefox-snap.pushdate
[2] https://www.imdb.com/title/tt0119345/ 😁
[3] https://firefoxci.taskcluster-artifacts.net/DTppit47RNmcBdHe72QGLw/0/public/logs/live_backing.log
[4] See attached log - the one hosted on Taskcluster is about to expire.
[5] https://searchfox.org/mozilla-central/rev/560bbebd595cf2f6a9861f3f67dd0739f4c4c559/taskcluster/docker/firefox-snap/Dockerfile#40

I don't remember, do we still use the snap repackages now that Canonical ships Firefox for us?

(In reply to Johan Lorenzo [:jlorenzo] from comment #5)

I don't remember, do we still use the snap repackages now that Canonical ships Firefox for us?

We still build locally for CI/testing, but ours don't ship.

FWIW, it appears that core20 is being used for the upstream snap builds as well, all the way down the line to ESR91.
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/tree/snapcraft.yaml#n7

Appears they've been on core20 since the initial repo setup.

Thanks for the information, Mike! 😃 Then, I'm looking into it to see if we can get repackages back on feet soon. The fact that core20 is used at Canonical is promising, thank you for the pointer, Ryan!

Upstream appears to be testing core22 as well:
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/log/?h=core22

(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)

FWIW, it appears that core20 is being used for the upstream snap builds as well, all the way down the line to ESR91.
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/tree/snapcraft.yaml#n7

Appears they've been on core20 since the initial repo setup.

It's true that we've been using older versions of core and gnome than Canonical to package Firefox, but this bug is about something different: we're using the wrong hard-coded versions of those snaps for snapcraft itself, because we're “installing” it in the Docker image by manually downloading the snap files and unpacking the squashfs images, then doing some hacks to (apparently?) compensate for how snapd would have set up the filesystem environment. Both the dependency information and the hacks (see the part of D153734 about making the Python interpreter work) have diverged from upstream, giving us this bug. (And none of this is pinned to specific revisions of anything, so re-running the “same” job that previously worked can fail, as seen here.)

However, I see that D153734 also updates the dependencies for packaging Firefox.

(For context, I ran into all of this a few weeks ago while trying and failing to get that Dockerfile to work locally. I meant to file some bugs about it, but at the time it wasn't clear if we were even going to continue running these repackaging jobs; for example, see bug 1297531 comment #10.)

https://hg.mozilla.org/mozilla-central/rev/c5c9d4450697

Do we need to land this on all branches?

Thank you for the thorough input, Jed! I agree with you, the original is because we hardcode the version of core for snapcraft itself. I also agree that we diverged from upstream. However, D153734 has brought us closer to upstream which includes the hack on the Python interpreter. It actually comes from the upstream repo of snapcraft[1]. I left more details about which part comes from upstream in the Phabricator revision.

Speaking of using these repacking jobs, I concur with you. It's not clear to me how they're used. Do you :jld and :gerard-majax use them? If not, I wouldn't be surprised if no one does actually and I'd be super happy to retire this job.

[1] https://github.com/snapcore/snapcraft/blob/60884d4e2c6db2441805efd1f625a682ce55af86/docker/Dockerfile#L41-L45

Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown

Beta/Release Uplift Approval Request

• User impact if declined: None to end-users. However our CI jobs will remain red on beta and release
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Low risk because we don't ship this binary anymore. It's just a way to make CI green again. Patch was tested in try.
• String changes made/needed: None
• Is Android affected?: No

(In reply to Johan Lorenzo [:jlorenzo] from comment #16)

Thank you for the thorough input, Jed! I agree with you, the original is because we hardcode the version of core for snapcraft itself. I also agree that we diverged from upstream. However, D153734 has brought us closer to upstream which includes the hack on the Python interpreter. It actually comes from the upstream repo of snapcraft[1]. I left more details about which part comes from upstream in the Phabricator revision.

Speaking of using these repacking jobs, I concur with you. It's not clear to me how they're used. Do you :jld and :gerard-majax use them? If not, I wouldn't be surprised if no one does actually and I'd be super happy to retire this job.

[1] https://github.com/snapcore/snapcraft/blob/60884d4e2c6db2441805efd1f625a682ce55af86/docker/Dockerfile#L41-L45

As I mentionned, having that in sync with Ubuntu's package and running on mozilla-central + autoland might make bug 1763188 much easier to fix.

Thanks for reminding me our conversation on Element, :gerard-majax! The long weekend made me forget about it 😅

Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown

Approved for 104.0b8

https://hg.mozilla.org/releases/mozilla-beta/rev/3997bd23e74e

Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown

Approved for 103.0.2 dot release

Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown

Also approved for the ESRs.

https://hg.mozilla.org/releases/mozilla-esr91/rev/22c240c2590a

https://hg.mozilla.org/releases/mozilla-esr102/rev/8e9222f1ef1a

I don't think I have any plans to use the CI repackage jobs themselves, but I do have some patches that use the snapcraft.yaml from those jobs to add a ./mach repackage snap command for repacking (and, optionally, installing as a Snap) a local build for development purposes.