We can make Docker image creation in image_builder faster by switching to python-zstandard. I have a patch in progress. Should have something up for review in hte next hour or so.

If I had infinite time, I would have made the compression level configurable so it is easier to change/experiment with. But I didn't feel like paging in the taskgraph code to implement this properly :) FWIW, the bump from level 3 to level 7 reduces the size of the desktop-build image by >50 MB (652 MB original). If we bump the gecko-image worker to a 2xlarge or a 4xlarge, we can likely increase the compression level further for more size reductions. Something else we can consider is periodically going through and recompressing existing Docker images with a higher compression level. Not sure how difficult that is since a lot of things are keyed off hashes of content. But it is certainly tempting to just have a beefy 16+ core machine sitting around and reducing our Docker image sizes by gigabytes. Definitely follow-up territory.

@gps, quick question: Is there any reason not to use the zstd binary? We are currently using it elsewhere, like docker-worker and in future tc-worker... For QEMU images which makes our bloated docker image look very small.. Also, if you want to make it configurable just put the compression level in an env var... Then maybe someday we can get Dustin to configure different commit levels with different compression level. Ie. on try image build time matters more than it does on level-3. And try will always prefer to use images built for level-3 if available in the index. So I think that's a good way to get the best of both worlds.

It's certainly possible to use the zstd binary. There's some magic environment variable you have to set when you build it to get multi-threaded support. Other than that, it does the same thing more or less. FWIW the Python doesn't add any noticeable overhead (execution time is dominated by compression). And, the Docker image bloat for adding the new files is minimal. We can cancel it out by using `apt-get --no-install-recommends` (which I didn't do because it breaks pre-existing processes in this image). A big advantage of the Python script approach is we don't write the raw Docker image to disk before we compress it. It's certainly possible to have `curl` pipe into `zstd`. But that wasn't implemented. So I just did it in inline Python with python-zstandard. I agree coding support for an environment variable is trivial. I should just do that and leave the door open. Using separate compression levels for different trust levels to optimize for different use cases is a terrific idea!

Comment on attachment 8851159 [details] Bug 1350447 - Use python-zstandard for Docker image compression; https://reviewboard.mozilla.org/r/123538/#review126436 ::: taskcluster/docker/image_builder/download-and-compress:35 (Diff revision 1) > + cctx = zstd.ZstdCompressor(threads=-1, level=7, write_checksum=True) > + cobj = cctx.compressobj() > + with open(path, 'wb') as fh: > + for raw in r.iter_content(zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE): > + # We need to print output periodically or the task may be > + # killed due to no output. This no-output killing is a Buildbot-only phenomenon. But periodic output is good for humans, too. Maybe somewhat less than once per second though?

Comment on attachment 8852283 [details] Bug 1350447 - Use lower compression level for level 1 tasks; https://reviewboard.mozilla.org/r/124508/#review127042 Level 3 "only" compressed at 162 MB/s on a Try run. And that's with multiple threads. I suspect Docker emitting the tar file is the slowest link here. We could likely increase the compression level for the "fast" setting. Fortunately, that's a trivial one-liner now and doesn't require updating any Docker images. So I'm inclined to experiment later.

Comment on attachment 8852283 [details] Bug 1350447 - Use lower compression level for level 1 tasks; https://reviewboard.mozilla.org/r/124508/#review127046 And https://public-artifacts.taskcluster.net/XsN7M0WzRraMnx9a-t7iYw/0/public/logs/live_backing.log shows it compressing at level 3 at 70 MB/s, which is in the realm of what level 10 can do. So I guess this is capable of being very I/O bound on Docker. I'm suddenly questioning whether this optimization is worth it at all.

Comment on attachment 8852280 [details] Bug 1350447 - Use --no-install-recommends in image_builder; https://reviewboard.mozilla.org/r/124500/#review127224

Comment on attachment 8852283 [details] Bug 1350447 - Use lower compression level for level 1 tasks; https://reviewboard.mozilla.org/r/124508/#review127226 This seems like a pretty detailed optimization, and as you've seen other parts are slower. But it's easy enough, so why not keep it :)

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4354ae67829f Use python-zstandard for Docker image compression; r=dustin https://hg.mozilla.org/integration/autoland/rev/b18964528c4b Use --no-install-recommends in image_builder; r=dustin https://hg.mozilla.org/integration/autoland/rev/179234cd2b88 Use lower compression level for level 1 tasks; r=dustin

https://hg.mozilla.org/mozilla-central/rev/4354ae67829f https://hg.mozilla.org/mozilla-central/rev/b18964528c4b https://hg.mozilla.org/mozilla-central/rev/179234cd2b88