ubuntu1804 - docker image optimizations
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(Not tracked)
People
(Reporter: egao, Assigned: egao)
References
Details
Attachments
(4 files)
Whatever the outcome of bug 1565332, one thing is true; Debian 10 and Ubuntu 18.04 both could be somewhat larger than the existing Ubuntu 16.04 docker image.
For example:
Ubuntu 18.04: 1264.467 mb
Debian 10: 1392.757 mb
Ubuntu 16.04: I recall it being ~900 mb at some point, though it isn't 100% certain
Now, since Ubuntu 16.04 is the default image on CI, it is often cached so does not need to be downloaded then loaded into docker.
Even then, I suspect we can do better than the 1.2GB size of the newer images.
In this bug, aim for the following:
- reduce the size of the docker image, with overachieve goal being 50%
- eliminate as many support and configuration files as possible
- reduce the number of steps in the setup phase
- reduce the time taken to build, with overachieve goal being somewhere in the mid-20min mark
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 1•5 years ago
|
||
Through some clever streamlining, I was able to reduce the size of Ubuntu 18.04 docker image greatly.
Before
[taskcluster 2019-11-20 20:40:30.663Z] Downloading artifact "public/image.tar.zst" from task ID: bwK_x_HlSPGc8U8MtbLcQA.
[taskcluster 2019-11-20 20:40:35.664Z] Download Progress: 20.63%
[taskcluster 2019-11-20 20:40:40.665Z] Download Progress: 43.83%
[taskcluster 2019-11-20 20:40:45.665Z] Download Progress: 65.72%
[taskcluster 2019-11-20 20:40:50.665Z] Download Progress: 86.41%
[taskcluster 2019-11-20 20:40:55.445Z] Downloaded artifact successfully.
[taskcluster 2019-11-20 20:40:55.445Z] Downloaded 1264.381 mb
[taskcluster 2019-11-20 20:40:55.445Z] Decompressing downloaded image
[taskcluster 2019-11-20 20:41:14.236Z] Loading docker image from downloaded archive.
[taskcluster 2019-11-20 20:44:25.525Z] Image 'public/image.tar.zst' from task 'bwK_x_HlSPGc8U8MtbLcQA' loaded. Using image ID sha256:493ea02d08a09b7cc4023bc0d1e3d9ef3464e83750512604e79a0c338e14adfa.
[taskcluster 2019-11-20 20:44:25.877Z] === Task Starting ===
After
[taskcluster 2019-11-20 22:20:49.993Z] Downloading artifact "public/image.tar.zst" from task ID: LtyijhkdQ_aoMRjMLd5niQ.
[taskcluster 2019-11-20 22:20:54.995Z] Download Progress: 34.08%
[taskcluster 2019-11-20 22:20:59.994Z] Download Progress: 79.06%
[taskcluster 2019-11-20 22:21:02.301Z] Downloaded artifact successfully.
[taskcluster 2019-11-20 22:21:02.301Z] Downloaded 736.608 mb
[taskcluster 2019-11-20 22:21:02.302Z] Decompressing downloaded image
[taskcluster 2019-11-20 22:21:13.654Z] Loading docker image from downloaded archive.
[taskcluster 2019-11-20 22:22:32.625Z] Image 'public/image.tar.zst' from task 'LtyijhkdQ_aoMRjMLd5niQ' loaded. Using image ID sha256:3420c675887b3751601d04c501fe5c93df064e2e2689dfea80eb8d9036b7f3b5.
For reference, the pushes:
before: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=277271951&revision=e391131f9afe2a1c82e33d11f0aafc452dd7106e
after: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=f4386735652d3e20d2709c32b66775bfab07dcd2
Image build time has been reduced from 35+ minutes to 17 minutes.
Assignee | ||
Comment 2•5 years ago
|
||
Assignee | ||
Comment 3•5 years ago
|
||
For the initial patch I have reduced the scope of work, since the changes were becoming quite large and variables untenable.
Specifically, the big gains produced by the --no-install-recommends
require some careful analysis that may end up costing some weeks, so I have decided to remove that from the scope of part 1 patch.
The potential gains are large, leading to half the size of any typical package installation (eg. python-pip), as shown below:
default:
Need to get 99.8 MB of archives.
After this operation, 346 MB of additional disk space will be used.
with --no-install-recommends
:
Need to get 8613 kB of archives.
After this operation, 27.3 MB of additional disk space will be used.
But this caused numerous problems with browser-chrome
, xpcshell
and other test suites.
Updated•5 years ago
|
Comment 5•5 years ago
|
||
bugherder |
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 6•5 years ago
|
||
First part of the streamlining has landed, I will investigate other opportunities to save some costs for mozilla in further rounds.
Assignee | ||
Comment 7•5 years ago
|
||
Renamed to optimize docker image.
Assignee | ||
Comment 8•5 years ago
|
||
Updated•5 years ago
|
Comment 10•5 years ago
|
||
bugherder |
Assignee | ||
Comment 11•5 years ago
|
||
Comment 12•5 years ago
|
||
Comment 13•5 years ago
|
||
bugherder |
Assignee | ||
Comment 14•5 years ago
|
||
Closing as this chapter of work is done.
Comment 15•4 years ago
|
||
I think this is meant to be a 'stderr into stdout' redirection
Comment 16•4 years ago
|
||
Comment 17•4 years ago
|
||
bugherder |
Description
•