The code in MozbuildObject._run_make defaults to multiprocessing.cpu_count() for the number of jobs used by builds. As builds on GCP workers seem to indicate, this may not be the best thing anymore: core count is growing faster than memory, and many processes during the build require much more than 1GB of memory.

The GCP workers have 32 cores and 32GB or memory, compared to 16 cores and 32GB of memory on the AWS workers. 32GB is clearly too tight for 32 cores, and I think we should automatically downscale the number of jobs based on the amount of memory available, if it doesn't allow for, say, 1.5GB of memory per job (I'm pulling this number out of thin air, it would be good to have a better idea of how bad things are)

We may not actually need to do this. What transpires from tracking memory use on the GCP workers is that it's only really a problem for GCC-based builds. For those, 32 jobs is too much, and so is 28. I have had good success with 24. The corresponding resource usage, with memory included looks like this: https://firefoxci.taskcluster-artifacts.net/FrWtqApyQYqmluxu4Y7j6w/0/public/build/build_resources.html
There is a spike of memory usage above 85%. It's not hard to imagine how this could get close to 100%. I also think this percentage doesn't include memory caches, and considering these workers don't have any swap, it's better to keep some margin.
Fun fact, because we end up starting sccache during configure, which is not governed by the make job server, rust doesn't limit itself, and steals CPU. Other fun fact, the awful long tail is ... in large part dump_syms.

Clang-based builds, however look like this: https://firefoxci.taskcluster-artifacts.net/TrLPPCIcTbu1fYIAGozU5Q/0/public/build/build_resources.html
Memory usage barely goes over 50%.

So I think for now, this is WONTFIX.

I'd like to add a version of this, an option to reduce default parallelism for just those gcc builds.

https://hg.mozilla.org/mozilla-central/rev/6cfc84b1d1fe