When running the export tier at default parallelism, and tracking the time taken by each python script, one can see that the slowest RunCbindgen takes about 15s.

However, when running at -j1, the slowest is 1.94s.

What's happening is that each cbindgen is invoking cargo metadata, each of which doesn't allow another instance of cargo to run at the same time, delaying any other one that starts.
(per https://github.com/eqrion/cbindgen/blob/c83137f3662e90d312b4b69abd84f5a08d4a00e9/src/bindgen/cargo/cargo_metadata.rs#L184)

Ironically, they all invoke the same cargo metadata command, and that command, when run alone, takes 1.4s, so is the longest part of cbindgen.

Note: In the hypothetical future where we leave it to Make to handle the dependencies between code and the headers cbindgen generates, we need to account for the fact that cbindgen is going to take jobserver slots forever if cargo build is running at the same time for the normal rust code compilation.

There's another cargo invocation that probably holds the lock too:
https://github.com/eqrion/cbindgen/blob/c83137f3662e90d312b4b69abd84f5a08d4a00e9/src/bindgen/cargo/cargo_expand.rs#L70

(In reply to Mike Hommey [:glandium] from comment #1)

There's another cargo invocation that probably holds the lock too:
https://github.com/eqrion/cbindgen/blob/c83137f3662e90d312b4b69abd84f5a08d4a00e9/src/bindgen/cargo/cargo_expand.rs#L70

That one we shouldn't be using, because cargo expand is a nightly-only feature to expand macros.

This needs https://github.com/eqrion/cbindgen/pull/538, and obviously a
cbindgen update, but I think it should address this issue.

I get similar times as before with ./mach build pre-export export, but
this is on a very-parallel machine, to the point where it may not
matter... It'd be worth profiling this before landing to ensure that
this is worth it.

I'd rather avoid requiring a cbindgen upgrade. The way I was going for was to set CARGO to a wrapper script that returns the metadata that would be created first. The main problem (which also applies to your approach), is to properly refresh the cached metadata when changes happen. I'm not quite sure whether Cargo.toml + Cargo.lock cut it.

FWIW, I don't think requiring a cbindgen upgrade is a big deal. We've been deferring them for a long while, see bug 1631929... So I wouldn't avoid one just for the sake of doing so.

(In reply to Mike Hommey [:glandium] from comment #4)

I'd rather avoid requiring a cbindgen upgrade. The way I was going for was to set CARGO to a wrapper script that returns the metadata that would be created first. The main problem (which also applies to your approach), is to properly refresh the cached metadata when changes happen. I'm not quite sure whether Cargo.toml + Cargo.lock cut it.

Updating any crate version updates the version and checksum in cargo.lock, so I think it does. I could be wrong of course. cbindgen doesn't care about much more than dependency versions..

FWIW, I don't think requiring a cbindgen upgrade is a big deal

I think it is a significant deal: it requires every contributors to update (Either with "cargo install" or with "mach bootstrap" and none of them are quick) and interrupting them in their tasks.

(In reply to Sylvestre Ledru [:Sylvestre] from comment #7)

FWIW, I don't think requiring a cbindgen upgrade is a big deal

I think it is a significant deal.

Sure, but nobody has got to update it in more than three months, and we have other reasons to want to do it other than this.

Backed out changeset 8f948dd74aba (bug 1646936) for SM and Toolchain failures

Backout: https://hg.mozilla.org/integration/autoland/rev/2b36508e934ad103f4cfd7d0b2b3dd40437a9b1f

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=8f948dd74abae461ee9688ac20769d3f180d1f33

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=307636906&repo=autoland&lineNumber=606

https://hg.mozilla.org/mozilla-central/rev/564eedd27f0a