The mach artifact install process, invoked at build-time during an artifact build here, does several things:

1. it guesses the automation job that can provide artifacts
2. it interprets local and remote state to find a candidate task
3. it fetches and caches artifacts from the candidate task
4. it ensures that the local object directory reflects the state of the candidate task (as far as we are able).

For historical reasons, everything here is done at build-time. I think this is mostly because we didn't have Python configure, so trying to do almost any of this at configure-time just wasn't feasible. But the world has moved on, and I propose that we move at least steps 1. and 2. into Python configure.

I see two advantages to doing this, the first major, the second minor:
a. we can provide the current (consuming) configure invocation with details of the artifact (consumed) configure invocation
b. we can remove some non-determinism and speed up ./mach build itself.

I envision changing artifact_builds from a simple boolean to some representation of the consumed config.status. This would allow artifact builds to, for example:

• ensure that critical settings such as whether the crashreporter is enabled are coherent (Bug 1748626, Bug 1728226)
• adopt upstream API key values

In the future, I think this could provide a path for consuming artifact builds to extract build inputs from consumed artifacts. For example, such as we'd like for Bug 1580919, and almost any place that bridges the compiled/interpreted gap.

The one disadvantage I see to doing this is that we pull some online (network requiring) and potentially slow configuration earlier in the build process into configure. But we're already doing a lot of that with --enable-bootstrap; this work needs to be done at some point; and by pulling forward we reduce the number of times we need to do the work.

I've done a little investigation into this, and the existing artifacts.py implementation doesn't need much from the build configuration. It's really just what's in _guess_artifact_job, which can be done -- after some small changes -- pretty early in /moz.configure. We might run the task-determining bits of artifacts.py, capture the result in some way -- perhaps always setting MOZ_ARTIFACT_TASK_ID, or perhaps using MOZ_ARTIFACT_URL of Bug 1750109 -- and then feed those details into the mach artifact install invocation.

glandium: mhentges: what do you think about this proposal?

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Mach Core' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

For historical reasons, everything here is done at build-time. I think this is mostly because we didn't have Python configure

Interestingly, python configure did exist when it was moved from config_status.py (where it was since the beginning) to Makefile.in in bug 1275111. It seems the reason to keep it out of configure back then was configure time telemetry. Considering auto-bootstrap in configure, we more or less don't care about that anymore (which is what you noted in your last paragraph)

(In reply to Mike Hommey [:glandium] from comment #4)

For historical reasons, everything here is done at build-time. I think this is mostly because we didn't have Python configure

Interestingly, python configure did exist when it was moved from config_status.py (where it was since the beginning) to Makefile.in in bug 1275111. It seems the reason to keep it out of configure back then was configure time telemetry. Considering auto-bootstrap in configure, we more or less don't care about that anymore (which is what you noted in your last paragraph)

Wow, I had completely forgotten that detail.

Can I assume that you're at worst neutral and hopefully +1 to this idea?

I'll give you a tentative 👍:

• I agree that having network shenanigans in configure is no longer a faux pas, especially now that we're bootstrapping in there
• Your point around being able to influence configure based on the settings of the "consumed" artifact makes sense, and I don't have the perspective to be able to offer alternative solutions for that problem - this seems to make sense

Thanks for pushing this forward :)

After sleeping on this for a few days, I don't think I have any objections. I'm curious what your thought are about how to consume the output for configure (config.status) as pseudo-input for local configure, when config items don't necessarily match inputs, and the complications from the lack of values in config.status that are equivalent to "False" in practice, cannot be translated into an input, while the input may have a different default in local builds. (also, you made me realize that the config.status artifacts contain the api keys, and I didn't think about that at all when I exposed config.status, and at least in theory, we should ensure they don't appear there)

(In reply to Mike Hommey [:glandium] from comment #7)

After sleeping on this for a few days, I don't think I have any objections. I'm curious what your thought are about how to consume the output for configure (config.status) as pseudo-input for local configure, when config items don't necessarily match inputs, and the complications from the lack of values in config.status that are equivalent to "False" in practice, cannot be translated into an input, while the input may have a different default in local builds.

I have not thought about such details in any meaningful way. I was intending to turn artifact_builds from a boolean to some structure like a BuildConfig produced from the upstream config.status, completely ducking the issue of trying to map some variable MOZ_DEBUG back to a meaningful setting or default like --enable-debug. That is, when inspecting the consumed build, you'd write code that looks more like moz.build inspecting CONFIG than it looks like moz.configure with its subtle graph of @depends functions.

(also, you made me realize that the config.status artifacts contain the api keys, and I didn't think about that at all when I exposed config.status, and at least in theory, we should ensure they don't appear there)

Mmm. Since fishing API keys was one of my use cases, that's unfortunate. We have no guarantee that said keys aren't inlined into compiled code, but they probably aren't and it's not worth a great deal of time or effort to make this work for artifact builds.

My motivation for considering this in the first place is to turn single-locale l10n repacks into artifact builds, and one of the differences in my WIP is exactly these API keys. But I'll be able to expose them to the l10n jobs in the same way they're exposed to the build jobs, so this isn't a hard blocker.