Currently, the bugbug reduction strategy doesn't care about the number of platforms selected (since on autoland we have all of them anyway) but just about the "cost" of the tasks.

For try, we should take into account not only the cost, but also the number of selected platforms.

We are switching to bugbug_disperse in bug 1638945 (which suffers from the same problem of scheduling too many platforms). There is some work needed in bugbug to support selecting platforms for the manifest-level model.

I'm not convinced the model should optimize for this. Builds are relatively cheap (and getting cheaper). Plus getting a bit of coverage on more platforms would likely find more regressions than limiting coverage to a few.

(In reply to Andrew Halberstadt [:ahal] from comment #2)

I'm not convinced the model should optimize for this. Builds are relatively cheap (and getting cheaper). Plus getting a bit of coverage on more platforms would likely find more regressions than limiting coverage to a few.

We can tune it however we want: e.g. run a manifest only on a single configuration if we are absolutely sure they're configuration-independent, run them on multiple configurations if we know they aren't or if we don't know yet (and, if we don't know yet, we can select the configurations in a way to collect more information for the future). Also, we can make sure that, if we select the same manifest across multiple pushes, it is run on a different configuration on each push.

The objective is not only to reduce the number of builds, but mainly to reduce the number of test executions for tests that are likely platform independent. E.g. if you had 30 platform-independent manifests and run them on 3 platforms, you'll nedlessly increase the total runtime by 3x.

Then, we'll also be able to use this work for bug 1637555.

Depends on D91605

Adding "leave-open" until we switch "mach try auto" to use the new strategy (which might require some tuning first).

https://hg.mozilla.org/mozilla-central/rev/6431da98cb37
https://hg.mozilla.org/mozilla-central/rev/07ff82216f70

Otherwise there is a mismatch between the group names in "groups" and "config_groups".

https://hg.mozilla.org/mozilla-central/rev/76a3adc2c6d8

https://hg.mozilla.org/mozilla-central/rev/7a1b37db8c74
https://hg.mozilla.org/mozilla-central/rev/fd47ecdfed71

Depends on D114363

By making the new strategy with bugbug's config selection and a low confidence threshold the
default for 'mach try auto'.

Depends on D114364

Backed out 3 changesets (Bug 1639164) for causing python flake8 and ci failures.
Backout link: https://hg.mozilla.org/integration/autoland/rev/00b9154f4315be73495d44b18af4d4b1a1a5b5ec
Push with failures, ci failure, flake failure, cram try failure.

(In reply to Alexandru Michis [:malexandru] from comment #22)

Backed out 3 changesets (Bug 1639164) for causing python flake8 and ci failures.
Backout link: https://hg.mozilla.org/integration/autoland/rev/00b9154f4315be73495d44b18af4d4b1a1a5b5ec
Push with failures, ci failure, flake failure, cram try failure.

I fixed the flake8 and tryselect test failures, which were trivial, but I still need to fix the test_mach_try_auto test.
The test is failing because we can now possibly schedule shippable builds. We were relying on bugbug_debug_disperse to filter out all non-debug platforms (shippable included), but the config selection alternative is not restricting to debug platforms anymore.
:ahal, any idea how to fix that?

I've added a filter in https://searchfox.org/mozilla-central/rev/3151f97de27730793c2e298716df760999423f26/taskcluster/taskgraph/target_tasks.py#373 to exclude shippable tasks.
There is still one problem left: we are now scheduling some build-signing tasks because they depend on xpcshell tasks. IDK why this wasn't the case with bugbug_debug_disperse.

The build-signing test was marked as failing until https://hg.mozilla.org/mozilla-central/rev/f07222b728fa426f56e269116c14b2c1e2987104.
I think it is passing with bugbug_debug_disperse just because of luck in the way the dispersion was done (and f07222b728fa426f56e269116c14b2c1e2987104 changed it). Indeed, if I increase the dispersion across more platform, the test fails.

So, I'll just mark it as failing.

Depends on D114365

Depends on D114489

https://hg.mozilla.org/mozilla-central/rev/3053350a4a94
https://hg.mozilla.org/mozilla-central/rev/99b4cc4801ae
https://hg.mozilla.org/mozilla-central/rev/90f0b462690c
https://hg.mozilla.org/mozilla-central/rev/fff1d447ffd1
https://hg.mozilla.org/mozilla-central/rev/7b6be863db85