https://github.com/mozilla-releng/releng-rfcs/pulls

This commit adds a new build for OSX since there is no current PGO build type for OSX.
And calls it shippable, this mirrors that of the nightly.

This was needed since when we have job-defaults and later on a test set for
run-on-projects: by-test-platform: ...
We were ending up with both the list of by-* being extended but also
any array in that list also being extended (like default: ['a', 'b']
was getting extended to also have the new values for default)

This is not only usually wrong but very very likely not what the author wanted.

When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.

This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.

Without this the shippable builds take >20 seconds to compute the prune through new_as_old_is_high_value, locally. With this it is near instant.

This does not change the substance of what is passed down to transforms.

raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.

TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.

Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.

This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.

I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.

We are adding shippable-qr to beta because of replacing nightly too.

Autoland and inbound should have the same sets of tasks.

beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.

central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable

This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.

Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.

Jmaher indicated we do not have the test capacity to incur this as a duplicated set

https://hg.mozilla.org/mozilla-central/rev/0fe5ffc10ff8
https://hg.mozilla.org/mozilla-central/rev/e7eeb55c71ba
https://hg.mozilla.org/mozilla-central/rev/1269ffd8a411
https://hg.mozilla.org/mozilla-central/rev/37f63cecb702
https://hg.mozilla.org/mozilla-central/rev/333ad0658d94
https://hg.mozilla.org/mozilla-central/rev/f59eb3b8d0b3
https://hg.mozilla.org/mozilla-central/rev/b764eae9ff5a
https://hg.mozilla.org/mozilla-central/rev/4a98810cec0a
https://hg.mozilla.org/mozilla-central/rev/b81435806374
https://hg.mozilla.org/mozilla-central/rev/5de8fab632bc
https://hg.mozilla.org/mozilla-central/rev/1fae9a0c3d4f
https://hg.mozilla.org/mozilla-central/rev/6dfa69851019
https://hg.mozilla.org/mozilla-central/rev/8f22b80c544e
https://hg.mozilla.org/mozilla-central/rev/dc7a3140939e
https://hg.mozilla.org/mozilla-central/rev/3e1475bc7fd5
https://hg.mozilla.org/mozilla-central/rev/ea10d6f34744
https://hg.mozilla.org/mozilla-central/rev/fa9cc5631313
https://hg.mozilla.org/mozilla-central/rev/a734011ca98e
https://hg.mozilla.org/mozilla-central/rev/2eed45e23116
https://hg.mozilla.org/mozilla-central/rev/f3040492fcbf
https://hg.mozilla.org/mozilla-central/rev/26580c78cfb0
https://hg.mozilla.org/mozilla-central/rev/046333a3b9eb

Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable

Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.

In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable

This will avoid needing to rebuild the shippable builds are part of nightly
graphs, while we work on nightly promotion.

This keeps load lower while continuing to only build full symbols on release branches, due to https://searchfox.org/mozilla-central/rev/a7315d78417179b151fef6108f2bce14786ba64d/taskcluster/taskgraph/transforms/build.py#125

On Wed, March 27, 2019, 1:44 PM GMT+2, by jwood@mozilla.com.
Revisions: D22832 diff 81613 ← D22833 diff 81614 ← D22834 diff 81615 ← D23122 diff 81616 ← D23123 diff 81617 ← D23124 diff 81618 ← D23125 diff 81619 ← D23248 diff 81620 ← D23383 diff 81621 ← D23126 diff 81622 ← D23128 diff 81623 ← D23129 diff 81624 ← D23382 diff 81625 ← D23437 diff 81626 ← D23130 diff 81627 ← D24699 diff 81628 ← D24700 diff 81629 ← D24829 diff 81630 ← D24831 diff 81631 ← D24999 diff 81645
Details: We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. (255, 'applying /tmp/tmpnLJo_c\npatching file taskcluster/ci/release-snap-repackage/kind.yml\nHunk #2 FAILED at 58\n1 out of 2 hunks FAILED -- saving rejects to file taskcluster/ci/release-snap-repackage/kind.yml.rej\nabort: patch failed to apply', '')

https://hg.mozilla.org/mozilla-central/rev/ece439cc2d6d
https://hg.mozilla.org/mozilla-central/rev/cc1547a13f15
https://hg.mozilla.org/mozilla-central/rev/d7d2f0b6d43a
https://hg.mozilla.org/mozilla-central/rev/129505a33947
https://hg.mozilla.org/mozilla-central/rev/0d73807e7e2b
https://hg.mozilla.org/mozilla-central/rev/9cf5c1259a85
https://hg.mozilla.org/mozilla-central/rev/28eabae429db
https://hg.mozilla.org/mozilla-central/rev/b19d25e352b7
https://hg.mozilla.org/mozilla-central/rev/4b333865518e
https://hg.mozilla.org/mozilla-central/rev/c919da7ce3d6
https://hg.mozilla.org/mozilla-central/rev/1e2f8a5f178e
https://hg.mozilla.org/mozilla-central/rev/1eae3d963d1b
https://hg.mozilla.org/mozilla-central/rev/9c524a5f06ea
https://hg.mozilla.org/mozilla-central/rev/b91304f4d69e
https://hg.mozilla.org/mozilla-central/rev/8a1e1068d6ba
https://hg.mozilla.org/mozilla-central/rev/f75cdebc4e7b
https://hg.mozilla.org/mozilla-central/rev/76562c9a4269
https://hg.mozilla.org/mozilla-central/rev/72988abda760
https://hg.mozilla.org/mozilla-central/rev/1de2be687d8f
https://hg.mozilla.org/mozilla-central/rev/dfcd5e5299f7
https://hg.mozilla.org/mozilla-central/rev/cc36ced7b323