As of Saturday August 29th, there are no more raptor-webext and raptor-browsertime tests running on mozilla-central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=browsertime

It seems to have started with this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=browsertime&revision=fdf95334aded3175771c724e785fb148f95ed10b

From the logs it looks like the browsertime tests are being optimized out with the push-interval-25 strategy:

[task 2020-09-01T03:02:50.248Z] Generating optimized task graph
[task 2020-09-01T03:02:50.249Z] optimize: build-docker-image-update-verify kept because of index-search (index-search)
[task 2020-09-01T03:02:50.249Z] optimize: fetch-gcc-7.4.0 removed because of dependents optimized
[task 2020-09-01T03:02:50.249Z] optimize: test-android-hw-p2-8-0-android-aarch64-shippable/opt-browsertime-tp6m-fenix-cold-amazon-wr-e10s removed because of push-interval-25 (push-interval-25)
[task 2020-09-01T03:02:50.249Z] optimize: build-docker-image-valgrind-build kept because of index-search (index-search)
[task 2020-09-01T03:02:50.249Z] optimize: build-docker-image-partner-repack kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: instrumented-build-android-api-16-shippable/opt removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-deb8-toolchain-build kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-system-symbols-mac kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-wgpu kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: fetch-nasm-2.14.02 removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: toolchain-linux64-gcc-7 removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-index-task kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-debian8-i386-packages kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-image_builder kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-deb10-toolchain-build kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: toolchain-linux64-dump-syms removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: packages-deb8-python-zstandard kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: toolchain-linux64-rust-size removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: build-android-api-16-shippable/opt removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: toolchain-linux64-binutils removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-debian10-test kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: packages-deb8-mercurial kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: fetch-wasi-sdk removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: packages-deb8-xz-utils kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: build-docker-image-condprof kept because of index-search (index-search)
[task 2020-09-01T03:02:50.250Z] optimize: fetch-mpfr-3.1.4 removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: packages-deb8-python3-defaults kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: toolchain-linux64-rust-android-1.43 removed because of dependents optimized
[task 2020-09-01T03:02:50.250Z] optimize: packages-deb8-32-xz-utils kept because of dependent tasks
[task 2020-09-01T03:02:50.250Z] optimize: test-android-hw-p2-8-0-arm7-api-16-shippable/opt-browsertime-tp6m-live-fenix-cold-amazon-wr-e10s removed because of push-interval-25 (push-interval-25)
[task 2020-09-01T03:02:50.251Z] optimize: build-docker-image-debian8-raw kept because of dependent tasks

This excerpt is from this cron task: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=cron&revision=93d8458a0c4dc91b24c34e2f446c76d80384c948&selectedTaskRun=MlWM__dxSg6EUSKxkIFiNA.0

:ahal, is it possible that your patches in that push from Bug 1659187, and Bug 1650406 could have caused this?

I don't think bug 1659187 would be the cause, if should only affect tasks that have the if-dependencies key set (and I definitely never applied it to raptor or browsertime), but I guess you never know. I'll still take a look.

Crap, this also affected the fuzzing builds. I'll ping decoder and see if he wants me to go backfill them all.

Talked to decoder, and manually adding the fuzzing builds with Add New Jobs should work for their purposes. I'll do that as soon as I get this patch up.

I'm also going to add some integration tests for central and autoland that ensure we don't break perf / fuzzing builds again. But I'll do this in a follow-up for expediency.

This also moves the tier 2 perf tasks to backstop pushes rather than every 25th
push (which we wanted to do anyway).

https://hg.mozilla.org/mozilla-central/rev/c400c4122d04