Backouts should be treated specially to only run the tasks that failed on the initial push (and possibly on pushes in-between the failed one and the backout).

Depends on D74407

I got nerd sniped by this problem and had to see if it would be terribly difficult to figure out which tasks failed on a previous push. Turns out it's not very. Now that I've satisfied my curiosity, I'm not going to have time to finish this bug up for awhile. So if anyone wants to take this and run with it please feel free!

Remaining work:

1. Figure out how to propagate the "backedoutrev" into the taskgraph, and use the existence of that value to replace appropriate optimizers with the one implemented here.
2. Ensure rev is the full 40 character hash (this value usually comes from sheriff annotations, so this may require an HGMO query)
3. Inspect all pushes in between the backout and the regression (I'd consider this follow-up fodder)

Oh, and:

4. Above patches are completely untested..

I put these patches up as a PoC, hoping someone else can run with this in the mean time.

Please verify the "every 10th push gets all tasks" still applies.

Comment on attachment 9146781 [details]
Bug 1636440 - [taskgraph] Make 'list_task_group_incomplete_tasks' more general purpose

Revision D74407 was moved to bug 1636271. Setting attachment 9146781 [details] to obsolete.

(In reply to Andrew Halberstadt [:ahal] from bug 1670002 comment #1)

Definitely a legit bug, but I'm curious to know why you'd want to run the full set of tasks on backouts? My intuition was that backouts should only run the things that were broken so that you can verify they got fixed.

1. When a backout gets pushed, 1+ issue is known but not necessarily everything which fails due to the backed out changes.
2. All missing jobs get manually triggered to verify a push is good for merging. Running only the failed tasks misses the unscheduled tasks with unknown results.
3. If tasks are omitted because they were successful in previous pushes, an overview which task ran when and which pushes have what coverage from their own or later builds and tests is missing. Armen was looking into this - I don't find a bug for it.