Closed Bug 1659187 Opened 4 years ago Closed 4 years ago

Implement an optimization so tasks only run when their dependencies are scheduled

Categories

(Firefox Build System :: Task Configuration, enhancement, P2)

enhancement

Tracking

(firefox82 fixed)

RESOLVED FIXED
82 Branch
Tracking Status
firefox82 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(8 files, 1 obsolete file)

(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details

We have a smattering of tasks such as build-signing, upload-symbols, etc which we typically only want to run when their associated build is explicitly scheduled.

E.g, if we optimize the linux64/opt away on autoland, we don't want linux64/opt-upload-symbols being the cause of bringing it back. Historically we've gotten around this by copying the dependencies' optimization over to the current task, e.g:
https://searchfox.org/mozilla-central/rev/50cb0892948fb4291b9a6b1b30122100ec7d4ef2/taskcluster/taskgraph/transforms/upload_symbols.py#78

Though this doesn't work if the optimization specified on the dependency relies on task specific metadata (like the label). This is specifically an issue with the bugbug based optimizers (but can theoretically happen with other ones as well).

Tom had started work on an optimizer that handles this properly, and optimizes a task away if its dependencies were also optimized over in:
https://phabricator.services.mozilla.com/D80508

I'm going to take it over and finish it off.

Blocks: 1660077
No longer blocks: 1660077

I got tripped up by the docs for Graph.visit_preorder/visit_postorder which
talks about nodes linking to and from other nodes, but makes no mention of
which direction the linking happens practically (in my mind I thought "linking
to" meant going from task -> dependent).

This blurb would have helped me catch my mistake.

This is needed as future revisions will refactor the optimization process to be
two pass. So we can't be sure if a task is optimized or not until the very end.

Depends on D87771

This test takes a long time and is annoying when it's included by default.

Depends on D87772

Depends on D87775

Attachment #9171233 - Attachment description: Bug 1659187 - [taskgrpah] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers → Bug 1659187 - [taskgraph] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers
Attachment #9171233 - Attachment description: Bug 1659187 - [taskgraph] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers → Bug 1659187 - [taskgrpah] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers
Attachment #9171233 - Attachment description: Bug 1659187 - [taskgrpah] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers → Bug 1659187 - [taskgraph] Add clarifying note to the TaskGraph docstring, r?#taskgraph-reviewers
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7f1a6c8793e9 [taskgraph] Add clarifying note to the TaskGraph docstring, r=taskgraph-reviewers,aki https://hg.mozilla.org/integration/autoland/rev/a6752cee069c [taskgraph] Move optimization reason logging to the _log_optimizations function, r=taskgraph-reviewers,aki https://hg.mozilla.org/integration/autoland/rev/11f1c7a2bfe1 [taskgraph] Require '--run-slow' for test_util_chunking.py, r=taskgraph-reviewers,aki
Keywords: leave-open
Attachment #9171239 - Attachment is obsolete: true
Attachment #9171236 - Attachment description: Bug 1659187 - [taskgraph] Implement an attribute to only run tasks when a specified dependency is present → Bug 1659187 - [taskgraph] Implement an attribute to only run tasks when a specified dependency is present, r?bhearsum
Attachment #9171236 - Attachment description: Bug 1659187 - [taskgraph] Implement an attribute to only run tasks when a specified dependency is present, r?bhearsum → Bug 1659187 - [taskgraph] Implement mechanism to only run tasks when a specified dependency is present, r?bhearsum

This removes boiler plate around adding new test cases.

Depends on D88482

Attachment #9171236 - Attachment description: Bug 1659187 - [taskgraph] Implement mechanism to only run tasks when a specified dependency is present, r?bhearsum → Bug 1659187 - [taskgraph] Implement a mechanism to only run tasks when a specified dependency is present, r?bhearsum
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f7912b1443a5 [taskgraph.test] Ensure logs are displayed when a test fails, r=taskgraph-reviewers,aki https://hg.mozilla.org/integration/autoland/rev/e1c247e7155a [taskgraph.test] Convert test_optimize.py to the pytest format and use parametrization, r=taskgraph-reviewers,aki https://hg.mozilla.org/integration/autoland/rev/e57bccb01008 [taskgraph.test] Automatically set dependencies based on edges in test_optimize.py, r=taskgraph-reviewers,bhearsum https://hg.mozilla.org/integration/autoland/rev/3fa9081e76a9 [taskgraph] Implement a mechanism to only run tasks when a specified dependency is present, r=bhearsum https://hg.mozilla.org/integration/autoland/rev/c132a4b8fa7a [ci] Use 'if-dependencies' for build-signing and repackage tasks r=bhearsum
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Keywords: leave-open
Resolution: --- → FIXED
Regressions: 1672505
Target Milestone: --- → 82 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: