Closed Bug 1455143 Opened 7 years ago Closed 7 years ago

Improvements to upload make target

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Version:
3 Branch
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox61 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla61
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: gps, Assigned: gps)

References

(Blocks 2 open bugs)

Details

Attachments

(11 files)

Bug 1455143 - Make hashlib required;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Remove hash name validation;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Use a global logger instance;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Handle directory inputs earlier;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Remove code for failing to obtain a hash;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Use a reasonable buffer size for reading files;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Use .write() instead of print >>;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Refactor checksumming to occur after upload.py;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Don't remove common test archive;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Generate test archives into final path;
(deleted), text/x-review-board-request
ted
: review+
Details
Bug 1455143 - Stop beetmoving deleted testing artifacts
(deleted), text/x-review-board-request
gps
: review+
mtabara
: review+
Details
The "upload" make target is taking a few minutes on Windows. We can do better.
Attachment #8969112 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969112 [details] Bug 1455143 - Make hashlib required; https://reviewboard.mozilla.org/r/237810/#review243576
Attachment #8969112 - Flags: review?(ted) → review+
Attachment #8969113 - Flags: review?(core-build-config-reviews)
Attachment #8969114 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969114 [details] Bug 1455143 - Use a global logger instance; https://reviewboard.mozilla.org/r/237814/#review243920
Attachment #8969114 - Flags: review?(ted) → review+
Attachment #8969116 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969116 [details] Bug 1455143 - Remove code for failing to obtain a hash; https://reviewboard.mozilla.org/r/237818/#review243964
Attachment #8969116 - Flags: review?(ted) → review+
Attachment #8969117 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969117 [details] Bug 1455143 - Use a reasonable buffer size for reading files; https://reviewboard.mozilla.org/r/237820/#review243966
Attachment #8969117 - Flags: review?(ted) → review+
Attachment #8969118 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969118 [details] Bug 1455143 - Use .write() instead of print >>; https://reviewboard.mozilla.org/r/237822/#review243968
Attachment #8969118 - Flags: review?(ted) → review+
Attachment #8969119 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969119 [details] Bug 1455143 - Refactor checksumming to occur after upload.py; https://reviewboard.mozilla.org/r/237824/#review243972
Attachment #8969119 - Flags: review?(ted) → review+
Attachment #8969115 - Flags: review?(core-build-config-reviews)
Attachment #8969113 - Flags: review?(core-build-config-reviews)
Attachment #8969114 - Flags: review?(core-build-config-reviews)
Attachment #8969112 - Flags: review?(core-build-config-reviews)
Attachment #8969440 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969440 [details] Bug 1455143 - Don't remove common test archive; https://reviewboard.mozilla.org/r/238194/#review243988
Attachment #8969440 - Flags: review?(ted) → review+
Attachment #8969441 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969441 [details] Bug 1455143 - Generate test archives into final path; https://reviewboard.mozilla.org/r/238196/#review243990 File a followup on fixing this for everything else? Aside from setup.exe, everything we pass in `UPLOAD_FILES` gets generated after the build.
Attachment #8969441 - Flags: review?(ted) → review+
Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4c987b999ff1 Make hashlib required; r=ted https://hg.mozilla.org/integration/autoland/rev/37fec6042438 Remove hash name validation; r=ted https://hg.mozilla.org/integration/autoland/rev/e6d309fde362 Use a global logger instance; r=ted https://hg.mozilla.org/integration/autoland/rev/2bbe78c011d3 Handle directory inputs earlier; r=ted https://hg.mozilla.org/integration/autoland/rev/c9cbbb881c78 Remove code for failing to obtain a hash; r=ted https://hg.mozilla.org/integration/autoland/rev/1f1186400490 Use a reasonable buffer size for reading files; r=ted https://hg.mozilla.org/integration/autoland/rev/1fa5254b9a69 Use .write() instead of print >>; r=ted https://hg.mozilla.org/integration/autoland/rev/12cfbcd2ccf4 Refactor checksumming to occur after upload.py; r=ted https://hg.mozilla.org/integration/autoland/rev/b604acdb1c78 Don't remove common test archive; r=ted https://hg.mozilla.org/integration/autoland/rev/7f69dd1ba3cb Generate test archives into final path; r=ted
These changes broke the beetmover jobs[1] in the latest nightly. The reason is, the artifacts aren't produced anymore in the build task under the "multi locale" folder. Therefore, ChainOfTrust bails out as they don't exist. My patch removes the artifacts offending artifacts. :gps, could you check I didn't miss one? :mtabara, this also removes the artifacts from the "ship" graphs, which is expected, I think. [1] Like https://tools.taskcluster.net/groups/FJ_74wd1Sl6jUpkObCBPGQ/tasks/Bph5uS4BQlinnQQ6snCcvg/runs/0/logs/public%2Flogs%2Fchain_of_trust.log#L219
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8969656 [details] Bug 1455143 - Stop beetmoving deleted testing artifacts https://reviewboard.mozilla.org/r/238450/#review244216 LGTM. I was gonna suggest a cleanup in beetmoverscript as well but I guess we still need these values within the templates mappings for the en-US counterparts.
Attachment #8969656 - Flags: review?(mtabara) → review+
Comment on attachment 8969656 [details] Bug 1455143 - Stop beetmoving deleted testing artifacts https://reviewboard.mozilla.org/r/238450/#review244282 Aside from robocop.apk removal, this seems reasonable. FWIW, we plan to follow up and generate most artifacts in their final location. We'll try to keep this file in mind when we do that. ::: taskcluster/taskgraph/transforms/beetmover.py (Diff revision 1) > "target.txt", > - "target.web-platform.tests.tar.gz", > - "target.xpcshell.tests.zip", > "target_info.txt", > - "mozharness.zip", > - "robocop.apk", My changes shouldn't have touched robocop.apk.
Attachment #8969656 - Flags: review?(gps) → review+
Comment on attachment 8969656 [details] Bug 1455143 - Stop beetmoving deleted testing artifacts https://reviewboard.mozilla.org/r/238450/#review244540 ::: taskcluster/taskgraph/transforms/beetmover.py (Diff revision 1) > "target.txt", > - "target.web-platform.tests.tar.gz", > - "target.xpcshell.tests.zip", > "target_info.txt", > - "mozharness.zip", > - "robocop.apk", Good catch. Fixed.
Pushed by jlorenzo@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d615164b2f6f Stop beetmoving deleted testing artifacts r=gps,mtabara
https://hg.mozilla.org/mozilla-central/rev/d615164b2f6f
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → FIXED
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: