We need these for staging releases, specifically devedition today.

14:58 <aki> hey rail! wondering if you need any help with dawn stuff 15:02 <rail> thanks for asking! one of the known things is that we need to create a fake previous release, otherwise we would need to change a lot in our automation. maybe something like a script that would download a CI build (probably just complete MARs) and upload it as a fake release 15:03 <rail> no need to generate l10n, we can reuse en-US MARs 15:03 <rail> also we may need to generate the cheksums files 15:03 <rail> SHA512sUMS 15:04 <aki> this is basically populating candidates/ ? 15:04 <rail> yeah 15:04 <rail> or releases, I forgot which one we use 15:04 <rail> we can copy 15:04 <rail> otherwise I'm just waiting for a CI build to start a release to uncover more issues :) 15:05 <rail> there is no bug for the fake release scrip, feel free to file one Guessing this would be boto3-based.

If the bits already live in the proper s3 bucket, we could also s3cmd -c CFG_PATH cp "s3://BUCKET/KEY1" "s3://BUCKET/KEY2" However, if we're based off the nightly-signing devedition tasks in https://treeherder.mozilla.org/#/jobs?repo=jamun&selectedJob=99521191 , we need to find the nightly signing tasks, find the target.complete.mar, and then upload to s3.

Not finding the complete mar in https://archive.mozilla.org/pub/devedition/tinderbox-builds/jamun-macosx64-devedition/ ; do we upload it?

wip https://github.com/escapewindow/staging-release-upload/blob/master/staging_upload.py

I manage to find the complete MARs attached as artifacts to the corresponding TC task (which is used by release promotion): https://tools.taskcluster.net/index/#gecko.v2.jamun.latest.firefox.macosx64-opt/gecko.v2.jamun.latest.firefox.macosx64-opt pointed me to https://tools.taskcluster.net/task-inspector/#SiTO4UE6TuaiPqLLN0QcuA/0 which has those.

Oops, s/firefox/devedition/ https://tools.taskcluster.net/index/#gecko.v2.jamun.latest.devedition.macosx64-opt/gecko.v2.jamun.latest.devedition.macosx64-opt https://tools.taskcluster.net/task-inspector/#IF_svEKNT86FMCTXdszCuA/0

I forked aki's script (comment 4) and started to adapt it for new locations.

download() is implemented

(New thumb-wrestle round!) This PR uploads the same mar for every locale (with one mar per architecture). At the end, it also uploads SHA512SUMS. It has been tested on this staging bucket[1]. We'll need to release beetmoverscript and mozapkpublisher on Pypi (the latter because of [2]). [1] https://console.aws.amazon.com/s3/buckets/mozilla-releng-beet-mover-dev/pub/devedition/candidates/54.0b1-candidates/build1 [2] https://github.com/mozilla-releng/mozapkpublisher/pull/29

Comment on attachment 8869094 [details] Implement upload() PR not really a hard r- but I wanted to point out I have some open questions in Github. Please r? me again when those have been replied to or addressed.

Comment on attachment 8869094 [details] Implement upload() PR Sorry about the confusion. I tested for one platform, and it turned out uploading 5 platforms at the same time was an issue. I investigated it and tweaked the network config to start uploading batch of 93 files (aka 1 platform at once). I detailed the context in the code. So here's the corrected version. I tested it twice, by looking at the following: * Number of files within each platform folder => 93 files per platform * Size of the devedition folder (27.7GB, which corresponds to 93*(sum of size of the 5 mars) * Number of lines in SHA512SUMS => 465 which equals to 93*5 * Format of the SHA512SUMS * Network usage which is about 27GB For reference, it took about 23 minutes to run the scriptm with a 200 Mbps upload connection. I'd recommend to run it either in a datacenter or behind an optic fiber connection.

Comment on attachment 8869094 [details] Implement upload() PR forgot to clear this after reviewing in github last week

Let's mark it as done. We ended up not using this unfortunately. :/ I just created a fake bouncer entry pointing to one of the betas to avoid any confusion because of the files published.