Now that we're inches away from having our beetmoverworkers deployed to production (that is: working on date project branch, uploading to a releng dev S3 bucket) we will need to massage a bit beetmoverscript to: * adapt the payload to cope with the nightly graph kinds * enrich the templates to more than just fennec (that is firefox/platform/etc) * adapt its output artifacts to serve them to balrogworker This bug tracks all that beetmoverscript work in order to have successful beetmover jobs scheduled in the nightly graph (bug 1305096)

@jlund r? Milestone - proper bits into place to tweak beetmoverwork to bind nicely with taskgraph as we discussed last week. I thought it'd be useful to land this first to unblock @kmoir while I work on a separate branch to massage the manifest for balrogworker. Once deployed we should be able to schedule beetmover tasks for signed and unsigned bits, at least for Fennec. I have few leftovers left to address later: * add more tests to ensure 100% coverage * docstrings + docs everywhere * add Travis support * make Travis deploy automatically to Pypi every time we tag a new release

@lundjordan f? (This is a PR in own fork of beetmoverscript) Next milestone: prepping the balrogworker manifest. I sort-of finished preparation of the manifest.json. I still have few TODOs to address but wanted you to have a look beforehand when you have a spare minute. I'm not sure yet as to how I'll get my hands on the actual url location but I'll find the answer soon I guess. Also, while glancing in the funsize-balrogworker code just realized you were right, it looks like there's gonna be a need to refactor some of that to cope with Fennec nightly contraints.

Comment on attachment 8805982 [details] Bug 1313154 - adjust beetmoverscript to cope with nightly graph constraints see PR

Comment on attachment 8806435 [details] Bug 1313154 - generate beetmover output to be used by balrogworker. r=jlund see PR

Comment on attachment 8805982 [details] Bug 1313154 - adjust beetmoverscript to cope with nightly graph constraints https://github.com/mozilla-releng/beetmoverscript/commit/8011b48fc0c468a8d3f6f2cace9cfc945a021d18

Added all other leftovers that can be postponed for now to the https://github.com/mozilla-releng/beetmoverscript/issues

Pushed https://pypi.python.org/pypi/beetmoverscript/0.0.2 and to our internal Puppet and also to Pypi. Github release tag is here - https://github.com/mozilla-releng/beetmoverscript/releases/tag/0.0.2

Checked-in puppet change as well to bump that version - https://hg.mozilla.org/build/puppet/rev/e05536c4067d

* Enriched the mimetypes with JSON as well - PR to beetmoverscript here https://github.com/mozilla-releng/beetmoverscript/commit/4b5e71f74eb152238d29f0584fc0999cbf82ad8d * Shipped 0.0.3 version of beetmoverscript * pushed to puppet to bump it in beetmoverworker too - https://hg.mozilla.org/build/puppet/rev/42640b01fb55

We now have fully working multi-locale Fennec nightlies on date - https://tools.taskcluster.net/push-inspector/#/GJMGcUAJQEuMHpGOYM5DNw/J1Qjsr2HTDG-NK7ds5QNNQ?_k=26nos1 \o/

Comment on attachment 8808304 [details] Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies. see PR comments. resetting r? for now.

Comment on attachment 8808304 [details] Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies. jlund r+'ed in github. Merged - https://github.com/mozilla-releng/beetmoverscript/commit/d37855d43a6d1b5fa811097a9efd77da565f9ff1

Most of the desktop nightlies got successful beetmoved to S3 buckets but we've hit some issues with gtest.zip, a rather large file (~500MB): --- 2016-11-07T23:26:00 INFO - application/zip 2016-11-07T23:26:00 INFO - 1478561190 2016-11-07T23:26:00 INFO - /net-mozaws-stage-delivery-firefox/pub/firefox/nightly/latest-date/firefox-52.0a1.en-US.linux-i686.gtest.tests.zip 2016-11-07T23:26:00 INFO - 2016-11-07 23:26:00,175 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 1 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,120 - beetmoverscript.script - INFO - 400 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,146 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?> 2016-11-07T23:26:59 INFO - <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>C22FA4E78B138A23</RequestId><HostId>....+.....=</HostId></Error> 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,152 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,167 - beetmoverscript.script - INFO - 400 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,167 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?> 2016-11-07T23:26:59 INFO - <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>8D5B1DC5A206467C</RequestId><HostId>.................=</HostId></Error> 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,167 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,168 - beetmoverscript.script - INFO - 400 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,168 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?><Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>964B2808FEFA1579</RequestId><HostId>..............................</HostId></Error> 2016-11-07T23:26:59 INFO - 2016-11-07 23:26:59,169 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry 2016-11-07T23:27:06 INFO - 2016-11-07 23:27:06,306 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry 2016-11-07T23:27:06 INFO - 2016-11-07 23:27:06,325 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2 2016-11-07T23:27:06 INFO - 2016-11-07 23:27:06,344 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2 2016-11-07T23:27:06 INFO - 2016-11-07 23:27:06,346 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2 2016-11-07T23:27:06 INFO - 2016-11-07 23:27:06,486 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2 --- Pushed a patch to increase the ExpireIn from presigned url validation to 1800 seconds instead of 30m and pushed 0.0.5 beetmoverscript to production. Awaiting to retest the changes. https://github.com/mozilla-releng/beetmoverscript/commit/cbe08d2efe85e384c7692f623c89c76352445112

Still getting the same error - the culprit is elsewhere. Dump from IRC: 01:11:35 <mtabara> I wish things were that easy. still the same issue, the culprit is elsewhere 01:12:25 <mtabara> something to do with a too large Content-Length we're setting before we send the files to S3, then aws waits for us to send more and timeouts after 20 secs for not receiving anything 01:12:27 <mtabara> https://github.com/aws/aws-sdk-js/issues/281#issuecomment-43798354 01:19:52 <mtabara> I give up, no more brainz. will debug in the morning 01:20:19 <mtabara> worst case scenario - we can remove the gtest.zip for now as it's not blocking tier-2 and readdress it later on before tier-1 01:20:25 <mtabara> it's the only one causing us troubles 01:20:31 <mtabara> s,one,artifact mtabara> the rest of them work nicely 01:24:00 <nthomas|away> You may be hitting the size where multi part upload is mandatory 01:24:42 <mtabara> is there a workaround for that? So the half-full side of the glass says we're technically done with desktop nightlies in terms of beetmoving. The half-empty says we still need to solve the gtest.zip or remove it for now to make the beetmover tasks green.

Temporarily will remove gtest.zip and crashtest-report-full.zip from files to be beetmoved since they're 0.5G and 1G respectively and we're hitting some issues. Might need to investigate more on multi-part upload or ditch these two files directly, we'll see. Shipped 0.0.6 to address the temporarily comment of the aforementioned packages.

Green tasks in desktop nightlies too! However, we've knocked down though the gtest and crashreport-full-test.zip as they were too large.

see PR for context :)

Comment on attachment 8808536 [details] Bug 1313154 - generate beetmover output to be used by balrogworker Addressed in the PR. Already merged it. https://github.com/mozilla-releng/beetmoverscript/commit/906fd17d68f7bb508fd5a4043b6a29f7b74e5e36

(In reply to Jordan Lund (:jlund) from comment #21) > Created attachment 8808536 [details] > Bug 1313154 - generate beetmover output to be used by balrogworker > > see PR for context :) Merged to production and deployed - manifest.json generation works nicely for Fennec - https://queue.taskcluster.net/v1/task/MN7-qxR-SaaqRxMOxW8YVQ/runs/1/artifacts/public/manifest.json

Comment on attachment 8809971 [details] Bug 1313154 - adapt beetmoverscript to cope with desktop nightlies. https://github.com/mozilla-releng/beetmoverscript/commit/1b7075a057452d2eac2c152c2c54ca1592cd9278 r+'ed by Callek.

Similar to 1315139, oremj granted our beetmover production user premissions to write to "net-mozaws-stage-delivery-{firefox,archive}" which backfills http://archive.mozilla.org/pub/{firefox,mobile}/nightly/. Now we're one puppet tweak away to pushing our artifacts in production buckets \o/

Switching the artifacts from "net-mozaws-stage-delivery-{firefox,archive}" to ""net-mozaws-prod-delivery-{firefox,archive}" which backfills http://archive.mozilla.org/pub/{firefox,mobile}/nightly.

Comment on attachment 8810666 [details] [diff] [review] Bug 1313154 - switch beetmoverworker to S3 production buckets. Seems fine to me, based on what I can see on the bucketlister for staging.

Comment on attachment 8811059 [details] [diff] [review] Bug 1313154 - switch beetmoverworker to S3 production buckets tight to just nightly subdirs https://hg.mozilla.org/build/puppet/rev/0563f4230c7f

Since now we're Fennec/Firefox single-locale Tier-2 ready with beetmoverscript/beetmoverworker working at full capacity, I'm closing this for now and will file a separate bug to track the l10n work for beetmoverworker/beetmoverscript.