Mihai and I met today to discuss what changes are needed in balrogworker. He discovered that balrogworker was never really prepared to do nightlies. At least it was never used anywhere so tweaks are needed. @Mihai, I took a look at how we do things in current fennec nightly world and compared that to current code in balrogworker. I can confirm that I am seeing the same thing you are and changes will need to be made. I think it's worth having some people sanity check this, including callek as he will be scheduling whatever balrogworker tasks are needed in the graph. First, this is how we currently do things in Nightly: snippet from recent current fennec nightly log[1]: 04:26:11 INFO - Calling Balrog submission script 04:26:11 INFO - retry: Calling run_command with args: (['python', '/builds/slave/m-cen-and-api-15-ntly-00000000/build/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-api-15-ntly-00000000/balrog_props.json', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-api-15-ntly-00000000/oauth.txt', '--verbose', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'ffxbld', '--url-replacement', 'http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub'],), kwargs: {}, attempt #1 04:26:11 INFO - Running command: ['python', '/builds/slave/m-cen-and-api-15-ntly-00000000/build/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-api-15-ntly-00000000/balrog_props.json', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-api-15-ntly-00000000/oauth.txt', '--verbose', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'ffxbld', '--url-replacement', 'http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub'] 04:26:11 INFO - Copy/paste: python /builds/slave/m-cen-and-api-15-ntly-00000000/build/tools/scripts/updates/balrog-submitter.py --build-properties /builds/slave/m-cen-and-api-15-ntly-00000000/balrog_props.json -t nightly --credentials-file /builds/slave/m-cen-and-api-15-ntly-00000000/oauth.txt --verbose --api-root https://aus4-admin.mozilla.org/api --username ffxbld --url-replacement http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub where balrog_props.json: 04:26:11 INFO - {'properties': {'appName': 'Fennec', 04:26:11 INFO - 'appVersion': '52.0a1', 04:26:11 INFO - u'basedir': u'/builds/slave/m-cen-and-api-15-ntly-00000000', 04:26:11 INFO - u'branch': u'mozilla-central', 04:26:11 INFO - u'buildername': u'Android armv7 API 15+ mozilla-central nightly', 04:26:11 INFO - u'buildid': '20161101030207', 04:26:11 INFO - u'buildnumber': 0, 04:26:11 INFO - u'builduid': '41ea543753304764ae96fc9ec016de46', 04:26:11 INFO - 'completeMarHash': u'485ce49d788ce436ff61ec899e3c3e0015c64319bb2840c770ffc8a13310ad26d63d0c52de7a85e956ac119f3e7ff2a6c8c0d4fc707af803eb5fcaa37ef728aa', 04:26:11 INFO - 'completeMarSize': u'33359218', 04:26:11 INFO - 'completeMarUrl': u'https://archive.mozilla.org/pub/mobile/nightly/2016/11/2016-11-01-03-02-07-mozilla-central-android-api-15/fennec-52.0a1.multi.android-arm.apk', 04:26:11 INFO - 'got_revision': u'2c773b971672', 04:26:11 INFO - 'hashType': 'sha512', 04:26:11 INFO - u'jsshellUrl': u'https://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1477994527/jsshell-android-arm.zip', 04:26:11 INFO - u'master': u'http://buildbot-master71.bb.releng.use1.mozilla.com:8001/', 04:26:11 INFO - u'packageFilename': u'fennec-52.0a1.en-US.android-arm.apk', 04:26:11 INFO - u'packageUrl': u'https://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1477994527/fennec-52.0a1.multi.android-arm.apk', 04:26:11 INFO - u'platform': u'android-api-15', 04:26:11 INFO - u'product': u'mobile', 04:26:11 INFO - u'project': u'', 04:26:11 INFO - u'repo_path': u'mozilla-central', 04:26:11 INFO - u'repository': u'', 04:26:11 INFO - u'revision': u'2c773b97167252cedcba0be0c7af9d4cab192ef5', 04:26:11 INFO - u'robocopApkUrl': u'https://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1477994527/robocop.apk', 04:26:11 INFO - u'scheduler': u'mozilla-central nightly', 04:26:11 INFO - u'script_repo_revision': u'production', 04:26:11 INFO - u'slavename': u'bld-linux64-spot-043', 04:26:11 INFO - 'sourcestamp': '2c773b97167252cedcba0be0c7af9d4cab192ef5', 04:26:11 INFO - 'stage_platform': 'android-api-15', 04:26:11 INFO - u'symbolsUrl': u'https://queue.taskcluster.net/v1/task/DYO6GvWoQNaBW04otYwRFQ/artifacts/public/build/fennec-52.0a1.en-US.android-arm.crashreporter-symbols-full.zip', 04:26:11 INFO - u'testPackagesUrl': u'https://queue.taskcluster.net/v1/task/DYO6GvWoQNaBW04otYwRFQ/artifacts/public/build/fennec-52.0a1.en-US.android-arm.test_packages.json', 04:26:11 INFO - u'testsUrl': u'https://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1477994527/fennec-52.0a1.multi.android-arm.talos.tests.zip', }} # NOTE: trimmed UploadFiles as that, along with many other properties above are probably not needed at all. diving into balrog-submitter.py, we can see it hits the following lines of code: https://dxr.mozilla.org/build-central/source/tools/scripts/updates/balrog-submitter.py?q=path%3Abalrog-submitter.py&redirect_type=single#63,72-74,84-86 that means a few things: 1) the only props needed are: ``` props.get('locale', 'en-US') props.get('extVersion', props['appVersion']) updateKwargs["completeInfo"] = [{ 'size': props['completeMarSize'], 'hash': props['completeMarHash'], 'url': props['completeMarUrl'], }] if "partialInfo" in props: updateKwargs["partialInfo"] = props["partialInfo"] props['platform'] props['buildid'] props['appName'] props['branch'] props['appVersion'] props['hashType'] ``` 2) it uses NightlySubmitterV4 3) nightly builds pass url_replacements that derives from 'http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub'. Ben will know more about why this is. 4) we only call submitter.run once 5) there are no partials for any Nightly build regardless of platform. I believe this is because we switched to funsize for nightlies. Rail can confirm and where to find the balrog submission for them. now let's contrast that to balrogworker: https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L198,L201 here we can see that we: 1) use very similar props from the manifest. albeit, with a few caveats: a) the manifest is a list and balrogworker appears to be able to handle multiple submissions in one go. This may be a nice feature in the future, but not important for us now. b) the manifest always appears to have partials. again, will be helpful but can probably be ignored for now. 2) it also uses NightlySubmitterV4 and a similar submitter.run() call 3) it doesn't know about url_replacements. We will probably have to add that. 4) like mentioned in 1a, it can handle multiple submitter.run() calls 5) like mentioned in 1b, it assumes we have partials So I have a few thoughts on all of this: We should make the most minimal changes and not worry about every edge case. I think you may want to start a third elif block here[2]. This block should handle just the nightly complete apk/mar and basically use the same props and run() call args that are used in production (balrog-submitter.py). Ignore partials altogether and this 'from_buildid' code copy ripped from releasepromotion. At least for now. When we get to partials/funsize, we may want to have multiple balrogworker tasks for each nightly funsize task. Or each nightly funsize task will append to the original balrog_props.json manifest and thus only have one balrogworker task that submits everything to balrog. Those details shouldn't freeze us from quick progress now though. iow - you may want to have something like: # pseudo code with temp comments elif "tcnightly_complete" in e: # tcnightly_complete comes from beetmoverscript (update_manifest) and could be temporary while we need to distinguish from `from_buildid` # doesn't include partials log.info("Nightly style balrog submission for Taskcluster Nightlies") complete_info = e['completeInfo'] complete_info[0]["url"] = verify_copy_to_s3(args, complete_info[0]['url'], '') submitter = NightlySubmitterV4(api_root=args.api_root, auth=auth, dummy=args.dummy, url_replacements=e['url_replacements']) return submitter, {'platform': e["platform"], 'buildID': e["buildid"], 'productName': e["appName"], 'branch': e["branch"], 'appVersion': e["appVersion"], 'locale': e["locale"], 'hashFunction': props['hashType'], 'extVersion': e["extVersion"], 'completeInfo': complete_info} While adding another block is dirty, I think it keeps scope to a minimal and allows us to combine them when we get to partials and more platforms. I think the manifest (balrog_props.json) appended by beetmoverscript, should be of len one as like production fennec Nightly builds, we only want to call submitter.run() once. There are some questions left hanging. I suggest pinging rail and ben for the parts above where I mentioned their names. [1] https://archive.mozilla.org/pub/mobile/nightly/2016/11/2016-11-01-03-02-07-mozilla-central-android-api-15/mozilla-central-android-api-15-nightly-bm71-build1-build0.txt.gz [2] https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L211 @mihai, does this help or create more confusion? :)

@jlund++: thanks for awesome introspective here - overall it makes sense. My plan is now: 1) all bits for beetmoverscript are now in place - so I'm just waiting for puppet to sync & update beetmoverworker-1 environment - once this is working, the morale should boost as we're in good spare for Friday deadline. 2) reread your comment to make sure I understand all the bits 3) adapt first the beetmoverscript - I'll tune the existing f?-ed PR and repush with the bits changed there 4) start working on funsize-balrogworker/blob/master/bin/balrogworker.py to accomodate with your comments. Thanks again!

Before I forget, we also need to discuss the credentials. funsize-balrogworker defaults[1] to env vars, unless instructed otherwise through the scriptworker's config.json[2]. Not sure I'm getting this right but we'll have to: - either change that call in config.json to pass the vars via cmd line (and feed them via puppet) - or feed env vars via puppet (not sure this is good practice). I'll investigate both to see what's the best approach. [1]: https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L225 [2]: https://hg.mozilla.org/build/puppet/file/tip/modules/balrog_scriptworker/templates/config.json.erb#l23

something like this should do it. need to test and setup things from beetmoverscript side.

(In reply to Jordan Lund (:jlund) from comment #1) Heyo! > 3) nightly builds pass url_replacements that derives from > 'http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub'. Ben > will know more about why this is. ni: bhearsum, do you remember or know why this is needed? > 5) there are no partials for any Nightly build regardless of platform. I > believe this is because we switched to funsize for nightlies. Rail can > confirm and where to find the balrog submission for them. ni: rail can you confirm this and point to partial gen (funsize) jobs for nightlies?

Yup, no more partials as a part of CI. All done via funsize.

Comment on attachment 8807418 [details] funsize-balrogworker PR 2 r+ - see PR for other details. Merged it - https://github.com/mozilla-releng/funsize-balrogworker/commit/5640518093cc56e82c2b8a9c13a9460d6fdff162

(In reply to Jordan Lund (:jlund) from comment #5) > (In reply to Jordan Lund (:jlund) from comment #1) > > Heyo! > > > 3) nightly builds pass url_replacements that derives from > > 'http://archive.mozilla.org/pub,http://download.cdn.mozilla.net/pub'. Ben > > will know more about why this is. > > ni: bhearsum, do you remember or know why this is needed? This might be a historical artifact. It appears to trace back to https://bugzilla.mozilla.org/show_bug.cgi?id=1144985, where we started replacing ftp.mozilla.org with download.cdn.mozilla.net (because ftp wasn't in s3, and couldn't handle the load). We did a wholesale s/ftp/archive/ as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1213721, with no special consideration for nightly mar URLs. At this point, it appears that archive.mozilla.org and download.cdn.mozilla.net are both ultimately backed by cloudfront, so they may be effectively the same: (balrog) ➜ balrog git:(master) host archive.mozilla.org archive.mozilla.org is an alias for d34chcsvb7ug62.cloudfront.net. d34chcsvb7ug62.cloudfront.net has address 52.84.140.217 (balrog) ➜ balrog git:(master) host download.cdn.mozilla.net download.cdn.mozilla.net is an alias for 2-01-2967-001e.cdx.cedexis.net. 2-01-2967-001e.cdx.cedexis.net is an alias for d2o47bxgqzkcc3.cloudfront.net. d2o47bxgqzkcc3.cloudfront.net has address 52.84.134.68 I think you should check with nthomas and/or oremj if you want a sanity check on which domain we should serve nightly updates from.

Comment on attachment 8808829 [details] Bug 1314517 - Refactor balrogworker deployment to be more like beetmover. https://reviewboard.mozilla.org/r/91560/#review91826 lgtm mihai! nice clean ups. great to have the scriptworker varients be impld in a similar way in puppet. nothing blocking afaict. You ran this in a test env though right? ::: manifests/moco-config.pp:427 (Diff revision 2) > - $balrog_scriptworker_taskcluster_access_token = secret("balrog_scriptworker_taskcluster_access_token") > $balrog_scriptworker_task_max_timeout = 1200 > $balrog_scriptworker_artifact_expiration_hours = 336 > $balrog_scriptworker_artifact_upload_timeout = 600 > $balrog_scriptworker_verbose_logging = false > - $balrog_scriptworker_base = "/builds/balrog" > + $balrog_scriptworker_root = "/builds/balrogworker" do we plan on blowing away current balrogworker nodes? iiuc, puppet won't delete /builds/balrog/ and old state.. ::: modules/balrog_scriptworker/manifests/init.pp:74 (Diff revision 2) > "six==1.10.0", > ]; > } > > git::repo { > - "balrogscript-clone": > + "balrogscript": we so should s/funsize-balrogworker/balrogscript/ at the actual github repo :P ::: modules/balrog_scriptworker/templates/config.json.erb:28 (Diff revision 2) > - "task_script": ["<%= scope.lookupvar("config::balrog_scriptworker_py27venv") %>/bin/python", > - "<%= scope.lookupvar("config::balrog_scriptworker_git_balrogscript_path") %>/bin/balrogworker.py", > + "task_script": ["<%= scope.lookupvar("config::balrog_scriptworker_root") %>/py27venv/bin/python", > + "<%= scope.lookupvar("config::balrog_scriptworker_root") %>/balrogscript/bin/balrogworker.py", > "--taskdef", "<%= scope.lookupvar("config::balrog_scriptworker_root") %>/work/task.json", > + "--balrog-api-root", "<%= @env_config["balrog_api_root"] %>", > + "--balrog-username", "<%= @env_config["balrog_username"] %>", > + "--balrog-password", "<%= @env_config["balrog_password"] %>", hm, I'm just thinking that this lives in `args` is passed throughout balrogworker.py itself. It would be possible to leak the pw by accident in the logs. We should probably save this password immediately after it is parsed in main() and then pop it out. similiar to: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/beet_mover.py#362 not sure if this is blocking but want to point this out.

Comment on attachment 8808829 [details] Bug 1314517 - Refactor balrogworker deployment to be more like beetmover. https://reviewboard.mozilla.org/r/91560/#review91826 > do we plan on blowing away current balrogworker nodes? iiuc, puppet won't delete /builds/balrog/ and old state.. The old balrogworker is now gone. Will recreate a new machine as soon as we redeploy. > we so should s/funsize-balrogworker/balrogscript/ at the actual github repo :P That's actually something I've been chasing for weeks, so glad I have a vouch on this, I'll proceed with hwine to change that :) > hm, I'm just thinking that this lives in `args` is passed throughout balrogworker.py itself. It would be possible to leak the pw by accident in the logs. > > We should probably save this password immediately after it is parsed in main() and then pop it out. similiar to: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/beet_mover.py#362 > > not sure if this is blocking but want to point this out. Good call - addressed it under https://github.com/mozilla-releng/funsize-balrogworker/pull/5

Comment on attachment 8808829 [details] Bug 1314517 - Refactor balrogworker deployment to be more like beetmover. https://reviewboard.mozilla.org/r/91560/#review92160 weee exciting! ::: modules/balrog_scriptworker/manifests/init.pp (Diff revisions 2 - 3) > source => "puppet:///modules/balrog_scriptworker/release.pubkey", > require => Git::Repo["balrogscript"], > owner => "${users::builder::username}", > group => "${users::builder::group}"; > } > - # requirement as part of scriptworker pentest bug 1298199#c23 not needed anymore?

Comment on attachment 8808829 [details] Bug 1314517 - Refactor balrogworker deployment to be more like beetmover. http://hg.mozilla.org/build/puppet/rev/d06ad23cf437

Balrogworker is claiming tasks successfully though hitting issues with sending proper details to Balrog: 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,775 - DEBUG - REQUEST STATS: {"url": "https://balrog-admin.stage.mozaws.net/api/releases/Fennec-date-nightly-20161111122445", "timestamp": 1478872310.775483, "method": "HEAD", "elapsed_secs": 0.15775418281555176, "status_code": 200} 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,776 - DEBUG - Balrog request to https://balrog-admin.stage.mozaws.net/api/releases/Fennec-date-nightly-20161111122445/builds/Android_arm-eabi-gcc3/en-US 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,776 - DEBUG - Data sent: {'alias': 'null', 'product': u'Fennec', 'hashFunction': u'sha512', 'data_version': '1', 'data': '{"buildID": "20161111122445", "platformVersion": "52.0a1", "displayVersion": "52.0a1", "completes": [{"fileUrl": "http://bucketlister-delivery.stage.mozaws.net/pub/mobile/nightly/2016/11/2016-11-11-12-24-45-date-android-api-15/fennec-52.0a1.multi.android-arm.apk", "hashValue": "4e79b0d0deb76eb1340ad6f46bc6a0496c1f87e1f8d4f24816ab5897bd0c339a000fb9365993c5b156c5a0426d44d1eeb3058132e33663f6a6c7fb211a2a4edc", "from": "*", "filesize": 33212642}], "appVersion": "52.0a1"}', 'schema_version': 4} 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,946 - ERROR - Caught HTTPError: {"data": ["Blob contains forbidden domain(s)"]} 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,946 - DEBUG - REQUEST STATS: {"url": "https://balrog-admin.stage.mozaws.net/api/releases/Fennec-date-nightly-20161111122445/builds/Android_arm-eabi-gcc3/en-US", "timestamp": 1478872310.946776, "method": "PUT", "elapsed_secs": 0.1696760654449463, "status_code": 400} 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,947 - DEBUG - retry: Caught exception: 2016-11-11T13:51:50 INFO - Traceback (most recent call last): 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/balrogscript/tools/lib/python/vendor/redo-1.4.1/redo/__init__.py", line 152, in retry 2016-11-11T13:51:50 INFO - return action(*args, **kwargs) 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/balrogscript/bin/../tools/lib/python/balrog/submitter/cli.py", line 314, in update_dated 2016-11-11T13:51:50 INFO - schemaVersion=schemaVersion, data_version=data_version) 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/py27venv/lib/python2.7/site-packages/balrogclient/api.py", line 223, in update_build 2016-11-11T13:51:50 INFO - return self.request(method='PUT', data=data) 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/py27venv/lib/python2.7/site-packages/balrogclient/api.py", line 111, in request 2016-11-11T13:51:50 INFO - return self.do_request(url, data, method) 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/py27venv/lib/python2.7/site-packages/balrogclient/api.py", line 129, in do_request 2016-11-11T13:51:50 INFO - req.raise_for_status() 2016-11-11T13:51:50 INFO - File "/builds/balrogworker/py27venv/lib/python2.7/site-packages/requests/models.py", line 837, in raise_for_status 2016-11-11T13:51:50 INFO - raise HTTPError(http_error_msg, response=self) 2016-11-11T13:51:50 INFO - HTTPError: 400 Client Error: BAD REQUEST for url: https://balrog-admin.stage.mozaws.net/api/releases/Fennec-date-nightly-20161111122445/builds/Android_arm-eabi-gcc3/en-US 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,949 - DEBUG - sleeping for 2.00s (attempt 9/10) Debugging now ...

mtabara> http://bucketlister-delivery.stage.mozaws.net/ is not whitelisted and not sure how to deal with this 20:10:10 <mtabara> since it's a staging bucklet 20:10:13 <~bhearsum> ah 20:10:28 <~bhearsum> that's interesting, we've not had to deal with that post-cloudops migration 20:10:52 <~bhearsum> do we want that domain whitelisted for stage permanently, or only temporarily? 20:11:03 <~bhearsum> ie: until we have a production bucket lister 20:11:43 <mtabara> our plan is: 1. make sure our balrogscript submits the correct release blob to http://bucketlister-delivery.stage.mozaws.net/ 2. switch to http://bucketlister-delivery.prod.mozaws.net/pub/mobile/nightly/ which if my understanding is correct backs-up http://archive.mozilla.org/pub/mobile/nightly/ 20:12:25 <~bhearsum> ok, so using bucketlister stage is a temporary measure 20:12:46 <~bhearsum> long term, both balrog stage and prod would want to use bucketlister prod - right? 20:13:40 <mtabara> yes, using bucketlister stage is a temporary measure 20:13:44 <~bhearsum> okay 20:14:11 <~bhearsum> so i'm going to ignore the general problem and see what we can do as a short term workaround... 20:15:04 <~bhearsum> those whitelists are defined in https://github.com/mozilla/balrog/blob/master/uwsgi/public.wsgi and admin.wsgi, and those files are shared between stage and prod 20:16:24 <mtabara> so should I add them https://github.com/mozilla/balrog/blob/master/uwsgi/admin.wsgi#L8 and do a PR? 20:16:48 <mtabara> well, not so much of "them" but "it" :) 20:16:57 <~bhearsum> maybe, but it's not acceptable to stage bucketlister stage to the whitelist for balrog prod for the real Fennec product - that's a security risk 20:17:45 <~bhearsum> one thing you might be able to do is whitelist bucketlister stage for some fake product names (eg: Fennec-bucketlisttest), and make sure you set that as the product when submitting to balrog stage 20:18:00 <~bhearsum> that'd require tweaking the submission scripts too probably 20:22:18 <mtabara> hm 20:22:51 <~bhearsum> not a great option, i know 20:23:31 <mtabara> so instead of "Fennec-mozilla-central-nightly-latest" id't be "Fennec-somethingXYZ-mozilla-central-nightly-latest"? 20:23:37 <mtabara> sorry 20:23:44 <mtabara> * "Fennec-date-nightly-latest" 20:23:47 <~bhearsum> no, you wouldn't need to adjust the Release name, just the product 20:24:25 <mtabara> ah, ok 20:24:28 <~bhearsum> in your pastebin you've got: 20:24:30 <~bhearsum> 2016-11-11T13:51:50 INFO - 2016-11-11 13:51:50,776 - DEBUG - Data sent: {'alias': 'null', 'product': u'Fennec', 'hashFunction': u'sha512', 'data_version': '1', 'data': '{"buildID": "20161111122445", "platformVersion": "52.0a1", "displayVersion": "52.0a1", "completes": [{"fileUrl": 20:24:34 <~bhearsum> "http://bucketlister-delivery.stage.mozaws.net/pub/mobile/nightly/2016/11/2016-11-11-12-24-45-date-android-api-15/fennec-52.0a1.multi.android-arm.apk", "hashValue": "4e79b0d0deb76eb1340ad6f46bc6a0496c1f87e1f8d4f24816ab5897bd0c339a000fb9365993c5b156c5a0426d44d1eeb3058132e33663f6a6c7fb211a2a4edc", "from": "*", "filesize": 33212642}], "appVersion": "52.0a1"}', 'schema_version': 4} 20:24:39 <~bhearsum> and it's only "product" that needs to change - nothing in the data nor the URL 20:25:08 <~bhearsum> let me investigate one other option too...

Whitelisting the admin.uwsgi allowed us to successfully submit release blobs to Balrog but not to retrieve them via the public Balrog staging as the public.uwsgi had to be addressed as well. Eventually we used the production buckets (that are backfilling archive.m.o which is already whitelisted in both staging/production) so we can now serve updates from Balrog staging. Since now we're Fennec/Firefox single-locale Tier-2 ready, I'm closing this for now and will file a separate bug tracking the l10n work for balrogworker.