Funsize will need to submit partial data to balrog. The current implementation requires complete update information. The patch relaxes this requirement and adds a script for partial submission. There are 2 highlights in the script: 1. Balrog credentials are set in the script environment. They will be set by Taskcluster via encrypted env variables (see http://docs.taskcluster.net/docker-worker/ Encrypted Environment Variables). I think this is better than baking the credentials into a docker image, but we should be careful dumping debug information to the logs. 2. A manifest used in the script looks like this: { "ACCEPTED_MAR_CHANNEL_IDS": "firefox-mozilla-release", "appName": "Firefox", "branch": "mozilla-release", "from_buildid": "20150217132925", "from_mar": "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0b10-candidates/build1/update/linux-i686/ach/firefox-36.0b10.complete.mar", "hash": "089b713674c84dbf0f12529b94748be57e8a61bbe0914a7f765b7ca9e8e49377aa908a3380dca126c2383126130886c59dde5ec0a05438f61c5030ed0bcaebd8", "locale": "ach", "mar": "Firefox-mozilla-release-36.0-20150217132925-20150219174507.partial.mar", "platform": "linux", "repo": "https://hg.mozilla.org/releases/mozilla-release", "revision": "59eefd8af5d7", "signed": true, "size": 3905978, "to_buildid": "20150219174507", "to_mar": "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0-candidates/build1/update/linux-i686/ach/firefox-36.0.complete.mar", "version": "36.0" }

Comment on attachment 8571940 [details] [diff] [review] balrog-submitter-partials-only-tools.diff Review of attachment 8571940 [details] [diff] [review]: ----------------------------------------------------------------- The cli.py changes look pretty straightforward and sane, I have a couple of higher level questions/comments though. (In reply to Rail Aliiev [:rail] from comment #0) > Created attachment 8571940 [details] [diff] [review] > balrog-submitter-partials-only-tools.diff > > Funsize will need to submit partial data to balrog. The current > implementation requires complete update information. > > The patch relaxes this requirement and adds a script for partial submission. > There are 2 highlights in the script: > > 1. Balrog credentials are set in the script environment. They will be set by > Taskcluster via encrypted env variables (see > http://docs.taskcluster.net/docker-worker/ Encrypted Environment Variables). > I think this is better than baking the credentials into a docker image, but > we should be careful dumping debug information to the logs. It kindof sucks to have this script use a different way of passing credentials than all of the rest. What do you think about using a credentials file like the other scripts? Is that possible with TC? > 2. A manifest used in the script looks like this: > > { > "ACCEPTED_MAR_CHANNEL_IDS": "firefox-mozilla-release", > "appName": "Firefox", > "branch": "mozilla-release", > "from_buildid": "20150217132925", > "from_mar": > "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0b10- > candidates/build1/update/linux-i686/ach/firefox-36.0b10.complete.mar", > "hash": > "089b713674c84dbf0f12529b94748be57e8a61bbe0914a7f765b7ca9e8e49377aa908a3380dc > a126c2383126130886c59dde5ec0a05438f61c5030ed0bcaebd8", > "locale": "ach", > "mar": > "Firefox-mozilla-release-36.0-20150217132925-20150219174507.partial.mar", > "platform": "linux", > "repo": "https://hg.mozilla.org/releases/mozilla-release", > "revision": "59eefd8af5d7", > "signed": true, > "size": 3905978, > "to_buildid": "20150219174507", > "to_mar": > "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0-candidates/ > build1/update/linux-i686/ach/firefox-36.0.complete.mar", > "version": "36.0" > } One of the things that I really dislike about the current scripts is that they depend on external config files (release configs, buildbot properties). I find that it makes them hard to understand, and even harder to test. How do you feel about actually passing this information to the script rather than putting it in a manifest? ::: scripts/updates/funsize-balrog-submitter.py @@ +52,5 @@ > + submitter.run( > + platform=manifest["platform"], buildID=manifest["to_buildid"], > + productName=manifest["appName"], branch=manifest["branch"], > + appVersion=manifest["version"], locale=manifest["locale"], > + hashFunction='sha512', extVersion=manifest["version"], Let's not hardcode "sha512" here. Make it an arg with a default of sha512, please.

We talked about this and decided that the library changes should go to this patch and the actual script will be part of funsize.

Comment on attachment 8572741 [details] [diff] [review] balrog-submitter-partials-only-tools.diff https://hg.mozilla.org/build/tools/rev/db91d2ba0bc5