We need this to start submitting nightly build data directly to the Balrog dev environment. Rail has been doing some work on this.

Ignore lib/python/vendor TODOs: * SSL verification * print messages for retry.py Todos should be done before we go live using production balrog server. buildbotcustom part incoming in a bit

Once we get the password I can start using it in staging!

Comment on attachment 608721 [details] [diff] [review] buildbotcustom Have to use pyton2.6, to be updated.

* use python2.6 (requests needs it) * move self.balrog_api_root to MozillaBuildFactory

TODO: update secrets.pp on master-puppet1 with real credentials.

Comment on attachment 608833 [details] [diff] [review] buildbotcustom Review of attachment 608833 [details] [diff] [review]: ----------------------------------------------------------------- ::: process/factory.py @@ +1802,5 @@ > + slavedest='buildprops_balrog.json', > + workdir='.', > + flunkOnFailure=False, > + )) > + credentialsFile = os.path.join(os.getcwd(), "BuildSlaves.py") For the Tuxedo factory, we pass this in from release.py. Can we do a similar thing here and either put it in misc.py or config.py?

Comment on attachment 608695 [details] [diff] [review] CLI tools using buildprops.json Review of attachment 608695 [details] [diff] [review]: ----------------------------------------------------------------- This is looking pretty good. There's just a few minor things to address before this can land. ::: lib/python/balrog/client/api.py @@ +7,5 @@ > + > +class API(object): > + > + url_template = \ > + '%(api_root)s/releases/%(name)s/builds/%(abi)s/%(locale)s' s/api/build_target/, since that's what we call it in most places. @@ +13,5 @@ > + verify = False > + auth = None > + > + def __init__(self, api_root='https://balrog.build.mozilla.org', > + auth=None, ca_certs=None, timeout=60, raise_exceptions=True): Can you add docstrings for at least auth and ca_certs? It's not immediately clear what's expected in them. @@ +16,5 @@ > + def __init__(self, api_root='https://balrog.build.mozilla.org', > + auth=None, ca_certs=None, timeout=60, raise_exceptions=True): > + self.api_root = api_root > + if ca_certs: > + self.verify = ca_certs Your usage of this variable is a bit weird...it defaults to "False" but then gets assigned a non-boolean value. I think it should always be the same type. Also, let's default to verifying certs! @@ +22,5 @@ > + self.timeout = timeout > + # raise exception by default > + self.config = dict(danger_mode=True) > + if not raise_exceptions: > + self.config = dict(danger_mode=False) I think these 3 lines can just be: self.conf = dict(danger_mode=raise_exceptions) ::: lib/python/balrog/client/cli.py @@ +15,5 @@ > + > + build_type = 'nightly' > + appName = None > + branch = None > + abi = None Rename to buildTarget, like in the other file. @@ +41,5 @@ > + if self.dummy: > + self.branch = '%s-dummy' % self.branch > + self.appVersion = props['appVersion'] > + self.name = '%s-%s-%s-%s' % (self.appName, self.branch, > + self.build_type, buildID) I think it would be good to put the generation of these names into another function, maybe in balrog.names? @@ +51,5 @@ > + } > + if not (props.get('completeMarFilename') or > + props.get('partialMarFilename')): > + raise UpdateException( > + 'completeMarFilename or/and partialMarFilename should be set.') Any case where the complete MAR is missing seems like an error case to me, regardless of whether or not we have a partial. Can we change this to error out only on the absence of the complete? ::: scripts/updates/balrog-client.py @@ +26,5 @@ > + logging_level = logging.INFO > + if options.verbose: > + logging_level = logging.DEBUG > + logging.basicConfig(stream=sys.stdout, level=logging_level, > + format="%(message)s") This needs to get run regardless of what options.verbose is set to.

* address comments * enable SSL by default * include Mozilla CA Cert

Comment on attachment 609418 [details] [diff] [review] CLI tools using buildprops.json Review of attachment 609418 [details] [diff] [review]: ----------------------------------------------------------------- Looks good overall! r=me with the two little things noted below fixed. ::: lib/python/balrog/client/cli.py @@ +6,5 @@ > +from release.platforms import buildbot2updatePlatforms > +from balrog.client.api import API > + > + > +def get_name(appName, branch, build_type, buildID, dummy=False): Since this is meant to handle both dated and latest cases, let's rename 'buildID' to 'suffix', to make it more clear. Another naming quibble: get_name should be get_blob_name or maybe get_nightly_blob_name. @@ +57,5 @@ > + 'from': '*', > + 'filesize': props['completeMarSize'], > + 'hashValue': props['completeMarHash'], > + 'fileUrl': props['completeMarUrl'] > + } Can you catch the potential KeyError's here and convert them to an UpdateException, similar to your original patch? It's good that we're always trying to set the complete now, but better to wrap failures to something with a better message IMO.

Comment on attachment 608835 [details] [diff] [review] puppet-manifests http://hg.mozilla.org/build/puppet-manifests/rev/ce8c4624b9cf

Comment on attachment 609418 [details] [diff] [review] CLI tools using buildprops.json http://hg.mozilla.org/build/tools/rev/da24e79c363f

Comment on attachment 609422 [details] [diff] [review] configs http://hg.mozilla.org/build/buildbot-configs/rev/261277b545d0

Comment on attachment 609420 [details] [diff] [review] buildbotcustom http://hg.mozilla.org/build/buildbotcustom/rev/c8085d85228b

Everything is pushed, waiting for reconfig and bug 739323

Bustage fix: http://hg.mozilla.org/build/buildbotcustom/rev/cca835304cdd

this was deployed during a reconfig today

Bug 739323 the dev instance was tweaked and now we need to use Geotrust's CA certificate.

Comment on attachment 610513 [details] [diff] [review] use GeoTrust's CA certificate http://hg.mozilla.org/build/tools/rev/f881687f0ed5

All one for an *initial* client.