Create a job which combines the checksums files from the builds and/or antivirus scan to produce the consolidated files SHA512SUMS, SHA1SUMs etc.

first step to this is to tell the checksums builder about beetmover s3 creds so we can use that instead of product-delivery bucket. note: I created the puppet secret files mentioned in this patch already

Review commit: https://reviewboard.mozilla.org/r/35695/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35695/

I'm going to land comment 2 on date as well as puppet patch while I implement releasetasks

Comment on attachment 8721516 [details] MozReview Request: Bug 1160385 - adds version, buildnum support from tc releasetasks on date: https://hg.mozilla.org/projects/date/rev/e0a91fd6584f

Comment on attachment 8721074 [details] [diff] [review] 160218_bug_1160385_relpro_checksums-puppet.patch on puppet prod: https://hg.mozilla.org/build/puppet/rev/308a4c385bad

Comment on attachment 8721520 [details] [diff] [review] 160219_bug_1160385_checksums-bbot-cfgs.patch I guess I need a review for this before I test it..

Comment on attachment 8721520 [details] [diff] [review] 160219_bug_1160385_checksums-bbot-cfgs.patch Review of attachment 8721520 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/project_branches.py @@ +198,5 @@ > + 'beetmover_candidates_bucket': 'mozilla-releng-beet-mover-dev', > + 'stage_product': { > + 'firefox': 'firefox', > + 'fennec': 'mobile', > + } can you add a trailing coma here

Comment on attachment 8721522 [details] [diff] [review] 160219_bug_1160385_checksums-bbotcustom.patch Review of attachment 8721522 [details] [diff] [review]: ----------------------------------------------------------------- conditional r+ ::: process/release.py @@ +1896,5 @@ > + 'slavebuilddir': normalizeName(checksums_buildername), > + 'factory': checksums_factory, > + 'properties': { > + 'branch': branch_name, > + 'platform': None, Can you add "product" here to make log_uploader.py work?

because because..

Review commit: https://reviewboard.mozilla.org/r/35917/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35917/

Review commit: https://reviewboard.mozilla.org/r/35919/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35919/

Comment on attachment 8722226 [details] MozReview Request: Bug 1160385 - Generate checksums for release promotion, r?rail https://reviewboard.mozilla.org/r/35919/#review32573

Almost there! http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date-firefox_chcksms/builds/0/steps/run_script/logs/stdio 23:15:55 INFO - Run as scripts/scripts/release/generate-checksums.py --stage-product firefox --bucket-name-prefix mozilla-releng-beet-mover-dev --credentials /builds/dev-beetmover-s3.credentials --tools-repo /tools/checkouts/build-tools Traceback (most recent call last): File "scripts/scripts/release/generate-checksums.py", line 262, in <module> myScript = ChecksumsGenerator() File "scripts/scripts/release/generate-checksums.py", line 93, in __init__ self.file_prefix = self._get_file_prefix() File "scripts/scripts/release/generate-checksums.py", line 140, in _get_file_prefix self.config["stage_product"], self.config["version"], self.config["build_number"] KeyError: 'version'

worked fine: https://tools.taskcluster.net/task-inspector/#TKxMRkHTTRGPBAxVZVhd2Q/0

bah, wrong task actually it failed: http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-date-firefox_chcksms/builds/0/steps/run_script/logs/stdio 01:51:41 INFO - ##### 01:51:41 INFO - ##### Running collect-individual-checksums step. 01:51:41 INFO - ##### 01:51:41 INFO - Running main action method: collect_individual_checksums 01:51:41 INFO - Connecting to S3 01:51:41 INFO - Connecting to bucket mozilla-releng-beet-mover-dev-firefox 01:51:42 FATAL - Uncaught exception: Traceback (most recent call last): 01:51:42 FATAL - File "/builds/slave/rel-date-fx_chcksms-0000000000/scripts/mozharness/base/script.py", line 1765, in run 01:51:42 FATAL - self.run_action(action) 01:51:42 FATAL - File "/builds/slave/rel-date-fx_chcksms-0000000000/scripts/mozharness/base/script.py", line 1707, in run_action 01:51:42 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 01:51:42 FATAL - File "/builds/slave/rel-date-fx_chcksms-0000000000/scripts/mozharness/base/script.py", line 1647, in _possibly_run_method 01:51:42 FATAL - return getattr(self, method_name)() 01:51:42 FATAL - File "scripts/scripts/release/generate-checksums.py", line 163, in collect_individual_checksums 01:51:42 FATAL - bucket = self._get_bucket() 01:51:42 FATAL - File "scripts/scripts/release/generate-checksums.py", line 156, in _get_bucket 01:51:42 FATAL - self.bucket = conn.get_bucket(self.bucket_name) 01:51:42 FATAL - File "/builds/slave/rel-date-fx_chcksms-0000000000/build/venv/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket 01:51:42 FATAL - return self.head_bucket(bucket_name, headers=headers) 01:51:42 FATAL - File "/builds/slave/rel-date-fx_chcksms-0000000000/build/venv/lib/python2.7/site-packages/boto/s3/connection.py", line 546, in head_bucket 01:51:42 FATAL - raise err 01:51:42 FATAL - S3ResponseError: S3ResponseError: 404 Not Found 01:51:42 FATAL - 01:51:42 FATAL - Running post_fatal callback... 01:51:42 FATAL - Exiting -1 01:51:42 INFO - Running post-run listener: copy_logs_to_upload_dir 01:51:42 INFO - Copying logs to upload dir... 01:51:42 INFO - mkdir: /builds/slave/rel-date-fx_chcksms-0000000000/build/upload/logs

http://hg.mozilla.org/projects/date/file/tip/testing/mozharness/scripts/release/generate-checksums.py#l131 adds "-firefox" to the bucket name. Probably need to move it to configs.

(In reply to Rail Aliiev [:rail] from comment #20) > http://hg.mozilla.org/projects/date/file/tip/testing/mozharness/scripts/ > release/generate-checksums.py#l131 adds "-firefox" to the bucket name. > Probably need to move it to configs. I can patch something up

v1

v2 over irc there was some concern for fennec. I believe v1 patch addressed both desktop/fennec in the current bbot process case. but v2 addresses the fennec relpro case. There are X amount of changes we need to make in order to support multiple products in relpro but since we are close to starting work on fennec, here is beetmover bucket plumbing..

Comment on attachment 8724917 [details] [diff] [review] checksums_use_full_bucket_support_relpro_fennec.patch Review of attachment 8724917 [details] [diff] [review]: ----------------------------------------------------------------- ::: process/release.py @@ +954,5 @@ > extra_args=[ > "--stage-product", releaseConfig["stage_product"], > "--version", releaseConfig["version"], > "--build-number", releaseConfig["buildNumber"], > + "--bucket-name-full", branchConfig["S3Bucket"], I bet this should be left as is, unless you uplift your patch everywhere, including esr38.

Comment on attachment 8724913 [details] [diff] [review] checksums_use_full_bucket.patch v2 is "more better" :P

Comment on attachment 8724372 [details] [diff] [review] buildbot_props.diff https://hg.mozilla.org/integration/mozilla-inbound/rev/aae846e4552b

it worked! https://tools.taskcluster.net/task-inspector/#9qXVsNpjSPa_pDo1Vas0hw/

Reopening to make sure we landed everything we need to m-c

https://hg.mozilla.org/integration/mozilla-inbound/rev/1ed6ec687c61 https://hg.mozilla.org/integration/mozilla-inbound/rev/b326f5187df6

https://hg.mozilla.org/mozilla-central/rev/941a62368cb7 https://hg.mozilla.org/mozilla-central/rev/c59c022943f6

https://hg.mozilla.org/mozilla-central/rev/1ed6ec687c61 https://hg.mozilla.org/mozilla-central/rev/b326f5187df6

Review commit: https://reviewboard.mozilla.org/r/49291/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/49291/

This is why it generates: https://ftp.stage.mozaws.net/pub/firefox/releases/46.0.1/SHA512SUMS TG: https://tools.taskcluster.net/task-graph-inspector/#gXXz_g6uSeq9MYTqtnOMVw/

Comment on attachment 8746185 [details] MozReview Request: Bug 1160385 - Generate beetmover checksums r=jlund a=release DONTBUILD https://reviewboard.mozilla.org/r/49291/#review46213 wow, this is neato. I like this approach :) some in line comments before stamping ::: testing/mozharness/scripts/release/beet_mover.py:254 (Diff revision 1) > + def _strip_prefix(self, s3_key): > + """Return file name relative to prefix""" > + # "abc/def/hfg".split("abc/de")[-1] == "f/hfg" > + return s3_key.split(self._get_template_vars()["s3_prefix"])[-1] > + > def upload_bits(self): it took a bit to understand changes made in this method we have many variables that are variations of `s3_key` and `downloaded_file`. I wonder if we can use less variables or change a name. `downloaded_file_info` and `beet_info_dest` stick out. could just be me :) ::: testing/mozharness/scripts/release/beet_mover.py:276 (Diff revision 1) > + s3_key = self.manifest['mapping'][locale][deliverable]['s3_key'] > downloaded_file = os.path.join(dirs['abs_work_dir'], self.get_filename_from_url(source)) > - self.upload_bit( > - source=downloaded_file, > - s3_key=self.manifest['mapping'][locale][deliverable]['s3_key'], > - bucket=bucket, > + hash_ = self.file_sha512sum(downloaded_file) > + size = os.path.getsize(downloaded_file) > + file_ = self._strip_prefix(s3_key) > + downloaded_file_info = '{}.beet'.format(downloaded_file) lol ::: testing/mozharness/scripts/release/generate-checksums.py:147 (Diff revision 1) > suffix = "firefox" > > return "{}-{}".format(self.config["bucket_name_prefix"], suffix) > > def _get_file_prefix(self): > - return "pub/{}/candidates/{}-candidates/build{}".format( > + return "pub/{}/candidates/{}-candidates/build{}/".format( hmm, self.file_prefix seems to be used all over this script. why does it need a trailing / now? I'm surprised it doesn't break anything else. ::: testing/mozharness/scripts/release/generate-checksums.py:186 (Diff revision 1) > > def find_checksums_files(): > self.info("Getting key names from bucket") > + checksum_files = {"beets": [], "checksums": []} > for key in bucket.list(prefix=self.file_prefix): > if key.key.endswith(".checksums"): hm, if we kept the '.checksums' files in the beet_mover yml template configs, won't they take precedence to the .beet equivalent you are generating that has proper file names?

(In reply to Jordan Lund (:jlund) from comment #39) > it took a bit to understand changes made in this method > > we have many variables that are variations of `s3_key` and `downloaded_file`. > > I wonder if we can use less variables or change a name. > `downloaded_file_info` and `beet_info_dest` stick out. > > could just be me :) I see what you mean. I can rearrange things to be more clear. > > def _get_file_prefix(self): > > - return "pub/{}/candidates/{}-candidates/build{}".format( > > + return "pub/{}/candidates/{}-candidates/build{}/".format( > > hmm, self.file_prefix seems to be used all over this script. why does it > need a trailing / now? I'm surprised it doesn't break anything else. I made this change just for the consistency with some other method which lists files under releases/$version/. Trailing slash was important there to prevent matching all betas when you try to use a prefix like releases/46.0 (which would match releases/46.0b*). In the current situation it is not a big deal, because we never go from build10 to build1 (the latter would catch the former without this fix). But as I said, this is not a big deal, I can revert thi hunk. > ::: testing/mozharness/scripts/release/generate-checksums.py:186 > (Diff revision 1) > > > > def find_checksums_files(): > > self.info("Getting key names from bucket") > > + checksum_files = {"beets": [], "checksums": []} > > for key in bucket.list(prefix=self.file_prefix): > > if key.key.endswith(".checksums"): > > hm, if we kept the '.checksums' files in the beet_mover yml template > configs, won't they take precedence to the .beet equivalent you are > generating that has proper file names? Nope, they shouldn't. You evaluate this condition for every file in the bucket separately.

Comment on attachment 8746185 [details] MozReview Request: Bug 1160385 - Generate beetmover checksums r=jlund a=release DONTBUILD Review request updated; see interdiff: https://reviewboard.mozilla.org/r/49291/diff/1-2/

Comment on attachment 8746185 [details] MozReview Request: Bug 1160385 - Generate beetmover checksums r=jlund a=release DONTBUILD https://reviewboard.mozilla.org/r/49291/#review46389 ++ ty, I could grep this in seconds.. :)

Comment on attachment 8746185 [details] MozReview Request: Bug 1160385 - Generate beetmover checksums r=jlund a=release DONTBUILD https://hg.mozilla.org/integration/mozilla-inbound/rev/eaecda9c7ac6 https://hg.mozilla.org/releases/mozilla-aurora/rev/97a2906dfe59 https://hg.mozilla.org/releases/mozilla-beta/rev/46c6778b335a https://hg.mozilla.org/releases/mozilla-release/rev/0f308d310cef https://hg.mozilla.org/releases/mozilla-esr45/rev/8de7191a6474