source code: https://github.com/mozilla-releng/beetmoverscript/ should be pretty similar to signingscript: all py3 based + deployed via puppet in releng infra. Probably best to start with 1 node. see Readme.md[1] for installation instructions like expected project layout, config files, and venv this should probably use a staging bucket (stage-fxbld?) for now while we work on date. Will also require tc-creds. no special scopes are defined. [1] https://github.com/mozilla-releng/beetmoverscript/blob/master/README.md

If I read the bug right, there's no task configuration to do here -- just deploying a scriptworker service.

Successfully tested beetmoverscript staging instance against a fake task - it downloaded all proper bits but failed to upload because I used fake S3 credentials. For future reference, the definition of the dummy task I played with- https://queue.taskcluster.net/v1/task/WzJpNc-uR7-Dg3ro8_tgsQ { "provisionerId": "test-dummy-provisioner", "workerType": "dummy-worker-mtabara", "schedulerId": "-", "taskGroupId": "WzJpNc-uR7-Dg3ro8_tgsQ", "dependencies": [ "YVq4WkdlTmSz4on_FwuGIw" ], "requires": "all-completed", "routes": [], "priority": "normal", "retries": 0, "created": "2016-10-18T22:04:07.675Z", "deadline": "2016-10-18T23:04:07.675Z", "expires": "2017-09-29T19:56:32.687Z", "scopes": [], "payload": { "maxRunTime": 600, "version": "52.0a1", "upload_date": 1472747174, "taskid_to_beetmove": "YVq4WkdlTmSz4on_FwuGIw", "template_key": "fennec_nightly_unsigned" }, "metadata": { "name": "Mihai Tabara's playground", "description": "beetmover fake task", "owner": "mtabara@mozilla.com", "source": "https://tools.taskcluster.net/task-creator/" }, "tags": {}, "extra": {} }

A note on the PR above. * I won't push to puppet until we have all secrets generated and stored in hiera * build-cloud-tools PR is upstream * all bits are prepared to roll out an beetmoverworker-1 instance

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review85710

We probably want to start using some scopes here, to restrict what can move what (nightlies can only move nightlies, releases can only move releases, CI only CI). We were brainstorming a way to get even more restricted access: only be able to move this specific nightly's bits around. I think that latter idea would be awesome but is likely outside the scope of the first phase.

fwiw - also pushed beetmoverscript to public Pypi - https://pypi.python.org/pypi

Comment on attachment 8802341 [details] add beetmoverworker cloud-tools configs See my comments in the PR.

(In reply to Rail Aliiev [:rail] from comment #10) > Comment on attachment 8802341 [details] > add beetmoverworker cloud-tools configs > > See my comments in the PR. Thanks for your comments, good catch on that! I'm refactoring some of the scripts for balrogworker as well so I'll have that in mind to tweak too. Will follow-up with patches & PR. Thanks.

Added a production TC client under https://tools.taskcluster.net/auth/clients/#project%252freleng%252fscriptworker%252fbeetmoverworker with the following scopes: project:releng:beetmoverworker:* queue:claim-task:scriptworker-prov-v1/beetmoverworker-* queue:poll-task-urls:scriptworker-prov-v1/beetmoverworker-* queue:worker-id:beetmoverworker-v1/beetmoverworker-*

Dropping a status-update in this bug: * staging instance working great * puppet changes for production are reviewed and ready to land * still blocked on Dev IAM S3 credentials before I can land (puppet changes will break if I don't add secrets to hiera beforehand) * once puppet changes land, I can rollout via invtools the production instance.

Comment on attachment 8802341 [details] add beetmoverworker cloud-tools configs https://github.com/mozilla-releng/build-cloud-tools/commit/7f57de3c33a021061b3e3811bcc402363a17ad88

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. http://hg.mozilla.org/build/puppet/rev/7e5a2522f625

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review87386 We talked in IRC about this patch and decided to backout all together in favour of the env-specific appraoch, where the secrets are defined per env (prod/dev) in moco-config.pp and used as variables in the ERB templates, similar to what we use for release-runner.

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review87504 Almost there! Soory for being too pedantic, but it's better to make it work properly from the beginning. :) ::: manifests/moco-config.pp:447 (Diff revision 4) > $balrog_scriptworker_git_balrogscript_repo = 'https://github.com/mozilla-releng/funsize-balrogworker.git' > > + # TC beetmover scriptworkers > + $beetmover_scriptworker_provisioner_id = "scriptworker-prov-v1" > + $beetmover_scriptworker_worker_group = "beetmoverworker-v1" > + $beetmover_scriptworker_worker_type = "beetmoverworker-v1" If you want to continue testing after you deploy things to production, I think you need to move at least worker group or type under the env section. ::: manifests/moco-nodes.pp:1175 (Diff revision 4) > } > > +# Beetmover scriptworkers > +node /beetmoverworker-.*\.srv\.releng\..*\.mozilla\.com/ { > + $aspects = [ 'maximum-security' ] > + $beetmoverworker_env = "dev" Maybe use somehthing like beetmoverworker-dev-... instead to distinguish from the future prod instances? ::: modules/beetmover_scriptworker/manifests/init.pp:70 (Diff revision 4) > + user => "${users::builder::username}", > + require => [ > + Class["packages::mozilla::git"], > + Python35::Virtualenv["${beetmover_scriptworker::settings::root}"] > + ]; > + } Please package this and move it the virtualenv definition. Otherwise you would get unpredictable deployment whenever you push to master. ::: modules/beetmover_scriptworker/manifests/init.pp:88 (Diff revision 4) > + owner => "${users::builder::username}", > + group => "${users::builder::group}", > + content => template("${module_name}/script_config.json.erb"), > + show_diff => false; > + '/root/certs.sh': > + ensure => absent; Can you add comments why you need to remove this file? ::: modules/beetmover_scriptworker/manifests/init.pp:93 (Diff revision 4) > + ensure => absent; > + } > + > + service { > + 'rpcbind': > + enable => false; The same here

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review87504 > Maybe use somehthing like beetmoverworker-dev-... instead to distinguish from the future prod instances? I don't quite get your point here. $beetmoverworker_env has to be generic, not dev/prod related. When we'll have a second machine, say, a production one, I'd be using $beetmoverworker_env="prod". If I change the var name it would not make sense in the modules/beetmover_scriptworker/manifests/init.pp where we have "$env_config = $config::beetmover_scriptworker_env_config[$beetmoverworker_env]". Ramping up a production instance would use the same init.pp files which should not change. Am a bit confused. Or were you referring to the FQDN? > Please package this and move it the virtualenv definition. Otherwise you would get unpredictable deployment whenever you push to master. Good catch \o/ No idea how I've missed this.

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review87660 ::: modules/beetmover_scriptworker/templates/script_config.json.erb:7 (Diff revisions 4 - 5) > "work_dir": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/work", > "artifact_dir": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/artifacts", > > "template_files": { > - "fennec_nightly_unsigned": "<%= scope.lookupvar("config::beetmover_scriptworker_beetmoverscript_dir") %>/beetmoverscript/templates/fennec_nightly_en_us_multi_unsigned.yml", > - "fennec_nightly_signed": "<%= scope.lookupvar("config::beetmover_scriptworker_beetmoverscript_dir") %>/beetmoverscript/templates/fennec_nightly_en_us_multi_signed.yml" > + "fennec_nightly_unsigned": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly_en_us_multi_unsigned.yml", > + "fennec_nightly_signed": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly_en_us_multi_signed.yml" These look a bit weird, but I have no better idea.

(In reply to Rail Aliiev [:rail] from comment #23) > Comment on attachment 8802334 [details] > Bug 1308042 - add beetmoverworker catalog. > > https://reviewboard.mozilla.org/r/86746/#review87660 > > ::: modules/beetmover_scriptworker/templates/script_config.json.erb:7 > (Diff revisions 4 - 5) > > "work_dir": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/work", > > "artifact_dir": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/artifacts", > > > > "template_files": { > > - "fennec_nightly_unsigned": "<%= scope.lookupvar("config::beetmover_scriptworker_beetmoverscript_dir") %>/beetmoverscript/templates/fennec_nightly_en_us_multi_unsigned.yml", > > - "fennec_nightly_signed": "<%= scope.lookupvar("config::beetmover_scriptworker_beetmoverscript_dir") %>/beetmoverscript/templates/fennec_nightly_en_us_multi_signed.yml" > > + "fennec_nightly_unsigned": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly_en_us_multi_unsigned.yml", > > + "fennec_nightly_signed": "<%= scope.lookupvar("config::beetmover_scriptworker_root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_nightly_en_us_multi_signed.yml" > > These look a bit weird, but I have no better idea. Yeah, I totally agree. Didn't know a better way either O_O

Pushed to default - http://hg.mozilla.org/build/puppet/rev/1ef060737d24

This is done now - leftover is to verify all bits are in place.

Clearing some of the fog with respect to S3 bucket deployment as I've been having some <Access Denied> errors eveer since I deployed this instance. oremj> for pub/mobile, you'll need to use the net-mozaws-stage-delivery-archive bucket for stage and net-mozaws-prod-delivery-archive for prod <mtabara> oremj: and for /pub/firefox I presume net-mozaws-stage-delivery-firefox for stage and net-mozaws-prod-delivery-firefox for prod? oremj> yes So to sum this up, we'll have to use the following for now (staging dogfooding from `date` branch): * net-mozaws-stage-delivery-archive for /pub/mobile * net-mozaws-stage-delivery-firefox for /pub/firefox When we'll be ready for escalating this to mozilla-central, we'll be using production S3 bucket: * net-mozaws-prod-delivery-archive for /pub/mobile * net-mozaws-prod-delivery-firefox for /pub/firefox Am a bit confused as to how we're supposed to toggle between two S3 buckets but I'll investigate now what's to be changed from our side to support that.

(In reply to Mihai Tabara [:mtabara] from comment #27) > Am a bit confused as to how we're supposed to toggle between two S3 buckets > but I'll investigate now what's to be changed from our side to support that. I forgot we would have different buckets for different products. so, briefly looking, I think we could have something like: s/context.config['s3']['bucket']/context.config['s3']['fennec']['bucket']/ here: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/script.py#L105 where 'fennec' comes from something like your infer_template_args() method in https://github.com/mozilla-releng/beetmoverscript/pull/1/files

Comment on attachment 8802334 [details] Bug 1308042 - add separate S3 buckets per app. https://reviewboard.mozilla.org/r/86746/#review89506 looks good to me. one nit in line ::: manifests/moco-config.pp:460 (Diff revision 6) > taskcluster_client_id => secret("beetmoverworker_dev_taskcluster_client_id"), > taskcluster_access_token => secret("beetmoverworker_dev_taskcluster_access_token"), > beetmover_aws_access_key_id => secret("stage-beetmover-aws_access_key_id"), > beetmover_aws_secret_access_key => secret("stage-beetmover-aws_secret_access_key"), > - beetmover_aws_s3_bucket => "net-mozaws-stage-delivery-firefox", > + beetmover_aws_s3_firefox_bucket => "net-mozaws-stage-delivery-firefox", > + beetmover_aws_s3_archive_bucket => "net-mozaws-stage-delivery-archive", I'm torn on this one. I think I'm leaning on this being called `beetmover_aws_s3_fennec_bucket` as that is a more explicit variable identifier, while the value of the bucket name doesn't matter that it uses the generic `archive` name.

(In reply to Jordan Lund (:jlund) from comment #30) > Comment on attachment 8802334 [details] > Bug 1308042 - add separate S3 buckets per app. > > https://reviewboard.mozilla.org/r/86746/#review89506 > > looks good to me. one nit in line > > ::: manifests/moco-config.pp:460 > (Diff revision 6) > > taskcluster_client_id => secret("beetmoverworker_dev_taskcluster_client_id"), > > taskcluster_access_token => secret("beetmoverworker_dev_taskcluster_access_token"), > > beetmover_aws_access_key_id => secret("stage-beetmover-aws_access_key_id"), > > beetmover_aws_secret_access_key => secret("stage-beetmover-aws_secret_access_key"), > > - beetmover_aws_s3_bucket => "net-mozaws-stage-delivery-firefox", > > + beetmover_aws_s3_firefox_bucket => "net-mozaws-stage-delivery-firefox", > > + beetmover_aws_s3_archive_bucket => "net-mozaws-stage-delivery-archive", > > I'm torn on this one. I think I'm leaning on this being called > `beetmover_aws_s3_fennec_bucket` as that is a more explicit variable > identifier, while the value of the bucket name doesn't matter that it uses > the generic `archive` name. Thanks! Addressed the comment pushed. Checked-in - https://hg.mozilla.org/build/puppet/rev/020e6d983ec1

Beetmoverworker doing its job finally! \o/ * dogfooding Fennec nightlies multi-locales in S3 - https://tools.taskcluster.net/push-inspector/#/GJMGcUAJQEuMHpGOYM5DNw/J1Qjsr2HTDG-NK7ds5QNNQ?_k=26nos1 * http://bucketlister-delivery.stage.mozaws.net/pub/mobile/nightly/

Since this is my second deployment in a couple of weeks, thought it'll be better to (re)dump some of the steps to have them off hand for future times. So, things to remember before deploying a staging node: 1. if using virtualenvs in the puppet catalogs, make sure Python packages coexist within the Puppet mirror, either py27 or py35; also, when adding new packages, make sure to pin existing puppet manifests to their current latest-python version to avoid unwanted side-effects. 2. feed in hiera any secrets before deploying any catalog - here goes s3, TC auth client, etc. 3. if deploying scriptworker-related modules, make sure the parent dir of your playground is 700 as storing secrets in the scriptworker configs needs protection 4. to iterate, can always nuke the playground and redo the catalog 5. out of my experience, it's nicer to have the verbose set to True in the first place and toggle back to false right before production Furthermore, things to remember before rolling out catalog to production instance(s): a) make sure staging catalog deployment works on a clean ec2 instance, rerun on staging as it is before landing in puppet repo! b) moco-config.pp - tweak TC configs to reflect production env + create them if necessary c) moco-nodes.pp - change to reflect high level of trust and security d) some of the config.json chain of trust vars may have to change depending on updates of scriptworker which is still under development e) prepare PR in build-cloud-tools f) adapt fqdn in moco-nodes accordingly if not anticipated already g) lookup for potential scopes that need to be addresed here h) add new secrets to hiera (S3, TC auth client)! i) push changes to puppet - always push to git mirror repo first to check the linters for any potential syntax errors j) follow-up push to hg.m.o source of truth! k) wait until all tests pass and machines don't break before landing build-cloud-tools - you may need to prepare the security group id beforehand l) use https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#AWS_machines to ramp up the instance(s) m) test the production instance n) make sure the S3 credentials are suitable for pushing to whatever bucket(s) is/are needed o) if all looks stable, after a while, add nagios for monitoring

Making a small naming correction here - beetmoverworker is the beetmover scriptworker that runs in our releng infra deployed via puppet, while beetmoverscript is the python package itself encompassing all the logic pieces - https://github.com/mozilla-releng/beetmoverscript/