Maple[1] uses dep-signing which makes pushapk_scriptworker failing. We need a staging instance of pushapk that can't push to prod, but supports dep-signing. [1] https://hg.mozilla.org/projects/maple/

mozapkpublisher will error out at the first error reported by Google Play. I introduced a mode that keeps every step but does not do anything with Google Play.

Status update It's been about year since I deployed pushapkworker-1 (bug 1307826). When I read the bug up, I realized a few things were missing. 1. There is no need to connect to inventory[1] to reserve a domain name (FQDN). For some reason, I thought I did something similar last year. Anyway, it wasn't needed last week. I just ran the script (attached), which handled the DNS subscription just fine. This script is a modified copy of the one Aki used to spawn dep-signing scriptworkers. I noticed he used the same configuration as the prod workers[2]. This means machines live under the same subnet and samesec group as production ones. Then, I did the same for dep-pushapkworker 2. You don't need to create new Chain of Trust GPG keys, if you set verify_chain_of_trust to true, but not the 2 other vars (namely sign_chain_of_trust and verify_cot_signature). This still lets the scriptworker instance to download upstream artifacts. 3. If the FQDN doesn't match any node in the production listing, you still may connect to the machine and attach it to a personal puppet environment. In this regard, I did: > ssh root@dep-pushapkworker-1.srv.releng.use1.mozilla.com -i /PATH/TO/PRIVATE/REPO/keys/aws-releng > puppet agent --test --server releng-puppet2.srv.releng.scl3.mozilla.com --environment jlorenzo Once this puppet run finished, I was able to connect with my regular handle. Then I discovered: A. Puppet is still running v3.7. This prevents the use of loops, unless you globally activate the "future parser"[4]. Then, I rewrote my puppet patch with switch statements to define the right variables. B. No need to store the APKs certificates on Hiera. They are definitely public, and they can be extracted from the APKs. For instance, I got the dep certificate via: > unzip -p target.apk META-INF/DEP.RSA | openssl pkcs7 -inform DER -print_certs -out nightly.pem I confirmed the certs are right ones, by diff'ing nightly and release against the ones already imported in the prod push-apk. I versioned the certificate in the puppet repo, directly. C. I will delete the "dev" TC client[5] in favor of a "dep" one. The rationale is: * we shouldn't have a dev instance that touches the production APKs. * we shouldn't have any non-production APK signed with the production keys. * non-production APKs are signed with "dep" * pushapkscript relies heavily on what key signed the APK. * ergo non-production pushapk_scriptworker must be called "dep" and there should no "dev" instance (for clarity). Code in puppet has been changed accordingly. D. Some parts of pushapkscript and mozapkpublisher print out they uploaded something on Google Play. In order to reduce the confusion, I made sure these log lines say they processed dummy things. I also added some forewords at the top of the execution of pushapkscript to recall what will happen whether we're on dep/prod or whether the commit option (which used to be called "dry-run") is turned on. [1] https://inventory.mozilla.org/en-US/core/search/#q=pushapkworker [2] https://github.com/mozilla-releng/build-cloud-tools/blob/bb5dda8b19b400e4ced61b49d0c33bcba21c6311/configs/depsigning-worker and https://github.com/mozilla-releng/build-cloud-tools/blob/bb5dda8b19b400e4ced61b49d0c33bcba21c6311/configs/signingworker. FIles are different but content is the same. [3] https://dxr.mozilla.org/build-central/rev/ca31b2a85d8fa313a245acde72021b8dc8744837/relabs-puppet/modules/signing_scriptworker/manifests/settings.pp#30 [4] https://docs.puppet.com/puppet/3.7/function.html#each and https://docs.puppet.com/puppet/3.7/experiments_future.html#enabling-the-future-parser [5] https://tools.taskcluster.net/auth/clients/project%2Freleng%2Fscriptworker%2Fpushapk%2Fdev

Comment on attachment 8930881 [details] [mozapkpublisher] push_apk.py: expose --do-not-contact-google-play Patch tested in integration with the rest of the stack in: https://tools.taskcluster.net/groups/dpTLsHJ2Ts6I-yy_4jKV1w/tasks/OIAbcNBRSyOEO1H0n_ftMQ/runs/8/logs/public%2Flogs%2Flive_backing.log

Bug 1420459 being done, I let dep-pushapk handle the task, with the right worker group and TC client[1]. Then, I deleted the old (and confusing) dev TC client[2]. What's left: 1. Land the mozapkpublisher/pushapkscript patches. 2. Release mozapkpublisher 0.5.0 and pushapkscript 0.4.0 3. Update the puppet patch to use these packages. 4. Get a review on the puppet patch. 5. Clean the staging mozapkpublisher==0.5.0-13a7be71ace8 and pushapkscript==0.4.0-2a9381d332a5 python packages from releng-puppet2 6. Land puppet in production and stop dep-pushapkworker-1 from being attached to my personal account. [1] https://tools.taskcluster.net/groups/H2ukXbn4Q1KMReCxVaYlMg/tasks/ISmfIBIkTfShf8eNYm0BKg/runs/0 [2] https://tools.taskcluster.net/auth/clients/project%2Freleng%2Fscriptworker%2Fpushapk%2Fdev - now a 404

I forgot: (In reply to Johan Lorenzo [:jlorenzo] from comment #11) > What's left: 0. Backport changes on the maple branch to m-c ... 7. Land the in-tree changes to autoland and let it reach m-c. 8. Profit!

Comment on attachment 8932053 [details] [pushapkscript] Enforce Google Play not being contacted if not a on known channel Addressed comments in the PR.

Comment on attachment 8930881 [details] [mozapkpublisher] push_apk.py: expose --do-not-contact-google-play :mtabara r+'d at [1]. Landed on master at [2] [1] https://github.com/mozilla-releng/mozapkpublisher/pull/48#pullrequestreview-79473400 [2] https://github.com/mozilla-releng/mozapkpublisher/commit/9b6e1e11f49afe154d446b9303006197946d8617

Comment on attachment 8932053 [details] [pushapkscript] Enforce Google Play not being contacted if not a on known channel Addressed tiny comments in the PR.

Comment on attachment 8932053 [details] [pushapkscript] Enforce Google Play not being contacted if not a on known channel https://github.com/mozilla-releng/pushapkscript/commit/7d5463f6d90c5a69993ca647610d63124da1834b

Comment on attachment 8930072 [details] [build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing https://reviewboard.mozilla.org/r/201250/#review209080 ::: modules/pushapk_scriptworker/manifests/init.pp:88 (Diff revision 6) > cot_job_type => 'pushapk', > > + sign_chain_of_trust => $pushapk_scriptworker::settings::sign_chain_of_trust, > + verify_chain_of_trust => $pushapk_scriptworker::settings::verify_chain_of_trust, > + verify_cot_signature => $pushapk_scriptworker::settings::verify_cot_signature, > + Interesting. I went with `env_config` since that seems more straightforward to me. If the switch statement works for you, that's cool.

Comment on attachment 8932437 [details] Bug 1412836 - Use dep-pushapk scritpworker for non m-c, m-b, m-r branches https://reviewboard.mozilla.org/r/203486/#review209092 This looks good. However, because we're switching from `dry-run` to `commit`, we're likely going to need to support both task definition formats until this patch rides the trains into m-r.

Great point, Aki. Thank you for spotting it. I made a follow up patch that still allows dry-run.

Comment on attachment 8932793 [details] [pushapkscript] Deprecate dry_run flag in task payload, instead of killing it Mihai reviewed this patch at https://github.com/mozilla-releng/pushapkscript/pull/31#pullrequestreview-79791380 Landed at: https://github.com/mozilla-releng/pushapkscript/commit/72e3c67cdf3f56cfef04204257e9e16bcb0b8634

Comment on attachment 8930072 [details] [build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing Landed on: * default https://hg.mozilla.org/build/puppet/rev/f785fcec174f9039797b276415dcde483d7002f6 * production: https://hg.mozilla.org/build/puppet/rev/a5ca7abd0121abf75e65832776deca2f28672feb

Comment on attachment 8923375 [details] [pushapkscript] Support artifacts signed by dep-signing Fixed silly linter issues on: * default: https://hg.mozilla.org/build/puppet/rev/bc00cd01968297289e882f4108d3962734f713ac * production: https://hg.mozilla.org/build/puppet/rev/a5eef6de0ad90396bef4e00ab4030c7628fd39f1

(In reply to Johan Lorenzo [:jlorenzo] from comment #11) Steps 0 to 6 done. All the patches landed worked on maple against the prod config of dep-pushapk. https://tools.taskcluster.net/groups/H2ukXbn4Q1KMReCxVaYlMg/tasks/ISmfIBIkTfShf8eNYm0BKg/runs/3 Time to get the in-tree patch landed.

Pushed by jlorenzo@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c5b5263f3c1c Use dep-pushapk scritpworker for non m-c, m-b, m-r branches r=aki

Comment on attachment 8932437 [details] Bug 1412836 - Use dep-pushapk scritpworker for non m-c, m-b, m-r branches The puppet patch didn't break the production instance of push-apk: https://tools.taskcluster.net/groups/LfAtlzm5Ro6nq0DNy4TbbA/tasks/DYTFuSrPQvmh39CGRB_cJw/runs/0/logs/public%2Flogs%2Flive_backing.log. See line: > 2017-11-30 00:59:46,903 - pushapkscript.script - WARNING - You will publish APKs to Google Play. This action is irreversible,if no error is detected either by this script or by Google Play. That line was added in attachment 8932053 [details]. Then, this bug is officially resolved.