Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=248419120&repo=mozilla-beta
Full log: https://queue.taskcluster.net/v1/task/ZtH_l64nTBqpIap6nM_y1Q/runs/0/artifacts/public/logs/live_backing.log

2019-05-26 11:59:06,537 - mozapkpublisher.common.googleplay - INFO - Uploading "/builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk"
2019-05-26 11:59:06,538 - googleapiclient.discovery - WARNING - media_mime_type argument not specified: trying to auto-detect for /builds/scriptworker/work/cot/dgurJDi3SzOEQC--9qROEg/public/build/target.apk
2019-05-26 11:59:06,593 - googleapiclient.discovery - INFO - URL being requested: POST https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/08285841811199333129/apks?alt=json&uploadType=media
2019-05-26 11:59:19,077 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "apkUpgradeVersionConflict"
Traceback (most recent call last):
File "/builds/scriptworker/bin/pushapkscript", line 11, in <module>
sys.exit(main())
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 95, in main
client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 164, in sync_main
loop.run_until_complete(_handle_asyncio_loop(async_main, context))
File "/tools/python3/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
await async_main(context)
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 54, in async_main
strings_file)
File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/googleplay.py", line 36, in publish_to_googleplay
skip_checks_fennec=bool(product_config.get('skip_checks_fennec')),
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 111, in push_apk
rollout_percentage,
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 135, in _upload_apks
edit_service.upload_apk(path)
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 66, in _transaction_required
return method(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 86, in upload_apk
media_body=apk_path
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/08285841811199333129/apks?alt=json&uploadType=media returned "APK specifies a version code that has already been used.">
exit code: 1

This issue has been known for a few years, but has never been an annoying-enough issue because:

• We hit occasionally on staging releases, and it was an expected error
• mozilla-central has always had a couple of pushes so we've always had new code to deploy

As a consequence, it's not an intermittent failure, per se, and it's not a regressions.

To me, the real fix would be to let the nightly decision task detect that a previous nightly graph was successfully resolved. However, that looks like an easy way to break existing workflow. Therefore, the easiest way is to just make the task not fail when the APKs have already been pushed. That's the paradigm we usually have on Taskcluster and we've implemented something similar in bug 1491262.

I'll get this fixed this cycle, because we'll see this bug more often once Fennec Nightly is built off mozilla-esr68.

r+'d by mhentges at https://github.com/mozilla-releng/build-puppet/pull/494#pullrequestreview-242584647
Landed on master at https://github.com/mozilla-releng/build-puppet/commit/bea1a9a7bead28c997c9d5f16bfda13b9dda2ce3

Deployed and tested: https://tools.taskcluster.net/groups/b-xfKZ8CRHSdf-yRym0gvA/tasks/QuFIZpdRRFmjpO4G3_PHVA/runs/1/logs/public%2Flogs%2Flive_backing.log#L211

push-apk doesn't error out anymore when an APK has already been shipped.