2017-06-22 22:16:52,656 - DEBUG - "POST /releases/Devedition-55.0b4-build2 HTTP/1.1" 200 0 2017-06-22 22:16:52,657 - ERROR - Failed to start release "Devedition-55.0b4-build2" promotion for graph f1Scc3S4Qnu-mS5ii2UkYw. Error(s): You didn't give the task-graph scopes allowing it define tasks on the queue. Traceback (most recent call last): File "release-runner.py", line 517, in main print(scheduler.createTaskGraph(graph_id, graph)) File "/builds/releaserunner/lib/python2.7/site-packages/taskcluster/client.py", line 455, in apiCall return self._makeApiCall(e, *args, **kwargs) File "/builds/releaserunner/lib/python2.7/site-packages/taskcluster/client.py", line 232, in _makeApiCall return self._makeHttpRequest(entry['method'], route, payload) File "/builds/releaserunner/lib/python2.7/site-packages/taskcluster/client.py", line 424, in _makeHttpRequest superExc=rerr TaskclusterRestFailure: You didn't give the task-graph scopes allowing it define tasks on the queue.

The release_eta changes (in releasetasks) don’t seem to add any new scopes, and anyway I left the eta out in ship-it. The TC client doesn’t seem to have changed recently, so TC bustage ?

More precisely, here's what TC returns: > 2017-06-23 05:33:58,909 - DEBUG - "PUT /v1/task-graph/rQhDIK5dR5aO-IHCtT6IjQ HTTP/1.1" 400 6579483 > 2017-06-23 05:33:59,854 - DEBUG - Received HTTP Status: 400 > 2017-06-23 05:33:59,855 - DEBUG - Received HTTP Headers: {'content-length': '6579483', 'via': '1.1 vegur', 'x-powered-by': 'Express', 'server': 'Cowboy', 'access-control-request-method': '*', 'connection': 'keep-alive', 'date': 'Fri, 23 Jun 2017 12:33:58 GMT', 'access-control-allow-origin': '*', 'access-control-allow-methods': 'OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT', 'content-type': 'application/json; charset=utf-8', 'access-control-allow-headers': 'X-Requested-With,Content-Type,Authorization,Accept,Origin'} > > 2017-06-23 05:33:59,880 - DEBUG - Received HTTP Payload: { > "message": "You didn't give the task-graph scopes allowing it define tasks on the queue.", > "error": [ > { > "message": "Unauthorized: Bad mac\n----\nmethod: defineTask\nerrorCode: AuthenticationFailed\nstatusCode: 401\ntime: 2017-06-23T12:33:56.361Z", > "error": { > "code": "AuthenticationFailed", > "message": "Unauthorized: Bad mac\n----\nmethod: defineTask\nerrorCode: AuthenticationFailed\nstatusCode: 401\ntime: 2017-06-23T12:33:56.361Z", > "requestInfo": { > "method": "defineTask", > "params": { > "taskId": "goQtnZ6XRHCPvUtaBaMVdg" > }, > "payload": { > "workerType": "gecko-images", > "retries": 5, > "expires": "3017-06-23T12:32:36.662Z", > "created": "2017-06-23T12:32:36.585Z", > "extra": { > "build_props": { > "product": "devedition", > "build_number": 2, > "platform": null, > "partials": "55.0b1build5,55.0b2build1,55.0b3build1", > "version": "55.0b4", > "branch": "mozilla-beta", > "mozharness_changeset": "d4290c564d34062be68d216be6cf0d41f37c8227", > "release_eta": null, > "locales": [ > null > ], > "revision": "d4290c564d34062be68d216be6cf0d41f37c8227" > }, > "notifications": { > "task-failed": { > "message": "Uh-oh! Generate funsize-balrog-submitter docker image failed.", > "ids": [ > "releasetasks" > ], > "subject": "Failed: Generate funsize-balrog-submitter docker image" > }, > "task-completed": { > "message": "Generate funsize-balrog-submitter docker image has completed successfully! Yay!", > "ids": [ > "releasetasks" > ], > "subject": "Completed: Generate funsize-balrog-submitter docker image" > }, > "task-exception": { > "message": "Uh-oh! Generate funsize-balrog-submitter docker image resulted in an exception.", > "ids": [ > "releasetasks" > ], > "subject": "Exception: Generate funsize-balrog-submitter docker image" > } > }, > "signing": { > "signature": "eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE0OTgyMjExNTcsInZlcnNpb24iOiIxIiwidGFza0lkIjoiZ29RdG5aNlhSSENQdlV0YUJhTVZkZyIsImV4cCI6MTQ5ODU2Njc1N30.MO5Jn95WAdEVd1XI8A48tQ_CoWZXzlrs19z357m9d8SKA9sHDgSUvDq6h3RYpi1ma9e7QW_VuCNDjggc5AcKBL4ct5TwpulzWFWiENnYCpaAwrmIgkt5jahXhG2a8D6DdtdWUCNPA4CuTYZWgnoYgX2JI-maolTX0VArqFKVbSQbtDhXYR0Nih2IOR-PCkCYu_5xxtXgDG6T2dV7-XxolTcRuI1Ijpb44mdRG2lCRkklC844rFlbB_2DyzT4cvtUCYt76nT4IsmXfK-_tdcreL1GWSjYluRe6ydjbB2Rogb_GiR4wmvG4aRl-7gNukawhZTmL4u2biT3M4-XS24Kvw" > }, > "treeherder": { > "machine": { > "platform": "linux64" > }, > "symbol": "Fb", > "groupSymbol": "Images", > "collection": { > "opt": true > }, > "build": { > "platform": "linux64" > } > }, > "treeherderEnv": [ > "staging", > "production" > ] > }, > "priority": "high", > "deadline": "2017-06-27T12:32:36.585Z", > "routes": [ > "tc-treeherder-stage.v2.mozilla-beta.d4290c564d34062be68d216be6cf0d41f37c8227.7444", > "tc-treeherder.v2.mozilla-beta.d4290c564d34062be68d216be6cf0d41f37c8227.7444", > "index.releases.v1.mozilla-beta.d4290c564d34062be68d216be6cf0d41f37c8227.devedition.55_0b4.build2.funsize_balrog_image", > "index.releases.v1.mozilla-beta.latest.devedition.latest.funsize_balrog_image" > ], > "payload": { > "artifacts": { > "public/image.tar.zst": { > "path": "/home/worker/workspace/artifacts/image.tar.zst", > "expires": "2018-06-23T12:32:36.585296Z", > "type": "file" > } > }, > "maxRunTime": 1800, > "image": "taskcluster/image_builder:1.3.0", > "features": { > "dind": true > }, > "env": { > "HASH": "d4290c564d34062be68d216be6cf0d41f37c8227", > "HG_STORE_PATH": "/home/worker/checkouts/hg-store", > "GECKO_HEAD_REV": "d4290c564d34062be68d216be6cf0d41f37c8227", > "IMAGE_NAME": "funsize-balrog-submitter", > "PROJECT": "mozilla-beta", > "GECKO_BASE_REPOSITORY": "https://hg.mozilla.org/mozilla-unified", > "GECKO_HEAD_REPOSITORY": "https://hg.mozilla.org/releases/mozilla-beta", > "CONTEXT_URL": "" > } > }, > "provisionerId": "aws-provisioner-v1", > "metadata": { > "owner": "release@mozilla.com", > "source": "https://github.com/mozilla/releasetasks", > "name": "Generate funsize-balrog-submitter docker image", > "description": "Generate funsize-balrog-submitter docker image" > }, > "schedulerId": "task-graph-scheduler", > "taskGroupId": "rQhDIK5dR5aO-IHCtT6IjQ", > "dependencies": [], > "requires": "all-completed", > "scopes": [], > "tags": {} > }, > "time": "2017-06-23T12:33:56.361Z" > } > } > },

It turns out to be a problem on TC's side. From :garndt in #taskcluster: > 17:00:21 <•garndt> Callek: jlorenzo bhearsum I'm deeply sorry for this and the wasted time around it. So the token was rotated and the heroku app updated with that token and there really shouldn't have been any fall out. But from what I can tell there are *2* places where that access token needs to be updated, and not just one. It was updated in the wrong/duplicate spot rather than in the spot the scheduler was looking. This is one of many reasons we > 17:00:21 <•garndt> need to deprecate the use of this service