Joel Maher ( :jmaher ) (UTC -8) Assignee
	Description • 7 years ago

{'worker-type': 'aws-provisioner-v1/gecko-{level}-b-linux', 'index': {'product': 'firefox', 'job-name': 'linux64-asan-debug'}, 'run': {'script': 'mozharness/scripts/fx_desktop_build.py', 'secrets': True, 'need-xvfb': True, 'actions': ['get-secrets build check-test update'], 'custom-build-variant-cfg': 'asan-tc-and-debug', 'using': 'mozharness', 'config': ['builds/releng_base_linux_64_builds.py', 'balrog/production.py'], 'tooltool-downloads': 'public'}, 'description': 'Linux64 Debug ASAN', u'needs-sccache': True, 'treeherder': {u'tier': 1, 'platform': 'linux64/asan', 'symbol': 'tc(Bd)', u'kind': u'build'}, u'extra': {u'chainOfTrust': {u'inputs': {u'docker-image': {u'task-reference': u'<docker-image>'}}}}, 'worker': {u'chain-of-trust': True, u'docker-image': {u'in-tree': u'desktop-build'}, 'max-run-time': 36000, 'env': {'TOOLTOOL_MANIFEST': 'browser/config/tooltool-manifests/linux64/asan.manifest', 'PERFHERDER_EXTRA_OPTIONS': 'debug asan'}}, u'name': 'linux64-asan/debug'}
{'worker-type': 'aws-provisioner-v1/gecko-{level}-b-macosx64', 'index': {'product': 'firefox', 'job-name': 'macosx64-noopt-debug'}, 'run': {'script': 'mozharness/scripts/fx_desktop_build.py', 'secrets': True, 'actions': ['get-secrets build update'], 'custom-build-variant-cfg': 'cross-noopt-debug', 'keep-artifacts': False, 'using': 'mozharness', 'config': ['builds/releng_base_mac_64_cross_builds.py', 'balrog/production.py'], 'tooltool-downloads': 'internal'}, 'description': 'MacOS X x64 No-optimize Debug', u'name': 'macosx64-noopt/debug', u'needs-sccache': True, 'treeherder': {'tier': 2, 'platform': 'osx-10-7-noopt/debug', 'symbol': 'tc(B)', u'kind': u'build'}, u'extra': {u'chainOfTrust': {u'inputs': {u'docker-image': {u'task-reference': u'<docker-image>'}}}}, 'worker': {u'chain-of-trust': True, 'docker-image': {'in-tree': 'desktop-build'}, 'max-run-time': 36000, 'env': {'TOOLTOOL_MANIFEST': 'browser/config/tooltool-manifests/macosx64/cross-releng.manifest'}}, 'run-on-projects': ['stylo', 'autoland', 'mozilla-inbound', 'mozilla-central', 'try']}
{'index': {'product': 'firefox', 'job-name': 'macosx64-opt'}, 'run': {'tooltool-downloads': 'internal', 'script': 'mozharness/scripts/fx_desktop_build.py', 'secrets': True, 'using': 'mozharness', 'config': ['builds/releng_base_mac_64_builds.py', 'balrog/production.py'], 'actions': ['get-secrets build update']}, u'name': 'macosx64/opt', u'needs-sccache': True, 'worker-type': 'buildbot-bridge/buildbot-bridge', 'treeherder': {'tier': 2, 'platform': 'osx-10-7/opt', 'symbol': 'tc(B)', u'kind': u'build'}, 'description': 'MacOS X x64'}


the job I am failing on:
{'index': {'product': 'firefox', 'job-name': 'macosx64-opt'}, 'run': {'tooltool-downloads': 'internal', 'script': 'mozharness/scripts/fx_desktop_build.py', 'secrets': True, 'using': 'mozharness', 'config': ['builds/releng_base_mac_64_builds.py', 'balrog/production.py'], 'actions': ['get-secrets build update']}, u'name': 'macosx64/opt', u'needs-sccache': True, 'worker-type': 'buildbot-bridge/buildbot-bridge', 'treeherder': {'tier': 2, 'platform': 'osx-10-7/opt', 'symbol': 'tc(B)', u'kind': u'build'}, 'description': 'MacOS X x64'}

and there is no 'worker' in there, unlike the other jobs.  I assume this broke accidentally, although with a unittest for it, I am curious how it failed:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/test/test_try_option_syntax.py#307


just doing this locally, I am able to move on:
-        if env:
+        if env and 'worker' in job:

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 1 • 7 years ago

:dustin, do you think a small change like above in build.py (https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/build.py?q=job_env+%3D+job%5B%27worker%27%5D%5B%27env%27%5D&redirect_type=single#46 ), would be ok?

Is there anyone who would know more history on this code or why some jobs don't have a 'worker'?  I would rather cleanup more gotchas?  Likewise, are the unittests run anywhere?  Possibly they need to be updated?

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 2 • 7 years ago

I think you want job.setdefault('worker', {})['env']

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 3 • 7 years ago

looking at this, that sets job['worker'] = {}, so job['worker']['env'] fails.  I see two options:
1) skip the env clause when there is no worker?  (does it make sense to setenv when there is no worker?)
2) set |job_env = {}| if 'worker' not in job


I like option 2.  I still don't understand what a worker is and why a job wouldn't have one.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 4 • 7 years ago

I might as well put code up for option #2

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 5 • 7 years ago

The worker key contains information about the worker that will execute the job, including the command and environment variables.  We should really be using setdefault all over the place, but it's easy to forget and when things disappear later, KeyErrors occur at the line where the setdefault is required :)

	Pulsebot
	Comment 6 • 7 years ago

Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b976d46590c
pushing to try with --setenv fails the decision task, don't assume worker exists on a job. r=dustin

	Ryan VanderMeulen [:RyanVM]
	Comment 7 • 7 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/5b976d46590c