"Create Interactive Task" on Treeherder for docker workers fails to set certain environment variables
Categories
(Tree Management :: Treeherder, defect)
Tracking
(Not tracked)
People
(Reporter: whimboo, Assigned: jmaher)
Details
Attachments
(1 file, 1 obsolete file)
(deleted),
text/x-phabricator-request
|
Details |
With bug 1749977 fixed I'm not able to at least start an interactive task and logging in the shell. But running the job fails because certain payload["env"]
variables aren't set.
Here the list of environment variables when I manually create an interactive task:
"payload": {
"env": {
"PYTHON": "python3",
"GECKO_PATH": "/builds/worker/checkouts/gecko",
"ENABLE_E10S": "false",
"MOZ_FETCHES": "[{\"artifact\": \"project/gecko/android-emulator/android-emulator-linux.tar.zst\", \"extract\": true, \"task\": \"G8xi-fKMSNOXThTMl5O37A\"}, {\"artifact\": \"project/gecko/android-system-images/android-system-images-linux.tar.zst\", \"extract\": true, \"task\": \"JfVCYi-xTKyRt0gdg2UO8g\"}, {\"artifact\": \"public/build/android-avd-linux.tar.zst\", \"extract\": true, \"task\": \"TPhaOrYCR9Ss4eGi3nWY_g\"}, {\"artifact\": \"public/build/fix-stacks.tar.zst\", \"extract\": true, \"task\": \"IHj_FG8iReu9My7C2WPjFw\"}, {\"artifact\": \"public/build/geckodriver.tar.gz\", \"extract\": true, \"task\": \"JhygPz6kTWmFGLJNmp5JaQ\"}, {\"artifact\": \"public/build/minidump_stackwalk.tar.zst\", \"extract\": true, \"task\": \"AP2mvfl3TOqzOJlknuC0pQ\"}]",
"NEED_COMPIZ": "true",
"WORKING_DIR": "/builds/worker",
"TRY_SELECTOR": "fuzzy",
"HG_STORE_PATH": "/builds/worker/checkouts/hg-store",
"MOZ_NODE_PATH": "/usr/local/bin/node",
"MOZ_SCM_LEVEL": "1",
"GECKO_HEAD_REV": "6e2c75862193eb7abdaa918ee6d6c2f1debd4b1f",
"MOZHARNESS_URL": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IRv3p8jxSVCelNE5dux7mA/artifacts/public/build/mozharness.zip",
"MOZ_AUTOMATION": "1",
"TOOLTOOL_CACHE": "/builds/worker/tooltool-cache",
"TRY_COMMIT_MSG": "",
"MOZ_FETCHES_DIR": "fetches",
"NEED_PULSEAUDIO": "true",
"SCCACHE_DISABLE": "1",
"MOZHARNESS_CONFIG": "android/android-x86_64.py web_platform_tests/prod_config_android.py",
"MOZHARNESS_SCRIPT": "web_platform_tests.py",
"MOZILLA_BUILD_URL": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IRv3p8jxSVCelNE5dux7mA/artifacts/public/build/geckoview-test_runner.apk",
"MOZHARNESS_ACTIONS": "get-secrets",
"TASKCLUSTER_CACHES": "/builds/worker/checkouts;/builds/worker/tooltool-cache",
"NEED_WINDOW_MANAGER": "true",
"TASKCLUSTER_VOLUMES": "/builds/worker/.cache;/builds/worker/checkouts;/builds/worker/tooltool-cache;/builds/worker/workspace",
"GECKO_BASE_REPOSITORY": "https://hg.mozilla.org/mozilla-unified",
"GECKO_HEAD_REPOSITORY": "https://hg.mozilla.org/try",
"MOZHARNESS_TEST_PATHS": "{\"web-platform-tests-wdspec\": [\"/_mozilla/webdriver\", \"/_mozilla/webdriver/get_window_handle\", \"/_mozilla/webdriver/protocol\", \"/webdriver/tests/accept_alert\", \"/webdriver/tests/bidi\", \"/webdriver/tests/delete_cookie\", \"/webdriver/tests/element_clear\", \"/webdriver/tests/execute_async_script\", \"/webdriver/tests/find_element_from_element\", \"/webdriver/tests/find_elements_from_element\", \"/webdriver/tests/fullscreen_window\", \"/webdriver/tests/get_computed_label\", \"/webdriver/tests/get_element_attribute\", \"/webdriver/tests/get_element_rect\", \"/webdriver/tests/get_element_text\", \"/webdriver/tests/get_timeouts\", \"/webdriver/tests/get_window_handles\", \"/webdriver/tests/is_element_enabled\", \"/webdriver/tests/minimize_window\", \"/webdriver/tests/new_window\", \"/webdriver/tests/print\", \"/webdriver/tests/send_alert_text\", \"/webdriver/tests/status\", \"/webdriver/tests/switch_to_window\"]}",
"TESTS_BY_MANIFEST_URL": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MazuoCEvTquzpFcIq9ZaQw/artifacts/public/tests-by-manifest.json.gz",
"EXTRA_MOZHARNESS_CONFIG": "{\"installer_url\": \"https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IRv3p8jxSVCelNE5dux7mA/artifacts/public/build/geckoview-test_runner.apk\", \"test_packages_url\": \"https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IRv3p8jxSVCelNE5dux7mA/artifacts/public/build/target.test_packages.json\"}",
"TASKCLUSTER_UNTRUSTED_CACHES": "1"
},
And when I use Treeherder to create the interactive task the list has been drastically reduced:
"env": {
"ACTION_INPUT": "{\"notify\":\"hskupin@mozilla.com\"}",
"HG_STORE_PATH": "/builds/worker/checkouts/hg-store",
"ACTION_TASK_ID": "\"BdMN5w-0TAOJK6-cav6QnA\"",
"GECKO_HEAD_REF": "6e2c75862193eb7abdaa918ee6d6c2f1debd4b1f",
"GECKO_HEAD_REV": "6e2c75862193eb7abdaa918ee6d6c2f1debd4b1f",
"MOZ_AUTOMATION": "1",
"ACTION_CALLBACK": "create-interactive",
"TASKCLUSTER_CACHES": "/builds/worker/checkouts",
"ACTION_TASK_GROUP_ID": "MazuoCEvTquzpFcIq9ZaQw",
"GECKO_BASE_REPOSITORY": "https://hg.mozilla.org/mozilla-unified",
"GECKO_HEAD_REPOSITORY": "https://hg.mozilla.org/try",
"PYTHONDONTWRITEBYTECODE": "1"
},
This means that a lot of environment variables as needed by mozharness and various tools is missing and cause jobs to hang like:
08:03:07 INFO - Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/f3ORh5dPSVuNY_sXnwbBqA/artifacts/public
/tests-by-manifest.json.gz to fetches/tests-by-manifest.json.gz
08:03:07 INFO - retry: Calling _download_file with args: (), kwargs: {'url': 'https://firefox-ci-tc.services.mozilla.com/api/que
ue/v1/task/f3ORh5dPSVuNY_sXnwbBqA/artifacts/public/tests-by-manifest.json.gz', 'file_name': 'fetches/tests-by-manifest.json.gz'}, at
tempt #1
Note that the above line incorrectly references the current task id instead the one from the Gecko decision task.
Assignee | ||
Comment 1•3 years ago
|
||
Updated•3 years ago
|
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/999cb8fe77d3 add environment variables to interactive tasks. r=releng-reviewers,gbrown
Comment 4•3 years ago
|
||
bugherder |
Reporter | ||
Comment 5•3 years ago
|
||
Thank you Joel. I had a look but sadly this still doesn't work for me. I rebased my patch to enable wdspec on Android and submitted a new try build. But when starting an interactive task the expected environment variables are still not present and the job fails to download the test manifests.
https://treeherder.mozilla.org/jobs?repo=try&revision=1fcf7e4298fcc38521aeec7f0ac2cece9e944408&selectedTaskRun=TY5FB9BiQw-iFS3XP8O0mQ.0
https://firefox-ci-tc.services.mozilla.com/tasks/TY5FB9BiQw-iFS3XP8O0mQ/definition
Assignee | ||
Comment 6•3 years ago
|
||
this was working for me on wdspec linux tests, I don't know why android is a problem. Can you verify this works for you on wdspec linux so I can know if this is android only or all things.
Reporter | ||
Comment 7•3 years ago
|
||
I tried re-triggering this Wdspec job:
https://firefox-ci-tc.services.mozilla.com/tasks/aAgRYqVvTPCMP7ia_eC88Q
And it contains definitely more environment variables. So it might be indeed specific to Android builds.
Nevertheless the variable TESTS_BY_MANIFEST_URL
is still wrong and points to the interactive task itself instead of the gecko decision task and as such the wdspec job of the created interactive build will still fail when resuming the process in the shell.
Reporter | ||
Comment 8•3 years ago
|
||
Hi Joel, do you have an update for a possible fix? It would be kinda helpful to finally being able to run interactive tasks for wdspec tests. Thanks.
Assignee | ||
Comment 9•2 years ago
|
||
Updated•2 years ago
|
Description
•