We have to release a new marionette driver and harness package to PyPI to allow consumers of Marionette to make use of the new window command, and have some bug fixes.

The last releases of both packages is already a while ago. So here the long cleaned-up list of changes:

Marionette driver:

% hg log -M -r 721395128428:: --template "{desc|firstline}\n" .
Bug 1440577 - Marionette has to use "eForceQuit" for in_app shutdown and restart requests r=whimboo
Bug 1434872 - [marionette] Firefox has to automatically dismiss "beforeunload" prompts. r=ato
Bug 1348145 - Remove getWindowPosition/setWindowPosition and getWindowSize/setWindowSize. r=whimboo
Bug 1466658 - [geckodriver] Overhaul of user preferences in prefs.rs. r=ato
Bug 1441204 - Revert removal of extensions.showMismatchUI pref. r=maja_zf
Bug 1467700 - Disable download of OpenH264 and Widevine plugins for Marionette and geckodriver. r=ato
Bug 1448040 - Remove HangMonitor/ChromeHangs r=Nika
Bug 1408509 - Add web frame, web window, and chrome element support to Marionette client. r=whimboo
Bug 1473262 - [marionette] Force client to always use IPv4 to connect to Marionette. r=ato
Bug 1452490 - Remove marionetteScriptFinished. r=whimboo
Bug 1420514 - Replace app.update.enabled with app.update.disabledForTesting in the test harness r=ato,chutten,jmaher,lina
Bug 1400233 - Drop ContentWebElement.LegacyIdentifier key from Marionette. r=ato,whimboo
Bug 1480105 - Using WebDriver:AcceptAlert instead of WebDriver:AcceptDialog in Marionette. r=whimboo
Bug 1482829 - Track Marionette logger verbosity with Log#manageLevelFromPref. r=whimboo
Bug 1213008 - Add Marionette client option to save screenshot to a file. r=ato
Bug 905404 - [marionette] Add missing preferences from mozprofile. r=ato
Bug 1433873 - Fix race condition in Marionette client for in_app quit and restart. r=ato
Bug 1497566 - Remove logcat functionality from mozrunner; r=automatedtester
Bug 1497094 - Add missing attribute is_shutting_down to Marionette python client r=ato
Bug 1500242 - [marionette] Don't modify DEFAULT_SHUTDOWN_TIMEOUT constant. r=ato
Bug 1500242 - [marionette] Obey the extended shutdown timeout for Firefox ASAN builds. r=ato
bug 1504223: marionette: stop duplicating capabilities in new session request; r=whimboo
bug 1502864: marionette: enable strict file interactability in Python client; r=whimboo
bug 1504478: marionette: use SingleTap and AcceptConnections in Python client; r=whimboo
Bug 1488826 - [marionette] Disable content blocking introduction panel. r=ato
Bug 1506643 - [marionette] Disable "about:newtab" page, and always load "about:blank". r=ato
Bug 1480544 - Allow Console API to log messages on stdout, r=bgrins
Bug 1506917 - Don't attempt to check for updates from Firefox's nsBrowserGlue.js when running tests. r=ted
Bug 1509256 - Clarify where to add preferences for geckodriver and Marionette. r=ato
Bug 1509256 - Fix application update preferences for backward compatibility in Marionette and geckodriver. r=ato,bytesized
Bug 1511311 - [marionette] GeckoInstance classes should not overwrite already set custom preferences. r=automatedtester
Bug 1508726 - [marionette] Use "app.update.disabledForTesting" to disable application updates. r=ato
Bug 1420372 - Implement Marionette check for a valid Firefox pid and unit tests. r=whimboo
bug 1510929: marionette: stop using scriptTimeout parameter; r=whimboo
Bug 1504756 - [marionette] Added opening a new browsing context to Marionette client. r=ato
Bug 1517442 - [marionette] Use current instead of default startup time in raise_for_port r=whimboo
Bug 1524243 - [marionette] Correctly handle script_timeout for WebDriver:{ExecuteScript,ExecuteAsyncScript}. r=jgraham
Bug 1524243 - [marionette] Correct calculation of timeout from milliseconds to seconds. r=jgraham
Bug 1515746 - [flake8] Unsupport subdir .flake8 files and use new 'per-file-ignores' config instead, r=egao
Bug 1522637 - Part 8: Allow 'element_stale' to work after process swap, r=ato
automatedtester
Bug 1528836 - [marionette] Allow check_for_port to raise exceptions. r=automatedtester
Bug 1529077 - [marionette] Encode Unicode error messages as UTF-8. r=ato
Bug 1531068 - [marionette] Remove remaining links to readthedocs from client documentation. r=me

Marionette harness:

% hg log -M -r 721395128428:: --template "{desc|firstline}\n" .
Bug 1458280 - Release marionette_driver 2.7.0 and marionette_harness 4.5.0; r=whimboo
Bug 1441433 - Move test_carets_* files into layout subdirectory. r=ato
Bug 1434872 - [marionette] Fix remote_page checks in bf_cache navigtion tests. r=ato
Bug 1434872 - [marionette] Firefox has to automatically dismiss "beforeunload" prompts. r=ato
Bug 1434907 - [marionette] Remove dependency to hidden.html in unit tests. r=whimboo
Bug 1141118 - [marionette] In shuffle mode log the seed value before the tests are run. r=whimboo
Bug 1461463 - [marionette] Add validation for known contexts. r=ato
Bug 1461463 - [marionette] Empty response value should be null and not {}. r=ato
Bug 1387678 - Update tests to not use shadow DOM like a frame; r=ato
Bug 1387678 - Reenable Shadow DOM tests; r=ato
Bug 1348145 - Remove getWindowPosition/setWindowPosition and getWindowSize/setWindowSize. r=whimboo
Bug 1417920 - [python-test] Use a global pytest.ini configuration file, r=davehunt
Bug 1408509 - Add web frame, web window, and chrome element support to Marionette client. r=whimboo
Bug 1468198 - remove tests that only run on b2g. r=smaug,mcmanus,whimboo
Bug 1470646 - Update Mn and Fxfn tests to use WebDriver conforming platformName. r=whimboo
Bug 1349223 - update marionette a11y test based on the new handling of aria-hidden. r=ato
Bug 1473262 - [marionette] Force client to always use IPv4 to connect to Marionette. r=ato
Bug 1264259 - [marionette] Add support for unhandledPromptBehavior capability. r=ato
Bug 1452490 - Remove marionetteScriptFinished. r=whimboo
Bug 1286882 - Make SVG basic shape elements unselectable. r=dholbert
Bug 1439995 - [marionette] Add user prompt checks for script execution commands. r=ato
Bug 1478979 - Disallow pageLoadStrategy to be null. r=whimboo
Bug 1470659 - Add setWindowRect capability to Marionette. r=whimboo
Bug 1319793 - [marionette harness tests] Default to not swallow stdout from pytest. r=whimboo
Bug 1451725 - Remove deprecated WebDriver Commands r=whimboo
Bug 1337899 - [marionette] - Fix construction of script src. r=ato
Bug 1478094 - [marionette] Update crash unit tests to make use of about:crashparent and about:crashcontent r=whimboo
Bug 1485580 - [marionette] Update unit tests for invalid timeout values. r=jgraham
Bug 1485730 - [marionette] Limit width and height of created canvas to 32767 pixels. r=ato
Bug 1213008 - Add Marionette client option to save screenshot to a file. r=ato
Bug 1398111 - [marionette] Never disable focus manager test mode for unit tests. r=ato
Bug 1433873 - Fix race condition in Marionette client for in_app quit and restart. r=ato
Bug 1488468 - Bump wptserve in marionette_requirements.txt to 2.0.0; r=davehunt
Bug 1493796 - [mozprocess] Revert poll() behavior on Windows due to regression. r=gbrown
Bug 1500242 - [marionette] Don't modify DEFAULT_SHUTDOWN_TIMEOUT constant. r=ato
Bug 1500242 - [marionette] Obey the extended shutdown timeout for Firefox ASAN builds. r=ato
Bug 1502190 - bump mozdevice requirements for raptor, marionette, mozrunner, r=gbrown
Bug 1416747 - [marionette] Hardening test_cli_arguments.py TestCommandLineArguments.test_startup_timeout. r=ato
Bug 1504117 - [mozdevice] Bump mozdevice requirements to 1.1.6 for raptor, marionette, mozrunner, r=gbrown
bug 1504174: marionette: delete unused test_window_fullscreen.py test; r=automatedtester
bug 1502864: marionette: enable strict file interactability in Python client; r=whimboo
bug 1504478: marionette: update tests to use non-deprecated commands; r=whimboo
bug 1492499: marionette: drop broken window maximization test; r=automatedtester
Bug 1506796 - use OpenBrowserWindow instead of window.open() to create a browser window, r=whimboo
Bug 1503019 - Part I, Remove dom.webcomponents.shadowdom.enabled r=smaug
Bug 1423709 - Add initial-scale=1 for tests supposing the page is rendered 1:1 scale. r=botond
Bug 1503776 - Add marionette-gpu job. r=jmaher
Bug 1507803 - [marionette] Allow to open a chrome window via the WindowManager mixin class. r=ato
Bug 1504201 - "Intermittent Linux Mn testing/marionette/harness/marionette_harness/tests/unit/test_screenshot.py TestScreenCaptureChr
Bug 1508726 - [marionette] Use "app.update.disabledForTesting" to disable application updates. r=ato
Bug 1436037 - [python] Create Windows python-test tasks, r=gps
Bug 1503674 - Remove unused microformats code. r=Gijs
Bug 1420372 - Implement Marionette check for a valid Firefox pid and unit tests. r=whimboo
Bug 1517587 - [marionette] Get/Set Window Rect should not return window state. r=ato
bug 1385706: marionette: fix WebDriver:TakeScreenshot to use viewport bounds; r=ato
Bug 1504756 - [marionette] Added opening a new browsing context to Marionette client. r=ato
Bug 1441437 - [marionette] Remove dependency to cssTransform.html and cssTransform2.html in test_visibility.py. r=ato
Bug 1512336 - [marionette] Fix hang in TestNavigate.test_set_location_through_execute_script. r=ato
Bug 1510693 - Tests for marionette's reftest runner r=jgraham,ato
Bug 1510695 - Fix URL comparisons when loading reftests r=jgraham
Bug 1510696 - Make width/height of reftests configurable over Marionette r=jgraham,ato
Bug 1519552 - Disable test_reftest.py on Android r=jgraham,whimboo
Bug 1423013 - Set explicit minimum-scale=1 to avoid overflow:hidden area reachable and visible. r=botond
Bug 1522413 - [marionette] Register event listeners before focusing the new window in WindowManagerMixin.open_chrome_window(). r=ato
Bug 1524243 - [marionette] Correctly handle script_timeout for WebDriver:{ExecuteScript,ExecuteAsyncScript}. r=jgraham
Bug 1524243 - [marionette] Correct calculation of timeout from milliseconds to seconds. r=jgraham
Bug 1487065 - Implement popup-notification as a Custom Element r=MattN
Bug 1515746 - [flake8] Unsupport subdir .flake8 files and use new 'per-file-ignores' config instead, r=egao
Bug 1526084 - In Mn test_capabilities.py, allow for multiple links to sdcard; r=whimboo
Bug 1526913 - In Mn test_cookies.py allow for some clock difference between host and device; r=whimboo
Bug 1525829 - Add buildID to returned capabilities list for Marionette; r=ato
Bug 1529077 - [marionette] Encode Unicode error messages as UTF-8. r=ato
Bug 675539 - Make tab discard functionality work on tab objects directly r=mconley

https://treeherder.mozilla.org/#/jobs?repo=try&revision=674463ee19f32873f15a406b7884db0b5973a395

Nearly all dependencies except moznetwork and moztest already
support Python3. Lets make those releases the minimum allowed
version.

Depends on D22105

Marking as leave-open for pushing packages to PyPI.

Backed out 2 changesets (Bug 1530794) for ab python package install failures.

Backout: https://hg.mozilla.org/integration/autoland/rev/a59527dafca1d132facf8830c35cf1239ab7f0c2

Push that started the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=d03269ca7469f0160cd550537aa6c187890e99ea&selectedJob=231975627

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231975627&repo=autoland&lineNumber=457

It's broken because the browsermob-proxy version as specified doesn't exist on the internal pypi mirror. I will reland the patch without the Python3 revision for now.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] (away 02/28 - 03/03) from comment #8)

It's broken because the browsermob-proxy version as specified doesn't exist on the internal pypi mirror. I will reland the patch without the Python3 revision for now.

Given that I lost my local commits from the appropriate bookmark I changed my mind and requested an upload of the browsermob-proxy v0.8.0 package to the internal pypi server. Once that is done I will re-land my patch.

Backed out 2 changesets (bug 1530794) for failing to install python package on a CLOSED TREE.

Backout link: https://hg.mozilla.org/integration/autoland/rev/61db2b186d69f0a894096bd7cab0777cc475a1bf

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&searchStr=os%2Cx%2C10.10%2Copt%2Care%2Cwe%2Cslim%2Cyet%2Ctests%2Cby%2Ctaskcluster%2Cwith%2Ce10s%2Ctest-macosx64%2Fopt-awsy-base-e10s%2Csy-e10s%28ab%29&revision=c15813f5f72f253dae60495b68e84ee71b16a65d&selectedJob=232118526

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=232118526&repo=autoland&lineNumber=457

Log snippet:

10:53:57 WARNING - Return code: 1
10:53:57 INFO - retry: Failed, sleeping 300 seconds before retrying
10:58:57 INFO - retry: Calling run_command with args: [['/Users/cltbld/tasks/task_1551868333/build/venv/bin/pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub', '/Users/cltbld/tasks/task_1551868333/build/tests/awsy']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x10dfc0c70>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10e45a6c0>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x7fb5cb576e00>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10e274408>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10df9caf0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x10df7b648>, 'level': 'critical'}], 'cwd': '/Users/cltbld/tasks/task_1551868333/build', 'env': {'MOZ_AUTOMATION': '1', 'VERSIONER_PYTHON_PREFER_32_BIT': 'no', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin', 'HOME': '/Users/cltbld', 'IDLEIZER_DISABLE_SHUTDOWN': 'true', 'LANG': 'en_US.UTF-8', 'XPCOM_DEBUG_BREAK': 'warn', 'Apple_PubSub_Socket_Render': '/private/tmp/com.apple.launchd.ykT7nFRQF3/Render', 'MOZ_NODE_PATH': '/usr/local/bin/node', 'VERSIONER_PYTHON_VERSION': '2.7', 'SHLVL': '1', 'XPC_FLAGS': '0x0', 'TMPDIR': '/var/folders/1y/m75xwyrn6xz14pfpnjwlqwt800000x/T/', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/autoland', 'NO_FAIL_ON_TEST_ERRORS': '1', 'SCCACHE_DISABLE': '1', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.XMkAjsYLip/Listeners', 'MOZ_NO_REMOTE': '1', 'MOZ_HIDE_RESULTS_TABLE': '1', 'GECKO_HEAD_REV': 'c15813f5f72f253dae60495b68e84ee71b16a65d', 'XPC_SERVICE_NAME': '0', 'SHELL': '/bin/bash', 'LC_ALL': 'en_US.UTF-8', '_': '/usr/local/bin/generic-worker', 'TASK_ID': 'NXg2fuybTqCoUXxW8FtYzA', '__CF_USER_TEXT_ENCODING': '0x1D:0:0', 'PWD': '/Users/cltbld', 'TASKCLUSTER_ROOT_URL': 'https://taskcluster.net'}}, attempt #5
10:58:57 INFO - Running command: ['/Users/cltbld/tasks/task_1551868333/build/venv/bin/pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub', '/Users/cltbld/tasks/task_1551868333/build/tests/awsy'] in /Users/cltbld/tasks/task_1551868333/build
10:58:57 INFO - Copy/paste: /Users/cltbld/tasks/task_1551868333/build/venv/bin/pip install --timeout 120 --no-index --find-links https://pypi.pub.build.mozilla.org/pub /Users/cltbld/tasks/task_1551868333/build/tests/awsy
10:58:57 INFO - Using env: (same as previous command)
10:58:57 INFO - Ignoring indexes: https://pypi.python.org/simple
10:58:57 INFO - Processing ./tests/awsy
10:58:58 INFO - Requirement already satisfied (use --upgrade to upgrade): marionette_harness in ./venv/lib/python2.7/site-packages (from awsy==0.0.1)
10:58:58 INFO - Collecting browsermob-proxy>=0.8.0 (from marionette_harness->awsy==0.0.1)
10:59:00 INFO - Could not find a version that satisfies the requirement browsermob-proxy>=0.8.0 (from marionette_harness->awsy==0.0.1) (from versions: 0.6.0, 0.7.1)
10:59:00 INFO - No matching distribution found for browsermob-proxy>=0.8.0 (from marionette_harness->awsy==0.0.1)
10:59:00 WARNING - Return code: 1
10:59:00 FATAL - Could not install python package: failed all attempts.
10:59:00 FATAL - Running post_fatal callback...
10:59:00 FATAL - Exiting -1
10:59:00 INFO - Running post-action listener: _resource_record_post_action
10:59:00 INFO - Running post-action listener: _start_resource_monitoring
10:59:00 INFO - Starting resource monitoring.
10:59:00 INFO - [mozharness: 2019-03-06 10:59:00.276468Z] Finished create-virtualenv step (failed)
10:59:00 INFO - Running post-run listener: _resource_record_post_run
[taskcluster 2019-03-06T10:59:00.419Z] Exit Code: 255
[taskcluster 2019-03-06T10:59:00.419Z] User Time: 1m0.614433s
[taskcluster 2019-03-06T10:59:00.419Z] Kernel Time: 19.059159s
[taskcluster 2019-03-06T10:59:00.419Z] Wall Time: 13m58.498122464s
[taskcluster 2019-03-06T10:59:00.419Z] Result: FAILED
[taskcluster 2019-03-06T10:59:00.419Z] === Task Finished ===
[taskcluster 2019-03-06T10:59:00.419Z] Task Duration: 13m58.498316237s
[taskcluster 2019-03-06T10:59:00.884Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2020-03-05T09:57:19.844Z
[taskcluster 2019-03-06T10:59:01.606Z] Uploading artifact public/test_info/resource-usage.json from file build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2020-03-05T09:57:19.844Z
[taskcluster:error] exit status 255

Same problem as before. I should have checked myself that the browsermob-proxy Python package is present on the internal pypi server before pushing my patch again. Sorry.

After speaking to CIDuty it's not clear how an update works for the internal pypi mirror, and if there is a problem with tooltool. Given that we don't necessarily need the 0.8.0 release of browsermob-proxy lets simply leave the dependency as is.

https://hg.mozilla.org/mozilla-central/rev/0826fd0e4d5c
https://hg.mozilla.org/mozilla-central/rev/bfa18558c4e7

Both packages have been deployed to PyPI:

Uploading distributions to https://upload.pypi.org/legacy/
Uploading marionette_driver-2.8.0-py2-none-any.whl
Uploading marionette_driver-2.8.0.tar.gz

Uploading distributions to https://upload.pypi.org/legacy/
Uploading marionette_harness-4.6.0-py2-none-any.whl
Uploading marionette-harness-4.6.0.tar.gz