In order to prevent hitting bug 1141519 when we run update testing, chmanchester came up with the idea of switching back to the main window before shutting down (IIUC).

I tested locally by specifying and incorrect update branch and observing the failure. Without this we hang, with we fail immediately.

I tested the patch and I started having a new issue. [1] I will look into detail tomorrow. After that, all remaining locales with hit the busy port issue: > MarionetteException: MarionetteException: localhost:2828 is unavailable. [1] 0:20.89 LOG: MainThread INFO starting httpd 0:20.90 LOG: MainThread ERROR Failure during execution of the update test. Traceback (most recent call last): File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/firefox_ui_harness/runners/update.py", line 70, in _run_tests FirefoxUITestRunner.run_tests(self, [manifest]) File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/firefox_ui_harness/runners/base.py", line 91, in run_tests BaseMarionetteTestRunner.run_tests(self, tests) File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/marionette/runner/base.py", line 767, in run_tests self.start_httpd(need_external_ip) File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/marionette/runner/base.py", line 853, in start_httpd self.httpd = self.create_httpd(need_external_ip) File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/marionette/runner/base.py", line 861, in create_httpd rv.start() File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/marionette/runner/httpd.py", line 27, in start self.base_url = self.get_url() File "/home/cltbld/temp/build-tools/release/updates/venv/lib/python2.7/site-packages/marionette/runner/httpd.py", line 42, in get_url return self._server.get_url(path) AttributeError: 'MozHttpd' object has no attribute 'get_url'

This is due to a recent change in the marionette client. Are you using package versions from pypi? We can do a release and grab a new one, but that one should still be compatible.

Oh, maybe we need a new mozhttpd. I'll look into this.

I tried with a fresh virtualenv and packages from pypi and it works ok.

What is the output of your pip freeze? I had to upload a couple of packages to our internal pypi. These are the differences with older runs: -marionette-client==0.9.2 -marionette-driver==0.3.1 +marionette-client==0.10 +marionette-driver==0.4 @@ -7 +7 @@ marionette-transport==0.4 -mozInstall==1.11 +mozInstall==1.12 I see mozhttpd 0.7 released: https://pypi.python.org/pypi/mozhttpd My run had 0.5. I will upload it.

I think we should include a requirements.txt for this project since I'm freaking out a bit of the idea that later down the road someone uploads a python package to our internal pypi and some time later the update verification would start failing. Do you want the requirements.txt on the tools repo? or as part of firefox-ui-tests?

On another note, the socket was left open. It seems that anytime something can go wrong, it will leave the socket open regardless of the reason that made that happen. I'm not familiar with sockets (and I will read more into it) but can we somehow catch/prevent it all? or sleep until we exit? or find other available ports?

This is my pip freeze: argparse==1.2.1 blessings==1.6 -e git+git@github.com:chmanchester/firefox-ui-tests.git@e9b30688d53f8b9caf61809528a31efd9fb64105#egg=firefox_ui_tests-origin/workaround_hang manifestparser==1.1 marionette-client==0.10 marionette-driver==0.4 marionette-transport==0.4 mozInstall==1.11 mozcrash==0.14 mozdevice==0.44 mozfile==1.1 mozhttpd==0.7 mozinfo==0.7 mozlog==2.10 moznetwork==0.24 mozprocess==0.22 mozprofile==0.23 mozrunner==6.7 moztest==0.7 mozversion==1.2 wsgiref==0.1.2 There is a requirements file already in firefox-ui-tests. The error here is that recent changes to marionette silently made mozhttpd 0.7 a requirement, while marionette's own requirements.txt only requires mozhttpd >= 0.5. I will fix this. Marionette is apparently prone to leaking this socket. However, the cases it does can accurately be described as bugs in marionette that are being addressed. We have found two cases, this does not occur anytime something goes wrong. I don't think it would be correct to allow this experience to undermine our confidence in these tools, but rather the conclusion for me is that a modest amount of additional engineering effort is required because this is a novel use case for marionette automation. Regarding a more robust workaround for unexpectedly leaking this socket, specifying a new port for each test run can be done with the "--address" option.

I'm doing another full run. These are our differences: -blessings==1.5.1 +blessings==1.6 +argparse==1.2.1 -mozInstall==1.12 +mozInstall==1.11 -mozprocess==0.21 +mozprocess==0.22 -mozrunner==6.6 +mozrunner==6.7 -mozversion==1.1 +mozversion==1.2

Comment on attachment 8594994 [details] patch with a workaround What I mentioned on the PR: My apologies about this delay. For some reason I saw a review request on the bug for ato and I thought it was waiting on him. Please ship it!

Comment on attachment 8594994 [details] patch with a workaround Oh dear, for some reason I thought this already landed. r=armenzg