We currently compile geckodriver on Windows, but we are not running any tests. We need to enable the Wd job on Windows platforms.

Via bug 1367477 I added the trychooser entry for wdspec tests, which should be available soon.

Oh, it should have been more an info than a dependency. Reverting it.

https://bugzil.la/1374263 made sure the geckodriver binary was included in the correct resource file on macOS. To enable running the Wd job (WPT WebDriver tests) on Windows on try we need to achieve the same effect there. nfroyd, do you have any pointers what files we need to tickle for this?

Probably browser/installer/package-manifest.in; searching for "MOZ_UPDATER" or "MOZ_CRASHREPORTER" or "pingsender" should provide enough examples to cargo-cult from?

I do see the geckodriver executable as being part of the target.common.tests.zip archive. So this should be basically all what we need to get the Wd jobs running on Windows. I will have a look if we can make that happen given that it drives me nuts we only run the tests on Linux.

I got the job created via Taskcluster on Windows, and also updated the mozharness configs for the correct location of the geckodriver.exe. But starting geckodriver.exe fails because VCRUNTIME140.DLL is missing on the AMI. Rob or Peete could one of you have a look at this?

Here the link to the try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f13e63589f56ea2fedd61397f68c409f56a9768&selectedJob=177444760

I triggered some more jobs. Maybe this only affects Windows 7 but not Windows 10 AMIs.

there are a few instances of vcruntime140.dll available on the gecko-1-b-win2012 worker type. here's a task that lists them all (using `C: && cd C:\ && dir vcruntime140.dll /s`): https://tools.taskcluster.net/groups/U2ePZVLXTDWZowA6YE2NcA https://taskcluster-artifacts.net/U2ePZVLXTDWZowA6YE2NcA/0/public/logs/live_backing.log

Rob, that's a build machine but not a test machine. In my case the worker type is gecko-t-win7-32.

reran on win 7 & 10. looks like it's not available on the win 7 workers but it is on the win 10

this is how we install it on windows 2012: https://github.com/mozilla-releng/OpenCloudConfig/blob/8a69ab1/userdata/Manifest/gecko-1-b-win2012.json#L388-L406 we could try something similar on windows 7 but i have no idea if it will work. note that the 32 bit installer (vcredist_vs2015_x86) creates: C:\Windows\SysWOW64\vcruntime140.dll while the 64 bit installer (vcredist_vs2015_x64) creates: C:\Windows\System32\vcruntime140.dll which suggests to me that there will be hurdles installing this on a 32 bit system.

Please note that the same problem is also visible on Windows 10.

testing vcredist install on beta workers (gecko-t-win7-32-beta, gecko-t-win10-64-beta) https://github.com/mozilla-releng/OpenCloudConfig/commit/dbb53cca3bb19e3328b447c43c5215c27d366408

Now it works better but we get dozen of `Test harness output was not a valid structured log message` failures: https://treeherder.mozilla.org/logviewer.html#?job_id=177621642&repo=try > 08:49:21 CRITICAL - platform win32 -- Python 2.7.14, pytest-unknown, py-1.5.2, pluggy-0.5.3.dev -- Z:\task_1525854760\build\venv\Scripts\python.exe I assume this is pytest, which is just logging to stdout via print? But I do not understand the following: > 08:49:21 CRITICAL - tests/web-platform/tests/infrastructure/webdriver/tests/test_load_file.py::test_load {"source": "web-platform-tests", "thread": "Thread-TestrunnerManager-1", "time": 1525855761591, "action": "log", "message": "Got command: u'log'", "level": "DEBUG", "pid": 4512} This is coming from TestrunnerManager and fails for the message `Got command:` which is logged via: > self.logger.debug("Got command: %r" % command) Maybe it is because the message doesn't contain a log level? > {"source": "web-platform-tests", "test": "/infrastructure/webdriver/tests/test_load_file.py", "thread": "Thread-TestrunnerManager-1", "time": 1525855761129, "action": "test_start", "pid": 4512} James could you please help?

Note that the reference without the log level comes from the raw.log while the reference with the level is from the log viewer.

Btw. the only reference I was able to find for such a failure message is in the `StructuredOutputParser` of mozharness and it seems that the action as contained in the log line is not valid: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/structuredlog.py?q=%22Test+harness+output+was+not+a+valid+structured+log+message%22&redirect_type=single#70-72 For the above cases it would mean "log" and "test_start", which seem to be fine? Also why does that only happen on Windows but not Linux?

We should be wrapping the output from pytest so that it ends up in the logger rather than going directly to stdout. I guess there is a Windows/Linux difference here; one thought that comes to mind is that it could be caused by the difference between multiprocessing on the two platforms (on Linux Python attempts to share memory in a copy-on-write sense, on Windows it sensibly doesn't).

MozReview-Commit-ID: BlFMMXt8WBU

MozReview-Commit-ID: FgwitlstFP7

James, as promised here is a current wdspec test job which shows the structured log problem: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c017fb9ec65c1b21aee317945eb76be5a23ddcc7&selectedJob=201636819 It would be great if we could have a look at this today.

With bug 1495007 fixed we are now able to enable the Wd test jobs on Windows. Here is a try build from James with the patch on the other patch applied: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c9cdf6610527685eb6e07e6bbcf8dff5354807d The two tests which are currently failing will be disabled on Windows. Also once the tests are running on Windows, we will go ahead and also enable headless tests on bug 1493907.

Ok, so I pushed another try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=499458b7919c489039f0515dc25dd18c9277ce70&selectedJob=202696459 Sadly I'm not able to disable those affected tests. James, mind having a look at those disable statements of the manifest files? I wonder why it doesn't work: https://hg.mozilla.org/try/rev/a9ae1b9c5abd18d6f9a339d2333815682fca2c08 I actually found this logic in some other manifest files so I wonder if that is Wd only, or doesn't work at all. Also please ignore the trailing space after the sub test name in the first manifest. It's not the problem here given the other test is also not getting disabled. Thanks.

The patch makes sure that the correct path to the geckodriver binary is used for the web-platform-tests script.

Depends on D7417

We solved the manifest question via IRC. It's all working now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b34cd477af88c4b69ebe8023cd68a2af453c1ca

Depends on D7418

Comment on attachment 9013594 [details] Bug 1370636 - [wdspec] Enable Wd jobs on Windows platforms. r?jgraham James Graham [:jgraham] has approved the revision.

Comment on attachment 9013592 [details] Bug 1370636 - [wdspec] Update mozharness configuration for geckodriver on Windows. r?jgraham James Graham [:jgraham] has approved the revision.

Comment on attachment 9013593 [details] Bug 1370636 - [wdspec] Temporarily disable failing tests on Windows. r?jgraham James Graham [:jgraham] has approved the revision.

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ec024b2fc13a [wdspec] Update mozharness configuration for geckodriver on Windows. r=jgraham

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9e11b103f7ec [wdspec] Temporarily disable failing tests on Windows. r=jgraham

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b9a19eb6f805 [wdspec] Enable Wd jobs on Windows platforms. r=jgraham

https://hg.mozilla.org/mozilla-central/rev/ec024b2fc13a https://hg.mozilla.org/mozilla-central/rev/9e11b103f7ec https://hg.mozilla.org/mozilla-central/rev/b9a19eb6f805