Matthew Gregan [:kinetik] Reporter
	Description • 10 years ago

They'll need to be reviewed during import to make sure they agree with our reading of the spec (and either the test or spec clarified), and until we've implemented the complete spec I expect many of them will fail and need to be disabled until the missing functionality is implemented.

	Matthew Gregan [:kinetik] Reporter
	Comment 1 • 10 years ago

More here (that need review): https://github.com/w3c/web-platform-tests/pulls/shishimaru

	cmtalbert
	Comment 2 • 10 years ago

James,

These look like a subset of the web-platform tests that you are already working on importing. Can you comment here with how best to proceed with this?

The current web platform tests are being run on Cedar while they are stabilized before being rolled out to all trees. You can see the cedar jobs here: https://tbpl.mozilla.org/?tree=Cedar&jobname=web-

	James Graham [:jgraham]
	Comment 3 • 10 years ago

So, in the medium term the right solution is to run these along with all the other web-platform-tests. This import will be kept up to date so we will get new tests as soon as they reach the upstream repository, and we have machinery for dealing with tests that fail without having to disable them (basically we store the expected result for every test that doesn't pass).

I saw some media source tests in the update that I just pulled, so you should be able to see them running on cedar in the near future.

The code for web-platform-tests is in review and the plan is to roll it out to a few more trees, particularly try, once that is done. It will be enabled on m-c once we are confident it isn't going to cause more than its fair share of intermittent oranges.

If you need a short term solution, it is possible to import web-platform-tests on top of Mochitests, but that has a number of disadvantages, notably that it is relatively hard to keep up to date. It also lacks some advanced features, e.g. any kind of non-static response, that may or may not be needed by your tests. I'm not exactly sure how that import process works, but Ms2ger should know.

Also, I strongly encourage you to do the upstream review for PRs that you want to land. Finding expert reviewers is probably the biggest bottleneck to getting tests landed in web-platform-tests.

	:Ms2ger (he/him; ⌚ UTC+1/+2)
	Comment 4 • 10 years ago

I'd been hoping to avoid updating my copy; let me know if this is urgent, though, and the referenced PR has been reviewed and landed upstream.

	James Graham [:jgraham]
	Comment 5 • 10 years ago

https://github.com/w3c/web-platform-tests/pull/1238 is a PR for importing more Blink tests into wpt. It would be great if someone who knows the spec could review.

	Karl Tomlinson (:karlt) Assignee
	Comment 6 • 10 years ago

I'm happy to look at that PR at some stage, but higher priority for me right now is running the tests we already have with MediaSource enabled.  Once I sort out the issues there, I'll likely have better understanding of the harness so as to review new tests.

	James Graham [:jgraham]
	Comment 7 • 10 years ago

OK. Is running then with that enabled something that you need help with on the automation side, or is it just about making code changes to ensure that the tests are stable?

	Karl Tomlinson (:karlt) Assignee
	Comment 8 • 10 years ago

You've been very helpful, thanks James.

I have a subset of tests running with stable results, but I've had to disable a large number of them because of an interaction with old-tests/submission/Microsoft/sandbox/sandbox_002.htm:

Allow autoplay for HTML5 Video inside iframe with sandbox attribute if sandbox='allow-scri
pts'.
------------------------------------------------------------------------------------------
-----
Expected PASS, got FAIL
event is undefined(stack: callback/<@http://web-platform.test:8000/old-tests/submission/Mi
crosoft/sandbox/sandbox_002.htm:22:17
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:798:20
callback@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.
htm:20:13
)

This is more than likely a bug in Gecko, which I'd like to investigate.

Is there a way to get more output from the tests?
i.e. can I get the harness to log every assert_* both when it passes and when it fails?

	Karl Tomlinson (:karlt) Assignee
	Comment 9 • 10 years ago

This includes changes to web platform test expected results.

Removal of subtests annotations in meta .ini files means those subtests are
passing.  Removal of a whole file means the whole test is passing.

	James Graham [:jgraham]
	Comment 10 • 10 years ago

Comment on attachment 8491345 [details] [diff] [review]
Enable Media Source Extensions for testing

Review of attachment 8491345 [details] [diff] [review]:
-----------------------------------------------------------------

FWIW the changes look like they match the expected syntax. I obviously don't know if the results are correct given the current state of the implemntation.

	Karl Tomlinson (:karlt) Assignee
	Comment 11 • 10 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/113ecae356de

	Ed Morley [:emorley]
	Comment 12 • 10 years ago

https://hg.mozilla.org/mozilla-central/rev/113ecae356de