Given Ringmark's use of browser sniffing, I suggest this bug be taken with a grain of salt.

Depend https://bugzilla.mozilla.org/show_bug.cgi?id=783301

Update. Last May, in an attempt to further understand the above claim that Ringmark uses browser sniffing, I sent the following email to Ms2ger: -------------------------------------------------------------- Hello, I've elected to contact you directly instead of adding unnecessary noise to the ticket filed on Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=748248) where you posted: > Given Ringmark's use of browser sniffing, I suggest this bug be taken with a grain of salt. Can you be more specific? The only "browser sniffing" that Ringmark ever used was to temporarily disable 2 tests that were incapable of running correctly in Opera. These tests have since been completely re-written to ensure that all modern browsers are capable of running them (this issue was iframes not firing an onload event in Opera). The Ringmark tests aren't supposed to be "spec tests" - they are designed to mimic the way developers use web technology, as closely as is reasonably possible. I'm reaching out to you to help improve the tests -------------------------------------------------------------- I have not received a response. Unequivocally, Ringmark does not "browser sniff". If there are any questions or concerns or any need for clarification, please feel free to contact me either directly or via this ticket.

David, Just a heads up—my company Bocoup is now a developer partner and I'm now working full time on Gaia development. As the defacto maintainer of Ringmark, I can attend to issues regarding the test suite with an increased degree of immediacy

Marking all tracking bugs which haven't been updated since 2014 as INCOMPLETE. If this bug is still relevant, please reopen it and move it into a bugzilla component related to the work being tracked. The Core: Tracking component will no longer be used.

I'm reopening this, as we had a regression in Ringmark Level 0 with Firefox 52. We should at least figure out a way to get Ringmark level 0 in our testing (automated or otherwise). Even if we don't like it, phone manufacturers use it as a standard part of their testing.

Facebook has abandoned the project. https://github.com/facebookarchive/rng.io

> Facebook has abandoned the project. https://github.com/facebookarchive/rng.io We still have major phone vendors using at least level 0 for testing, so we should not regress in that area. We did have a regression - bug 1347164 - and had we been at least looking at this test, we would have found it. So the request here is that we do level 0 testing, even knowing that Facebook is no longer maintaining it.

Seems like if there's valid tests within this suite, we should at a minimum be trying to get them included in the web-platform-tests or something?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9) > Seems like if there's valid tests within this suite, we should at a minimum > be trying to get them included in the web-platform-tests or something? ni'ing annevk on what it would take for this to happen. And overlap with existing tests, etc.

Do other browsers care about these tests? If they do, adding them to web-platform-tests seems like a good idea and I recommend filing an issue there as a first step. Migrating these tests would be quite a bit of work though by the looks of it. Are there resources for that? (I can certainly take a look at areas I'm familiar with and port those, once we've decided this is actually something worth doing.) Note that we're getting closer to code coverage reporting too, which might help us make a decision one way or another.

(In reply to Anne (:annevk) from comment #11) > Do other browsers care about these tests? If they do, adding them to > web-platform-tests seems like a good idea and I recommend filing an issue > there as a first step. https://github.com/w3c/web-platform-tests/issues/5167 Mike: you mentioned major phone vendors use the test suite. Could you elaborate about this in the github issue if that can help with convincing this is worthwhile?

Will do.

(In reply to Anne (:annevk) from comment #11) > Do other browsers care about these tests? If they do, adding them to > web-platform-tests seems like a good idea and I recommend filing an issue > there as a first step. > > Migrating these tests would be quite a bit of work though by the looks of > it. Are there resources for that? (I can certainly take a look at areas I'm > familiar with and port those, once we've decided this is actually something > worth doing.) > This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests)

> This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests) I'm wondering how much overlap there is between Ringmark and the current web-platform-tests? There's more information about web-platform-tests here: https://developer.mozilla.org/en-US/docs/Mozilla/QA/web-platform-tests Did you (Boucoup) ever do any automation around Ringmark, or was there an expectation of a manual test?

(In reply to Mike Kaply [:mkaply] from comment #15) > > This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests) > > I'm wondering how much overlap there is between Ringmark and the current > web-platform-tests? I would certainly hope that there was, but Ringmark was never intended to be exhaustive spec-centric testing, FB's instruction was to test features that were important to specific use cases where the mobile web was struggling against native. > > There's more information about web-platform-tests here: > > https://developer.mozilla.org/en-US/docs/Mozilla/QA/web-platform-tests I'm familiar, but thank you none-the-less :) > > Did you (Boucoup) ever do any automation around Ringmark, or was there an > expectation of a manual test? The rng.io website is just running a set of generic tests in the Ringmark project and then doing something interesting with the results. This work was done about 5 years ago, and FB had a specific set of goals that it wanted us to achieve; at the time automation was not one of those goals, but I'm confident that these tests could easily be repurposed (given resources)