After speaking with Clint Talbert and Mark Finkle it's been suggested that developing a new Mozmill test type specifically for Metro mode would be prudent. Since the UI is dramatically different it will take a lot of resources to refactor tests which work in both modes, an approach which may not be wise. I'd like to use this bug report to be the starting point for discussion around the idea of developing a Mozmill Metro test type and to act as a metabug for implementation of whatever solution we land on.

In terms of developing new tests it might be best to align with the Metro team. As I understand it they are developing user stories in short sprints and Virgil is conducting manual testing aligned with those sprints. It might be best to align automation development with these sprints as well.

Lets move this over to mozmill-automation. That's the place where we have to start to get a testrun script implemented. Then we can land individual tests for metro mode. Anthony, as I can remember you were experimenting with the testrun scripts to check how those work with the Metro mode. Is there any code we could use for our scripts or was it just playing around? It would also be great if you could file issues for mozmill-ci and the mozmill-dashboard, and reference those issue links here. We can then start early next month to get all the stuff implemented and running through mozmill-ci.

So the dashboard work will be covered by: https://github.com/mozilla/mozmill-dashboard/issues/47 And Mozmill-CI here: https://github.com/mozilla/mozmill-ci/issues/215

I have made no progress in getting Mozmill to work in metro mode via code.

I will most likely start next week on this bug.

Clint sent me this file which is necessary to start Firefox in metro mode.

I'm not sure how mozmill works but if it's hard to get it to work in metro mode, you could try running the tests by first launching C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Simulator\11.0\Microsoft.Windows.Simulator.exe which basically relaunches win8 inside a desktop window. Another idea to launch the metro browser in a consistent way would be to just run metrotestharness.exe which we build on normal metro builds for talos test running. This info may not be helpful, but just thought I'd mention it.

Thank you Brian. The metrotestharness executable I already got from Clint. that one is used by the mochitest harness and should serve us well. Also thank you for pointing out the other way with the simulator. I'm not sure if that will work if it launches a 'virtual desktop' in it and how much control we have over it in terms of automation. I will get started on this by next week. I will make sure to reach out if i run into troubles.

So I did the first initial steps on this and got the metro mode launched by the above helper application. I only had to download the additional msvcr110.dll. But whatever we want to do here with Mozmill, we are blocked by bug 842605. Both Mozmill and JSBridge are add-ons which have to be enabled inside of Firefox. Without them there is no way for us to get tests running with Mozmill. As I can see the dependent bug is targeted for v2 of metro as of now. I'm not sure what that means but lets get feedback from Jim and Asa. If nothing unblocks us here from allowing us to test Metro with Mozmill, we will have to target Marionette as test framework.

(In reply to Henrik Skupin (:whimboo) from comment #9) > So I did the first initial steps on this and got the metro mode launched by > the above helper application. I only had to download the additional > msvcr110.dll. > > But whatever we want to do here with Mozmill, we are blocked by bug 842605. > Both Mozmill and JSBridge are add-ons which have to be enabled inside of > Firefox. Without them there is no way for us to get tests running with > Mozmill. > > As I can see the dependent bug is targeted for v2 of metro as of now. I'm > not sure what that means but lets get feedback from Jim and Asa. > > If nothing unblocks us here from allowing us to test Metro with Mozmill, we > will have to target Marionette as test framework. v2 basically means retriage this bug after v1 rolls out. When we work on add-ons is hard to say. I'm not sure where on the priority list it will land. You can load add-ons that aren't desktop specific (tied to dekstop xul or features) manually with a some tweaks to settings. We might be able to figure out a way to get what you need for mozmill enabled somehow. How do Mozmill and JSBridge interact? Do they have UI in the browser or are they completely hidden?

(In reply to Jim Mathies [:jimm] from comment #10) > You can load add-ons that aren't desktop specific (tied to dekstop xul or > features) manually with a some tweaks to settings. We might be able to > figure out a way to get what you need for mozmill enabled somehow. How do > Mozmill and JSBridge interact? Do they have UI in the browser or are they > completely hidden? There is no UI involved anymore for Mozmill. So both are working completely in the background. https://github.com/mozilla/mozmill/tree/hotfix-1.5/jsbridge/jsbridge/extension https://github.com/mozilla/mozmill/tree/hotfix-1.5/mozmill/mozmill/extension If you need some more information please let me know. Otherwise I hope that we can get it working. What do you think how long it would take? I ask because we have a metro work week for automation next week. Thanks.

(In reply to Henrik Skupin (:whimboo) from comment #11) > (In reply to Jim Mathies [:jimm] from comment #10) > > You can load add-ons that aren't desktop specific (tied to dekstop xul or > > features) manually with a some tweaks to settings. We might be able to > > figure out a way to get what you need for mozmill enabled somehow. How do > > Mozmill and JSBridge interact? Do they have UI in the browser or are they > > completely hidden? > > There is no UI involved anymore for Mozmill. So both are working completely > in the background. > > https://github.com/mozilla/mozmill/tree/hotfix-1.5/jsbridge/jsbridge/ > extension > https://github.com/mozilla/mozmill/tree/hotfix-1.5/mozmill/mozmill/extension > > If you need some more information please let me know. Otherwise I hope that > we can get it working. What do you think how long it would take? I ask > because we have a metro work week for automation next week. Thanks. What's the testing process like when you launch the browser with either of these? I'm curious how QA works with them so I can get a feel for what we might be able to do.

Hm, that might be interesting. So we install JSBridge and Mozmill by copying the extension into the profile: https://github.com/mozilla/mozbase/blob/master/mozprofile/mozprofile/addons.py#L199 But now i probably see the issue. Where does Firefox Metro has its profile location? The helper application I was using here does not allow us to specify a profile folder, right? Means we create the profile and prepare it but it's not getting used. How can I instruct the metro starter application to use a specific folder as profile?

Command line params passed to the metrotestharness get passed to firefox after it starts up. This works fine in our automated testing environment and local builds, it's untested as a stand alone launcher. You can try it though, for example: metrotestharness.exe -url www.mozilla.org If you can't get a profile passed in via this, it should dump some debug info out that we can use to tweak the harness to get it working. For addins, those are disabled by default through default prefs, but your test profile should be able to override those and maybe get the addin loaded. Is there a faq or wiki page we can use to walk through the steps a QA person would take to get all this running locally?

Hi Juan, please see Comment #14 from Jim to provide some insight.

We had to jump through some hoops to get dom inspector working - https://wiki.mozilla.org/Firefox/Windows_8_Integration#DOM_Inspector We might have better luck with these testing addins. The wiki above lists some of the prefs you have to change. We also ran into some problems with the addins database which we can probably work around. We just haven't had time to look into what's going on under the hood.

Jim, I have problems to pass-through the custom profile location to Firefox. Whenever I use '-profile %path%', Firefox opens the given path as URL. In the test.ini file it's stored correctly.

(In reply to Henrik Skupin (:whimboo) from comment #17) > Jim, I have problems to pass-through the custom profile location to Firefox. > Whenever I use '-profile %path%', Firefox opens the given path as URL. In > the test.ini file it's stored correctly. Should look something like this - INFO | metrotestharness.exe | args: '-no-remote -profile c:\users\cltbld~1.t-w\appdata\local\temp\tmpcnskbq/ about:blank' Here's a log of our automation test servers doing this - https://bug864940.bugzilla.mozilla.org/attachment.cgi?id=747508

The metro starter application is outdated and doesn't support a lot of the new features. I will attach the latest version of the metrotestharness.exe file, which can be used to start a metro build.

Jim, also what's the best way to debug Firefox? Is the remote debugger these days? I haven't used it so far. I need it to figure out what's missing in mozmill to get it fully running in metro. Using the default profile I'm already able to get both addons installed and enabled. JSBridge can successfully connect but Mozmill doesn't start. Looks like something is broken in the initialization code.

(In reply to Jim Mathies [:jimm] from comment #18) > INFO | metrotestharness.exe | args: '-no-remote -profile > c:\users\cltbld~1.t-w\appdata\local\temp\tmpcnskbq/ about:blank' That's what we do. The command looks like: ['c:\\mozilla\\metro\\metrotestharness.exe', '-firefoxpath', 'c:\\mozilla\\metro\\firefox\\firefox.exe', 'http://www.mozilla.org', '-profile', 'c:\\mozilla\\metro\\profile'] And the app also prints it correctly: INFO | metrotestharness.exe | firefoxpath: 'c:\mozilla\metro\firefox\firefox.exe' INFO | metrotestharness.exe | args: 'http://www.mozilla.org -profile c:\mozilla\metro\profile' INFO | metrotestharness.exe | Launching browser... INFO | metrotestharness.exe | App model id='AF59DC8FB41F101B' INFO | metrotestharness.exe | Harness process id: 644 INFO | metrotestharness.exe | Writing out tests.ini to: 'c:\mozilla\metro\firefox\tests.ini' But whether the profile is recognized nor the website is loaded. Hm.

(In reply to Henrik Skupin (:whimboo) from comment #21) > (In reply to Jim Mathies [:jimm] from comment #18) > > INFO | metrotestharness.exe | args: '-no-remote -profile > > c:\users\cltbld~1.t-w\appdata\local\temp\tmpcnskbq/ about:blank' > > That's what we do. The command looks like: > > ['c:\\mozilla\\metro\\metrotestharness.exe', '-firefoxpath', > 'c:\\mozilla\\metro\\firefox\\firefox.exe', 'http://www.mozilla.org', > '-profile', 'c:\\mozilla\\metro\\profile'] > > And the app also prints it correctly: > > INFO | metrotestharness.exe | firefoxpath: > 'c:\mozilla\metro\firefox\firefox.exe' > INFO | metrotestharness.exe | args: 'http://www.mozilla.org -profile > c:\mozilla\metro\profile' > INFO | metrotestharness.exe | Launching browser... > INFO | metrotestharness.exe | App model id='AF59DC8FB41F101B' > INFO | metrotestharness.exe | Harness process id: 644 > INFO | metrotestharness.exe | Writing out tests.ini to: > 'c:\mozilla\metro\firefox\tests.ini' > > But whether the profile is recognized nor the website is loaded. Hm. I found the problem here, there's a bug in the handling of command line params, our code down app runner assumes the first argv is the path to the exe, but when we process the ini file, we don;t append that to the new arg list. So the first param you pass gets ignored. In the test runs this was -no-remote which didn't really effect anything. I'll file a bug on this and get it fixed, but for the time being, you can just pass a dummy param to the harness, like: metrotestharness.exe -firefoxpath (path) -dummy -profile (path) For debugging, depends on what your setup is. I use remote debugging on a tablet, others have dual monitors running win8. You can also remote debug using a vm, although we haven't worked with that in a while - https://wiki.mozilla.org/Firefox/Windows_8_Integration#Remote_Debugging_with_a_VM Once you have a debugger setup, there's a handy startup delay in browserapp here - http://mxr.mozilla.org/mozilla-central/source/browser/app/nsBrowserApp.cpp#34 that will give you time to get a debugger attached. Of course this requires a custom build, which you can do through try if need be.

Updated harness launching executable I got from Jim which now correctly handles the list of command line options which have been passed in, even in a scrambled order.

FYI I got Mozmill working with Firefox Metro yesterday while sitting in the plane. There are only two little issues with the harness executable we will have to get fixed and I will file bugs for it. When it comes to Mozmill we will also have to do some more updates especially for handling tabs and so on. Also I will create a new mozrunner application class for Firefox Metro so we can support those builds down to the base. We have a metro work week this week in MV. So if you are around, stop by in #fail and say hello.

(In reply to Henrik Skupin (:whimboo) from comment #24) > We have a metro work week this week in MV. So if you are around, stop by in > #fail and say hello. That's the "Fail" conference room on the 3rd floor. "#fail" sounds like a dismal IRC channel.

As of now you can use the following example script to launch Firefox Metro and get Mozmill to execute the test. Before you do this make sure to increase the maxVersion in the install.rdf file for mozmill to 24.*, otherwise it will be disabled.

Jim, why have you removed bug 864940? It's also something we need for us. Or is this bug specific for RelEng purposes only?

(In reply to Henrik Skupin (:whimboo) from comment #27) > Jim, why have you removed bug 864940? It's also something we need for us. Or > is this bug specific for RelEng purposes only? That bug is specific to getting the browser registered properly on our automation slaves so they can run mochitests in metro. I could be wrong but I don't think slave config has anything to do with MozMill (?).

No, it hasn't. That's true. But i want to have the same feature for us. So lets add it to the see also field.

Hopefully by today we will be able to release mozmill 2.0 rc4. This version will have built-in support for MetroFirefox. Once it is available via pypi, we can work on the testrun scripts.

Ok, so the problem we have now is that which kind of tests we want to have for MetroFirefox. For now we are most likely ok with functional tests. But once we can update the builds, we also might want to have update tests. With that in mind we cannot store the tests under tests/metro given that we would need another extra layer of sub folders. So we probably might want to consider the option to have something like: |-data |-firefox |-lib |-tests |-lib |-metrofirefox |-lib |-tests With that we can separate tests and library modules for separate applications. There would be no need to mix them up in a folder. Also our testrun scripts could select the appropriate files dependent on the application to use. For global libraries which apply to more than one product we can have a lib folder in the root of the repository. Downside here is that we have to massively change the repository layout. I have known that we would have to do it at some point, and it looks like that it's now. What do you think?

Sounds more organized this way, but for the global library we would need to check each method and move it accordingly. So will we keep the same names for them? In a more complex way, we would have a test that uses both global lib and the one specific to it, it will import toolbars.js for example, from 2 libs.

Global libs will not be that much. At least the assertions and screenshot ones should be there.

Well the screenshot functionality is in Mozmill proper now, so we'll probably won't need it in a global lib. Sounds good to separate Desktop Firefox from Metro Firefox since they have few things in common UI-wise.

(In reply to Henrik Skupin (:whimboo) from comment #31) > |-data > |-firefox > |-lib > |-tests > |-lib > |-metrofirefox > |-lib > |-tests I like this proposal.

This is the github issue related to this bug: https://github.com/mozilla/mozmill-automation/issues/89

Nothing gets tracked anymore for automation on bugzilla. So this bug has to be closed as written in my response to the automation list.