Let's make it work.

Can we get a nightly binary from pvt/releng yet?

It's not ready yet, you have to build it from mozilla-central with ac_add_options --enable-application=b2g/dev in your mozconfig. Having tests running and being green is a condition to have nightly builds.

(In reply to Zac C (:zac) from comment #1) > Can we get a nightly binary from pvt/releng yet? I think it's in the process of getting built on tbpl. Meanwhile, I'm pulling new gecko master and rebuilding, I can share the result :)

Yes it is already being built in Fig branch over here: https://tbpl.mozilla.org/?tree=Fig And possibly try builds also.

OK thanks Alexandre. With access to pre-built binaries we can run tests locally if/when you need us to.

One will need also the fix from bug 1016030: https://github.com/ochameau/mozilla-central/commit/1bdb789f58847347b627193a5fbec2dc6766b852

Until we fix mozrunner/gaiarunner, Produce a profile: > $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests Run mulet: > $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html Please make sure it has been built with marionette, of course. Then start the tests: > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log

Attached is the log of the current state of running the tests.

(In reply to Alexandre LISSY :gerard-majax from comment #7) > Until we fix mozrunner/gaiarunner, > > Produce a profile: > > $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests > > Run mulet: > > $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html > > Please make sure it has been built with marionette, of course. > > Then start the tests: > > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log One addition to the final command line: "--restart" The log in comment #8 shows the test failing because this argument is missing. The tests are designed to work with --restart only although in some cases they will run/pass without it.

Fine, but: > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g//mulet-uitests.log That does start my mulet build, but it does not seems to be passing the app-arg values.

Do you mean that --app-arg worked without restart, but after that it did not?

(In reply to Zac C (:zac) from comment #11) > Do you mean that --app-arg worked without restart, but after that it did not? No, it just cannot properly handle the "-chrome chrome://...". > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html . This, however, should work but now I'm getting marionnette errors ...

Running: > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html . Shows an error on http://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#225, i.e., |Services.prefs.getCharPref('b2g.system_startup_url');|

Proper startup: > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

Could you provide a patch against gaia ui test runner codebase in order to automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which I imagine is the default and isn't required to be passed) ? It will have no effect on b2g desktop as the chrome uri is the same, but it will make it work automagically on mulet.

(In reply to Alexandre Poirot (:ochameau) from comment #15) > Could you provide a patch against gaia ui test runner codebase in order to > automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which > I imagine is the default and isn't required to be passed) ? Yes that's correct, when you use --binary you don't need to use --app=b2gdesktop

Thanks, now tests are properly starting.

So now the first test passes but it seems none of the subsequent success, they all fail with this error.

(In reply to Alexandre LISSY :gerard-majax from comment #18) > Created attachment 8447942 [details] > getDisplayedApp error > > So now the first test passes but it seems none of the subsequent success, > they all fail with this error. OK I think you just need to do `python setup.py develop` to pick up some code changes we made in the last couple of weeks (and before that, wipe or start a fresh virtualenv). Locally these test are running and passing well for me. I'll run the Travis test set while I'm out at lunch.

(In reply to Zac C (:zac) from comment #19) > (In reply to Alexandre LISSY :gerard-majax from comment #18) > > Created attachment 8447942 [details] > > getDisplayedApp error > > > > So now the first test passes but it seems none of the subsequent success, > > they all fail with this error. > > OK I think you just need to do `python setup.py develop` to pick up some > code changes we made in the last couple of weeks (and before that, wipe or > start a fresh virtualenv). > Locally these test are running and passing well for me. I'll run the Travis > test set while I'm out at lunch. Sorry, but I read in your mind one week ago and I already did all of this: > (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ which gaiatest /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest > (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ ls -hal /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest > -rwxr-xr-x 1 alex alex 346 juin 27 13:49 /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest With a gaia master uptodate of course.

And yet, after doing another python setup.py develop in the very same virtualenv, tests are much more happy. I think we can now hack this and start having mulet+ui tests run somewhere.

Just for the record: > $ GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g/mulet-uitests.log

Great! It's also running well for me locally, too.

What is the state of this bug? Can we run UI tests out of gecko/gaia master? If yes, we should be able to close this bug and start asking to run this suite on TBPL!

(In reply to Alexandre Poirot [:ochameau] from comment #24) > What is the state of this bug? > Can we run UI tests out of gecko/gaia master? > If yes, we should be able to close this bug and start asking to run this > suite on TBPL! Sure, I had issues the other day when running it, but I'm in the process of doing it right now. BTW, when doing this I noticed that the gaia-ui-test code living in gaia still does not have the fix from bug 1032799.

(In reply to Alexandre LISSY :gerard-majax from comment #25) > (In reply to Alexandre Poirot [:ochameau] from comment #24) > > What is the state of this bug? > > Can we run UI tests out of gecko/gaia master? > > If yes, we should be able to close this bug and start asking to run this > > suite on TBPL! > > Sure, I had issues the other day when running it, but I'm in the process of > doing it right now. > BTW, when doing this I noticed that the gaia-ui-test code living in gaia > still does not have the fix from bug 1032799. But according to Jonathan, this is a non issue (cf bug 1032799 comment 10).

This is the result for the following call: > GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --type=b2g --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html gaiatest/tests/tbpl-manifest.ini SUMMARY ------- passed: 96 failed: 5 todo: 5 FAILED TESTS ------- test_sms_add_contact.py test_sms_add_contact.TestSmsAddContact.test_sms_add_contact test_sms_contact_input_validation.py test_sms_contact_input_validation.TestContactValidation.test_sms_contact_validation test_sms_contact_match.py test_sms_contact_match.TestContactMatch.test_contact_match test_mms_add_subject.py test_mms_add_subject.TestMmsAddSubject.test_mms_add_subject test_inbox_to_settings.py test_inbox_to_settings.TestSettingsFromInbox.test_settings_from_inbox SUITE-END | took 1362s

Here is a patch to start running them on taskcluster. It looks like I'm almost there: https://treeherder.allizom.org/#/jobs?repo=try&revision=377680343ef3 I just get an exception: https://taskcluster-artifacts.s3-us-west-2.amazonaws.com/DxMtKT2yT3uqDAClly55Wg/0/public/logs/live_backing.log?AWSAccessKeyId=AKIAIZOQG4W6WG3PH3RQ&Expires=1424698422&Signature=PZ8XnZB7ao%2FVu1r%2F4hB%2FGP8U1JI%3D Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method Marionette.__del__ of <marionette.marionette.Marionette object at 0x2f5e710>> ignored 11:05:41 ERROR - Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method GeckoRuntimeRunner.__del__ of <mozrunner.base.browser.GeckoRuntimeRunner object at 0x2f5e9d0>> ignored

Gip is being killed. No need to continue this.

Agreed.