There have been a lot of errors on b2g.hamachi.mozilla-aurora.marketplace, most of which have been a JavascriptException that is thrown during the launch of the Marketplace app, which Marionette then catches and causes the launch to fail. This is a non-critical JS exception, but it causes the test to fail. A sample stack trace is: test_marketplace_feedback_user (test_marketplace_feedback_login.TestMarketplaceFeedback) ... ERROR ====================================================================== ERROR: None ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 143, in run testMethod() File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/marketplacetests/tests/test_marketplace_feedback_login.py", line 24, in test_marketplace_feedback_user self.marketplace.launch() File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/marketplacetests/marketplace/app.py", line 48, in launch Base.launch(self, launch_timeout=120000) File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/gaiatest-0.21.2-py2.7.egg/gaiatest/apps/base.py", line 26, in launch self.app = self.apps.launch(self.name, launch_timeout=launch_timeout) File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/gaiatest-0.21.2-py2.7.egg/gaiatest/gaia_test.py", line 75, in launch result = self.marionette.execute_async_script("GaiaApps.launchWithName('%s')" % name, script_timeout=launch_timeout) File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script filename=os.path.basename(frame[0])) File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 577, in _send_message self._handle_error(response) File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 612, in _handle_error raise JavascriptException(message=message, status=status, stacktrace=stacktrace) JavascriptException: TypeError: widgetFrame is null stacktrace: @app://system.gaiamobile.org/js/cost_control.js, line 70 TEST-UNEXPECTED-FAIL | test_marketplace_feedback_login.py test_marketplace_feedback_login.TestMarketplaceFeedback.test_marketplace_feedback_user | ---------------------------------------------------------------------- Ran 1 test in 74.436s FAILED (errors=1) We saw something similar in the gaia-ui-tests job awhile back and had to wait for a developer to address it. Because this repo is small and focussed on testing Marketplace functionality, as opposed to general gaia reliability, it was suggested that maybe we should just ignore these errors. I believe that Bebe proposed a similar approach at the time and it wasn't favoured, but I discussed this with Zac on IRC and we thought it might be appropriate for this repo.

Comment on attachment 8357840 [details] Pull request See comments in pull request.

Comment on attachment 8357840 [details] Pull request Bob, sorry I don't have time to run/review this thoroughly but I'm switching my r? to a f+ for the principle of the pull and I'll let you guys hash out the details of code syntax.

Comment on attachment 8357840 [details] Pull request +1 to some of Dave's comments

Comment on attachment 8357840 [details] Pull request Newer version of the PR.

Comment on attachment 8357840 [details] Pull request f+ only, no time to review sorry.

Comment on attachment 8357840 [details] Pull request I've not run the tests but the patch looks okay to me. I still think we're fixing (hacking) this in the wrong place though as the exception mentioned is not part of Marketplace.

If the failure isn't related to marketplace I don't understand why other aurora tests aren't failing.

(In reply to Zac C (:zac) from comment #8) > If the failure isn't related to marketplace I don't understand why other > aurora tests aren't failing. True, but what has cost_control.js got to do with Marketplace? I think we need more investigation here.

After all of this discussion and debate, the job seems to have stabilized. We haven't seen a JavascriptException in a couple of days, so let's hold off on this change for now. If things are still stable by the end of the week we should consider abandoning this.

Comment on attachment 8357840 [details] Pull request LGTM

Merging in

Merged in https://github.com/mozilla/marketplace-tests-gaia/commit/c16f83ff799e4cfdf7243a7c699c42ccaf7fe362

This shouldn't have been merged yet based on comment 10. I'll leave it to Bob to decide if we want to back this out though.

Sorry about that guys I did not seen comment 10. I backed out the commit and we will wait until next wee as Bob suggested.

re-evaluate this pull next week

Hmm, we did see one of these Javascript Exceptions [1] the day after those last comments were posted. So I guess the problem has not been eradicated, but it is definitely less frequent than we were seeing early last week. In the past 22 builds we've only seen one test fail with this Javascript exception, so it sounds like maybe we're better off not merging this hack just yet. What do you think, Bebe and davehunt? [1] http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-aurora.marketplace/25/HTML_Report/?

My preference is not to implement the workaround but to fix the issue. Also, if we implement the workaround I still think it should be in gaia, because as far as I can tell the issue is not specific to Marketplace.

In the last 30 builds (1 week) we saw 11 failed builds from these only 2 builds failed with a JavascriptException() I would not want to implement a hack for this. Let's log a bug to the Marketplace project so they could investigate the issue. Please reopen this if we see the fail in a more often case