=> https://treeherder.mozilla.org/#/jobs?repo=gaia-master

Current investigations: Last working: https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=cddb9f610cbe03d0ca39d81bbdce46a0fca841ab B2G comes from: Task: FXGKxu6MRoaLMp7MkKCZww https://tools.taskcluster.net/task-inspector/#FXGKxu6MRoaLMp7MkKCZww/ gecko: 22c34579ae0720e7d3dc39a22b9d33f13bc0198b http://hg.mozilla.org/mozilla-central/rev/22c34579ae0720e7d3dc39a22b9d33f13bc0198b https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=22c34579ae07 First not working: https://tools.taskcluster.net/task-inspector/#uqg8vcjOSIihPKp76Tp6tw/4 B2G comes from: Task: 0jnopxNmTWOyauKICIhGww https://tools.taskcluster.net/task-inspector/#0jnopxNmTWOyauKICIhGww/ gecko: 8a6045d14d6bd348a3b5bfeb55a9321e680cc93e http://hg.mozilla.org/mozilla-central/rev/8a6045d14d6bd348a3b5bfeb55a9321e680cc93e https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8a6045d14d6b

Locally, I reproduce fairly well but only if I use VERBOSE=1. My full command line is: VERBOSE=1 DISPLAY=:1 TEST_FILES=apps/calendar/test/marionette/modify_event_view_test.js make test-integration-test (I predownloaded b2g, precreated the profile, and preexecuted the in-memory X server Xephyr). And I noticed this message: message: 'JavaScriptError: TypeError: window.MARIONETTE_LOG_GRABBER is undefined\nRemote Stack:\n\ninline javascript, line 65\nsrc: " return window.MARIONETTE_LOG_GRABBER.grabAndClearLogs();"' }

(In reply to Julien Wajsberg [:julienw] from comment #1) > Current investigations: > > Last working: > https://treeherder.mozilla.org/#/jobs?repo=gaia- > master&revision=cddb9f610cbe03d0ca39d81bbdce46a0fca841ab > > B2G comes from: > Task: FXGKxu6MRoaLMp7MkKCZww > https://tools.taskcluster.net/task-inspector/#FXGKxu6MRoaLMp7MkKCZww/ > gecko: 22c34579ae0720e7d3dc39a22b9d33f13bc0198b > http://hg.mozilla.org/mozilla-central/rev/ > 22c34579ae0720e7d3dc39a22b9d33f13bc0198b > https://treeherder.mozilla.org/#/jobs?repo=mozilla- > central&revision=22c34579ae07 > > > > First not working: > https://tools.taskcluster.net/task-inspector/#uqg8vcjOSIihPKp76Tp6tw/4 Grrr, bad copy/paste First not working: https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=22fba58ea99fc8212de8d4d44f366773b439dd20 > > B2G comes from: > Task: 0jnopxNmTWOyauKICIhGww > https://tools.taskcluster.net/task-inspector/#0jnopxNmTWOyauKICIhGww/ > gecko: 8a6045d14d6bd348a3b5bfeb55a9321e680cc93e > http://hg.mozilla.org/mozilla-central/rev/ > 8a6045d14d6bd348a3b5bfeb55a9321e680cc93e > https://treeherder.mozilla.org/#/jobs?repo=mozilla- > central&revision=8a6045d14d6b

In nearly all failures we have errors like this: TEST-START | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu Covers entire screen TEST-UNEXPECTED-FAIL | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu "before each" hook Error: timeout of 180000ms exceeded at null.<anonymous> (/home/tester/git_checkout/node_modules/mocha/lib/runnable.js:139:19) at Timer.listOnTimeout [as ontimeout] (timers.js:112:15) TEST-UNEXPECTED-FAIL | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu "after each" hook TypeError: Cannot call method 'send' of undefined at Object.send (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:463:36) at Object.Client._sendCommand (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:512:21) at Object.closeDriver (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:774:14) at process._tickCallback (node.js:419:13) Exit code 2 So first is a timeout in "before each", and then the infamous "Cannot call method 'send' of undefined" error in "after each" (but I guess it's more a consequence of the previous issue). That's why I tried to increase the timeout when waiting for the boot in attachment 8652333 [details], because I saw the related error "Never saw webapps-registry-ready yes". But I still see issues. Should we increase it even more ? Maybe in TBPL environment 5 sec is too low ? I admit I don't really know how it's used.

So, I've been approaching this issue from various angles and here's what I've ruled out so far. * random glib assertions are red herrings, they are apparently _not_ critical. :/ * process-content.js not loading is apparently ok. * backing out the apparent gaia checkin that first was broken doesn't change anything. this isn't the changeset we're looking for if it's coming from a gaia commit. * reproducing this locally on ubuntu 14 is not easy, reproducing on mac os x is easier as julien pointed out, with or without using Xephyr. Why that is? No idea yet.

Appears to be caused by a gecko change afaict. I have a clean backout with tests that are passing, going to go ahead and push that to b2g-inbound to see if it has any effect there.

Here's my try run that I'm basing my assumptions on -- https://treeherder.mozilla.org/#/jobs?repo=try&revision=da5d408ba9a9 Using custom gaia revision (latest off of master).

Except that blew up Marionette. Bug 1197146 depends on it. https://treeherder.mozilla.org/logviewer.html#?job_id=2614453&repo=b2g-inbound

Comment on attachment 8652516 [details] [gaia] nullaus:bug1198172 > mozilla-b2g:master landed to help with debugging.

OK, the best line of thinking now is to simply standardize the environment between gecko and gaia. I have a pull request to update gaia-taskenv to use tester:0.0.5 as it's base image and customize only the bits we require since we don't use mozharness. There's still some hiccups to be worked out since there was overlap and differences in how these two environments were set-up and their expectations of where they may be running things. Hopefully these issues will get worked out and it will enable us to re-open. *fingers crossed*

Things are looking positive, I'm tentatively saying we should be re-opening in the next hour.

Commit gaia-master: https://github.com/mozilla-b2g/gaia/commit/4bf0cf91219175304ef906448f7dfbcf269fc033 Green things a plenty!