We can't re-enable running the make test-perf jobs in Jenkins against device because they're currently timing out after three hours. When running interactively this doesn't seem to be a problem. Entries appear in the console log, but after about 12 minutes there's nothing, right up until when we time out (forced by Jenkins configuration). During this time of apparent inactivity I checked what processes are running and get the following items of interest: node /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/bin/../node_modules/.bin/marionette-mocha --timeout 10s --ui tdd --profile-builder /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile_builder.js --profile-base /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile.js --host marionette-device-host /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/setup.js /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/tests/performance/perf.js tests/performance/startup_events_test.js tests/performance/startup_test.js -R /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/tests/reporters/jsonmozperf.js /usr/bin/nodejs /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/node_modules/mocha/bin/_mocha /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/node_modules/marionette-js-runner/lib/runtime.js --timeout 10s --ui tdd --profile-builder /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile_builder.js --profile-base /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile.js --host marionette-device-host /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/setup.js /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/tests/performance/perf.js tests/performance/startup_events_test.js tests/performance/startup_test.js -R /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/tests/reporters/jsonmozperf.js --reporter /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/node_modules/mocha-json-proxy/reporter.js

A failed Jenkins build can be found here (requires VPN): http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest/704/

Console log from Jenkins.

JSON output from Jenkins.

Two things I see that are not needed: Since bug 917717 has completely been fixed + npm install exec-sync is note need. Also on the line before, + adb forward tcp:2828 tcp:2828 Is not needed either. The host runner takes care of everything and the *local* port isn't actually 2828. These shouldn't impact, but not having them would simplify it.

Also, + cd gaia + echo MARIONETTE_RUNNER_HOST=marionette-device-host + echo MOZPERFOUT=/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/perf.json + MOZPERFOUT=/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/perf.json make test-perf I don't get that: where do we set MARIONETTE_RUNNER_HOST so that we get it propagated to the |make test-perf|? (maybe there is a redirect to local.mk, but I don't see it here.

(In reply to Hubert Figuiere [:hub] from comment #4) > Two things I see that are not needed: > > Since bug 917717 has completely been fixed > + npm install exec-sync > is note need. > > Also on the line before, > + adb forward tcp:2828 tcp:2828 > Is not needed either. The host runner takes care of everything and the > *local* port isn't actually 2828. > > These shouldn't impact, but not having them would simplify it. Fixed.

(In reply to Hubert Figuiere [:hub] from comment #5) > Also, > > + cd gaia > + echo MARIONETTE_RUNNER_HOST=marionette-device-host > + echo > MOZPERFOUT=/var/jenkins/workspace/b2g.hamachi.mozilla-central.master. > mozperftest/perf.json > + > MOZPERFOUT=/var/jenkins/workspace/b2g.hamachi.mozilla-central.master. > mozperftest/perf.json make test-perf > > I don't get that: where do we set MARIONETTE_RUNNER_HOST so that we get it > propagated to the |make test-perf|? > (maybe there is a redirect to local.mk, but I don't see it here. We echo it to local.mk but this is not shown in the console log. Here's what we have: echo "MARIONETTE_RUNNER_HOST=marionette-device-host" > local.mk echo "MOZPERFOUT=${WORKSPACE}/perf.json" >> local.mk I will schedule a new build as soon as Jenkins is running again (it's currently paused for unrelated issues) but I don't expect the results to be any different.

OK I was just wondering what it was doing there. Sounds alright.

(In reply to Dave Hunt (:davehunt) from comment #0) > We can't re-enable running the make test-perf jobs in Jenkins against device > because they're currently timing out after three hours. When running > interactively this doesn't seem to be a problem. When you say "running interactively" do you mean on the same machine that run Jenkins or on your local machine? I think the issue boils down to being a different environment, but I just don't know what and why.

New build is running as http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest/706/ (In reply to Hubert Figuiere [:hub] from comment #9) > (In reply to Dave Hunt (:davehunt) from comment #0) > > We can't re-enable running the make test-perf jobs in Jenkins against device > > because they're currently timing out after three hours. When running > > interactively this doesn't seem to be a problem. > > When you say "running interactively" do you mean on the same machine that > run Jenkins or on your local machine? I think the issue boils down to being > a different environment, but I just don't know what and why. Nope, I logged into the same box and ran the tests against the same device. Didn't make a lot of sense to me really, perhaps we could add some debugging?

Is there a way I could have a "sandbox" machine in which I could 1. have everything installed 2. modify the actual code to see if I can debug it when trying to run it from inside Jenkins. Looks like I have to dive down into it to see what's going on :-/ Thanks

(In reply to Hubert Figuiere [:hub] from comment #11) > Is there a way I could have a "sandbox" machine in which I could 1. have > everything installed 2. modify the actual code to see if I can debug it when > trying to run it from inside Jenkins. > > Looks like I have to dive down into it to see what's going on :-/ > > Thanks I've gone ahead and cloned the original job, and given Hub a brief overview of how to customize the new, custom job: http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/

There are a couple of debugging approaches I can suggest: Before 'make test-perf' in the Jenkins script, add: adb logcat > logcat.txt & which may give us some data about what was going on when it hung. Also, we can try app-specific runs, e.g., APP=browser make test-perf, in case the hang is app-specific.

I'll deal with it.

I'm attaching a logcat from the last failed run; maybe it will give some insight into what's going wrong.

(In reply to Stephen Donner [:stephend] from comment #12) > (In reply to Hubert Figuiere [:hub] from comment #11) > > Is there a way I could have a "sandbox" machine in which I could 1. have > > everything installed 2. modify the actual code to see if I can debug it when > > trying to run it from inside Jenkins. > > > > Looks like I have to dive down into it to see what's going on :-/ > > > > Thanks > > I've gone ahead and cloned the original job, and given Hub a brief overview > of how to customize the new, custom job: > http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central. > master.mozperftest.bug962040/ Thanks Stephen. I've enabled this job and disabled the triggers so it must be triggered manually. I've also removed the data submission steps so we're just gathering data and not attempting to send it to DataZilla. I've also reduced the timeout to 60 minutes so we don't have to wait three hours.

This appears to have resolved itself. It could be that bug 942840 was the cause, and now that Stephen has sorted out the headsets, we've fixed it... Hopefully we'll have some results in datazilla soon. I had to add the adb port forwarding back in because b2gperf requires that, and we had a failed job despite the results looking good.

It seems that there are still errors popping out in the JSON which cause JSON error parsing. *sigh*

I updated marionette-device-host to 0.0.2 and with that commit it actually stop breaking the JSON: https://github.com/hfiguiere/gaia/commit/958df5804019431c61bd636898f89dc88aebb4a8 Will submit a PR for that

Comment on attachment 8365562 [details] Pull request r=me but please merge on a green travis (I restarted the failing job)

It appears this job started timing out again since Jan 24, 2014 5:37:29 PM. This was on b2g-1 (previous builds were on b2g-0). Stephen, is there anything different about b2g-1? Is it possible it still has the wrong headset?

(In reply to Dave Hunt (:davehunt) from comment #24) > It appears this job started timing out again since Jan 24, 2014 5:37:29 PM. > This was on b2g-1 (previous builds were on b2g-0). Stephen, is there > anything different about b2g-1? Is it possible it still has the wrong > headset? I swapped in the original OEM Alcatel-Lucent headset/earbuds on all Hamachis, and have double-checked it for B2G-1, too. Sorry, can't think of what's different with this particular device.

Do we have other runs that pass or fail?

(In reply to Hubert Figuiere [:hub] from comment #26) > Do we have other runs that pass or fail? I'm not 100% sure what you mean. The job hasn't passed recently; you can see the list of jobs at http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest/. A few (the ones with red dots) failed; most of them are gray because they got stuck and were killed after 3 hours by Jenkins. The most recent logcat, from Jan 27, is here: http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest/ws/gaia/logcat.txt The Jenkins job that Dave set up for you, http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/, is faring better...the runs are completing, and generating some data, but a lot of errors as well. See the latest result file, from yesterday: http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/21/artifact/perf.json What can we do to help?

Looking at http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/21/artifact/perf.json I'll ignore the fm test for now. I notice this failure: "stack": "{ name: 'StaleElementReference',\n stack: 'StaleElementReference: (10) Stale element reference\\nRemote Stack:\\n<none>\\n at Error.MarionetteError (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/error.js:67:13)\\n at Object.Client._handleCallback (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:474:19)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:508:21\\n at TcpSync.send (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/drivers/tcp-sync.js:94:10)\\n at Object.send (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:455:36)\\n at Object.Client._sendCommand (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:501:19)\\n at Object.Element._sendCommand (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/element.js:49:21)\\n at Object.displayed (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/element.js:281:19)\\n at Object.Client.waitForSync (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:669:9)\\n at Object.Client.waitFor (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:648:60)\\n at Object.MarionetteHelper.waitForElement (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-helper/index.js:131:12)\\n at Object.PerfTestApp.launch (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/app.js:53:24)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/apps/settings/test/performance/rendering_wifi_list_test.js:37:11\\n at Object.PerformanceHelper.task (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/performance_helper.js:163:7)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/performance_helper.js:149:16\\n at Object.Client.waitForSync (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:666:18)\\n at Object.Client.waitFor (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:648:60)\\n at Object.Helper.delay (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/js-marionette/helper.js:14:12)\\n at Object.PerformanceHelper.delay (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/performance_helper.js:171:24)\\n at trigger (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/performance_helper.js:148:14)\\n at Object.PerformanceHelper.repeatWithDelay (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/tests/performance/performance_helper.js:153:7)\\n at Context.<anonymous> (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/apps/settings/test/performance/rendering_wifi_list_test.js:35:23)\\n at Test.Runnable.run (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runnable.js:211:32)\\n at Runner.runTest (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:372:10)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:448:12\\n at next (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:297:14)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:307:7\\n at next (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:245:23)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:269:7\\n at Hook.Runnable.run (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runnable.js:213:5)\\n at next (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:257:10)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runner.js:269:7\\n at done (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runnable.js:185:5)\\n at /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/mocha/lib/runnable.js:197:9\\n at Object.executeHook (/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/gaia/node_modules/marionette-client/lib/marionette/client.js:370:18)\\n at process._tickCallback (node.js:415:13)',\n type: 'StaleElementReference',\n constructorName: 'Error',\n expected: null,\n actual: null }", The "StaleElementReference" might be the cause of the timeouts. I have no idea why we get them nor what they mean, but they come from marionette. Any idea?

A StaleElementReference means you're attempting to access an HTMLElement which is no longer present. It could be that some unexpected transition has occurred, or that the element you're operating on has disappeared due to some new logic in the app, etc.

Bob, is this something you can potentially help out with?

If we can test with this patch....

Can someone test whether this patch fixes it ? Or do we need to land it ? Also, we'll need a separate bug number for reenabling it, Hubert.

Jonathan, I have run it on my jenkins instance and it did pass, but we have seen that it did before. Do you think we can try it on the others?

So, this test passes using hub's job at http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/56/consoleFull; I think that's about the best testing we can do without landing. Let's land this and see what happens.

Comment on attachment 8372432 [details] Blacklist camera for now. r=me but please do the actions I asked on github also be careful to not land the first commit.

Uh oh I didn't realize I had push the other commit with it.

See bug 971771 to reenable it.

merged https://github.com/mozilla-b2g/gaia/commit/29c5494ec589ecc67e975402496f4b7e42880bc3 Now we need to see if the tests run better on Jenkins.

(In reply to Hubert Figuiere [:hub] from comment #38) > merged > > https://github.com/mozilla-b2g/gaia/commit/ > 29c5494ec589ecc67e975402496f4b7e42880bc3 > > Now we need to see if the tests run better on Jenkins. We should probably change the repo and branch in http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/ to gaia and master, right? (Right now, they're yours, for testing purposes.)

(In reply to Julien Wajsberg [:julienw] from comment #35) > Comment on attachment 8372432 [details] > Blacklist camera for now. > > r=me but please do the actions I asked on github Seems like my comments disappared on github? Here are the comments I asked: * add a "bug 971771" comment in front of the "camera" blacklist * file another bug about having a configurable list of disabled apps, but I just filed Bug 972240

Sorry looks like I missed some. Good call on filing that bug, this is something we need to address in the medium term.

Yeah I missed the comment in the review. Here we go.

Comment on attachment 8376061 [details] Add the missing comment r=me

What's the status of this? I see the jobs appear to be completing, but are failing due to invalid JSON.

I had the same question.... I don't see where the JSON parsing error occurs (ie what is causing it to fail)

seems like we have extra ',' in the JSON. *sigh*

We get exceptions thrown because of some errors in the reporter. The console output is quite explicit about what's going on. I'll submit a patch to make sure we have a more robust error handling.

It times out again. http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.master.mozperftest.bug962040/71/console (this is with my patch)

I've disable the main job again due to the timeouts.

It seems that my test instance turned green this morning. I don't really know what happened. sdonner has been triggering them.

This should fix the invalid JSON errors encountered. Also make the VERBOSE mode more useful.

Comment on attachment 8379864 [details] Fix the JSON errors r=me with one question

When the test hangs, I seems to get the following error in logcat: E/GeckoConsole( 8712): [JavaScript Error: "sandbox is null" {file: "chrome://marionette/content/marionette-listener.js" line: 612}]

Merged (with comments address) https://github.com/mozilla-b2g/gaia/commit/20f016be0b1348f424f3489c1364a321afca930c

The latest results looks promising. Lot of green on the jenkins job.

There have been a couple of timeouts but most are now succeeding within the 60 minutes. I did notice that the data is not being submitted due to missing revision details, which should be addressed by updating the version of b2gperf in use. I've tagged the repo and bumped to the latest (0.18) which should address this. Looking at the memory data and the mozperf.py script, it does not appear that any of this will be submitted without some changes to the latter. As this is unrelated to this bug, I'll comment on bug 962238 with the details.

Okay, data is now being submitted! See https://datazilla.mozilla.org/b2g/?branch=master&device=hamachi&range=90&test=startup_time I've increased the timeout to 90 minutes so that the jobs comfortably complete in that time.

It seems we still frequently time out. When a build is successful it completes in ~50 minutes, so any that are still running at 90 minutes are aborted. Here's the details of the last 20 builds, where four timed out. Build Duration Slave #1163 1 hr 43 min b2g-1 #1162 1 hr 33 min b2g-1 #1161 49 min b2g-1 #1160 49 min b2g-1 #1159 48 min b2g-1 #1158 48 min b2g-1 #1157 45 min b2g-1 #1156 51 min b2g-1 #1155 50 min b2g-1 #1154 43 min b2g-1 #1153 49 min b2g-1 #1152 48 min b2g-1 #1151 48 min b2g-1 #1150 1 hr 0 min b2g-1 #1149 51 min b2g-1 #1148 47 min b2g-1 #1147 1 hr 33 min b2g-1 #1146 1 hr 33 min b2g-1 #1145 49 min b2g-1 #1144 49 min b2g-1

Running a simple test locally (just for the SMS app) showed me that B2G crashed. In that case the performance test times out. So maybe we have a more severe issue here. Since you use tbpl-produced builds, do you think we can have a meaningful entry in crash-stats?

That mean we should have a mechanism to detect that we crashed... and return an error to fail.

Filled bug 980973 for that.

Could we close this one and file new issues for new problem? Thanks.

See bug 980541 instead for the crash detection

The original issue has been fixed, new bugs will be raised for any remaining issues.

v1.3t uplift https://github.com/mozilla-b2g/gaia/commit/1245ead1d9901d5e2bbbb8cc7c4cd21d8518c6d0 https://github.com/mozilla-b2g/gaia/commit/43160685c876c44605b89fc219031a8531f53c1e https://github.com/mozilla-b2g/gaia/commit/ca097e6e46860cc20fea42ec21cd8cc61c160c5f https://github.com/mozilla-b2g/gaia/commit/860b14450c4343967012b9ba4a8be9f26b730bb5