After we land bug 837139, we can make a generic startup measurement. This will allow us to have a report for free for all apps.

The work in progress is at https://github.com/Rik/gaia/tree/838527-performance-tests.

You can now run |make test-perf| or |APP=browser make test-perf| or |APPS='communications/dialer browser make test-perf| Issues I ran into: - You need to setup the cost control app once before testing it. - Sometimes the contacts test will timeout. If it does, the next app tested will fail.

Comment on attachment 714003 [details] [diff] [review] Proposed patch Review of attachment 714003 [details] [diff] [review]: ----------------------------------------------------------------- Wonder if we could not move the Performance Helper in the performance/ subdirectory too ? otherwise if works perfectly, this is awesome. ::: Makefile @@ +435,5 @@ > INJECTED_GAIA = "$(MOZ_TESTS)/browser/gaia" > > TEST_PATH=gaia/tests/${TEST_FILE} > > +ifeq ($(APPS),) did you try |ifndef APPS| after all ? @@ +440,1 @@ > ifneq ($(APP),) and here |ifdef APP| ::: tests/js/xpc.js @@ +7,5 @@ > + 'system', // reboots the phone > + 'system/camera', // copy of the camera app > + ]; > + if (excludedApps.indexOf(window.mozTestInfo.appPath) !== -1) { > + console.log('No test for ' + window.mozTestInfo.appPath); nit: "'" + window.mozTestInfo.appPath + "' is an excluded app, skipping tests." Also, I wonder if we could not set something in Mocha or mocha instead of polluting the global scope ? Maybe we can wait for James to do that though ? ::: tests/performance/startup_test.js @@ +19,5 @@ > + entryPoint: entryPoint > +}; > + > + > +suite(window.mozTestInfo.appPath, function() { I like to have a ">" between the suite name and test name. I usually add it at the end of my suite names but we may do this in our Reporter instead maybe ? @@ +39,5 @@ > + }); > + > + test('startup time', function() { > + this.timeout(100000); > + yield device.setScriptTimeout(150000); I'd like a comment explaining these 2 timeouts: first for Mocha for the whole test, second one for the remote script execution. BTW it seems strange your scriptTimeout is greater than the Mocha timeout ;) @@ +44,5 @@ > + > + for (var i = 0; i < PerformanceHelper.kRuns; i++) { > + yield IntegrationHelper.delay(device, PerformanceHelper.kSpawnInterval); > + yield app.launch(); > + yield app.close(); I wonder if we can not put app.close in teardown instead (or additionally).

(In reply to Julien Wajsberg [:julienw] from comment #3) > Wonder if we could not move the Performance Helper in the performance/ > subdirectory too ? Agreed, moving. > did you try |ifndef APPS| after all ? > and here |ifdef APP| I didn't try. But now I did and it works :) > nit: "'" + window.mozTestInfo.appPath + "' is an excluded app, skipping > tests." Done. > Also, I wonder if we could not set something in Mocha or mocha instead of > polluting the global scope ? Maybe we can wait for James to do that though ? Maybe we could but I'm reluctant to fiddle now cause I think it's better to land this soon and improve later. > I like to have a ">" between the suite name and test name. I usually add it > at the end of my suite names but we may do this in our Reporter instead > maybe ? Done. > I'd like a comment explaining these 2 timeouts: first for Mocha for the > whole test, second one for the remote script execution. > BTW it seems strange your scriptTimeout is greater than the Mocha timeout ;) I included them without thinking :). Values fixed and comments added. > I wonder if we can not put app.close in teardown instead (or additionally). I'm getting a Script Timeout error when trying this. We should make the runner more resilient but I don't think it should block landing this.

Comment on attachment 714463 [details] [diff] [review] Proposed patch v2 r=me

Comment on attachment 714463 [details] [diff] [review] Proposed patch v2 We probably need these tests in v1-train too. no actual app code is changed here, except performance_helper.

Comment on attachment 714463 [details] [diff] [review] Proposed patch v2 a=tests

I was not able to uplift this bug to v1-train. If this bug has dependencies which are not marked in this bug, please comment on this bug. If this bug depends on patches that aren't approved for v1-train, we need to re-evaluate the approval. Otherwise, if this is just a merge conflict, you might be able to resolve it with: git checkout v1-train git cherry-pick -x e184b751c191fd132326e7afdd3b379834d92f02 <RESOLVE MERGE CONFLICTS> git commit

v1-train: 35b7ad0295225ed6cf4568a692cf85fb66c2b9a6