The mechanisms behind Gaia performance tests, `make test-perf`, currently support running tests against individual applications that have clearly defined launch lifecycles, but don't have a well-defined approach for testing against other types of apps like the home screen and system. We need to make changes to `make test-perf` to support running these tests.

Also activities. See bug 1007421.

... and keyboard apps, but I don't know if the bug already exists.

Maybe this should block bug 940164, would be nice to have some bare minimum perf indicators before doing the next wave of refactorings

This patch enables tests to be run against the system app. This introductory patch is meant only for doing manual tests using `make test-perf` and as such, will not report any results to Datazilla. There are a few markers introduced into the System and Vertical Home apps, and is captured by storing a value on an object rather than events as with other performance events, as using events was very buggy with the reboot process. Usage is: APP=system TEST_BOOT=2 make test-perf The JSON reported back currently contains 3 measurements: mozSystemLoadEnd - When the System app finishes its execution of window.onload mozOsLogoEnd - When the System app removes the boot logo mozHomescreenStart - When the Homescreen has first JS initialization I have reserved TEST_BOOT=1 if we decide that this test should start reporting the Datazilla in the future. A Datazilla-compatible implementation is a larger time effort, and this patch is merely to enable one-off measurements of the System app launch from boot. Since the Flame's timing implementation can be buggy at boot, these timestamps are measured from performance.requestStart. For a closer measurement of how long the boot process took, you can look at the test start, end, and duration, e.g.: "start": "2014-10-09T18:26:55.874Z", "end": "2014-10-09T18:27:23.413Z", "duration": 27539 but this is not a precise measurement of boot, just a close estimate.

Another note, the Datazilla-compatible implementation would include taking advantage of the RUNS variable, so this test will only run once.

Comment on attachment 8502676 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007 Not super-happy about the shim, but I think this is fine for now. We should probably plan on making that shim thing real at some point :) Thanks for the patch!

Yeah, I'm not happy about it either, and I've got some things in the works with Web API to change it. :)

Comment on attachment 8502676 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007 I'm not the expert here, but this seems to give us what we want, so fb+ for that!

Comment on attachment 8502676 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007 James gave a verbal OK on this in IRC.

This causes the unit tests to fail, probably because window.mozPerformance needs to be shimmed or something. Please fix and make sure we have a green run before landing.

Hmm, weird, window.mozPerformance IS shimmed in shared/js/performance_testing_helper.js. I'll look into why the unit test is failing.

Looks like the unit tests for the failing files use their own requires for the performance_testing_helper. I have since updated these files and recommitted. Waiting for a green try run before continuing.

Try run: https://treeherder.mozilla.org/ui/#/jobs?repo=gaia-try&revision=9a284030a4c0

This test has issues running as-is, see bug 1069863 comment 33. I have a local script running to work around these issues and gather data, but this test should not be used in any official capacity until bug 1069863 is resolved.