Our starting moment for make test-perf essentially occurs at document-element-inserted [1], which is after app touch, iframe instantiation, and splash screen. We need to move this starting moment to occur right after app touch. The timestamp captured at [2] appears to be the best existing candidate as tracing through the code shows that this would occur close to 0ms after touch. [1] https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/performance_helper_atom.js#L115 [2] https://mxr.mozilla.org/mozilla-central/source/dom/apps/src/Webapps.js#477

We currently programmatically launch the app. "App Touch" as you put it is when the homescreen get a tap. That mean instrumenting the homescreen. I'd rather instrument that separately.

The goals [1] we have defined for the launching of applications is the aggregate of time between the app launch, whether user-initiated or simulated, and the invocation of "moz-app-visually-complete". The aggregate goal is to be within 1000ms, and is what we are basing our release acceptance on [2]. So the only way we can back up these goals with our measurements is to get the start time as close to user-perceived app launch as possible. [1] https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines#Goals [2] https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance#2.0

Do we launch the apps directly, or do we inject a tap to launch? Also, where does the elapsed-time measurement happen? Typically, for a metric, you'd anchor this sort of timing on the impulse on the script side (i.e. the tap or call to a launch method). But that assumes the test is measuring the elapsed time between impulse and and the event independently. The big reason to do it on the system side is if the events contain their own timing result, but if we're measuring the timeline from the test script then it might not be necessary to change on the system side.

We launch the app directly as said in Comment 1.

Thanks for clarifying. Programmatically could also mean user event injection, which uses a different code path. If the test code intercepts the events directly, might make sense for the timeline to be calculated on the test side, starting at the programmatic call. If nothing else, could be a crutch while we tune the rest.

This patch captures the launch creation timestamp and passes it along for use in the tests. The startup tests have been modified to allow overloading of the results to provide a launch time delta.

Comment on attachment 8460768 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22058/files This is expected to slow down the performance result since you move the start point earlier. Please note this.

Alive is correct, this will show an increase in the start time for all applications in the *new* launch events being captured by make test-perf, not the ones currently being used in b2gperf.

Comment on attachment 8460768 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22058/files Requesting uplift to 2.0 as it is important for meeting our release performance acceptance criteria and providing more accurate numbers for the criteria. [Feature/regressing bug #]: bug 996038 [User impact if declined]: none [Describe test coverage new/current, TBPL]: These changes only affect numbers gathered for a single performance test [Risks and why]: Low, as there are no user-perceived changes [String/UUID change made/needed]: n/a

Merged https://github.com/mozilla-b2g/gaia/commit/be8a862c815422acbc5f8839cf1e5080742db6cc

There is a regression, Need a followup.

Comment on attachment 8461464 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22114 This patch needs the same gaia-approval-v2.0 for the same reason as the other patch.

Comment on attachment 8461464 [details] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22114 that's it. r=me

Merged https://github.com/mozilla-b2g/gaia/commit/6cc134a8d8b19ee2a30fea06ccb0013b36169c5a

v2.0: https://github.com/mozilla-b2g/gaia/commit/28f807f3f86f98779b40e2fb9ea7f20da00890b6 v2.0: https://github.com/mozilla-b2g/gaia/commit/99a285a504dee269eefb1503599a7705f2529989