We want to measure the time it takes to load the content of about:newtab in order to avoid any potential decrease in performance when about:newtab becomes remote. There are a couple of potential ways we can do this: 1) A talos test: PROS: * Mozilla knows and trusts talos and we use it for other performance testing * Easily runs on all platforms via treeherder CONS: * Might be hard to test the impact of performance each time we change the remotely hosted html/js (talos has no network access) * Testing jank on a page seems more difficult to do here than with option (2) 2) Web latency benchmark (http://google.github.io/latency-benchmark/): PROS: * Can have a continuous integration test that runs with every change * Can easily test interactions on the page (e.g. jank of drag/drop) CONS: * Not used Mozilla-wide. Can we really trust its results? (need to look into it) * May take more work to setup tests on multiple platforms I think it would actually be good to have both types of tests.

Note: the WIP is for measurements for loading the current about:newtab. This will look a little different after bug 1021654 since we'll have to wait for a message from the content process for when load is complete.

Joel, does this look like something we might be able to add to Talos? Also, please let me know if there's a better way to do this, I got this idea from the existing sessionstore test :)

we were just talking about this on IRC yesterday (related to bug 1179377)- we have tpaint which does measure the time for mozafterpaint in a new window, and we were talking about measuring just the time for a new window as well as just the time for a chrome page (like about:home). Maybe :avih could chime in here.

Like we discussed on IRC, the main issue here is probably that the about:newtab page becomes remote, and talos is just not equipped (read: explicitly forbidden) to access remote locations. Another issue specific to your patch is that you measure the "load" event, but this event doesn't necessarily mean "the page is ready for the user" (e.g. depending on the page itself, it could do some more formatting and run other code after the load event and before the page becomes fully available for the user to interact with). So there are basically 2 issues to solve: 1. How to deal with remote content - maybe we can have a local talos copy of the content somehow which will get updated once in a while? (specifically, e.g. when the actual online content changes?). Even if we do manage that, having local content is not the same as having remote content, especially when the user has low bandwidth connection or connection reliability issues. 2. Make sure we measure when it counts - which isn't _necessarily_ the load event (though it might be - depends on the page itself) - maybe by embedding some kind of "mark" within the page to indicate when it's ready and which the test could identify somehow and measure? And another thing which is not really an issue but still worth considering: 3. The issue of measuring user-facing about:* pages (newtab, preferences, home, etc) did come up recently and I do think it's valuable to measure, so it's probably better to have a new test which measures several about:* pages rather than a test dedicated to about:newtab.

Thanks for the responses Avi and Joel! (In reply to Avi Halachmi (:avih) from comment #5) > 1. How to deal with remote content - maybe we can have a local talos copy of > the content somehow which will get updated once in a while? (specifically, > e.g. when the actual online content changes?). Even if we do manage that, > having local content is not the same as having remote content, especially > when the user has low bandwidth connection or connection reliability issues. This is a really good point and it's pretty much what we're considering doing - periodically changing the local js/html that mimics what would be fetched from the server. We would probably only need to change it once in a while when we make major changes. (In theory, we would also have other tests mentioned in comment 0 #2 to account for intermediate smaller changes). We also plan to pre-fetch the remote files (bug 1180291) so that connection/bandwidth issues are not a factor in about:newtab load time. > 2. Make sure we measure when it counts - which isn't _necessarily_ the load > event (though it might be - depends on the page itself) - maybe by embedding > some kind of "mark" within the page to indicate when it's ready and which > the test could identify somehow and measure? Thanks for pointing this out. I think in the case of about:newtab, we're done 'loading' around here: https://dxr.mozilla.org/mozilla-central/source/browser/base/content/newtab/page.js#235 And perhaps we can send a custom event when load is complete that the test listens for. > 3. The issue of measuring user-facing about:* pages (newtab, preferences, > home, etc) did come up recently and I do think it's valuable to measure, so > it's probably better to have a new test which measures several about:* pages > rather than a test dedicated to about:newtab. So as you mentioned, each page will be ready for the user at a different time, so I believe each of the about:* pages will need to set their own 'markers' for when they are ready. I imagine the only thing in common when measuring speed of an about:* page loading is that they might send the same custom event to a test. In which case, would it be fair to start with an about:newtab test and later add on tests for other about:* pages? Or has someone else already started working on tests for about:* pages?

(In reply to Marina Samuel [:emtwo] from comment #6) > We also plan to pre-fetch the remote files (bug 1180291) so that > connection/bandwidth issues are not a factor in about:newtab load time. OK, while off topic for this specific bug, did you consider making it an addon? After all, addons already have their update mechanism so you won't have to write a new one which pre-fetches content etc. > > 2. Make sure we measure when it counts... > > Thanks for pointing this out. I think in the case of about:newtab, we're > done 'loading' around here: > https://dxr.mozilla.org/mozilla-central/source/browser/base/content/newtab/ > page.js#235 Actually, by looking at the code I'm not sure that's the point where it's "done", since it still does two things after it reaches this point: this.reportLastVisibleTileIndex(); and gIntro.showIfNecessary(); . I'd say we should consider it done once it has nothing else to do. > And perhaps we can send a custom event when load is complete that the test > listens for. That sounds good to me. > > 3. The issue of measuring user-facing about:* pages (newtab, preferences, > > home, etc) did come up recently and I do think it's valuable to measure ... > > So as you mentioned, each page will be ready for the user at a different > time, so I believe each of the about:* pages will need to set their own > 'markers' for when they are ready. I imagine the only thing in common when > measuring speed of an about:* page loading is that they might send the same > custom event to a test. Well, we could define a new custom event which will be used on all about:* pages when they're "done". > In which case, would it be fair to start with an about:newtab test and later > add on tests for other about:* pages? Or has someone else already started > working on tests for about:* pages? Sounds good to me. We can start with about:newtab and add the other pages later. Currently there's no active work on other about:* pages load time.

Putting this bug aside for now until more of bug 1176429 dependencies are closed.