I got tired of the state of our pageload tests. So, taking a cue from both the reftests and rhelmer's stand-alone pageload stuff, I've created a pageload component. It works much the same way as the reftests do -- you run from the command line, and it gives output via dump(). The input is a list of URLs, one per line. Comments are lines starting with #; it's rather simplistic in this regard (comments at the end of lines aren't allowed etc. this could be improved, but it's not important). There are a number of command line options, with -tp being the main one: -tp <file> Run pageload perf tests on given manifest -tpfilter str Only include pages from manifest that contain str -tpcycles n Loop through pages n times -tpstart n Start at index n in the manifest -tpend n End with index n in the manifest -tpformat f1,f2,.. Report format(s) to use One major TODO is that currently the tests run without the browser chrome; a command line option should be added to run pages through the browser chrome. Note that this means that we don't need a "pageload server" -- the manifest file can just contain http:// URLs that point to a specific machine; that machine can then be just a normal web server (or rather, a fast web server light lighttpd or thttpd).

Yeah, running with the browser chrome is key; it typically gives a slowdown in Tp. Up to 20% last I checked.

Comment on attachment 271217 [details] [diff] [review] pageloader component Looks great! >diff -r 8aa5063cc092 layout/tools/pageloader/report.js >+ * Contributor(s): >+ * Rob Helmer <rhelmer@mozilla.com> This was originally written by Darin for tp2, I just moved it around a little to make it easier to use.

... now with: -tpchrome Test with normal browser chrome -tprender Run render-only benchmark for each page (Yes, this means that Trender goes away, too.) There's a bit of a problem with getting -tprender results when running with -tpchrome... I'm not calling redraw() on the right window, and I can't figure out why (returns instantly/all times are 0), but that can be fixed later. The numbers for the full pageset that we use with tp2 with and without -tpchrome are pretty sadmaking... in some cases the browser chrome hit is much more than 20%. Finally, a useful tool to compare results between runs is at http://people.mozilla.com/~vladimir/tools/compare.html -- just copy and paste the JSON blobs into the input boxes and hit compare.

We should probably file bugs on the pages where the browser chrome has a particularly large effect...

Will one be able to run this with a normal, without-debug, without-test-flags, without-building-FF-oneself, in a binary shipped by Mozilla as the real product? I would assume so, as it would not make sense to have performance tests runnable only on debug builds. But then there is that "taking a cue from reftest" above. I wanted to confirm this feature would not require a debug/test build, as reftest does. Thanks.

Reftest has nothing to do with debug, but it is conditioned on tests being enabled. The binaries we ship are built with tests disabled. I don't see why we should make the --disable-tests flag disable only some tests. If we want to revisit the decision about whether to ship tests with our product (or whether to produce nightly builds that have tests), that seems like a separate issue. (But if we shipped them with the product, we'd have to do things like security reviews.) (That said, there's nothing stopping you from dropping reftest.jar and reftest.manifest into any nightly build.) If you want to discuss further whether we should be shipping tests, I think that belongs in a separate forum rather than a bug about one particular new test.

The question was not about reftest. It was about this feature. I understand the rationale for why reftest is not enabled by default. I think the question still stands. Will I be able to take a normal nightly and run with the -tp flag? I can see reasons for both, though I think it should be runnable in all builds. But I am just asking what has been decided as of now.

Added -tpwidth and -tpheight, and resizing is working correctly for both chrome and chromeless windows. Marking sicking's review from irc: 14:53 < sicking> r/sr=me

Checked in, issues as followups as they come up