as part of the AWFY -> Talos migration we have many tests that need to be run in talos/raptor/jsshell-bench. One test is assortedDOM: https://arewefastyet.com/#machine=36&view=breakdown&suite=assorteddom this runs in a browser (not shell). here is the source for the test: https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/misc-desktop/tests/assorted These are not large (<200KB), and we can add them in-tree. :bz recommended making these into new perf-reftest style tests, I assume that will be straightforward and we can figure that out. The one quirk here by doing that is perf-reftests run in talos and we don't have a good method for running talos against google chrome. raptor does and maybe we could port perf-reftest tests from talos->raptor if all APIs are compatible on the backend (can we do this in compliant web extension code only)

I'm a lot more interested in the Firefox numbers for these than the Chrome numbers, fwiw.

:rwood, you had done a lot of the work on getting perf-reftest tests to run, is this something you can pick up in the next week or two?

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #2) > :rwood, you had done a lot of the work on getting perf-reftest tests to run, > is this something you can pick up in the next week or two? That won't be easy - perf-reftests use their own test code and measure / calculate their own values (in each test), it's not a like a pageload test that Raptor currently supports i.e. capturing specific event(s). Also they run sets of two pages in one test and then compare the values and that is the actual result. It would be a fair amount of work in Raptor, definitely not something quick. If perf-reftests aren't really required on google chrome then I would recommend leaving them in Talos. If there is a requirement to having them on google chrome then maybe we could schedule some time to look into this further on i.e. for raptor M2?

:rwood, I think keeping it on talos is a good and reasonable plan. We can look in detail at these tests and see if there is a reasonable path forward. lets look next week.

I had a closer look at the tests - from what I can tell they don't have to be compared against a baseline or anything (like a perf-reftest) they are standalone (more like perf-reftest-singletons). In which case yes I will see if I can port them to Raptor. I should be able to get them to work like other benchmarks in Raptor - that report their own results via the webext benchmark-relay.js etc.

Had a closer look at the assorted dom tests and ran them locally via the arewefastyet repo. I see that all of the small tests are already combined into one benchmark which is run via /hosted/index.html. I think the best way to port this to raptor is to take a snapshot of this benchmark and modify it a bit to get it running like the other benchmarks (autorun, and post results to the raptor webext). Attaching a screenshot of the tests as run currently in arewefastyet for reference.

I added the assorted-dom benchmark source to the perf-automation repo [1] so now with this patch the new raptor benchmark (raptor-assorted-dom) can be run locally (but not yet in production until the fetch task is done in Bug 1481930). [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1481930#c1

Comment on attachment 8998962 [details] Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark Thanks for the review, I made the changes and updated the phabricator diff (but didn't get an email for some reason). https://phabricator.services.mozilla.com/D3046?id=7600

Comment on attachment 8998962 [details] Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark Joel Maher ( :jmaher ) (UTC+2) has approved the revision.

Comment on attachment 8998962 [details] Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark Sorry :jmaher, one more update - added taskcluster configs and switched to geometric mean. Thanks!

Comment on attachment 8998962 [details] Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark https://phabricator.services.mozilla.com/D3046?id=7903

Comment on attachment 8998962 [details] Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark Joel Maher ( :jmaher ) (UTC+2) has approved the revision.

Note: It's red on try due to the same issue as the 'ugl' job (fetch tasks not working at the moment) so I set it to tier 3 so it can land.

Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a2876bd19572 migrate AWFY assortedDOM tests to a raptor benchmark r=jmaher