I have started creating this measure locally, but after speaking with Alice I think it makes sense to file the bug now. Two things to watch here (due to a11y): 1. Once our a11y engine is instantiated (required for the a11y talos measure), it will effect subsequent performance measures unless FF is restarted. 2. When this lands, we shouldn't run it against OSX (where we don't build with accessibility enabled by default).

How do I create a cvs diff against talos trunk? (I can't cvs add - no write access)

Alice please advise if this is looks okay. For the config, I copied dromaeo.. and called the tests ta11y.

Comment on attachment 452713 [details] [diff] [review] WIP >Index: page_load_test/a11y/a11y.manifest >=================================================================== >RCS file: page_load_test/a11y/a11y.manifest >diff -N page_load_test/a11y/a11y.manifest >--- /dev/null 1 Jan 1970 00:00:00 -0000 >+++ page_load_test/a11y/a11y.manifest 21 Jun 2010 14:01:47 -0000 >@@ -0,0 +1,3 @@ >+static.html >+dhtml.html >+tablemutation.html Note to self: I forgot the %'s here.

I'm not seeing these tests actually cycle - do they need to be run against a particular build or with a particular pref set? Also, I don't see your updates to sample.config in your patch.

Upon looking at the console it is dying in initAccessibility().

Right I didn't copy my sample config over from my standalone tree. I'll do that. (In reply to comment #6) > Upon looking at the console it is dying in initAccessibility(). Oh dear, what platform are you testing on? Note: Mac release builds are not expected to work.

Alice, please let me know how this goes. (Find me on IRC if you can 'davidb')

Can you give me a list of branches/platforms that this should be run on?

Sure. I think for now we want only Windows and Linux trunk.

Testing on talos-r3-fed-002.

The initial manual run looked good, I'm going to put together the necessary buildbot/graph server patches so that we can do a full staging run. Would this be okay only run on moz-central win/linux? Are you interested in 64bit results at all?

Initial manual test run on linux: i,page,median,mean,min,max,runs 0,static.html,1,1,1,14,14|1|1|1|1 1,dhtml.html,44,44,41,57,57|41|46|42|47 2,tablemutation.html,37.5,38.25,35,47,40|35|47|35|43 Do the results look in line with your expectations? Everything is loading in under a second (as the measure is in ms)...

(In reply to comment #12) > The initial manual run looked good, I'm going to put together the necessary > buildbot/graph server patches so that we can do a full staging run. > > Would this be okay only run on moz-central win/linux? Yes that's exactly right thanks. > Are you interested in 64bit results at all? If there are no gotchas, yes please.

(In reply to comment #13) > Initial manual test run on linux: > > i,page,median,mean,min,max,runs > 0,static.html,1,1,1,14,14|1|1|1|1 > 1,dhtml.html,44,44,41,57,57|41|46|42|47 > 2,tablemutation.html,37.5,38.25,35,47,40|35|47|35|43 > > Do the results look in line with your expectations? Everything is loading in > under a second (as the measure is in ms)... I'm not sure. Here is what I get on my OS X debug accessibility build: i,page,median,mean,min,max,runs 0,static.html,4,3.6666666666666665,3,5,5|3|4|4 1,dhtml.html,132,126.66666666666667,109,157,109|139|157|132 2,tablemutation.html,106,112,105,125,106|125|125|105 RETURN: ta11y: 55.0 I suppose the fact it is debug might be the difference. The static.html test looks too quick doesn't it... do you recommend I make it more involved -- take longer?

Actually the static.html used to take a lot longer -- we've had some recent perf wins. I'll modify it. Alice are you sure a11y was invoked in your manual tests (the build had accessibility enabled)?

How is accessibility enabled? Is there a pref that needs setting? I feel like I'm missing the whole test set up here...

(In reply to comment #17) > How is accessibility enabled? Is there a pref that needs setting? Most build configurations default to building an accessibility enabled firefox. There is a pref to force accessibility off, but that defaults to false. You can check a build with https://addons.mozilla.org/en-US/firefox/addon/2407/ > I feel like I'm missing the whole test set up here... I'll be around all week 8-4 EST davidb on IRC :)

I wasn't clear actually. To check if accessibility is available you can try evaluating this in the js console: Components.classes["@mozilla.org/accessibleRetrieval;1"]. getService(nsIAccessibleRetrieval); To check if accessibility is available and 'activated' use: Components.manager.QueryInterface(CI.nsIServiceManager) .isServiceInstantiatedByContractID( "@mozilla.org/accessibilityService;1", CI.nsISupports); To ensure you are running the tests on a relevant build it should be enough to perform the first check.

Okay, I tried: Components.classes["@mozilla.org/accessibleRetrieval;1"]. getService(nsIAccessibleRetrieval); And the nsIAccessibleRetrieval was undefined. So that would be why my test results were so low. I'm using a Firefox3.6 tinderbox build. Is accessibility only on in the nightlies? Only release builds?

I'm not familiar with the config details for the tinderbox builds but generally we don't build-in accessibility on OS X. If you test with a windows or linux build (perhaps a nightly), you should be good to go.

I've been working on linux exclusively after your first warning about osx... guess that I'll have to try some more builds until one works? Do we build in accessibility on all branches?

Ugh paste error - sorry! Please try: Components.classes["@mozilla.org/accessibleRetrieval;1"].getService(Components.interfaces.nsIAccessibleRetrieval);

And: Components.manager.QueryInterface(Components.interfaces.nsIServiceManager).isServiceInstantiatedByContractID("@mozilla.org/accessibilityService;1",Components.interfaces.nsISupports); To see if a11y is active.

I'm going to drop the static.html test since we nailed that one already in our performance work.

Updated patch. I added but didn't test the a11y init fail path.

Looks to be running okay, but could you update the patch for sample.config to call the test 'a11y' instead of 'accessibility' - it'll fit on the waterfall better.

(In reply to comment #27) > Looks to be running okay, but could you update the patch for sample.config to > call the test 'a11y' instead of 'accessibility' - it'll fit on the waterfall > better. Yep! Sorry for delay.

Graph server db update applied to graphs-stage.

Comment on attachment 460355 [details] [diff] [review] add a11y to talos buildbot-configs Have you explicitly tested 1.9.1 and 1.9.2? I'd like to be sure that this works there prior to landing. I think talos-pool is obsolete -- you can probably omit that one line change. Looks fine otherwise.

Comment on attachment 460355 [details] [diff] [review] add a11y to talos buildbot-configs Flipping to r- based on this comment: (In reply to comment #10) > Sure. I think for now we want only Windows and Linux trunk. Except, we'll want it for all branches except 1.9.1 and 1.9.2 since m-c project branches and try need to track trunk's configuration.

Graph server db update applied to production graphs.

Comment on attachment 460559 [details] [diff] [review] [checked in]add a11y to graph server changeset: 315:4ba11ef8b1d3

Comment on attachment 460516 [details] [diff] [review] [checked in]patch 1.3 Checking in sample.config; /cvsroot/mozilla/testing/performance/talos/sample.config,v <-- sample.config new revision: 1.48; previous revision: 1.47 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.js,v done Checking in page_load_test/a11y/a11y.js; /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.js,v <-- a11y.js initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.manifest,v done Checking in page_load_test/a11y/a11y.manifest; /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.manifest,v <-- a11y.manifest initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/dhtml.html,v done Checking in page_load_test/a11y/dhtml.html; /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/dhtml.html,v <-- dhtml.html initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/tablemutation.html,v done Checking in page_load_test/a11y/tablemutation.html; /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/tablemutation.html,v <-- tablemutation.html initial revision: 1.1 done

Comment on attachment 460626 [details] [diff] [review] [checked in]add a11y to talos buildbot-configs (take 2) exclude 1.9.1/1.9.2 changeset: 2766:603bfc3a61c8

Bustage fix for missing graph server config in buildbot-configs: changeset: 2767:ab93026d32e5

Already seeing successful green runs on mozilla-central.

Alice, thanks for your help and patience!

Don't forget to check 'Scrape' on the tinderbox pages after deploying things like this.