I just noticed this on some of our tab close subtests: Windows XP[1]: ~17% - ~26% regression Windows 7[2]: ~22% - ~35% regression Windows 8[3]: ~3% - ~13% regression Linux 12.04 (32-bit)[4]: ~22% - ~25% regression Regression range falls in these changesets: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?startID=36567&endID=36568 My money is on Bug 940262. I'll back that out and see if that helps. [1]: https://datazilla.mozilla.org/?start=1384984809&stop=1385299503&product=Firefox&repository=Mozilla-Inbound-Non-PGO&os=win&os_version=5.1.2600&test=tart&project=talos [2]: https://datazilla.mozilla.org/?start=1384984809&stop=1385299503&product=Firefox&repository=Mozilla-Inbound-Non-PGO&os=win&os_version=6.1.7601&test=tart&project=talos [3]: https://datazilla.mozilla.org/?start=1384984809&stop=1385299503&product=Firefox&repository=Mozilla-Inbound-Non-PGO&os=win&os_version=6.2.9200&test=tart&project=talos [4]: https://datazilla.mozilla.org/?start=1384984809&stop=1385299503&product=Firefox&repository=Mozilla-Inbound-Non-PGO&os=linux&os_version=Ubuntu%2012.04&test=tart&project=talos

I also find it strange that tree management didn't catch such a meaningful regression. :/

m-c baseline: https://tbpl.mozilla.org/?tree=Try&rev=3c33fbd1521d 940262 backout: https://tbpl.mozilla.org/?tree=Try&rev=e133be561c91

(In reply to Mike Conley (:mconley) from comment #1) > I also find it strange that tree management didn't catch such a meaningful > regression. :/ As discussed on IRC, most .close subtests regressed decidedly, while few .open subtests improved slightly. The average (which is used for the regression detection) regressed by less than 2% on XP, while the noise level is such that the detection on that case was not definitive enough, hence not flagged automatically. This graph view explains why the detection-by-average was not flagged (visible, but possibly too noisy for automatic flagging): http://graphs.mozilla.org/graph.html#tests=[[293,131,37]]&sel=1382999328180,1385586000327.9958,0.35294117647058876,5.470588235294118&displayrange=30&datatype=running I'm thinking of reducing the noise level a bit, possibly by doing something with the (knowingly and expectedly) noisy .error subtests values. Not sure what though yet.

Possibly, we should also add automatic detection per subtests, which should have clearly detected the regressions on this case. I believe there was at least some plan for this, but I don't recall if there's a bug for it. Joel?

Matt Brubeck has done a lot of work on the regression detection so he may also know.

We did get two alerts for this regression, both on 32-bit Linux: https://groups.google.com/d/topic/mozilla.dev.tree-management/kOVu4EwajKw/discussion https://groups.google.com/d/topic/mozilla.dev.tree-management/NhqveC39DK4/discussion It looks like the regression on other platforms/trees was below the 2% threshold for alerts. Since TART is relatively stable, we could except it from this threshold here: http://hg.mozilla.org/graphs/file/0b168c847cc2/server/analysis/analysis.cfg.template#l23 I'm not aware of any plans to enable per-subtest regression analysis in the graph server. Datazilla seems like the best way to get per-subtest visibility, and it should replace the graph server in the long term.

(In reply to Mike Conley (:mconley) from comment #2) > m-c baseline: https://tbpl.mozilla.org/?tree=Try&rev=3c33fbd1521d > 940262 backout: https://tbpl.mozilla.org/?tree=Try&rev=e133be561c91 Early results from Linux looks like it's a bingo. Marking as blocking bug 940262.

And pretty consistent for the other OS's as well (in this compare talos, old is the baseline, new is the backout, so we're expecting a performance increase): http://compare-talos.mattn.ca/?oldRevs=3c33fbd1521d&newRev=e133be561c91&server=graphs.mozilla.org&submit=true

Tim, can you take a look? And/or should we just back this out pending a new patch that doesn't regress TART? :-(

avi: with the current implementation of graph server we won't be able to detect by subtest. Datazilla is designed for this, but we are not there and won't be in the next month or two.

What's the story on this, Tim? Are you making any headway, or should we just back out bug 940262?

Current fx-team: https://tbpl.mozilla.org/?tree=Try&rev=5123a7f0d683 With patch: https://tbpl.mozilla.org/?tree=Try&rev=97942ae4ab92

Early TART results show a perf gain of about -2.2%. Which is almost exactly the regression introduced by bug 940262.

Looking good. http://perf.snarkfest.net/compare-talos/index.html?oldRevs=5123a7f0d683&newRev=97942ae4ab92&submit=true

Landed: https://hg.mozilla.org/integration/fx-team/rev/4fda28f9e259 Backed out because I forgot the [Australis] prefix: https://hg.mozilla.org/integration/fx-team/rev/5ba7ac937c2c Re-landed with the proper commit message: https://hg.mozilla.org/integration/fx-team/rev/729b91a1c16f