here is a graph server: http://graphs.mozilla.org/graph.html#tests=[[309,53,25]]&sel=none&displayrange=30&datatype=running I did some retriggers to find the offending changeset: https://tbpl.mozilla.org/?tree=Mozilla-Beta&fromchange=6bba830d1d04&tochange=68838f520b63&jobname=Windows%207%2032-bit%20mozilla-beta%20pgo%20talos%20svgr here is the changeset in question: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=f7faeaf19dfa as this is on beta, we could be shipping with this in 2 weeks- lets get a decision made on what we want to do.

Actually, you have less than 2 weeks. beta 8 gtb is today (Monday), beta 9 Thursday... Maybe a backout of the changeset responsible.

Since this is the first Firefox version where we try to annotate/file each and every regression, we're now exposed to more explicit/visible regressions than we would have in the past. See bug 990644 for a discussion on how we should handle this new system of explicit regressions bugs. I believe that in the past, a similar regression (though maybe unnoticed or less visible) would not have blocked us and would not have been backed out. This wasn't necessarily a good thing, but it's important to understand that this new regressions visibility is not something we have experience in dealing with. My suggestion, until this process is settled and streamlined, is to not block/backoput patches unless their regression is very meaningful. In other words, live with this new visibility but not change our handling until we have a process we're relatively happy with. Lukas, what say you?

Thanks for the clarification for the Mozilla newbie that I am :) My point was mainly to say that you don't have much time if you want to do something with regard to this bug. I will trust your judgment about a potential backout or not.

Tracking all of current branches since I guess all are affected.

Yes, we're new to this process - it's a wontfix for 29 for sure and in future bugs like this we'll have to work on finding owners for potential backout decisions (made in bug) earlier in the cycle.

Matt, what are the next steps here? Was this an expected result?

Matthew, could you help here? Thanks

Too late now for FF30, we've gone to build with FF30's final beta. Targeting FF31 for further investigation and answers to comment 6.

Just like 30, too late for 31.

There was a 0.5% expected regression from bug 973855 but this then landed with bug 990387 and the regression was larger. It's possible that something else changed before the patch in bug 973855 landed which made it have a larger impact or that bug 990387 also caused a regression.