Matt Brubeck (:mbrubeck) Reporter
	Description • 11 years ago

There was a 2.3% regression in Kraken on mozilla-inbound yesterday which showed up on Windows XP:
http://graphs.mozilla.org/graph.html#tests=[[232,131,1]]&sel=1366844366870,1366998531576&displayrange=7&datatype=running

It's not clear if other platforms showed any regression.  Windows 7, for example, is too noisy to tell clearly, though some statistical analysis might help answer the question:
http://graphs.mozilla.org/graph.html#tests=[[232,131,12],[232,131,1]]&sel=1366844366870,1366998531576&displayrange=7&datatype=running

Here's is the regression range:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=16ebd3f796c1&tochange=ac39efa583f7

The most relevant changes seem to be bug 864727 and bug 863898.

dev-tree-management thread:
https://groups.google.com/d/topic/mozilla.dev.tree-management/eYbx4xVIDzQ/discussion

	Boris Zbarsky [:bzbarsky]
	Comment 1 • 11 years ago

The code that was changed in those bugs had better not be exercised in Kraken.  ;)

Are these pgo builds or regular builds?  It's at least _possible_ that changes in one part of the code affect pgo of other parts of the code...

	Jan-Ivar Bruaroey [:jib] (needinfo? me)
	Comment 2 • 11 years ago

Perhaps Bug 865544 - It landed on inbound 14:34:05 PDT and adds, I believe, an extra QI to all webidl object creations?

	Andrew McCreight [:mccr8]
	Comment 3 • 11 years ago

Nah, that only affects JS-implemented WebIDL, which we currently have 0 of. :)

	Matt Brubeck (:mbrubeck) Reporter
	Comment 4 • 11 years ago

(In reply to Boris Zbarsky (:bz) from comment #1)
> Are these pgo builds or regular builds?

Regular (non-PGO) builds.

(In reply to Jan-Ivar Bruaroey [:jib] from comment #2)
> Perhaps Bug 865544 - It landed on inbound 14:34:05 PDT and adds, I believe,
> an extra QI to all webidl object creations?

No, the regression definitely happened well before bug 865544 landed.  There are 12 test runs on inbound *before* that landing that are all regressed.

The regression range seems very clear (an abrupt increase of about 7x the standard deviation), but even if I widen it by several pushes there are no /js/src changes that could plausibly be the cause.

I retriggered the Talos run on the changeset before the regression, to check for external (e.g. infrastructure) causes:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=16ebd3f796c1&jobname=5.1.*dromaeojs

	Matt Brubeck (:mbrubeck) Reporter
	Comment 5 • 11 years ago

(In reply to Matt Brubeck (:mbrubeck) from comment #4)
> I retriggered the Talos run on the changeset before the regression, to check
> for external (e.g. infrastructure) causes:
> https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=16ebd3f796c1&jobname=5.1.
> *dromaeojs

The retrigger appears to rule out any external causes.  From the email:

    Previous: avg 3326.633 stddev 10.983 of 12 runs up to revision 16ebd3f796c1
    New     : avg 3402.575 stddev 9.847 of 12 runs since revision ac39efa583f7

The retrigger on the pre-regression changeset had a result of 3327.20.

	Boris Zbarsky [:bzbarsky]
	Comment 6 • 11 years ago

That really makes no sense.  There should be no WebIDL anything during Kraken tests....

Matt, do you know where the source for the relevant tests lives, or who would know that?  Also, whether we have per-test data or just an overall number?

	Matt Brubeck (:mbrubeck) Reporter
	Comment 7 • 11 years ago

(In reply to Boris Zbarsky (:bz) from comment #6)
> Matt, do you know where the source for the relevant tests lives, or who
> would know that?

The test code is here:
https://hg.mozilla.org/build/talos/file/e405acebbbf9/talos/page_load_test/kraken

The main "harness" code is here:
http://hg.mozilla.org/build/talos/file/e405acebbbf9/talos/pageloader/chrome/pageloader.js

> Also, whether we have per-test data or just an overall number?

Search for "ai-astar;" in the log to see the raw data.

Before:
https://tbpl.mozilla.org/php/getParsedLog.php?id=22244100&tree=Mozilla-Inbound

After:
https://tbpl.mozilla.org/php/getParsedLog.php?id=22245152&tree=Mozilla-Inbound

At a glance, audio-beat-detection and audio-fft look significantly slower, while the other tests look basically unchanged.

	Boris Zbarsky [:bzbarsky]
	Comment 8 • 11 years ago

Thanks!

audio-beat-detection certainly regressed by a good bit there, order of 15%+.

I'm going to look into what's going on there; a regression that big ought to be easy to hunt down...

	Boris Zbarsky [:bzbarsky]
	Comment 9 • 11 years ago

And of course the regression is completely absent for me locally on Mac...

I guess I'll try to see if I can bisect on XP builds on try or something.  :(

	Boris Zbarsky [:bzbarsky]
	Comment 10 • 11 years ago

https://tbpl.mozilla.org/?tree=Try&fromchange=5fb85709482f&tochange=a6506f0ce611

	Boris Zbarsky [:bzbarsky]
	Comment 11 • 11 years ago

Well, if I believe the numbers there the problem is one of the first 4 parts of bug 864727.  I guess I'll try pushing those and see how things look.

	Boris Zbarsky [:bzbarsky]
	Comment 12 • 11 years ago

So if my try pushes:

https://tbpl.mozilla.org/?tree=Try&rev=cf38c3a92bc3
https://tbpl.mozilla.org/?tree=Try&rev=e1db69603c0d
https://tbpl.mozilla.org/?tree=Try&rev=7237e06b977a

are to be believed, the regression comes from this changeset: http://hg.mozilla.org/integration/mozilla-inbound/rev/3011f288bfe7

I really have no idea why this would affect the Kraken script, much less only on XP.

And if I look at current inbound builds the audio-beat-detection times are back down to their "pre-regression" levels...

	Matt Brubeck (:mbrubeck) Reporter
	Comment 13 • 11 years ago

(In reply to Boris Zbarsky (:bz) from comment #12)
> And if I look at current inbound builds the audio-beat-detection times are
> back down to their "pre-regression" levels...

Indeed.  The un-regression range is:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=1936616a7f41&tochange=11f11d9ca555
(Bug 798165 and other cycle-collection changes by mccr8)

Feel free to close this bug as WORKSFORME unless someone feels it is useful to dig deeper for understanding...

	Boris Zbarsky [:bzbarsky]
	Comment 14 • 11 years ago

The un-regression range is just as nonsensical as the regression range.  :(

	Andrew McCreight [:mccr8]
	Comment 15 • 11 years ago

Haha.  If anything, I'd expect my nsTimeout patch to make things slower.  Though maybe we're more likely to inline the nsTimeout AddRef and Release. :)

	Boris Zbarsky [:bzbarsky]
	Comment 16 • 11 years ago

The point is, during the parts of kraken that are being timed your nsTimeout stuff should not be exercised at all.