This regression was seen in succession on m-i, m-c, and aurora, but each alert was blamed on a different changeset: Message: 7 Date: Sun, 04 Nov 2012 17:41:11 -0000 From: nobody@cruncher.build.mozilla.org To: dev-tree-management@lists.mozilla.org Subject: Talos Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Inbound Message-ID: <20121104174111.DE0771042A2@cruncher.srv.releng.scl3.mozilla.com> Content-Type: text/plain; charset="us-ascii" Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Inbound -------------------------------------------------------------------------------- Previous: avg 25203936.667 stddev 742.541 of 30 runs up to revision e5d4cfce296c New : avg 28804040.000 stddev 427.785 of 5 runs since revision 58b4bd7b5065 Change : +3600103.333 (14.3% / z=4848.353) Graph : http://mzl.la/VqvEX0 Changeset range: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=e5d4cfce296c&tochange=58b4bd7b5065 Changesets: * http://hg.mozilla.org/integration/mozilla-inbound/rev/58b4bd7b5065 : Jonathan Kew <jkew@mozilla.com> - bug 808288 - render missing glyphs as hexboxes in <canvas> text. r=bas : http://bugzilla.mozilla.org/show_bug.cgi?id=808288 Bugs: * http://bugzilla.mozilla.org/show_bug.cgi?id=808288 - <canvas> text drawing does not handle missing glyphs properly -------------- Message: 6 Date: Mon, 05 Nov 2012 09:40:40 -0000 From: nobody@cruncher.build.mozilla.org To: dev-tree-management@lists.mozilla.org Subject: Talos Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) mobile Message-ID: <20121105094040.4C878104126@cruncher.srv.releng.scl3.mozilla.com> Content-Type: text/plain; charset="us-ascii" Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) mobile ----------------------------------------------------------------------- Previous: avg 25203900.000 stddev 763.386 of 30 runs up to revision e9a9341e27ed New : avg 28803880.000 stddev 601.664 of 5 runs since revision 5c6b71348e20 Change : +3599980.000 (14.3% / z=4715.804) Graph : http://mzl.la/TsAlJj Changeset range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e9a9341e27ed&tochange=5c6b71348e20 Changesets: * http://hg.mozilla.org/mozilla-central/rev/5571f8eb4c0a : Ms2ger <ms2ger@gmail.com> - Bug 807926 - Don't handle outer windows in nsWindowSH::NewResolve; r=bholley : http://bugzilla.mozilla.org/show_bug.cgi?id=807926 * http://hg.mozilla.org/mozilla-central/rev/647f0ed44f2f : Ms2ger <ms2ger@gmail.com> - Bug 801991 - Remove the ability to generate custom quickstubs; r=peterv : http://bugzilla.mozilla.org/show_bug.cgi?id=801991 * http://hg.mozilla.org/mozilla-central/rev/8f56f3f96a79 : Ms2ger <ms2ger@gmail.com> - Bug 803520 - Don't use JSCList for JSRuntime::contextList; r=Waldo : http://bugzilla.mozilla.org/show_bug.cgi?id=803520 * http://hg.mozilla.org/mozilla-central/rev/20965bd69e28 : Ms2ger <ms2ger@gmail.com> - Bug 804005 - Update dom/imptests python to be py3k-compatible; r=jhammel : http://bugzilla.mozilla.org/show_bug.cgi?id=804005 * http://hg.mozilla.org/mozilla-central/rev/fdb936f5d986 : Ms2ger <ms2ger@gmail.com> - Bug 629876 - Throw when drawImage's sw/sh arguments are zero; r=bz : http://bugzilla.mozilla.org/show_bug.cgi?id=629876 * http://hg.mozilla.org/mozilla-central/rev/aea5ae9d8762 : Ms2ger <ms2ger@gmail.com> - Bug 709954 - Fix assertion failure "!cx->isExceptionPending()" with invalid regexp in <input pattern>; r=bz : http://bugzilla.mozilla.org/show_bug.cgi?id=709954 * http://hg.mozilla.org/mozilla-central/rev/40e08e6b0fda : Ms2ger <ms2ger@gmail.com> - Bug 807410 - Part a: Pass the global object to the nsJSContext constructor; r=jst : http://bugzilla.mozilla.org/show_bug.cgi?id=807410 * http://hg.mozilla.org/mozilla-central/rev/8e21cdfce89c : Ms2ger <ms2ger@gmail.com> - Bug 807410 - Part b: Pass the "GC on destruction" flag to the nsJSContext constructor; r=jst : http://bugzilla.mozilla.org/show_bug.cgi?id=807410 * http://hg.mozilla.org/mozilla-central/rev/cac8275e885b : Ms2ger <ms2ger@gmail.com> - Bug 807465 - Less QIs in nsXULPopupManager::UpdateMenuItems; r=Enn : http://bugzilla.mozilla.org/show_bug.cgi?id=807465 * http://hg.mozilla.org/mozilla-central/rev/5c6b71348e20 : Ms2ger <ms2ger@gmail.com> - Bug 808305 - Pass a gfx::Rect instead of a gfxRect in InvalidateCanvasContent; r=Bas : http://bugzilla.mozilla.org/show_bug.cgi?id=808305 Bugs: * http://bugzilla.mozilla.org/show_bug.cgi?id=803520 - Don't use JSCList for JSRuntime::contextList * http://bugzilla.mozilla.org/show_bug.cgi?id=709954 - "Assertion failure: !cx->isExceptionPending()" with invalid regexp in <input pattern> * http://bugzilla.mozilla.org/show_bug.cgi?id=804005 - Update dom/imptests python to be py3k-compatible * http://bugzilla.mozilla.org/show_bug.cgi?id=801991 - Remove the ability to generate custom quickstubs * http://bugzilla.mozilla.org/show_bug.cgi?id=807465 - Less QIs in nsXULPopupManager::UpdateMenuItems * http://bugzilla.mozilla.org/show_bug.cgi?id=808305 - Pass a gfx::Rect instead of a gfxRect in InvalidateCanvasContent * http://bugzilla.mozilla.org/show_bug.cgi?id=807410 - Pass some things to the nsJSContext constructor instead of using setters * http://bugzilla.mozilla.org/show_bug.cgi?id=629876 - Throw when drawImage's sw/sh arguments are zero * http://bugzilla.mozilla.org/show_bug.cgi?id=807926 - Don't handle outer windows in nsWindowSH::NewResolve ------------ Message: 4 Date: Tue, 06 Nov 2012 02:20:59 -0000 From: nobody@cruncher.build.mozilla.org To: dev-tree-management@lists.mozilla.org Subject: Talos Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Aurora Message-ID: <20121106022059.687B01042A2@cruncher.srv.releng.scl3.mozilla.com> Content-Type: text/plain; charset="us-ascii" Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Aurora ------------------------------------------------------------------------------- Previous: avg 25204003.333 stddev 910.280 of 30 runs up to revision e070197f4c89 New : avg 28804420.000 stddev 303.315 of 5 runs since revision e070197f4c89 Change : +3600416.667 (14.3% / z=3955.286) Graph : http://mzl.la/SvalhS Changeset range: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=e070197f4c89&tochange=e070197f4c89

Shouldn't we be more concerned that the Ts shutdown data looks totally bogus? 25204003 milliseconds to shutdown?

(In reply to Mark Finkle (:mfinkle) from comment #1) > Shouldn't we be more concerned that the Ts shutdown data looks totally > bogus? 25204003 milliseconds to shutdown? I think :jmaher was looking into that.

Now on mozilla-beta also: Message: 2 Date: Wed, 07 Nov 2012 03:02:29 -0000 From: nobody@cruncher.build.mozilla.org To: dev-tree-management@lists.mozilla.org Subject: Talos Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Beta Message-ID: <20121107030229.72A45104126@cruncher.srv.releng.scl3.mozilla.com> Content-Type: text/plain; charset="us-ascii" Regression Ts Shutdown increase 14.3% on Android 2.2 (Native) Mozilla-Beta ----------------------------------------------------------------------------- Previous: avg 25203886.667 stddev 886.190 of 30 runs up to revision 9de15654e484 New : avg 28803920.000 stddev 1669.431 of 5 runs since revision 6e071059b71c Change : +3600033.333 (14.3% / z=4062.370) Graph : http://mzl.la/RHxcHR Changeset range: http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=9de15654e484&tochange=6e071059b71c Changesets: * http://hg.mozilla.org/releases/mozilla-beta/rev/1fe8fd75e5d5 : Gian-Carlo Pascutto <gpascutto@mozilla.com> - Bug 782106 - Disable the per-client randomization in the url-classifier. r=dcamp a=akeybl : http://bugzilla.mozilla.org/show_bug.cgi?id=782106 * http://hg.mozilla.org/releases/mozilla-beta/rev/6e071059b71c : Gian-Carlo Pascutto <gpascutto@mozilla.com> - Bug 807847 - Noise entries are erroneously entered into the misscache. r=dcamp a=akeybl : http://bugzilla.mozilla.org/show_bug.cgi?id=807847 Bugs: * http://bugzilla.mozilla.org/show_bug.cgi?id=782106 * http://bugzilla.mozilla.org/show_bug.cgi?id=807847 - Noise entries are erroneously entered into the misscache

>I think :jmaher was looking into that. Bug 797339 and perhaps others.

As discussed in bug 797339 recently, these times are based on milliseconds-since-the-epoch. A difference of approx 3600000 is 1 hour: 60x60x1000. The regression is not logically connected with any changeset, but corresponds roughly to the daylight-savings-time change!!

Wow, nice job tracking that down. But seriously, wtf. I feel like talos really needs an overhaul.

I would say this is more a factor of our machine management vs talos. For the desktop we do not experience any of these problems and the desktop has the date/time set correctly. Actually every time there is a time change we have a series of unittests fail for a brief period of time (some of those might be fixed or disabled now). We have been working on given talos an overhaul, unfortunately it still has a good ways to go.

Bug 797339 is the bug to fix.