Back in 2011, in bug 630428, we were having test failures on WinXP slaves because they didn't know about the 2007 changes to when Daylight Saving Time started and ended. https://tbpl.mozilla.org/php/getParsedLog.php?id=23665562&tree=Mozilla-Central https://tbpl.mozilla.org/php/getParsedLog.php?id=23665299&tree=Mozilla-Central https://tbpl.mozilla.org/php/getParsedLog.php?id=23655610&tree=Mozilla-Central are the same sort of failures, and when you look at a random test log for a random test, and look at the timestamp for the buildstep, which comes from the master, and then look at the timestamps from mozharness at the start of every line, which come from the slave, you see ========= Started 'c:/mozilla-build/python27/python -u ...' (results: 0, elapsed: 6 mins, 2 secs) (at 2013-05-31 22:45:16.552677) ========= ... 23:45:05 INFO - MultiFileLogger online at 20130531 23:45:05 in C:\slave\test indicating that either some, most, or all the t-xp32-ix- slaves are running an hour in the future. Are they running in Alaskan time, are they set to the wrong time, or have they set themselves into DST twice because they thought it started again on the second Sunday in April?

t-xp32-ix-051 https://tbpl.mozilla.org/php/getParsedLog.php?id=23666558&tree=Mozilla-Inbound

t-xp32-ix-032 https://tbpl.mozilla.org/php/getParsedLog.php?id=23666909&tree=Mozilla-Inbound t-xp32-ix-037 https://tbpl.mozilla.org/php/getParsedLog.php?id=23666956&tree=Mozilla-Inbound

t-xp32-ix-004 https://tbpl.mozilla.org/php/getParsedLog.php?id=23666555&tree=UX t-xp32-ix-081 https://tbpl.mozilla.org/php/getParsedLog.php?id=23655928&tree=UX t-xp32-ix-048 https://tbpl.mozilla.org/php/getParsedLog.php?id=23655577&tree=UX

t-xp32-ix-123 https://tbpl.mozilla.org/php/getParsedLog.php?id=23655899&tree=UX At least on UX, this seems to have started on Friday.

t-xp32-ix-029 https://tbpl.mozilla.org/php/getParsedLog.php?id=23714801&tree=Mozilla-Inbound t-xp32-ix-056 https://tbpl.mozilla.org/php/getParsedLog.php?id=23714642&tree=Mozilla-Central

t-xp32-ix-087 https://tbpl.mozilla.org/php/getParsedLog.php?id=23715147&tree=Mozilla-Inbound

Setting up a fix on the windows side to fix this releng domain wide

For reference, we deployed this file to our rev3 minis in the past: WindowsXP-KB931836-x86-ENU.exe [1] We hit the issue in bug 630428. [1] http://hg.mozilla.org/build/opsi-package-sources/file/5ffc5be3d1fe/winxp-daylight-savings/CLIENT_DATA/winxp-daylight-savings.ins

Let's re-open if it happens again.

We're back into another period where it can happen, and it's happening again, starting around https://tbpl.mozilla.org/php/getParsedLog.php?id=24803916&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=24821557&tree=Mozilla-Aurora https://tbpl.mozilla.org/php/getParsedLog.php?id=24821503&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=24820653&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=24821340&tree=Mozilla-Inbound

Q, did this fix ever deploy/happen? (c.f. c#7 and c#8)

Removing buildduty whiteboard tag because this isn't actionable by buildduty.

https://tbpl.mozilla.org/php/getParsedLog.php?id=24835005&tree=Try https://tbpl.mozilla.org/php/getParsedLog.php?id=24834997&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=24911316&tree=Mozilla-Aurora https://tbpl.mozilla.org/php/getParsedLog.php?id=24911912&tree=Mozilla-Aurora https://tbpl.mozilla.org/php/getParsedLog.php?id=24911395&tree=Fx-Team

https://tbpl.mozilla.org/php/getParsedLog.php?id=24926358&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=24926529&tree=Mozilla-Inbound

If we're going to spam it, we might as well spam it right.

The kb fixes were deployed and the problems with machines having the wrong time zones ( some were in mdt ) were corrected. Time for deep under the hood dives if this is an ongoing issue.

I've checked t-xp32-ix-084 and the time is 8:15am (on my side if 11:15am) which seems correct. The machine has Service Pack 3. Maybe I don't have privileges to see more but the KB931836 fix does not show up on the list installed updates (I checked "show updates"). Q, would you mind checking if I'm doing anything wrong? please :) Can I please get some instructions on how to login as root on those machines? I couldn't switch users with VNC and RDP was not doing anything. (In reply to Q from comment #24) > The kb fixes were deployed and the problems with machines having the wrong > time zones ( some were in mdt ) were corrected. Time for deep under the hood > dives if this is an ongoing issue. IIUC, this happens once few times a year because of end of the month timings. ################# I've looked at all the logs from comment 10 and onwards and here are the slaves that hit it. slave: t-xp32-ix-039 slave: t-xp32-ix-015 slave: t-xp32-ix-038 slave: t-xp32-ix-089 slave: t-xp32-ix-035 slave: t-xp32-ix-067 slave: t-xp32-ix-076 slave: t-xp32-ix-074 slave: t-xp32-ix-084

(In reply to Armen Zambrano G. [:armenzg] (back in July 7th) from comment #25) > > I've looked at all the logs from comment 10 and onwards and here are the > slaves that hit it. > slave: t-xp32-ix-039 > slave: t-xp32-ix-015 > slave: t-xp32-ix-038 > slave: t-xp32-ix-089 > slave: t-xp32-ix-035 > slave: t-xp32-ix-067 > slave: t-xp32-ix-076 > slave: t-xp32-ix-074 > slave: t-xp32-ix-084 My apologies, this list was not complete when I hit submit. I got lazy after looking at several logs and saw it not relevant to have a complete list.

KB955839 superseded and replaced KB931836 ( which should be part of sp3 already). I am forcing a repush of KB955839 to make sure all slaves have it. I will ping you out of bug to walk through root log in instructions.

Ran reports across the domain and KB955839 is on the XP slaves.

I don't know what to look into :/

can you come up with a simple test script that I can set the time on an xp slave to a potential problem windows and run to replicate the error?

From the previous instance (adjusted for the way we seem to have made javascript: URLs less useful in the intervening years), in Firefox, Tools menu -> Web Developer -> Error Console, paste new Date(1301643000000).toDateString() in the text input and press enter, good should print "Fri Apr 01 2011" and bad should print "Thu Mar 31 2011".

(In reply to Phil Ringnalda (:philor) from comment #31) > From the previous instance (adjusted for the way we seem to have made > javascript: URLs less useful in the intervening years), in Firefox, Tools > menu -> Web Developer -> Error Console, paste > > new Date(1301643000000).toDateString() > > in the text input and press enter, good should print "Fri Apr 01 2011" and > bad should print "Thu Mar 31 2011". A command-line alternative would be: python -c "import datetime;print datetime.datetime.fromtimestamp(1301643000)" Should yield "2011-04-01 00:30:00" if the DST settings are correct. Q: is that more useful for scripting?

Status: * waiting on needinfo from Q

Running a loop to identify offenders so far only t-xp32-ix-091 is a problem.

This last script from coop with some modification for print formatting is working like a charm

All XP machines tested only three problematic systems reported: * t-xp32-ix-091 - Stuck on a reinstall does not have all of the correct patches will hand over to Mark to investigate. * t-xp32-ix-099 - Someone manually changed the "automatically adjust for daylight savings time" option in time date - fixed and remote correction steps put into timezone scripts * t-xp32-ix-043 - Offline

(In reply to Q from comment #36) > * t-xp32-ix-091 - Stuck on a reinstall does not have all of the correct > patches will hand over to Mark to investigate. Filed bug 914795 > * t-xp32-ix-043 - Offline Filed bug 914793 Work on this is happening over in bug 913618 now.

Thanks coop!