b2g_b2g-inbound_win32_gecko-debug build on 2014-08-29 00:38:43 PDT for push afe145022092 slave: b-2008-ix-0130 https://tbpl.mozilla.org/php/getParsedLog.php?id=47011046&tree=B2g-Inbound configure:22686: checking for Unicode NSIS version 2.46 or greater configure: error: To build the installer you must have the latest MozillaBuild or Unicode NSIS version .46 or greater in your path. *** Fix above errors and then restart with\ "c:/builds/moz2_slave/b2g-in-w32_g-d-000000000000000/build/mozmake.exe -f client.mk build" c:/builds/moz2_slave/b2g-in-w32_g-d-000000000000000/build/client.mk:355: recipe for target 'configure' failed

https://tbpl.mozilla.org/php/getParsedLog.php?id=47007387&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=47011651&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=47011046&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=47010662&tree=B2g-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=46979824&tree=Thunderbird-Esr31 https://tbpl.mozilla.org/php/getParsedLog.php?id=46949663&tree=Thunderbird-Esr31

So at this point in time it is not clear what the source of this problem could be. Speaking to Mike Hommey, he believes that NSIS is a build dependency and should be part of the Windows image. However, the affected slaves often do not have tracking bugs, and it would therefore appear as though they have not recently been reimaged. Some of these slaves were running jobs successfully a few hours before, so it appears something external may have changed. However, if something external has changed, it does not appear to be affecting all the Windows build slaves, since others are not exhibiting this problem. Currently the only reasonable plan we have is to disable offending slaves, until we can work out what is causing this problem.

I presume this is related to bug 989531 which rolled out yesterday.

it sounds like things are fixed up with this: https://bugzilla.mozilla.org/show_bug.cgi?id=989531#c10 https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=b-2008-ix-0138 has been hit this a lot prior to fix. if it greens up, we should resolve this bug and re-enable disabled slaves

According to https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=build&type=b-2008-ix&name=b-2008-ix-0114 I rebooted it at 8:02pm, and it failed this way on a 8:28pm build. I'm disabling the rest of the appaarently-still-affected slaves, too - we've got far too many Windows build slaves, we don't need these broken ones running over the weekend.

0138 wasn't actually any better, it was just doing a good job of only taking jobs where nobody would look at the way it failed.

Sounds like we need a deeper dive into this ?

Unless b-2008-ix-0122 was some separate bustage that just happens to have the same visible symptom, I'd certainly say we do, since it successfully built yesterday, and wound up broken today.

hit b-2008-ix-0137 on Aug 28 during Thunderbird 31.1.0 release: http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/release-comm-esr31-win32_repack_1%2F10/builds/1

Is there a machine we can use for testing for this? I think we take a machine install NSIS 3.0a2 and then add it to the path, and see if we get the same error. If that works then we can just push that out through a GPO.

hit b-2008-ix-0114 on Aug 28 during Thunderbird 31.1.0 release repack: http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/release-comm-esr31-win32_repack_1%2F10/builds/2

(In reply to Mark Cornmesser [:markco] from comment #14) > Is there a machine we can use for testing for this? b-2008-ix-0114 has been disabled, with this bug given as the reason. You should be able to use that.

b-2008-ix-0114 has been set up with the files for nsis 3.0a2 and the directory has been added to the path on the machine. If this works, I have a GPO ready to push out these changes.

Slight delay in the testing, since earlier I reenabled 0014 instead of 0114, but now 114 is enabled, waiting for it to take a job where this would have hit.

(In reply to Phil Ringnalda (:philor) from comment #18) > Slight delay in the testing, since earlier I reenabled 0014 instead of 0114, > but now 114 is enabled, waiting for it to take a job where this would have > hit. didn't work out -> https://tbpl.mozilla.org/php/getParsedLog.php?id=48158116&tree=Mozilla-Inbound configure: error: To build the installer you must have the latest MozillaBuild or Unicode NSIS version .46 or greater in your path. disabled that slave again

I went back looked at the GPO and the file copy was missing some of the sub-directories. I have corrected this. However, am I miss interpreting "your path"? I have added nsis 3.0a2 to the system variable for path. Is that where it needed to be added?

Getting mass failures now. All trees closed.

The GPO changes were backed out and things seem to have gone back to normal. Reopened at 12:10 MVT.

(In reply to Mark Cornmesser [:markco] from comment #20) > However, am I miss interpreting "your path"? I have added nsis 3.0a2 to the > system variable for path. Is that where it needed to be added?

Hi guys, Should we close this bug now, or is there more to do? Many thanks, Pete

haven;t seen any new slaves so i think its ok to close from my pov

We rolled back the change that pushed out NSIS. The deployment is still broken (path issues that someone needs to resolve and pass onto relops to fix the GPO), so that bug still needs to be fixed before we can deploy. I'm not sure if that should be tracked here or in a different bug, though.

(In reply to Amy Rich [:arich] [:arr] from comment #26) > We rolled back the change that pushed out NSIS. The deployment is still > broken (path issues that someone needs to resolve and pass onto relops to > fix the GPO), so that bug still needs to be fixed before we can deploy. I'm > not sure if that should be tracked here or in a different bug, though. Thanks Amy! Since bug 989531 is tracking that rollout then, and these symptoms have been solved by the current rollback, I'll close this for now, and it can always be reopened if we roll out again, and cause the same problem. Pete