A while back I submitted my patch to bug 305039 to the try server (accidentally) and the build failed. I don't have the link to the error log on hand, but the culprit was a missing makensisu.exe (Unicode NSIS executable). Once bug 305039 lands, all Windows try server builds will fail. We need to have Unicode NSIS on the try server. Since Unicode NSIS is included in MozillaBuild, upgrading the machine(s) to the latest MozillaBuild release should do the trick.

Ouch... if there are any build machines not running the latest release of MozillaBuild then they need to be upgraded as soon as possible. cc'ing a bunch of people to give them a heads up Thanks for catching this Ehsan

(In reply to comment #1) "any build machines" should be "any *windows* build machines"

urgh. We just recently setup additional try slaves, and afaik they were identical to the existing try slaves and production slaves. Please specify what version of MozillaBuild / NSIS you need. Do you know if the version of NSIS on current production machines is ok?

MozillaBuild 1.2 and 1.3 both include Unicode NSIS, but I have been testing with MozillaBuild 1.3, which ships Unicode NSIS 2.33 (and NSIS 2.22). We need to make sure that exact versions of Unicode NSIS is present on build slaves, because I've been testing the patch with that version (and I suspect the same is true for Robert). I have not tested it on production machines yet (the patch has not yet landed) but if they include MozillaBuild 1.3, then all should be fine.

I believe all the other build machines have MozillaBuild 1.2 on them, which includes the same version of unicode NSIS as 1.3. Either 1.2 or 1.3 would be fine.

1) Ted: Glad to hear MozillaBuild1.2 (and the included NSIS v2.22) is sufficient. From our ref platform docs, https://wiki.mozilla.org/ReferencePlatforms/Win32#Install_MozillaBuild_1.2, NSIS v2.22 is already being used on our production win32 build slaves, and in our new replacement try slaves. So, hopefully we are already all good. 2) Ehsan: You mention you tried this "a while back". Would you mind submitting your patch to the try server again? I ask because we recently added more try slaves to improve overall try server throughput. Those new try slaves were identical to the production build slaves. While doing that we also replaced the old un-maintained try slaves with new slaves, making all the slaves identical to our existing production slaves. So, afaict, all those try slaves should already have NSIS v2.22 on them, and your patch should be ok. (/me crosses his fingers!)

From a random windows build slave, $ which makensisu /d/mozilla-build//nsis-2.33u/makensisu.exe and the path contains ...:/d/mozilla-build//nsis-2.22:/d/mozilla-build//nsis-2.33u:... so I'd agree with John that it's worth resubmitting the build to the try server. Also, $ which makensis /d/mozilla-build//nsis-2.22/makensis.exe $ which makensisw.exe /d/mozilla-build//nsis-2.22/makensisw.exe I see makensisw is referenced in the v1.5 patch on bug 305039, is that executed in the build or on the client machine ? Please confirm that the copy from 2.22 is OK, or let us know if the path order needs switching.

(In reply to comment #7) > From a random windows build slave, > $ which makensisu > /d/mozilla-build//nsis-2.33u/makensisu.exe > and the path contains > ...:/d/mozilla-build//nsis-2.22:/d/mozilla-build//nsis-2.33u:... > so I'd agree with John that it's worth resubmitting the build to the try > server. OK, I have just submitted a patch to see how it goes, but it looks like that the problem should have been fixed. > Also, > $ which makensis > /d/mozilla-build//nsis-2.22/makensis.exe > $ which makensisw.exe > /d/mozilla-build//nsis-2.22/makensisw.exe > I see makensisw is referenced in the v1.5 patch on bug 305039, is that executed > in the build or on the client machine ? Please confirm that the copy from 2.22 > is OK, or let us know if the path order needs switching. makensisw is a GUI front-end to NSIS, which we're not using. The non-Unicode NSIS version (or its existence for that matter) won't make a difference at all because it's not used once the Unicode installer patch lands.

The submitted build failed with exactly the same error as I was seeing before: <http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1230375206.1230380592.27609.gz#err7>

Huh. I actually got as far as patching this, since as you can see in the log every time the tryserver buildbot spews the environment, the problem is that the PATH doesn't include nsis-2.33u, and then there's that stray 'd:\\sdks\\v6.0\\bin' in http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/tools/buildbotcustom/env.py&rev=1.10&mark=38,55-56#29 which winds up being output at the start of INCLUDE, but then I realized that that *doesn't* show up in the mozilla-central tinderbox logs, and that they actually have a different path which does include nsis-2.33u, and which looks like the default msys shell path that mozilla-build produces, so I guess the actual problem is that the tryserver *uses* env.py and as a result is very different from the tinderboxes, not that env.py is wrong.

But assuming we're happy with the tryserver building with a rather different environment than the actual build slaves, this ought to fix Ehsan's problem (and the problem he didn't know he was going to have, yet, where the unit tests would have failed trying to build the uninstaller).

(In reply to comment #11) > Created an attachment (id=354566) [details] > Fix env.py paths, v.1 > > But assuming we're happy with the tryserver building with a rather different > environment than the actual build slaves, this ought to fix Ehsan's problem > (and the problem he didn't know he was going to have, yet, where the unit tests > would have failed trying to build the uninstaller). Strange. The try win32 slaves are identical to the production win32 slaves afaik, and so NSIS should be good enough as is. From irc, next week Aki will reconfirm what I claimed in comment#6. Our leading suspicion right now is that the slaves are identical, but that the try master has some slightly different config - in which case we'll have to bring the masters back in sync with each other! After all, the whole idea of the try setup is to "try" code out before landing in production, :-) so these are supposed to be identical.

Comment on attachment 354566 [details] [diff] [review] Fix env.py paths, v.1 Actually, I wouldn't have been saving Ehsan from the unit testers, because I can't figure out what they are using. It looks quite a bit like tools/buildbot-configs/testing/unittest/mozbuild.py, but the bogus MOZILLABUILD=D:\mozilla-build MOZILLABUILDDRIVE=C: combo in the logs doesn't seem to come from anything in CVS.

While it would be lovely if everything had the identical config (of build, leaktest, nightly, unittest, and try, only build and leaktest actually build with the same paths), in the meantime Ehsan's stuck. Since there's no case where we want to have nsis-2.22 and not nsis-2.33u in the path (mozilla-build goes to great lengths to rename the 2.33u binary just so that they can both be in the path, and that's what build, leaktest, nightly, and every developer already uses), the shotgun approach ought to fix *this* bug.

Comment on attachment 355602 [details] [diff] [review] Shotgun It appears this will only affect the unit tests, which may already be rolled into moz2, and the try server.

I have tested this and the above "shotgun" patch against staging-try and staging-master. I've avoided changing the base env.py too much since I'm unaware of the full scope.

Comment on attachment 356017 [details] [diff] [review] Make the tryserver win32 env match the 1.9.1/m-c win32 env Checking in env.py; /cvsroot/mozilla/tools/buildbotcustom/tryserver/env.py,v <-- env.py new revision: 1.2; previous revision: 1.1 done

Comment on attachment 355602 [details] [diff] [review] Shotgun Checking in mozbuild.py; /cvsroot/mozilla/tools/buildbot-configs/testing/unittest-stage/mozbuild.py,v <-- mozbuild.py new revision: 1.9; previous revision: 1.8 done Checking in leaktest/mozbuild.py; /cvsroot/mozilla/tools/buildbot-configs/testing/leaktest/mozbuild.py,v <-- mozbuild.py new revision: 1.5; previous revision: 1.4 done Checking in unittest/mozbuild.py; /cvsroot/mozilla/tools/buildbot-configs/testing/unittest/mozbuild.py,v <-- mozbuild.py new revision: 1.26; previous revision: 1.25 done Checking in env.py; /cvsroot/mozilla/tools/buildbotcustom/env.py,v <-- env.py new revision: 1.11; previous revision: 1.10 done I didn't check in the seamonkey part, that's up to KaiRo

tryserver-master config has been updated and reconfig'ed. production-master will take downtime due to bug 472333.

(In reply to comment #18) > I didn't check in the seamonkey part, that's up to KaiRo I *did* mention it was a shotgun patch, didn't I? Once I looked beyond the context in the patch, the seamonkey part is dead code, pulling from CVS, from before the comm-central switch. All the SeaMonkey tinderboxes already have nsis-2.33u in their paths.

I pushed my patch again to the try server, and this time it built fine on Windows. I'm not sure if we should mark this as FIXED yet or give it more time.

I saw that, a pretty happy green :) But it's only fixed, and not FIXED, and you can't push to mozilla-central yet, until they take some downtime, close the tree, and update the buildbot production-master, because the unit test builds would still turn red on you right now (they don't build the installer, but they do build the uninstaller since everything does, and you're calling makensisu.exe there).

The production-master has picked up this change, and I'm seeing nsis-2.33u in the PATHs. Resolving.

Fixed 1.9.1 as per bhearsum on IRC

What's the best way to verify this patch for m-c and 1.9.1?

Since Unicode NSIS patch has been landed, any Windows builds which succeed now is a testimony that this is FIXED.

We had several tryserver builds in the within the last year on 1.9.2 and 1.9.1 which were all fine. Marking this bug as verified fixed.