The Mozmill fallback update tests started failing on Nightly IT with today's build. It appears that the Italian locale is being offered the wrong fallback update: https://aus3.mozilla.org/update/3/Firefox/21.0a2/20130313042013/WINNT_x86-msvc/it/aurora/Windows_NT%206.1/default/default/update.xml?force=1 Mozmill report: http://mozmill-crowd.blargon7.com/#/update/report/2a6536e9db9f5f44ed48c58510b605d1 Jenkins link: http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-central_update/1377/console The error can be reproduced locally by running the Mozmill tests and after update Firefox cannot be opened (I have attached the screenshot with the error received when trying to open Firefox after update).

Confirmed manually, here is the update I'm offered on fallback (br instead of it): http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-15-03-09-43-mozilla-central-l10n/firefox-22.0a1.br.win32.complete.mar

Looking at the update.xml snippet it seems like there's a typo: https://aus3.mozilla.org/update/3/Firefox/22.0a1/20130314030914/WINNT_x86-msvc/it/nightly/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-15-03-09-43-mozilla-central-l10n/firefox-22.0a1.br.win32.complete.mar"

I'm disabling nightly updates while we investigate this.

So, looking at logs I see: ========= Started find filepath (results: 0, elapsed: 0 secs) (at 2013-03-15 08:24:59.857282) ========= 'bash' '-c' 'find build/mozilla-central/obj-firefox/dist/update -maxdepth 1 -type f -name *.complete.mar' in dir e:\builds\moz2_slave\m-cen-w32-l10n-ntly-0000000000\. (timeout 1200 secs) build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.br.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.et.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.he.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.it.win32.complete.mar and we end up taking the first one as the complete MAR. So, either something isn't cleaning up like it did before, or we changed something in our buildbot code that's making us less choosey about the MAR.

Looks like this is related to bug 784848 13:14 < bhearsum> i see "'sh' '-c' 'if [ -d mozilla-central/dist/update ]; then rm -rf mozilla-central/dist/update; fi'" is yesterday's l10n nightly.... 13:14 < bhearsum> coop: ah, whic his why https://github.com/mozilla/build-buildbotcustom/commit/1115642ad5573710d40280c6c603d034af3e1a81#L1R2439 got added, i guess? 13:15 < bhearsum> i see this in today's nightly: "'sh' '-c' 'if [ -d obj-firefox/dist/update ]; then rm -rf obj-firefox/dist/update; fi'" 13:15 < bhearsum> which gets run in in dir e:\builds\moz2_slave\m-cen-w32-l10n-ntly-0000000000\build (timeout 1200 secs) 13:15 <@coop> yes 13:15 < bhearsum> so that dir probably doesn't exist...needs to be mozilla-central/obj-firefox 13:15 <@coop> that's not the correct dir, yeah 13:15 < bhearsum> boo

After this gets fixed and a new set of nightlies get run we need to chmod the snippet directory to enable nightlies again: chmod 775 /opt/aus2/incoming/2/Firefox/mozilla-central should do it.

Comment on attachment 725595 [details] [diff] [review] Specify the proper path to the dist dir for pre-build cleanup https://hg.mozilla.org/build/buildbotcustom/rev/84f077434979

In production now, although bug 851720 may prevent aurora updates from happening tonight.

Updates for nightly still seem to be off. Can we turn them back on now?

(In reply to Reed Loden [:reed] from comment #10) > Updates for nightly still seem to be off. Can we turn them back on now? I just asked about this and was told that bug 851793 also need to be fixed first.

We'll track further issues via bug 852106.