Our mozmill test for crashreporter has failed a few times on Linux due to the fact that sometimes the crashreporter is not enabled inside the application.ini file of the build (bug 880220). Here I found the 'it' build which shows this issue: ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla-central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2 We also have seen it in May 31st, June 6th and June 22th, but for the older ones there are no l10n builds on ftp anymore. It never happened with en-US, only it, fr, de (these are the one we test so far, could be other locales as well).

Ted and Ben, do you have any idea why it's not always enabled in localized builds?

This would require me to know way more about the l10n process than I currently do. I can tell you that application.ini is informational only nowadays, its contents are compiled into the browser binary directly, so you should sanity check that crash reporting is actually enabled in these builds (you can use my Crash Me Now extension: http://people.mozilla.com/~tmielczarek/crashme/ )

If there's actually a problem here, it's probably a question for Axel.

That's not something I can help with. I suspect that for some or all l10n repacks, MOZILLA_OFFICIAL doesn't match with what en-US is built with. That is, if the repacks actually rebuild the application.ini, which would be packaging code I don't understand. Reading Ted's comment, that probably doesn't have an impact on whether crash reporting actually works or not, though.

I found the difference from enabled or not in that application.ini file, but it is certainly disabled for those builds where our test has failed (see the link in first comment as example). We check that using nsICrashReporter: http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional/testInstallation/testBreakpadInstalled.js#l45 Shouldn't the crashreporter be enabled on all builds by default? Even localized? I need this information to know how to further proceed with our test, if we need to update the way we handle it.

Yes, it definitely should be.

(In reply to Andreea Matei [:AndreeaMatei] from comment #0) > Here I found the 'it' build which shows this issue: > ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla- > central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2 Ben, do we have the repack logs still available for that l10n build? If yes, is there something suspicious in there? This bug occurs rather rarely, so I could imaging something is broken with the builder, and it incorrectly writes out the application.ini file.

(In reply to Henrik Skupin (:whimboo) [away 06/28 - 07/07] from comment #7) > (In reply to Andreea Matei [:AndreeaMatei] from comment #0) > > Here I found the 'it' build which shows this issue: > > ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla- > > central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2 > > Ben, do we have the repack logs still available for that l10n build? If yes, > is there something suspicious in there? This bug occurs rather rarely, so I > could imaging something is broken with the builder, and it incorrectly > writes out the application.ini file. No. Looks like we only keep a weeks worth of l10n builds/logs.

Ok, so when it happens again, we will make sure to reply immediately. That will hopefully help us to nail down the reported issue.

I can reproduce again with today's it, de and fr builds from here: ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/

Setting needinfo flag for Ben so we do not miss to check for the logs of those builds until they will get deleted.

I checked all 4 files currently available for it, only linux64 is missing the line "Enabled = 1" in CrashReporter section.

I wonder if 23.0 builds are also affected. But for now asking blocking for 24.0 and 25.0.

If I'm not mistaken, based on the buildid in application.ini this should be the build log.

(In reply to Andreea Matei [:AndreeaMatei] from comment #10) > I can reproduce again with today's it, de and fr builds from here: > ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/ Logs are here: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/ Grab 'em while you can.

I rechecked and the request came through the job #495. So the correct build log has been already attached by flod. Ben, can you please have a look at and check if something is suspicious?

(In reply to Henrik Skupin (:whimboo) from comment #16) > I rechecked and the request came through the job #495. So the correct build > log has been already attached by flod. Ben, can you please have a look at > and check if something is suspicious? I don't have anymore insight into this than you do -- none of the build steps failed, the job was green. I know only a little bit of the l10n and packaging parts of the build system, and nothing in the logs stands out to me.

Interestingly, the build id and sourcestamp are different between the it 64 build, the it 32 build, and en-US 32/64. I get three different ones, actually. 64 it: fx_revision 5976b9c673f8 buildid 20130715211256 32 it: fx_revision 5e191a26d909 buildid 20130715100109 en-US 32 and 64: [App] Vendor=Mozilla Name=Firefox Version=25.0a1 BuildID=20130715030202 SourceRepository=http://hg.mozilla.org/mozilla-central SourceStamp=ff0a372e3170 ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384} Ben, any idea?

Possibly of note, there were three nightlies yesterday: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-03-02-02-mozilla-central-l10n/ https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-10-01-09-mozilla-central-l10n/ https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/

It might not be related to the number of builds per day, given that the dates from the original comment do not have multiple nightly builds per day. So it should be something different.

(In reply to Ben Hearsum [:bhearsum] from comment #19) > Possibly of note, there were three nightlies yesterday: > https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-03-02-02- > mozilla-central-l10n/ > https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-10-01-09- > mozilla-central-l10n/ > https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56- > mozilla-central-l10n/ Ah probably grabbed random stuff then. Which was triggered by the fact that there aren't 32bit Italian builds on https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/ More dark to stab in.

I know bug 858953 is for updates only but I wonder if that also has an effect in how we rebuild for all of our locales.

It looks like some of the l10n builds are repackaged from debug builds and others are not. The de build is a debug build, while sv-SE is an opt build. For some reason the en-US build in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ is a debug build at the time that some l10n builders fetch the nightly.

The ASAN build is in that directory too, maybe it's grabbing that?

None of the following grabbed the ASAN build: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-03-02-02-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-it-bm62-build1-build648.txt.gz https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-10-01-09-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-it-bm62-build1-build662.txt.gz https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-it-bm58-build1-build495.txt.gz

(In reply to Hasse from comment #23) > It looks like some of the l10n builds are repackaged from debug builds and > others are not. The de build is a debug build, while sv-SE is an opt build. > > For some reason the en-US build in > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla- > central/ is a debug build at the time that some l10n builders fetch the > nightly. Can you point me at a specific repack that was based on a debug build?

The sv-SE build is repackaged from this build: 2013-07-16 03:26:54 URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [34679000/34679000] -> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1] And de is repackaged from this build: 2013-07-16 03:27:53 URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [33885535/33885535] -> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1] Notice the different file size. Could it be that a debug build is uploaded into this directory sometime after 2013-07-16 03:26:54? The build log for the de build (debug): https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-de-bm61-build1-build682.txt.gz The sv-SE build log (opt): https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-sv-SE-bm64-build1-build562.txt.gz

(In reply to Hasse from comment #27) > The sv-SE build is repackaged from this build: > 2013-07-16 03:26:54 > URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla- > central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [34679000/34679000] -> > "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1] This matches the filesize of https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 > And de is repackaged from this build: > 2013-07-16 03:27:53 > URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla- > central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [33885535/33885535] -> > "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1] And this is https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1373947976/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 And indeed, in that log I see: Running post-upload command: post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i 20130715211256 --revision 5976b9c673f8 --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated The "release-to-latest" part is causing the builds to upload to https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/, which is causing this problem. This is actually a really big issue, because it means that anyone who downloads "Nightly" in this window will end up with a static analysis build that never updates.

Interesting, Dave Hunt has seen the same some weeks ago when trying to install a nightly through http://nightly.mozilla.org/. He also got a debug build served! I think that's somewhat really problematic and would need investigation. We shouldn't offer debug builds to our nightly users.

Ben, which builds from which branches would that affect? Only nightly or also aurora?

--- /home/bhearsum/tmp/a.txt 2013-07-16 11:41:46.041622353 -0400 +++ /home/bhearsum/tmp/b.txt 2013-07-16 11:40:21.653621016 -0400 @@ -15288,7 +15288,7 @@ C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} - C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ionmonkey-linux64-dbg-st-an -b ionmonkey -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} + C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ionmonkey-linux64-dbg-st-an -b ionmonkey -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {} DisconnectStep {'alwaysRun': True, 'command': ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {} @@ -17439,7 +17439,7 @@ C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} - C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} + C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {} DisconnectStep {'alwaysRun': True, 'command': ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {} @@ -18532,7 +18532,7 @@ C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} - C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-oak', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir oak-linux64-dbg-st-an -b oak -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} + C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-oak', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir oak-linux64-dbg-st-an -b oak -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-oak', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-oak', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {} DisconnectStep {'alwaysRun': True, 'command': ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {} @@ -19426,7 +19426,7 @@ C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {} - C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ux', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ux-linux64-dbg-st-an -b ux -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} + C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ux', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ux-linux64-dbg-st-an -b ux -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">, 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1', 'UPLOAD_HOST': 'stage.mozilla.org', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa', 'UPLOAD_TO_TEMP': '1', 'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {} C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ux', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {} C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">, 'CCACHE_COMPRESS': '1', 'CCACHE_DIR': '/builds/ccache', 'CCACHE_UMASK': '002', 'DISPLAY': ':2', 'HG_SHARE_BASE_DIR': '/builds/hg-shared', 'IS_NIGHTLY': 'yes', 'LC_ALL': 'C', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_OBJDIR': 'obj-firefox', 'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an', 'MOZ_UPDATE_CHANNEL': 'nightly-ux', 'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin', 'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py', 'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org', 'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/', 'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa', 'SYMBOL_SERVER_USER': 'ffxbld', 'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {} DisconnectStep {'alwaysRun': True, 'command': ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}

Any branch that runs static analysis builds. Looks like that's most mozilla-central based branches. My fix will be in production today, though, so this shouldn't be an issue starting with the next set of nightlies.

I did some more grepping and found more debug-ey builds that are doing this, too.

Comment on attachment 776474 [details] [diff] [review] disable other uploads in production

Shouldn't be a problem starting with the next set of nightlies.

I have to thank everyone on this bug who was involved in getting this critical issue fixed. I never thought that with our mozmill test failure such an important underlying issue would open. It's great to see that team work!

While doing regression tests for a bug, which I found in Nightly 26.0a1, I also tested the following build: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.linux-x86_64.tar.bz2 As it turns out it is a debug build again. So the underlying issue here has not been fully fixed.

When you check the folder itself you will notice that this linux x86_64 build has been uploaded again two days later: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/ All builds are from Aug 19th but the linux 64bit one is from Aug 21st: The json file says: "buildid": "20130819030205", "cc": "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang", "cxx": "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang++", So it's really a static analysis build again.

I have checked all the September builds of Nightly on ftp.mo but wasn't able to find any other instance of this problem. Looks like it's still happening very rarely.

(In reply to Henrik Skupin (:whimboo) from comment #39) > I have checked all the September builds of Nightly on ftp.mo but wasn't able > to find any other instance of this problem. Looks like it's still happening > very rarely. In the future, can you please file new bugs for issues that recur after so long? This almost got lost because it was already assigned to me. We added a new static analysis build type in bug 910919, but I don't see it running anywhere but try, so I'm not convinced that's the issue. I look into this more later/tomorrow.

(In reply to Henrik Skupin (:whimboo) from comment #38) > When you check the folder itself you will notice that this linux x86_64 > build has been uploaded again two days later: > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03- > 02-05-mozilla-central/ > > All builds are from Aug 19th but the linux 64bit one is from Aug 21st: > > The json file says: > > "buildid": "20130819030205", > "cc": > "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang", > "cxx": > "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang++", > > So it's really a static analysis build again. An upload from a recent dbg-st-an build doesn't make it look like these builds are uploading to latest-mozilla-central: sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-linux64-dbg-st-an', '-p', 'firefox', '-i', '20130826171944', '--revision', 'c0d1baa50a64', '--release-to-tinderbox-dated-builds', '/tmp/tmp.PIhFcIablt/', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.tar.bz2', '/tmp/tmp.PIhFcIablt/linux-x86_64/xpi/firefox-26.0a1.en-US.langpack.xpi', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.txt', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.json', '/tmp/tmp.PIhFcIablt/host/bin/mar', '/tmp/tmp.PIhFcIablt/host/bin/mbsdiff', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.checksums'] http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1377562784/firefox-26.0a1.en-US.linux-x86_64.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1377562784/firefox-26.0a1.en-US.langpack.xpi

The log from the build you linked to tells a different story: sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-linux64-dbg-st-an', '-b', 'mozilla-central', '-p', 'firefox', '-i', '20130819030205', '--revision', 'c8c9bd74cc40', '--release-to-tinderbox-dated-builds', '--release-to-dated', '/tmp/tmp.LrSlcDIeqe/', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.tar.bz2', '/tmp/tmp.LrSlcDIeqe/linux-x86_64/xpi/firefox-26.0a1.en-US.langpack.xpi', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.txt', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.json', '/tmp/tmp.LrSlcDIeqe/host/bin/mar', '/tmp/tmp.LrSlcDIeqe/host/bin/mbsdiff', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.checksums'] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.linux-x86_64.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.langpack.xpi Note that "-b mozilla-central" and "--release-to-dated" are present here, but not in the more recent post_upload.py command I pasted in my previous comment. Still digging into this.

Looks like something happened around August 11th or 12th. This grepping shows that "release-to-dated" only started showing up on the 12th: -bash-4.1$ for i in `find . -iname "*-mozilla-central"`; do zgrep -L release-to-dated $i/mozilla-central-linux64-dbg-st-an-nightly*.gz; done ./2013-08-01-03-02-23-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm66-build1-build24.txt.gz ... ./2013-08-11-03-02-25-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm62-build1-build2.txt.gz -bash-4.1$ for i in `find . -iname "*-mozilla-central"`; do zgrep -l release-to-dated $i/mozilla-central-linux64-dbg-st-an-nightly*.gz; done ./2013-08-12-03-02-09-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm58-build1-build26.txt.gz And then around the 31st of August these files start appearing altogether: gzip: ./2013-08-31-03-02-24-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory gzip: ./2013-08-31-03-17-59-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory gzip: ./2013-08-31-03-14-59-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory ... gzip: ./2013-09-29-03-02-02-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory gzip: ./2013-09-30-03-02-05-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory gzip: ./2013-09-30-07-12-02-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory Given that, I don't think this is an issue anymore...I'd still like to get to the bottom of it though.

I'm still baffled by this, but I don't have any more time to investigate it right now. I really do think it's fixed again based on the logs from the past month, so I'm closing this. Please file a new bug and reference this one if we hit it again.