Excerpt from production-master:~cltbld/update.log: Starting run at Wed Mar 24 19:50:01 PDT 2010 ./mozilla-central-macosx-opt-unittest-mochitests-2 2181 1/79 0.00% complete ETA in 0 seconds ... ./mozilla-1.9.2-linux-unittest-reftest 134 7/7 7.10% complete ETA in 3253 seconds ./mozilla-central-win32-nightly 398 1/6 7.11% complete ETA in 3253 seconds Traceback (most recent call last): File "/tools/buildbotcustom/buildbotcustom/bin/update_from_files.py", line 300, in <module> updated = updateFromFiles(session, options.master, options.name, builders, last_time, options.times) File "/tools/buildbotcustom/buildbotcustom/bin/update_from_files.py", line 220, in updateFromFiles db_build = model.Build.fromBBBuild(session, build, builder_name, master.id) File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 536, in fromBBBuild b.updateFromBBBuild(session, build) File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 471, in updateFromBBBuild self.properties = Property.fromBBProperties(session, build.getProperties()) File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 104, in fromBBProperties if props[prop.name] == prop.value and props.getPropertySource(prop.name) == prop.source: File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/properties.py", line 42, in __getitem__ rv = self.properties[name][0] KeyError: u'completeSnippetFilename' which is blocking a lot of builds being dumped into the db. We do seems to get some data, but the last compile job from pm01 was mozilla-1.9.2-linux64-xulrunner at 2010-03-24 17:32:33 PDT.

Backed off the master_cleanup.py job from -t4 to -t6 to give us a little time to deal with this without losing data.

catlee: any insight here? Is there one build missing a key and it's preventing all db updates from happening?

I'm not really sure. If it was just one build, I would have expected it to have been purged a while ago, and then we'd get back up and running.

Perhaps not purged yet because I changed that to 6 days ? Also noticed that pm02 is getting backed up reprocessing the same builds each time it runs, about 20 mins to run and more than running at once. I think because the timestamp isn't updated when we error on the bogus jobs. Didn't change the the purge time there. Last time we hit this there was something bogus in the db ?

So the problem stems from the fact that mysql does case insensitive matching on in the query, so when we search for 'completeSnippetFilename', we get back a property from the db called 'completesnippetFilename', which doesn't exist in the build.

Comment on attachment 435097 [details] [diff] [review] Check property name before accessing it changeset: 671:4acda6a63f0a