Started after the prod deploy of the read DB host fix. This is likely just another case where we should use the master DB instead (and the read host fix just exposed the existing race condition): https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/3000207132?original_error_id=3000830047 Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 44, in submit_bug_comment submit_bug_comment.retry(exc=e, countdown=(1 + submit_bug_comment.request.retries) * 60) File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 680, in retry raise ret Retry: Retry in 60s: ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034}) I believe it comes from: https://github.com/mozilla/treeherder-service/blob/cae85ae8c4cc5cb9fdecd86ffbdcf2367782b4d6/treeherder/etl/tbpl.py#L114 bug_job_map = jm.get_bug_job_map_detail(self.job_id, self.bug_id)

Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/443706689202db51b4023ee43257993394f867b9 Bug 1127940 - Pass 'who' directly to BugzillaBugRequest to avoid DB race We did not previously pass the username of the person who made a failure classification to BugzillaBugRequest. As a result, BugzillaBugRequest has to fetch the bug mapping again to find it out. After the fix for our master-read DB setup, this fetch of the bug mapping was being performed against the read host, which occasionally did not yet have the requested mapping (due to replication delay) causing an ObjectNotFoundException. The 'get_bug_job_map_detail' query could have been switched to use the master host to avoid the race, however the more efficient fix is just to pass the username to BugzillaBugRequest directly, like we already do for OrangeFactorBugRequest, avoiding the query entirely. https://github.com/mozilla/treeherder-service/commit/e61738b5a9db338b10e296cd6474786a7c93d2e2 Bug 1127940 - Remove the now unused get_bug_job_map_detail() BugzillaBugRequest was the only consumer, so we can now remove it.