Closed
Bug 1127940
Opened 10 years ago
Closed 10 years ago
submit_bug_comment "ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})" due to read host DB race
Categories
(Tree Management :: Treeherder, defect, P2)
Tree Management
Treeherder
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Assigned: emorley)
References
Details
Attachments
(1 file)
Started after the prod deploy of the read DB host fix.
This is likely just another case where we should use the master DB instead (and the read host fix just exposed the existing race condition):
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/3000207132?original_error_id=3000830047
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 44, in submit_bug_comment submit_bug_comment.retry(exc=e, countdown=(1 + submit_bug_comment.request.retries) * 60)
File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 680, in retry raise ret
Retry: Retry in 60s: ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})
I believe it comes from:
https://github.com/mozilla/treeherder-service/blob/cae85ae8c4cc5cb9fdecd86ffbdcf2367782b4d6/treeherder/etl/tbpl.py#L114
bug_job_map = jm.get_bug_job_map_detail(self.job_id, self.bug_id)
Assignee | ||
Comment 1•10 years ago
|
||
Attachment #8557245 -
Flags: review?(mdoglio)
Assignee | ||
Updated•10 years ago
|
Summary: submit_bug_comment "ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})" likely due to read DB use → submit_bug_comment "ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})" due to read host DB race
Updated•10 years ago
|
Attachment #8557245 -
Flags: review?(mdoglio) → review+
Comment 2•10 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder-service
https://github.com/mozilla/treeherder-service/commit/443706689202db51b4023ee43257993394f867b9
Bug 1127940 - Pass 'who' directly to BugzillaBugRequest to avoid DB race
We did not previously pass the username of the person who made a failure
classification to BugzillaBugRequest. As a result, BugzillaBugRequest
has to fetch the bug mapping again to find it out. After the fix for our
master-read DB setup, this fetch of the bug mapping was being performed
against the read host, which occasionally did not yet have the requested
mapping (due to replication delay) causing an ObjectNotFoundException.
The 'get_bug_job_map_detail' query could have been switched to use the
master host to avoid the race, however the more efficient fix is just to
pass the username to BugzillaBugRequest directly, like we already do for
OrangeFactorBugRequest, avoiding the query entirely.
https://github.com/mozilla/treeherder-service/commit/e61738b5a9db338b10e296cd6474786a7c93d2e2
Bug 1127940 - Remove the now unused get_bug_job_map_detail()
BugzillaBugRequest was the only consumer, so we can now remove it.
Assignee | ||
Updated•10 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•