https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4f4e53-4592e47c-394e-11e6-bd66-b82a72d22a14 Traceback (most recent call last): File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__ File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 838, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 420, in _event_process_exit File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1260, in maintain_pool File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1252, in _maintain_pool File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _repopulate_pool File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 415, in _create_worker_process File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1068, in _create_worker_process File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 137, in start File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/forking.py", line 105, in __init__ File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 295, in run File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 367, in workloop File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.66.0.49/newrelic/hooks/application_celery.py", line 66, in wrapper File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__ File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/workers/taskset.py", line 24, in inner File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 36, in inner File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 126, in store_failure_lines File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/task.py", line 676, in retry File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/utils/__init__.py", line 248, in maybe_reraise File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 120, in store_failure_lines File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/failureline.py", line 52, in store_failure_lines File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/failureline.py", line 71, in create File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 392, in bulk_create File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 937, in _batched_insert File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__ File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.66.0.49/newrelic/hooks/database_dbapi2.py", line 22, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler django.db.utils:IntegrityError: (1062, "Duplicate entry '30041743-3734' for key 'failure_line_job_log_id_2bb6c3dc51971773_uniq'") https://github.com/mozilla/treeherder/blob/5d4b9ecb5568b41fc2bf41a261fb28305611adff/treeherder/log_parser/failureline.py#L68-L72

In addition to hitting this exception, we're also retrying it. James, would you mind adding store-failure-lines to the tasks using the retry wrapper to prevent this? :-) eg: https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4eda72-f24618c8-3951-11e6-bd66-b82a72d22a14

Comment on attachment 8764750 [details] [treeherder] mozilla:store_failure_lines_integrity > mozilla:master Awesome, thank you :-)

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/005c06ebdf6a30dfcd3300f97e1f28664bb07a36 Bug 1281809 - Handle IntegrityError when storing failure lines. (#1615) If it happens that the failure lines for a log are partially stored, retry storing only the missing lines. In the case that this doesn't fix the problem, don't try to retry at the celery level.

Don't forget to update the bugs :-)