https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/5c731e-9e600abe-3e28-11e6-bd66-b82a72d22a14 Traceback (most recent call last): File "/data/www/treeherder.allizom.org/venv/bin/celery", line 14, in <module> File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/__main__.py", line 30, in main File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__ File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 838, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 420, in _event_process_exit File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1260, in maintain_pool File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1252, in _maintain_pool File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _repopulate_pool File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 415, in _create_worker_process File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1068, in _create_worker_process File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/process.py", line 137, in start File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/forking.py", line 105, in __init__ File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 295, in run File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 367, in workloop File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.66.0.49/newrelic/hooks/application_celery.py", line 66, in wrapper File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__ File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/workers/task.py", line 28, in inner File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/task.py", line 676, in retry File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/utils/__init__.py", line 248, in maybe_reraise File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/workers/task.py", line 20, in inner File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/autoclassify/tasks.py", line 30, in detect_intermittents File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.66.0.49/newrelic/api/function_trace.py", line 110, in literal_wrapper File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/autoclassify/management/commands/detect_intermittents.py", line 27, in handle File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/autoclassify/management/commands/detect_intermittents.py", line 69, in add_new_intermittents File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/models.py", line 758, in set_classification File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/models.py", line 821, in elastic_search_insert File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/search.py", line 46, in save File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/elasticsearch_dsl/document.py", line 241, in save File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/elasticsearch_dsl/utils.py", line 382, in full_clean File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/elasticsearch_dsl/utils.py", line 376, in clean_fields elasticsearch_dsl.exceptions:ValidationException: {'subtest': [ValidationException('Value required for this field.',)]}

James, could you take a look at this? :-) I'll also add this to the retry blacklist in bug 1284289.

Pretty sure that the right thing here is just to make the subtest not required. That will mean rebuilding the data though (I think).

Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/df61b3a2af3f13a4b40c69bc72faad1218e7c871 Bug 1283253 - Don't require a subtest name in elasticsearch. (#1700)