We've been getting this traceback every morning when `index_latest_mkt_stats` (https://github.com/mozilla/zamboni/blob/master/scripts/crontab/crontab.tpl#L70) runs: Task mkt.stats.tasks.index_finance_total_by_currency with id 7f0e07cf-2e05-4398-a4b9-006f53c6ad28 raised exception: 'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[RCHLDB4pQTOAKAPlFkdhDA][addons][2]: RemoteTransportException[[Mekano][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[uWWAHmxVR6mCeaFTdEqezw][addons][4]: RemoteTransportException[[Sawyer, Sam][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }\',)' Task was called with args: [[368172L, 368160L, 368158L, 368156L, 368152L, 368149L, 368147L, 368136L, 368127L, 368111L, 368090L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_total_by_currency', 'task_id': '7f0e07cf-2e05-4398-a4b9-006f53c6ad28', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}. The contents of the full traceback was: Traceback (most recent call last): File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task R = retval = task(*args, **kwargs) File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped return fun(*args, **kw) File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 98, in index_finance_total_by_currency index_finance_total_by_currency.retry(args=[addons], exc=exc) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry maybe_reraise() File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 94, in index_finance_total_by_currency if not already_indexed(Contribution, data): File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 278, in already_indexed return list(model.search().filter(**data).values_dict(data.keys()[0])) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 208, in __iter__ return iter(self._do_search()) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 184, in _do_search hits = self.raw() File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 199, in raw hits = es.search(qs, self.index, self.type._meta.db_table) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search return self._query_call("_search", body, indexes, doc_types, **query_params) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call response = self._send_request('GET', path, body, querystring_args) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request raise_if_error(response.status, decoded) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error raise excClass(msg, status, result) SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[RCHLDB4pQTOAKAPlFkdhDA][addons][2]: RemoteTransportException[[Mekano][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[uWWAHmxVR6mCeaFTdEqezw][addons][4]: RemoteTransportException[[Sawyer, Sam][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }

Still happening. Just got 75 tracebacks for this: Date: July 12, 2012 8:44:09 PM PDT Task mkt.stats.tasks.index_finance_daily with id f26bfd69-a4dd-4754-9841-1b673c297e49 raised exception: 'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[6g7X35lVRZu74QMX30-0Bw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][2]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][4]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][3]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[6g7X35lVRZu74QMX30-0Bw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)' Task was called with args: [[5270587L, 5270585L, 5270583L, 5270581L, 5270579L, 5270577L, 5270575L, 5270573L, 5270571L, 5270569L, 5270567L, 5270565L, 5270563L, 5270561L, 5270559L, 5270557L, 5270555L, 5270553L, 5270551L, 5270549L, 5270547L, 5270545L, 5270543L, 5270541L, 5270539L, 5270537L, 5270535L, 5270533L, 5270531L, 5270529L, 5270527L, 5270525L, 5270523L, 5270521L, 5270519L, 5270517L, 5270515L, 5270513L, 5270511L, 5270509L, 5270507L, 5270505L, 5270503L, 5270501L, 5270499L, 5270497L, 5270495L, 5270493L, 5270491L, 5270489L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': 'f26bfd69-a4dd-4754-9841-1b673c297e49', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}. The contents of the full traceback was: Traceback (most recent call last): File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task R = retval = task(*args, **kwargs) File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped return fun(*args, **kw) File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 142, in index_finance_daily index_finance_daily.retry(args=[ids], exc=exc) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry maybe_reraise() File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 137, in index_finance_daily if not already_indexed(Contribution, data): File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 356, in already_indexed return list(model.search().filter(**data).values_dict(data.keys()[0])) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 208, in __iter__ return iter(self._do_search()) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 184, in _do_search hits = self.raw() File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 199, in raw hits = es.search(qs, self.index, self.type._meta.db_table) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search return self._query_call("_search", body, indexes, doc_types, **query_params) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call response = self._send_request('GET', path, body, querystring_args) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request raise_if_error(response.status, decoded) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error raise excClass(msg, status, result) SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[6g7X35lVRZu74QMX30-0Bw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][2]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][4]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][3]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[6g7X35lVRZu74QMX30-0Bw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

Got 165 tracebacks in two days. We should fix this.

92 celery tracebacks at 8:44 PM today: Task mkt.stats.tasks.index_finance_daily with id 8a70d2f3-a305-4fc9-ac16-e79cb3ffeb11 raised exception: 'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[3nuG5ezdR7iEbeeJQ2fDeQ][addons][4]: RemoteTransportException[[Goldbug][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][2]: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[ejqupHrYTMm7PbAUAfr-ZQ][addons][0]: RemoteTransportException[[Blitzschlag, Baron Von][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)' Task was called with args: [[5259873L, 5259861L, 5259857L, 5259853L, 5259845L, 5259843L, 5259827L, 5259815L, 5259795L, 5259779L, 5259771L, 5259741L, 5259703L, 5259697L, 5259671L, 5259647L, 5259573L, 5259559L, 5259503L, 5259443L, 5259437L, 5259429L, 5259397L, 5259335L, 5259331L, 5259297L, 5259281L, 5259279L, 5259273L, 5258777L, 5258769L, 5258763L, 5258747L, 5258721L, 5258717L, 5258713L, 5258683L, 5258679L, 5258641L, 5258621L, 5258581L, 5258573L, 5258561L, 5258539L, 5258527L, 5258517L, 5258505L, 5258491L, 5258451L, 5258447L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': '8a70d2f3-a305-4fc9-ac16-e79cb3ffeb11', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}. The contents of the full traceback was: Traceback (most recent call last): File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task R = retval = task(*args, **kwargs) File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped return fun(*args, **kw) File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 143, in index_finance_daily index_finance_daily.retry(args=[ids], exc=exc) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry maybe_reraise() File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 138, in index_finance_daily if not already_indexed(Contribution, data): File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 357, in already_indexed return list(model.search().filter(**data).values_dict(data.keys()[0])) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 207, in __iter__ return iter(self._do_search()) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 183, in _do_search hits = self.raw() File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 198, in raw hits = es.search(qs, self.index, self.type._meta.db_table) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search return self._query_call("_search", body, indexes, doc_types, **query_params) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call response = self._send_request('GET', path, body, querystring_args) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request raise_if_error(response.status, decoded) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error raise excClass(msg, status, result) SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[3nuG5ezdR7iEbeeJQ2fDeQ][addons][4]: RemoteTransportException[[Goldbug][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][2]: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[ejqupHrYTMm7PbAUAfr-ZQ][addons][0]: RemoteTransportException[[Blitzschlag, Baron Von][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

let's try this https://github.com/mozilla/zamboni/commit/67f7629d3b70d3406c6796ef5c0be97c5eb26925

104 tracebacks again: ask mkt.stats.tasks.index_finance_daily with id 4e06ed1e-de81-426e-92b3-5a13192c9024 raised exception: 'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[-ik3Ji4eTBabTGX4GdVVmg][addons][4]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][2]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][0]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][1]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][3]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)' Task was called with args: [[5287963L, 5287961L, 5287959L, 5287957L, 5287955L, 5287953L, 5287951L, 5287949L, 5287947L, 5287945L, 5287943L, 5287941L, 5287939L, 5287937L, 5287935L, 5287933L, 5287931L, 5287929L, 5287927L, 5287925L, 5287923L, 5287921L, 5287919L, 5287917L, 5287915L, 5287913L, 5287911L, 5287909L, 5287907L, 5287905L, 5287903L, 5287901L, 5287899L, 5287897L, 5287895L, 5287893L, 5287891L, 5287889L, 5287887L, 5287885L, 5287883L, 5287881L, 5287879L, 5287877L, 5287875L, 5287873L, 5287871L, 5287869L, 5287867L, 5287865L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': '4e06ed1e-de81-426e-92b3-5a13192c9024', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}. The contents of the full traceback was: Traceback (most recent call last): File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task R = retval = task(*args, **kwargs) File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped return fun(*args, **kw) File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 143, in index_finance_daily index_finance_daily.retry(args=[ids], exc=exc) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry maybe_reraise() File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 138, in index_finance_daily if not already_indexed(Contribution, data): File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 357, in already_indexed return list(model.search().filter(**data).values_dict(data.keys()[0])) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 207, in __iter__ return iter(self._do_search()) File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 183, in _do_search hits = self.raw() File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 198, in raw hits = es.search(qs, self.index, self.type._meta.db_table) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search return self._query_call("_search", body, indexes, doc_types, **query_params) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call response = self._send_request('GET', path, body, querystring_args) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request raise_if_error(response.status, decoded) File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error raise excClass(msg, status, result) SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[-ik3Ji4eTBabTGX4GdVVmg][addons][4]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][2]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][0]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][1]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][3]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

CC'ing ES folks to help take a look. We're getting hammered a lot on -dev/production with this.

Could you comment out this job in the crontab if we can't figure this out.

Last time we got this was 2 days ago - maybe before the new ES cluster. Has this happened since?

I also get like 30 failures on mkt/stats/tests/test_views.py. Kevin, can you please take a look at this? If you need help running some queries on -dev, holler at your boi.

k

https://github.com/mozilla/zamboni/commit/e554a47985c97d79b76acc0d987324255a9c2ba1 Took another stab at it. I don't really have a way of debugging this traceback.

feel free to add some extra logging if you think it will help you understand the problem when it happens again in prod

closing it for now until it's reverified as still broken.

I didn't see this error last night. So I think we're good. I'll keep it open for a few more days and verify.