Tests fail in Ci like 20% of the time with:

______________ TestIntegrationSuperSearchFields.test_test_mapping ______________
Traceback (most recent call last):
  File "/app/socorro/external/es/super_search_fields.py", line 410, in test_mapping
    crashes_sample = es_connection.search(
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/__init__.py", line 530, in search
    _, data = self.transport.perform_request('GET', _make_path(index,
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 78, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(503, 'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/socorro/unittest/external/es/test_super_search_fields.py", line 253, in test_test_mapping
    assert self.api.test_mapping(mapping) is None
  File "/app/socorro/external/es/super_search_fields.py", line 422, in test_mapping
    raise BadArgumentError(
socorro.lib.BadArgumentError: Indexing existing data in Elasticsearch failed with the new mapping. Error is: TransportError(503, 'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]')

Example:

https://app.circleci.com/pipelines/github/mozilla-services/socorro/1504/workflows/d1069fdc-40d4-41f5-8c91-9df4a63853be/jobs/7097

This is irksome. We should fix it.

Grabbing this to look into soon. Every time it fails, I have to rerun things which makes PR and landing things take longer.

willkg merged PR #6031: "bug 1759080: attempt to reduce intermittent ci fail" in f7d70f3.

I'll keep this open for a week. If the problem happens again, then this either didn't fix it at all or didn't fix it enough.

I haven't seen this pop up in the last week, so I think we're good here.