On bug 1300812 I made download_unpack() reliable against network issues. Unfortunately, there's a case when the fetch data is correct, however, unzipping turns out to be a bad zip file. I logged this in the code, however, it should be fatal() logging (the script stops) rather than exception() (We get a traceback but continue). When discussing this with gps we decided not to retry fetching the file again since we might have a bad upload. This will hopefully fix bug 1300345 forever.

Looking at: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=36399206#L604 I was surprised to see the job continues with a bad zip file. [1] [task 2016-09-23T19:44:42.643603Z] 19:44:42 INFO - retry: Calling unzip with args: (), kwargs: {'file_object': <addinfourl at 44969136 whose fp = <socket._fileobject object at 0x2ae37d0>>, 'verbose': False, 'extract_dirs': ['bin/*', 'certs/*', 'config/*', 'mach', 'marionette/*', 'modules/*', 'mozbase/*', 'tools/*', 'mochitest/*'], 'extract_to': '/home/worker/workspace/build/tests'}, attempt #1 [task 2016-09-23T19:46:24.106847Z] 19:46:24 ERROR - File is not a zip file [task 2016-09-23T19:46:24.108076Z] 19:46:24 ERROR - Traceback (most recent call last): [task 2016-09-23T19:46:24.108317Z] 19:46:24 ERROR - [task 2016-09-23T19:46:24.108703Z] 19:46:24 ERROR - File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 484, in unzip [task 2016-09-23T19:46:24.109072Z] 19:46:24 ERROR - with zipfile.ZipFile(compressed_file) as bundle: [task 2016-09-23T19:46:24.109418Z] 19:46:24 ERROR - [task 2016-09-23T19:46:24.109783Z] 19:46:24 ERROR - File "/usr/lib/python2.7/zipfile.py", line 714, in __init__ [task 2016-09-23T19:46:24.110154Z] 19:46:24 ERROR - self._GetContents() [task 2016-09-23T19:46:24.110513Z] 19:46:24 ERROR - [task 2016-09-23T19:46:24.110903Z] 19:46:24 ERROR - File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents [task 2016-09-23T19:46:24.111276Z] 19:46:24 ERROR - self._RealGetContents() [task 2016-09-23T19:46:24.111628Z] 19:46:24 ERROR - [task 2016-09-23T19:46:24.111991Z] 19:46:24 ERROR - File "/usr/lib/python2.7/zipfile.py", line 763, in _RealGetContents [task 2016-09-23T19:46:24.112334Z] 19:46:24 ERROR - raise BadZipfile, "File is not a zip file" [task 2016-09-23T19:46:24.112648Z] 19:46:24 ERROR - [task 2016-09-23T19:46:24.112978Z] 19:46:24 ERROR - BadZipfile: File is not a zip file [task 2016-09-23T19:46:24.432277Z] 19:46:24 INFO - Downloading and extracting to /home/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/* from https://queue.taskcluster.net/v1/task/ZKTjfv1LSGeClK3WZUP9YQ/artifacts/public/build/target.mochitest.tests.zip

Somebody pushed old code to inboud, thus, the spike of intermittents. <armenzg> Armen Zambrano Gasparnian can someone confirm this with me? 3:45 PM did someone push to try code almost two weeks old? 3:45 PM https://hg.mozilla.org/integration/mozilla-inbound/file/72c8ea085d7227204666b528d001f34c1cf4b113/testing/mozharness/mozharness/base/script.py#L483-501 3:46 PM I'm so confused 3:46 PM <philor> people push old revs to try constantly 3:46 PM <armenzg> Armen Zambrano Gasparnian philor: but this is inbound? https://hg.mozilla.org/integration/mozilla-inbound/rev/72c8ea085d7227204666b528d001f34c1cf4b113 3:46 PM did they use -f? 3:46 PM https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=72c8ea085d7227204666b528d001f34c1cf4b113&selectedJob=36399206 3:47 PM oh wow what a waste of my time 3:47 PM I've been trying to remove all these intermittent badzipfile issues 3:47 PM and I thought I got it all nailed down and it suddently spikes