https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb5-24168bec-2856-11e6-b947-b82a72d22a14 Triggered by: https://treeherder.allizom.org/api/project/mozilla-inbound/logslice/?job_id=26525415&start_line=4000&end_line=4300 Traceback (most recent call last): ... File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/api/logslice.py", line 75, in list File "/usr/lib64/python2.7/gzip.py", line 464, in readline File "/usr/lib64/python2.7/gzip.py", line 268, in read File "/usr/lib64/python2.7/gzip.py", line 303, in _read File "/usr/lib64/python2.7/gzip.py", line 197, in _read_gzip_header exceptions:IOError: Not a gzipped file Not sure if this is a regression from bug 1275903 or not?

The job in question: https://treeherder.allizom.org/logviewer.html#?job_id=26525415&repo=mozilla-inbound#L4151 Raw log: https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/public%2Flogs%2Flive_backing.log However this redirects: $ curl -I https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/pub lic%2Flogs%2Flive_backing.log HTTP/1.1 303 See Other ... Location: https://public-artifacts.taskcluster.net/ExYcsXIVTlO9lv8iaqan7Q/0/public/logs/live_backing.log

There's also this: https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb7-1566a282-284c-11e6-b947-b82a72d22a14 exceptions:OSError: [Errno 13] Permission denied: '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/tmp0Q46ih'

We need to hold off on the next prod deploy until this is resolved - loading log content doesn't work for any log at the moment on stage, eg: buildbot gzipped log: https://treeherder.allizom.org/logviewer.html#?job_id=26544334&repo=mozilla-inbound#L29211 http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464859494/mozilla-inbound-linux-bm72-build1-build1838.txt.gz TC log: https://treeherder.allizom.org/logviewer.html#?job_id=26550075&repo=mozilla-inbound#L4070 https://queue.taskcluster.net/v1/task/arRoybJ2TpS0Qfjx-APc5w/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Hmm, not sure how this could be happening, I thought we fixed this up pretty good. Regardless, I'll take this one.

Comment on attachment 8759261 [details] [treeherder] wlach:1277499 > mozilla:master The implementation could be more elegant, but I think this is o.k. Bonus: unit tests!

(In reply to Ed Morley [:emorley] from comment #2) > There's also this: > > https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/ > 553eb7-1566a282-284c-11e6-b947-b82a72d22a14 > > exceptions:OSError: [Errno 13] Permission denied: > '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/ > log_cache/tmp0Q46ih' This turned out to because the '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/' directory was owned by newrelic, instead of treeherder. I chown'ed it back. Do you have any idea how this could have happened Ed?

(In reply to William Lachance (:wlach) from comment #7) > This turned out to because the > '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/ > log_cache/' directory was owned by newrelic, instead of treeherder. I > chown'ed it back. Do you have any idea how this could have happened Ed? Should add this was only on the *second* webhead (th-stage-webapp-2). The other webheads had the correct permissions set (which was this API worked intermittently)

Ah the 2nd stage webhead was the replaced one (bug 1260707).

Pull request has landed in master: https://github.com/mozilla/treeherder/commit/a48c9ea1e8a4f6d1b0199d6403f8533b7aa933a4

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/0c1ccce689b84a2f550387b6f37a5d6ff8c25371 Bug 1277499 - Don't test same functionality multiple times in logslice tests https://github.com/mozilla/treeherder/commit/89d6e6e84c0b5fe83c259f8c1ccce3a03350a414 Bug 1277499 - Handle non-gzipped files in logslice API