Closed
Bug 1343831
Opened 8 years ago
Closed 8 years ago
[Alert] Cloudamqp - Queue total messages alarm: treeherder-prod log_parser 2017-03-02
Categories
(Tree Management :: Treeherder: Infrastructure, enhancement, P1)
Tree Management
Treeherder: Infrastructure
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Assigned: emorley)
References
(Depends on 1 open bug)
Details
CloudAMQP just alerted about a log parser backlog.
Checking New Relic shows a spike in transaction time:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1488455025&tw%5Bstart%5D=1488451425#id=5b224f746865725472616e73616374696f6e2f43656c6572792f6c6f672d706172736572222c22225d
...with some logs taking upto 200s to parse. eg:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1488455025&tw%5Bstart%5D=1488451425#id=5b224f746865725472616e73616374696f6e2f43656c6572792f6c6f672d706172736572222c22225d&app_trace_id=3a76d3f5-ff3b-11e6-a5be-f8bc124256a0_29280_30847
For that example, the log URL is:
https://queue.taskcluster.net/v1/task/MUxLsC8LQDWI6A6HkTBkzw/runs/0/artifacts/public/logs/live_backing.log
...and the the log is 316MB.
|
Assignee | |
Comment 1•8 years ago
|
||
Things that would helps:
1) Bug 1295997 - Add limit to size of log we will parse
2) Bug 1294544 - Treeherder should limit the task time for log parsing so we don't get a backlog on long logs
|
|
Assignee | |
Updated•8 years ago
|
|
|
Assignee | |
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•