RelMan pointed that we've started to see funsize jobs failing with this lately. Subsequent rereuns eventually go green like in this case.

We've seen occurrences of the same issue in bug 1411358 but that seems too broad.

Jcristau also suggested that these timeouts should be less permisive. A normal task usually takes <10 min yet the timeout is set to 3600. This should help increase the end-to-end time we wait to see the jobs green. Something similar to what we did in bug 1531072.

Interesting, something inside make_incremental_update.sh is breaking, but we're not getting the logs. I will update the timeout as suggested and try to improve the reporting.

https://hg.mozilla.org/mozilla-central/rev/817014bcd372

Turns out 600s isn't enough for Linux64 Asan partial genertion, where the job may take up to 36 minutes. Windows asan is ~20 minutes.

jlund pointed out we usually specify parameters like a timeout elsewhere, like the kind config. My fix was only a quick followup to resolve the bustage in the next nightly.

https://hg.mozilla.org/mozilla-central/rev/1cc8b60d8a6b

https://tools.taskcluster.net/groups/Jj8j2VzfQ-WCPqsqZHPGbA/tasks/MOuYO-Y2QyyvH0ft4ry44A/runs/0/logs/public%2Flogs%2Flive.log had an error in today's nightly, and it looks to have been diffing libxul.so:

2019-03-05 12:15:29,381 - WARNING - target.partial-1.mar: diffing "libxul.so"
2019-03-05 12:24:15,463 - WARNING - target.partial-1.mar: patch "libxul.so.patch" "libxul.so"

Each of the four partials took nine minutes to process that file. Caching seems disabled.

https://hg.mozilla.org/mozilla-central/rev/a91196fe2eb3#l2.64 introduced a default level, which was a string, not an int, so the caching has been broken for a while.

Backed out changeset ce3dfcdb5861 (Bug 1532236) for linting opt failure in partials.py CLOSED TREE

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=ce3dfcdb58615574d7f7d19e863877311d933caf&selectedJob=231926130

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231926130&repo=autoland&lineNumber=295

Backout: https://hg.mozilla.org/integration/autoland/rev/9fca85ee3084599c0119d126840eb2062f65003d

https://hg.mozilla.org/mozilla-central/rev/bc4e03f4ea20

https://hg.mozilla.org/mozilla-central/rev/f6d891b25f43

Found in postmortem triaging.
sfraser: can we close this?

There are still a few errors to sort out with this, sorry.

Reinsert awscli for partials, which is needed for caching. Also update packages and fix the metrics recording

Backed out for lint failure at mbsdiff_hook.sh

push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=7b2ae2ea04957e8f18250ffc62563f301e5335f8&selectedJob=233055627

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=233055627&repo=autoland&lineNumber=280

Backout: https://hg.mozilla.org/integration/autoland/rev/8076b05b8631a55cc9a9922821e975c4ea4434af

https://hg.mozilla.org/mozilla-central/rev/e108f9ad99a5

https://trello.com/c/roEzdOJx/172-partial-hu-linux-task-failing

looks like this is failing at least some of the time on new 67 beta (devedition)

aws-cli isn't found. Do we need to uplift given this missed the merge a few hours earlier?

I think we have enabled s3 caching for releases between 66 and 67, where previously we had no caching at all. See the scopes on the last 66 beta and deved 67.0b1. I don't know if that was deliberate or not. We did have local caching for releases until bug 1501113.

I think it'll either be a case of increasing the task timeout (even just to 20 minutes) or adding the caching in, as some of the files now take a lot longer to diff than they did at this point last year.

I'd vote for an uplift.