+++ This bug was initially created as a clone of Bug #1204281 +++ There is considerable history in bug 1204281; I have cloned and closed that bug because it was simply too big -- too many comments, too much ancient history. I expect these failures to continue. Task timeouts are the last resort whenever a task is hung or otherwise runs for too long. The immediate goal is to address timeouts which recur in the same task frequently.

Bug 1339568 is the single biggest cause of these failures.

Waiting for updates on bug 1339568, which depends on bug 1357466, which had no progress in the last months.

Still waiting for updates on bug 1339568.

Bug 1339568 remains the biggest contributer of failures reported in this bug.

Update: there are 65 failures for the last 7 days

Some tests have been disabled in bug 1339568, reducing mochitest-media-e10s-2 shutdown hangs considerably. I expect to still see some intermittent failures reported here from other causes.

In the last week there have been 31 total failures. The failures occur on debug and opt build types and on the following platforms: - android-4-3-armv7-api16: 14 - android-api-16-gradle: 5 - Android 4.2 x86: 2 - Linux x64: 2 - linux64-stylo-disabled: 2 - android-4-0-armv7-api16: 1 - Linux: 1 - linux32-stylo-disabled: 1 - linux64-ccov: 1 - linux64-qr: 1 - windows-mingw32-32: 1 Here is a recent log file: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=150857059&lineNumber=9563 And a relevant snippet from it: [task 2017-12-08T21:22:31.523Z] 21:22:31 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/xpcom/build' 9560 [task 2017-12-08T21:22:31.524Z] 21:22:31 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/xpcom/build' 9561 [task 2017-12-08T21:22:31.557Z] 21:22:31 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/xpcom/build' 9562 9563 [taskcluster:error] Task timeout after 7200 seconds. Force killing container. 9564 [taskcluster 2017-12-08 21:22:56.375Z] === Task Finished === 9565 [taskcluster 2017-12-08 21:22:56.383Z] Unsuccessful task run with exit code: -1 completed in 7596.094 seconds :gbrown, any updates on this?

The latest increase in frequency is due to failing Android builds. I cannot tell if that is an on-going problem, or a temporary glitch but will continue to monitor.

Most recent failures have been Android tests, for a variety of reasons. Some show intermittent download failures/retries -- hopefully a temporary condition. I don't see enough consistency to warrant further investigation right now.

Last week there were a variety of tasks failing here. The biggest group is linux32/debug jsreftests -- bug 1430668.

Some recent android opt mochitest failures here - bug 1433560 should help. Also some test-verify failures - bug 1431125.

Recent failures seem to be mostly bug 1321605 -- probably nothing more I can do there. There may also be some fallout from android mochitest run-by-manifest -- keeping an eye on that.

There are 83 failures in the past 7 days. Platforms: most of them on android-4-3-armv7-api16 debug/ opt and linux64-jsdcov opt; and a few occurrences on Linux/Linux x64 debug, linux32-stylo-disabled debug, linux64-jsdcov opt, linux64-qr debug and windows10-64-ccov debug. Recent failure log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=165586428&lineNumber=5170877

(In reply to OrangeFactor Robot from comment #37) > Platform breakdown: > * linux64-jsdcov: 91 > * android-4-3-armv7-api16: 11 The linux64-jsdcov problem is apparent here -- working on that in bug 1442823.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/c291e7dfe010 Increase taskcluster max-run-time for Android mochitest-chrome and Android/opt mochitest; r=me,a=test-only

There are 105 failures in the past week. Platforms: - Android 4.2 x86 opt - android-4-0-armv7-api16 opt and debug - android-4-3-armv7-api16 opt and debug - android-5-0-aarch64 opt - Linux opt and debug (many occurrences) - Linux x64 opt, debug and asan (many occurrences) - linux32-stylo-disabled debug - linux64-ccov opt - linux64-jsdcov opt (many occurrences) - linux64-noopt debug - linux64-qr opt and debug - linux64-stylo-disabled opt and debug - windows10-64-ccov debug Recent log failure: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=168753923&lineNumber=1537 Relevant part of the log: [task 2018-03-17T23:01:53.204Z] 23:01:53 INFO - Successfully installed browsermob-proxy-0.6.0 firefox-puppeteer-52.1.0 marionette-driver-2.5.0 marionette-harness-4.3.0 wptserve-1.4.0 [task 2018-03-17T23:01:53.253Z] 23:01:53 INFO - Return code: 0 [task 2018-03-17T23:01:53.255Z] 23:01:53 INFO - Installing None into virtualenv /builds/worker/workspace/build/venv [task 2018-03-17T23:01:53.256Z] 23:01:53 INFO - error resolving pypi.pvt.build.mozilla.org (ignoring): [taskcluster:error] Task timeout after 5400 seconds. Force killing container. [task 2018-03-17T23:01:53.260Z] 23:01:53 INFO - retry: Calling run_command with args: [['/builds/worker/workspace/build/venv/bin/pip', 'install', '--timeout', '120', '-r', '/builds/worker/workspace/build/tests/config/marionette_requirements.txt', '--no-index', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x7f7cc33d4490>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f7cc3490300>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x158aa80>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f7cc488fbb8>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f7cc3493188>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f7cc48a7b28>, 'level': 'critical'}], 'cwd': '/builds/worker/workspace/build/tests/config', 'env': {'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.2', 'TASKCLUSTER_INSTANCE_TYPE': 'm3.large', 'TASKCLUSTER_WORKER_TYPE': 'gecko-t-linux-large', 'MOZ_AUTOMATION': '1', 'MOZ_SOURCE_CHANGESET': 'efce78e62b6dac195e7cb4898684da54155d1661', 'MOCHITEST_FLAVOR': 'plain', 'LOGNAME': 'worker', 'USER': 'worker', 'HOME': '/builds/worker', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builds/worker/bin', 'DISPLAY': ':0', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'MOZ_NODE_PATH': '/usr/local/bin/node', 'RUN_ID': '0', 'HG_STORE_PATH': '/builds/worker/checkouts/hg-store', 'MOZILLA_BUILD_URL': 'https://queue.taskcluster.net/v1/task/cjvNZxdBSKKfmKHtz6MnGQ/artifacts/public/build/target.tar.bz2', 'TASKCLUSTER_PORT': 'tcp://172.17.0.2:80', 'MOZHARNESS_SCRIPT': 'desktop_unittest.py', 'TASKCLUSTER_NAME': '/laughing_khorana/taskcluster', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/mozilla-central', 'SCCACHE_DISABLE': '1', 'TASKCLUSTER_PORT_80_TCP_PORT': '80', 'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/mozilla-central', 'GECKO_HEAD_REV': 'efce78e62b6dac195e7cb4898684da54155d1661', 'MOZHARNESS_URL': 'https://queue.taskcluster.net/v1/task/cjvNZxdBSKKfmKHtz6MnGQ/artifacts/public/build/mozharness.zip', 'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified', 'LC_ALL': 'en_US.UTF-8', 'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp', 'TASK_ID': 'OgTQKS-PS-udVgG1SWL7lA', 'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.2:80', 'OLDPWD': '/builds/worker/workspace', 'HOSTNAME': 'taskcluster-worker', 'SHLVL': '1', 'PWD': '/builds/worker/workspace', 'NEED_PULSEAUDIO': 'true', 'ENABLE_E10S': 'false', 'NEED_WINDOW_MANAGER': 'true', 'MOZHARNESS_CONFIG': 'unittests/linux_unittest.py remove_executables.py', 'TASKCLUSTER_WORKER_GROUP': 'us-west-1', 'TASKCLUSTER_PUBLIC_IP': '54.183.87.60'}}, attempt #1 [task 2018-03-17T23:01:53.261Z] 23:01:53 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/pip', 'install', '--timeout', '120', '-r', '/builds/worker/workspace/build/tests/config/marionette_requirements.txt', '--no-index', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org'] in /builds/worker/workspace/build/tests/config [task 2018-03-17T23:01:53.261Z] 23:01:53 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/pip install --timeout 120 -r /builds/worker/workspace/build/tests/config/marionette_requirements.txt --no-index --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host pypi.pub.build.mozilla.org [task 2018-03-17T23:01:53.261Z] 23:01:53 INFO - Using env: {'DISPLAY': ':0',

Some failures are from 'try -p all -u all' runs which have triggered jsdcov/dev-tools failures -- carry over from bug 1442823.

Many of the recent logs on linux and android show pypi failures, similar to bug 1445580.

Over the last 7 days there are 81 failures on this bug. These happen on Android 4.2 x86, android-4-0-armv7-api16, android-4-3-armv7-api16, android-5-0-aarch64, Linux, Linux x64, linux64-qr, Here is a recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=170655917&lineNumber=12538 [taskcluster:error] Task timeout after 7200 seconds. Force killing container.

See comment 0 and bug dependencies. Investigation and mitigation is on-going.

Recent failures on Android and Linux are frequent and irregular: Not associated with particular tests or test tasks. Some are related to network failures or delays during 'pip install': [task 2018-04-03T18:13:55.354Z] 18:13:55 INFO - Collecting jsonschema==2.5.1 [task 2018-04-03T18:15:55.610Z] 18:15:55 INFO - Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fba9e73e3d0>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub [task 2018-04-03T18:17:56.643Z] 18:17:56 INFO - Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fba9e73e550>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub [task 2018-04-03T18:19:58.106Z] 18:19:58 INFO - Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fba9e73e6d0>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub [task 2018-04-03T18:22:00.205Z] 18:22:00 INFO - Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fba9e73e850>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub [task 2018-04-03T18:24:04.445Z] 18:24:04 INFO - Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7fba9e73e9d0>, 'Connection to pypi.pub.build.mozilla.org timed out. (connect timeout=120.0)')': /pub [task 2018-04-03T18:26:04.821Z] 18:26:04 INFO - Could not find a version that satisfies the requirement jsonschema==2.5.1 (from versions: ) [task 2018-04-03T18:26:04.822Z] 18:26:04 INFO - No matching distribution found for jsonschema==2.5.1

Updates: There have been 141 failures in the last 7 days. Summary: Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container. / [taskcluster:error] Task timeout after 5400 seconds. Force killing container. / [taskcluster:error] Task timeout after 7200 seconds. Force killing container. The most affected build types are debug and opt, but there are also over 10 failures on asan & pgo. Failures per platforms: Linux: 32 Linux x64: 32 android-4-3-armv7-api16: 31 linux64-qr: 19 linux64-ccov: 18 android-4-0-armv7-api16: 5 android-5-0-aarch64: 2 linux32-nightly: 1 Android 4.2 x86: 1

Bug 1451432 remains my biggest current concern here.

Over the last 7 days there are 99 failures present on this bug. These happen on Android 4.2 x86, android-4-3-armv7-api16, android-5-0-aarch64, Linux, Linux x64, linux64-ccov, linux64-qr, Windows 7. Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=176126037&lineNumber=25270 [task 2018-04-28T15:11:00.833Z] 15:11:00 INFO - TEST-START | /referrer-policy/unsafe-url/meta-referrer/same-origin/http-https/iframe-tag/generic.swap-origin-redirect.http.html [task 2018-04-28T15:11:00.844Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 28 (0x7fa601cc9800) [pid = 15232] [serial = 17] [outer = (nil)] [url = http://web-platform.test:8000/referrer-policy/generic/subresource/document.py?redirection=no-redirect&cache_destroyer=1524928253055] [task 2018-04-28T15:11:00.849Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 27 (0x7fa601508000) [pid = 15232] [serial = 14] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.853Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 26 (0x7fa602492800) [pid = 15232] [serial = 3] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.855Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 25 (0x7fa609723000) [pid = 15232] [serial = 2] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.858Z] 15:11:00 INFO - PID 15175 | --DOCSHELL 0x7fa601552800 == 8 [pid = 15232] [id = {62097795-cefa-4626-92b1-e2b01134ebcf}] [task 2018-04-28T15:11:00.862Z] 15:11:00 INFO - PID 15175 | --DOCSHELL 0x7fa5ff39c800 == 7 [pid = 15232] [id = {907aed2f-82ba-484c-a1a7-209937e276ec}] [task 2018-04-28T15:11:00.864Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 24 (0x7fa5ff365000) [pid = 15232] [serial = 12] [outer = (nil)] [url = http://web-platform.test:8000/referrer-policy/generic/subresource/document.py] [taskcluster:error] Task timeout after 7200 seconds. Force killing container. [task 2018-04-28T15:11:00.874Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 23 (0x7fa609720c00) [pid = 15232] [serial = 7] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.875Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 22 (0x7fa609721000) [pid = 15232] [serial = 6] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.876Z] 15:11:00 INFO - PID 15175 | --DOMWINDOW == 21 (0x7fa6098bf000) [pid = 15232] [serial = 9] [outer = (nil)] [url = about:blank] [task 2018-04-28T15:11:00.878Z] 15:11:00 INFO - PID 15175 | --DOCSHELL 0x7fa601555000 == 6 [pid = 15232] [id = {f6b879e5-be92-4156-adc9-5489f3b17629}] [taskcluster 2018-04-28 15:11:01.679Z] === Task Finished === [taskcluster 2018-04-28 15:11:01.685Z] Unsuccessful task run with exit code: -1 completed in 7740.784 seconds

Update: There have been 62 failures in the last 7 days, 230 total failures in the last 21 days. Depends on: 1243080, 1246165, 1321605, 1404190, 1431125, 1448049, 1451432 Blocks: 1306635 Summary: Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container. / [taskcluster:error] Task timeout after 5400 seconds. Force killing container. / [taskcluster:error] Task timeout after 7200 seconds. Force killing container. Failure per platform and build type: - Android 4.2 x86 / opt: 2 - android-4-0-armv7-api16 / debug & opt :3 - android-4-3-armv7-api16 / debug & opt: 17 - android-5-0-aarch64 / debug & op: 1 - Linux x64 / pgo, debug & asan: 8 - linux64-ccov / opt: 8 - linux64-qr / opt & debug: 9 - linux64-asan-reporter / opt: 1 - Linux / debug: 13 Recent log with the failure: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=177354933&lineNumber=1772

I expect these failures to continue and this bug to remain open indefinitely. Task timeouts are the last resort whenever a task is hung or otherwise runs for too long. My immediate goal is to address timeouts which recur in the same task frequently and identify other related issues in dependent bugs. I expect infrequent failures on Android especially due to bug 1321605. There are some infrequent failures on Android that can be addressed with more chunks - bug 1461393. Recent failures on Android and Linux are frequent and irregular: Not associated with particular tests or test tasks. I'm hoping those might be addressed in bug 1457694. (Recent network failures and delays during 'pip install' appear to have been resolved.)

Over the last 7 days there are 124 failures present on this bug. These happen on android-4-0-armv7-api16, android-4-2-x86, android-4-3-armv7-api16, linux32, linux64, linux64-ccov, linux64-nightly, linux64-noopt, linux64-qr, windows7-32. Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?job_id=180438377&repo=mozilla-inbound&lineNumber=1806 [task 2018-05-27T02:38:21.245Z] 02:38:21 INFO - TEST-OK | testSessionOOMRestore | took 335841ms [task 2018-05-27T02:38:21.246Z] 02:38:21 INFO - TEST-START | Shutdown [task 2018-05-27T02:38:21.247Z] 02:38:21 INFO - Passed: 97 [task 2018-05-27T02:38:21.249Z] 02:38:21 INFO - Failed: 0 [task 2018-05-27T02:38:21.250Z] 02:38:21 INFO - Todo: 0 [task 2018-05-27T02:38:21.250Z] 02:38:21 INFO - SimpleTest FINISHED [task 2018-05-27T02:38:42.819Z] 02:38:42 INFO - INFO | automation.py | Application ran for: 0:06:10.045962 [task 2018-05-27T02:38:42.821Z] 02:38:42 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmp_o252qpidlog [task 2018-05-27T02:38:43.458Z] 02:38:43 INFO - /data/tombstones does not exist; tombstone check skipped [task 2018-05-27T02:38:50.701Z] 02:38:50 INFO - INFO | automation.py | Application pid: 5599 [task 2018-05-27T02:39:03.345Z] 02:39:03 INFO - SimpleTest START [task 2018-05-27T02:39:03.347Z] 02:39:03 INFO - TEST-START | testSessionPrivateBrowsing [taskcluster:error] Task timeout after 3600 seconds. Force killing container. [taskcluster 2018-05-27 02:39:11.369Z] === Task Finished === [taskcluster 2018-05-27 02:39:11.372Z] Unsuccessful task run with exit code: -1 completed in 3601.941 seconds

linux32-debug/jittest-4 and android-debug/xpcshell-11 are frequent and might be actionable. Otherwise, see comment 92.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5989424834d2 Increase Android/debug xpcshell max-run-time; r=me,a=test-only

As an update, there are 76 failures in the last 7 days as it follows: - 36 on linux64 (asan-reporter, ccov and qr) - 29 on android. Mostly on 4-3-armv7-api16 but there are also 3 on 4-0-armv7-api16 and 2 on 4-2-x86 - 11 on linux32

In the last 7 days, there are 111 failures on this bug. They occur mostly on linux 32, 64, 64-ccov, 64-qr, all builds type. Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=183726361&repo=mozilla-inbound&lineNumber=24638

In the last 7 days there are 127 failures: - 51 failures on linux64 / asan, debug, pgo & lto - 47 failures on android-em-4-3-armv7-api16 / opt & debug - 11 failures on linux 32 / debug, pgo & opt - 7 failures on linux64-ccov / debug - 6 failures on osx-cross / opt & debug - 2 failures on android-4-0-armv7-api16 / debug - 2 failures on linux64-qr / debug - 1 failure on linux64-noopt / debug As per comment 92, these will keep occuring. :gbrown is there no way to decrease their rate on the most common platforms they occur: linux64 and android?

(In reply to Andreea Pavel [:apavel] from comment #134) > In the last 7 days there are 127 failures: > > - 51 failures on linux64 / asan, debug, pgo & lto > - 47 failures on android-em-4-3-armv7-api16 / opt & debug > - 11 failures on linux 32 / debug, pgo & opt > - 7 failures on linux64-ccov / debug > - 6 failures on osx-cross / opt & debug > - 2 failures on android-4-0-armv7-api16 / debug > - 2 failures on linux64-qr / debug > - 1 failure on linux64-noopt / debug > > As per comment 92, these will keep occuring. > > :gbrown is there no way to decrease their rate on the most common platforms > they occur: linux64 and android? These are certainly too frequent, but I don't know what to do about it. There are multiple causes, even on one platform. Bug 1470177 eliminated a common cause of android timeouts, but the weekly stats hardly changed. Bug 1457694 was an attempt to address a common scenario I (may) see on linux and android, but that hasn't made any progress and I'm not sure how to make progress on it. :( I will keep monitoring failures and acting on those cases that I can.

:gbrown thanks for looking into this!

(In reply to Geoff Brown [:gbrown] from comment #137) > Bug 1457694 was an attempt to address a common scenario I (may) see on linux > and android, but that hasn't made any progress and I'm not sure how to make > progress on it. :( Bug 1457694 has been confirmed as a real issue and is progressing, I think!

Update: There has been a total of 119 failures in the last 7 days( and 339 failures in the last 21 days). - 2 failures on android-4-0-armv7-api16 debug - 1 failure on android-4-2-x86 opt - 1 failure on android-em-4-2-x86 opt - 19 failures on android-em-4-3-armv7-api16 debug/opt - 17 failures on linux32 debug/opt - 51 failures on linux64 asan/debug/lto/opt/pgo - 10 failures on linux64-ccov debug - 10 failures on linux64-qr debug/opt - 1 failure on linux64-noopt debug - 1 failure on osx-10-10 opt - 3 failures on osx-cross opt - 1 failure on windows10-64 asan - 2 failures on windows7-32 debug There are different type of tests running, but all of them have this error in common: [taskcluster:error] Task timeout after 3600 seconds. Force killing container. [taskcluster 2018-07-18 12:00:30.241Z] Unsuccessful task run with exit code: -1 completed in 3924.674 seconds

(In reply to OrangeFactor Robot from comment #163) > 135 failures in 138 pushes (0.978 failures/push) were associated with this > bug yesterday. > > Platform breakdown: > * android-em-4-3-armv7-api16: 105 robocop is perma-failing - https://bugzilla.mozilla.org/show_bug.cgi?id=1451513#c19.

(In reply to Geoff Brown [:gbrown] from comment #164) > robocop is perma-failing - > https://bugzilla.mozilla.org/show_bug.cgi?id=1451513#c19. That was corrected with a backout and subsequent re-landing - great!

(In reply to Geoff Brown [:gbrown] from comment #150) > Bug 1457694 has been confirmed as a real issue and is progressing, I think! That bug hasn't been marked as resolved, but some changes have landed in related bugs and I am seeing far fewer of those issues. Failure rates in this bug have recently been significantly lower than we've seen in months.

Thanks :aryx! Yes, it looks like bug 1480494 is the biggest contributor to current failures here.

Update: There have been 98 failures in the last week. Failures per platform and build type: linux64 / debug, asan, opt, lto: 39 osx-10-10 / opt & debug: 12 android-em-4-3-armv7-api16 / debug & opt: 11 linux64-ccov / debug: 11 linux32 / debug & opt: 7 osx-cross / opt & asan: 7 windows2012-32 / debug & pgo: 2 windows7-32 / opt & pgo: 3 android-em-4-2-x86 / debug: 2 linux64-noopt / debug: 1 linux64-qr / debug: 1 android-4-0-armv7-api16 / opt: 1 android-5-0-aarch64 / opt: 1 Summary: Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container. / [taskcluster:error] Task timeout after 5400 seconds. Force killing container. / [taskcluster:error] Task timeout after 7200 seconds. Force killing container.

There have been 118 failures in the last week. The most affected platform / build: linux64-ccov / debug. Recent relevat log file: https://treeherder.mozilla.org/logviewer.html#?job_id=197122988&repo=mozilla-central&lineNumber=180429 Summary: Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container. / [taskcluster:error] Task timeout after 5400 seconds. Force killing container. / [taskcluster:error] Task timeout after 7200 seconds. Force killing container. Blocks: 1306635 Depends on: 1115253, 1243080, 1321605, 1404190, 1431125, 1457694, 1480348, 1487877, 1487886

linux64/ccov devtool and android/debug mochitest failure are being addressed with more test chunks. linux64 mochitest-media failures are reportedly 1480348. There are also some build task timeouts, which seem quite random -- I don't know what to do about those.

Update: There have been 37 failures in the last week. Failures per platform and build type: - linux64 / debug, opt, asan, pgo, lto: 16 - linux32 / debug & opt: 10 - android-em-4-3-armv7-api16 / debug & opt: 5 - linux64-qr / debug: 2 - linux64-ccov / debug: 1 - osx-10-10-dmd / opt: 1 - osx-cross / opt: 1 - osx-cross-noopt / debug: 1 Recent relevant log file: https://treeherder.mozilla.org/logviewer.html#?job_id=199174505&repo=mozilla-inbound&lineNumber=40467 Summary: Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container. / [taskcluster:error] Task timeout after 5400 seconds. Force killing container. / [taskcluster:error] Task timeout after 7200 seconds. Force killing container.

(In reply to Geoff Brown [:gbrown] from comment #199) > linux64/ccov devtool and android/debug mochitest failure are being addressed > with more test chunks. > > linux64 mochitest-media failures are reportedly 1480348. > > There are also some build task timeouts, which seem quite random -- I don't > know what to do about those. Many of those (apparent) build task timeouts are in rusttests; I haven't investigated, but note bug 1489277 might be tangentially related.

There was a recent spike in linux64-ccov/asan build timeouts after 60 minutes; these are "build-linux64-asan-fuzzing-ccov/opt (bocf)" builds, on mozilla-central only. The most recent builds have run close to 50 minutes, so I won't take action until/unless this issue returns.