Closed Bug 1476893 Opened 6 years ago Closed 5 years ago

Intermittent gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output

Categories

(Core :: Panning and Zooming, defect, P5)

Product:
Core
Component:
Panning and Zooming
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla73
Tracking Flags:
Tracking Status
firefox73 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: hiro)

References

Details

(Keywords: intermittent-failure, Whiteboard: [retriggered][stockwell unknown])

Attachments

(1 file, 2 obsolete files)

Bug 1476893 - Split helper_bug1280013.html into its own test group. r=tnikkel
(deleted), text/x-phabricator-request
Details
Filed by: ebalazs [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=188882070&repo=autoland https://queue.taskcluster.net/v1/task/JCOPe5VIQeG1B8KXw3sYXw/runs/0/artifacts/public/logs/live_backing.log WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output [task 2018-07-19T07:51:57.091Z] 07:51:57 INFO - INFO | automation.py | Application ran for: 0:21:22.108770 [task 2018-07-19T07:51:57.092Z] 07:51:57 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpHehGx0pidlog [task 2018-07-19T07:51:57.408Z] 07:51:57 INFO - Contents of /data/anr/traces.txt: [task 2018-07-19T07:51:57.410Z] 07:51:57 INFO - ----- pid 2466 at 2018-07-19 00:51:42 ----- [task 2018-07-19T07:51:57.410Z] 07:51:57 INFO - Cmd line: org.mozilla.fennec_aurora [task 2018-07-19T07:51:57.410Z] 07:51:57 INFO - JNI: CheckJNI is on; workarounds are off; pins=0; globals=254 [task 2018-07-19T07:51:57.410Z] 07:51:57 INFO - DALVIK THREADS: [task 2018-07-19T07:51:57.410Z] 07:51:57 INFO - (mutexes: tll=0 tsl=0 tscl=0 ghl=0) [task 2018-07-19T07:51:57.411Z] 07:51:57 INFO - "main" prio=5 tid=1 NATIVE [task 2018-07-19T07:51:57.412Z] 07:51:57 INFO - | group="main" sCount=1 dsCount=0 obj=0x414c9578 self=0x2a00d090 [task 2018-07-19T07:51:57.412Z] 07:51:57 INFO - | sysTid=2466 nice=0 sched=0/0 cgrp=apps handle=1073811452

Note that the increase in frequency here is a result of the emulator upgrade in bug 1556058.

I tried to identify the subtest that's timing out, but in all the instances I've checked so far, it's the very first one, and often that subtest doesn't even get very far, making me wonder if this is some sort of issue with the environment / infrastructure rather than the test itself.

The emulator log from around the failure shows EGL errors similar to those in bug 1516298. And the try push with snorp's fix from that bug also shows no failures in test_group_zoom, so likely the two failures have the same root cause, which snorp's fix addresses.

This seems to still be happening even after snorp's fix.

No longer depends on: 1564846

The screenshot in these failures shows the test harness page, whereas I would expect it to show the page with the subtest, since that's supposed to open in a new browser window/tab. So maybe it's a GV bug where new tabs don't open properly or aren't put in the foreground.

Hi Joel, I read the conversation within the patch and wanted to ask what to do about this failure. It hasn't reach the disable-recommended queue and it looks like it won't, however the average in 30 days is ~100 failures. https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-22&endday=2019-07-22&tree=trunk&bug=1476893

Looked in OF in the last 3 months and it seems that there are higher spikes in June towards July.

There are 32 total failures in the last 7 days on:
*debug: android-em-7-0-x86_64, linux64-qr,
*opt: android-em-7-0-x86_64

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257641266&repo=autoland&lineNumber=6013

[task 2019-07-22T06:44:25.509Z] 06:44:25 INFO - 2507 INFO TEST-START | gfx/layers/apz/test/mochitest/test_group_zoom.html
[task 2019-07-22T06:44:35.718Z] 06:44:35 INFO - GECKO | "subtest":"helper_bug1280013.html | Main-thread scroll position is still at 0","status":"PASS","js_source":"TestRunner.js"}
[task 2019-07-22T06:44:45.935Z] 06:44:45 INFO - GECKO | ⰲ겿
[task 2019-07-22T06:44:56.150Z] 06:44:56 INFO - GECKO | 겿
[task 2019-07-22T06:45:06.368Z] 06:45:06 INFO - GECKO |
[task 2019-07-22T06:51:38.173Z] 06:51:38 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-07-22T06:51:38.276Z] 06:51:38 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-07-22T06:51:38.276Z] 06:51:38 INFO - TEST-INFO | started process screentopng
[task 2019-07-22T06:51:38.597Z] 06:51:38 INFO - TEST-INFO | screentopng: exit 0
[task 2019-07-22T06:51:52.950Z] 06:51:52 WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output
[task 2019-07-22T06:51:52.950Z] 06:51:52 INFO - remoteautomation.py | Application ran for: 0:08:00.338450
[task 2019-07-22T06:51:53.615Z] 06:51:53 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmp4QHyWT/481f262f-9964-eef5-6bef-a45406499930.dmp /builds/worker/workspace/build/symbols
[task 2019-07-22T06:52:10.681Z] 06:52:10 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/481f262f-9964-eef5-6bef-a45406499930.dmp
[task 2019-07-22T06:52:10.681Z] 06:52:10 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/481f262f-9964-eef5-6bef-a45406499930.extra
[task 2019-07-22T06:52:10.686Z] 06:52:10 WARNING - PROCESS-CRASH | gfx/layers/apz/test/mochitest/test_group_zoom.html | application crashed [@ libc.so + 0x8c66a]
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash dump filename: /tmp/tmp4QHyWT/481f262f-9964-eef5-6bef-a45406499930.dmp
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Operating system: Android
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - CPU: amd64
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - family 6 model 6 stepping 3
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 4 CPUs
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - GPU: UNKNOWN
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash reason: SIGABRT
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash address: 0x2c4e
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Process uptime: not available
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Thread 0 (crashed)
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 0 libc.so + 0x8c66a
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000051f6c51
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsi = 0x00007fff4e947f20 rdi = 0x000000000000001f
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rbp = 0x00000000051f6c51 rsp = 0x00007fff4e947ea8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r10 = 0x00000000051f6c51 r11 = 0x0000000000000246
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r12 = 0x00000000051f6c51 r13 = 0x0000772e13329e60
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rip = 0x0000772e16fcd66a
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: given as instruction pointer in context
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 1 libc.so + 0x2a0e6
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947eb0 rip = 0x0000772e16f6b0e6
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 2 libutils.so + 0x184e7
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ed0 rip = 0x0000772e13c2b4e7
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 3 system@framework@boot.art + 0x50f2c8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ed8 rip = 0x0000000070fbe2c8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 4 libart.so + 0x14cd91
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ee0 rip = 0x0000772e12c58d91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 5 boot-framework.oat + 0x1000
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f08 rip = 0x0000000072f54000
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 6 dalvik-main space (deleted) + 0x62b60
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f20 rip = 0x0000000012c62b60
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 7 boot-framework.oat + 0x1e48820
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f78 rip = 0x0000000074d9b820
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 8 libart.so + 0x14cd91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f80 rip = 0x0000772e12c58d91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 9 libart.so + 0x39c37f
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947fc0 rip = 0x0000772e12ea837f
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 10 libart.so + 0x6e3c40
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947fc8 rip = 0x0000772e131efc40
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 11 dalvik-jit-code-cache (deleted) + 0x18c7000
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e947fd8 rip = 0x0000772e00000000
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 12 libutils.so + 0x1837b
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948020 rip = 0x0000772e13c2b37b
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 13 system@framework@boot-framework.art + 0x3f5b10
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948048 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 14 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948050 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 15 libandroid_runtime.so + 0xff856
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948060 rip = 0x0000772e13db9856
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 16 system@framework@boot-framework.art + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948068 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 17 boot-framework.oat + 0x1e17220
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948080 rip = 0x0000000074d6a220
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 18 system@framework@boot-framework.art + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948098 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 19 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480a0 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 20 system@framework@boot-framework.art + 0x3f5790
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480a8 rip = 0x0000000071695790
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 21 dalvik-jit-code-cache (deleted) + 0x18c7007
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480c0 rip = 0x0000772e00000007
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 22 system@framework@boot.art + 0x50f2c8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480c8 rip = 0x0000000070fbe2c8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 23 system@framework@boot-framework.art + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948110 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 24 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948118 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 25 boot-framework.oat + 0x1e19034
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948120 rip = 0x0000000074d6c034
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 26 system@framework@boot-framework.art + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948128 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 27 libart.so + 0x1611c0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948160 rip = 0x0000772e12c6d1c0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 28 system@framework@boot.art + 0x1fd518
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948170 rip = 0x0000000070cac518
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 29 dalvik-main space (deleted) + 0x16c7f0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948190 rip = 0x0000000012d6c7f0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 30 dalvik-main space (deleted) + 0x671d0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948198 rip = 0x0000000012c671d0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 31 dalvik-main space (deleted) + 0x204a00
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481a0 rip = 0x0000000012e04a00
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 32 system@framework@boot-framework.art + 0x3ad778
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481a8 rip = 0x000000007164d778
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 33 boot-framework.oat + 0x1e160c4
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481e0 rip = 0x0000000074d690c4
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 34 system@framework@boot-framework.art + 0x39bd40
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481e8 rip = 0x000000007163bd40
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 35 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481f8 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 36 dalvik-main space (deleted) + 0x62b60
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948200 rip = 0x0000000012c62b60
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 37 system@framework@boot.art + 0x3412c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948208 rip = 0x0000000070df02c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 38 system@framework@boot.art + 0x1fd518
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948210 rip = 0x0000000070cac518
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 39 system@framework@boot.art + 0x206668
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948218 rip = 0x0000000070cb5668
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 40 boot-framework.oat + 0x1e12ab0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948220 rip = 0x0000000074d65ab0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 41 system@framework@boot-framework.art + 0x32c2f0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948228 rip = 0x00000000715cc2f0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 42 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948230 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 43 system@framework@boot.art + 0x3405c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948258 rip = 0x0000000070def5c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 44 dalvik-main space (deleted) + 0xa1840
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948270 rip = 0x0000000012ca1840
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 45 dalvik-main space (deleted) + 0x204a00
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948280 rip = 0x0000000012e04a00
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 46 system@framework@boot-framework.art + 0x419fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482a0 rip = 0x00000000716b9fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 47 boot-framework.oat + 0x165ee53
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482b0 rip = 0x00000000745b1e53
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 48 system@framework@boot-framework.art + 0x419fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482b8 rip = 0x00000000716b9fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 49 dalvik-main space (deleted) + 0x5fe90
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482c0 rip = 0x0000000012c5fe90
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - 50 dalvik-main space (deleted) + 0x77090
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - rsp = 0x00007fff4e9482c8 rip = 0x0000000012c77090
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - Found by: stack scanning

Flags: needinfo?(jmaher)
Whiteboard: [stockwell needswork]

:apavel, without hitting the limit, we need to be more conservative on disabling, this can stick around for a while.

it looks like the linux64-qr failures are different (or at least more informative):
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257120829&repo=mozilla-inbound&lineNumber=25781

:jgilbert- not sure if you are the backup for :kats; there is an assertion in the failures on linux64-qr ^, and that looks to be the root cause, any chance you know if that is a quick fix?

Flags: needinfo?(jmaher) → needinfo?(jgilbert)

Thank you.

I'm not seeing anything obvious.
:jbonisteel, can you route this?

Flags: needinfo?(jgilbert) → needinfo?(jbonisteel)
Flags: needinfo?(jbonisteel) → needinfo?(botond)

It looks like the failures are all on one of two platforms: linux64-qr, and android-em-7-0-x86_64.

They seem to be two different issues, because they're failing in different parts of the test: on android, the failure happens very soon after the test starts, while on linux64-qr, it's usually a subtest somewhere in the middle that fails.

The linux64-qr failure spiked on the 18th, but seems to be back to a low frequency now; probably not worth investigating at this frequency.

That leaves the android failure; I confirmed that bug 1564846 does not help with that (I got 2/60 failures on Try both with [1] and without [2] that patch).

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1b4fbea2922a19f42a6e4ce993124b5fff4ab69
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a51059db6a7530e69801ed51e514d42e588c3d99

It looks like Kats' most recent suggestion was:

(In reply to Kartikaya Gupta (email:kats@mozilla.com) (away until Feb-2020) from comment #49)

The screenshot in these failures shows the test harness page, whereas I would expect it to show the page with the subtest, since that's supposed to open in a new browser window/tab. So maybe it's a GV bug where new tabs don't open properly or aren't put in the foreground.

Snorp, do you have any ideas for how an issue like this can be investigated?

It would be really good not to disable this test, as it provides important test coverage for APZ.

Flags: needinfo?(botond) → needinfo?(snorp)

Not aware of anything right now. The patch on 1564846 bounced, but like you said, that didn't seem to help anyway.

Flags: needinfo?(snorp)
Attachment #9076555 - Attachment is obsolete: true

(In reply to Joel Maher ( :jmaher ) (UTC-4) (on leave, back November 5th) from comment #54)

:apavel, without hitting the limit, we need to be more conservative on disabling, this can stick around for a while.

it looks like the linux64-qr failures are different (or at least more informative):
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257120829&repo=mozilla-inbound&lineNumber=25781

:jgilbert- not sure if you are the backup for :kats; there is an assertion in the failures on linux64-qr ^, and that looks to be the root cause, any chance you know if that is a quick fix?

This does not seem to go away and there is no fix since then. History looks like this: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-07-06&endday=2019-11-03&tree=trunk&bug=1476893

Recent failures are on android (last 30 days) so I made a patch for this to be disabled just on that.

This bug failed 44 times in the last 7 days. Occurs on android-em-7-0-x86_64 on debug and opt build types.

Log:
https://hg.mozilla.org/integration/autoland/rev/ee2997888f54ee12b6505d0af8a216076a351643

The failure rate seems to have spiked recently -- it went from 18 failures in 7 days (comment 77) to 16 failures in a single day (comment 78).

Is it possible to get a regression window for this spike?

Retrigger range: https://tinyurl.com/u99ufm3

Next shift will continue.

Whiteboard: [stockwell unknown] → [retriggered][stockwell needs:work]

Retrigger range: https://tinyurl.com/u24nmcp

Hiro, hi, I saw in bug 1562757 you mentioned that this test is flaky. Is this something you can look at ?

Flags: needinfo?(hikezoe.birchill)

I honestly don't think I can do better work than Botond can do in this areas, that's said, I know he's been super busy so I am happy to help him and snorp. Hope I can reproduce the failure locally!

Assignee: nobody → hikezoe.birchill
Status: NEW → ASSIGNED
Flags: needinfo?(hikezoe.birchill)

Thanks, Hiro, appreciated :)

Meanwhile I will work on getting test_group_touchevents.html re-enabled in bug 1591519. I think both of these tests are important for APZ test coverage on Android.

Just an update. I could reproduce the failure locally and saw a suspicious message in adb log

Choreographer: Skipped 84 frames! The application may be doing too much work on its main thread

Just before this message there are a bunch of errors in JS;

E Web Content: [JavaScript Error: "TypeError: can't access dead object"]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #87)

Just an update. I could reproduce the failure locally and saw a suspicious message in adb log

Choreographer: Skipped 84 frames! The application may be doing too much work on its main thread

Just before this message there are a bunch of errors in JS;

E Web Content: [JavaScript Error: "TypeError: can't access dead object"]

These two may or may not be related to this issue. It pretty often happens even in the case where the test passed.

As far as I can tell, there are at least two stuck points;

  1. at await promiseAllPaintsDone in waitUntilApzStable
  2. at yield synthesizeNativeTouchDrag(subframe, 10, 200, 0, -300) in helper_bug1280013.html

Though I have no clue for 2), 1) is fixed by bug 1564846 at least on my local Android emulator and a try in bug 1564846 comment 10.
I've already asked snorp to land the fix in bug 1564846.

Hiroyuki I saw that James did not reply on bug 1564846, could you land the fix?

There are 39 total failures in the last 7 days on android-em-7-0-x86_64 debug and opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277968433&repo=autoland&lineNumber=3944

[task 2019-11-25T09:44:00.641Z] 09:44:00 INFO - 945 INFO TEST-START | gfx/layers/apz/test/mochitest/test_group_zoom.html
[task 2019-11-25T09:49:28.956Z] 09:49:28 INFO - Failed to get top activity, retrying, once...
[task 2019-11-25T09:50:43.825Z] 09:50:43 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-11-25T09:50:43.928Z] 09:50:43 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-11-25T09:50:43.929Z] 09:50:43 INFO - TEST-INFO | started process screentopng
[task 2019-11-25T09:50:44.222Z] 09:50:44 INFO - TEST-INFO | screentopng: exit 0
[task 2019-11-25T09:50:58.838Z] 09:50:58 WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output
[task 2019-11-25T09:50:58.838Z] 09:50:58 INFO - remoteautomation.py | Application ran for: 0:07:21.595455
[task 2019-11-25T09:50:59.502Z] 09:50:59 INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aSzGEdqwT7uVqIXqJXOCCA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-11-25T09:51:01.685Z] 09:51:01 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpoIZ9hp/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp /tmp/tmpRKSUAN
[task 2019-11-25T09:51:04.148Z] 09:51:04 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp
[task 2019-11-25T09:51:04.148Z] 09:51:04 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/5f74dab3-85aa-a649-c8af-2b28478a4d00.extra
[task 2019-11-25T09:51:04.151Z] 09:51:04 WARNING - PROCESS-CRASH | gfx/layers/apz/test/mochitest/test_group_zoom.html | application crashed [@ libc.so + 0x8c66a]
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - Crash dump filename: /tmp/tmpoIZ9hp/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - Operating system: Android
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - CPU: amd64
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - family 6 model 6 stepping 3
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - 4 CPUs
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - GPU: UNKNOWN
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Crash reason: SIGABRT
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Crash address: 0x23c8
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Process uptime: not available
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Thread 0 (crashed)
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - 0 libc.so + 0x8c66a
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000051fe175
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rsi = 0x00007fff2e2f9af0 rdi = 0x000000000000001f
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rbp = 0x00000000051fe175 rsp = 0x00007fff2e2f9a78
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - r10 = 0x00000000051fe175 r11 = 0x0000000000000246
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - r12 = 0x00000000051fe175 r13 = 0x00007a30c6129f40
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rip = 0x00007a30c9b5b66a
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: given as instruction pointer in context
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 1 libc.so + 0x2a0e6
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9a80 rip = 0x00007a30c9af90e6
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 2 libutils.so + 0x184e7
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9aa0 rip = 0x00007a30c94eb4e7
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 3 system@framework@boot-framework.art + 0x3d5350
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ac8 rip = 0x0000000071675350
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 4 boot-framework.oat + 0x1000
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ad8 rip = 0x0000000072f54000
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 5 dalvik-main space (deleted) + 0x5d700
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9af0 rip = 0x0000000012c5d700
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 6 libart.so + 0x2d39f9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b10 rip = 0x00007a30c5bdf9f9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 7 boot-framework.oat + 0x15bc3f0
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b18 rip = 0x000000007450f3f0
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 8 libart.so + 0x4f1be9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b20 rip = 0x00007a30c5dfdbe9
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 9 system@framework@boot-framework.art + 0x2a9f48
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b38 rip = 0x0000000071549f48
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 10 system@framework@boot-framework.art + 0x3d5350
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b50 rip = 0x0000000071675350
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 11 libart.so + 0x39c37f
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b90 rip = 0x00007a30c5ca837f
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 12 libart.so + 0x6e3c40
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b98 rip = 0x00007a30c5fefc40
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 13 libutils.so + 0x1837b
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9bf0 rip = 0x00007a30c94eb37b
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 14 system@framework@boot-framework.art + 0x3f5b10
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c18 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 15 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c20 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 16 libandroid_runtime.so + 0xff856
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c30 rip = 0x00007a30c9684856
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 17 system@framework@boot-framework.art + 0x3f5b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c38 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 18 boot-framework.oat + 0x1e17220
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c50 rip = 0x0000000074d6a220
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 19 system@framework@boot-framework.art + 0x3f5b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c68 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 20 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c70 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 21 system@framework@boot-framework.art + 0x3f5790
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c78 rip = 0x0000000071695790
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 22 system@framework@boot.art + 0x50f2c8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c98 rip = 0x0000000070fbe2c8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 23 system@framework@boot-framework.art + 0x3f5b10
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ce0 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 24 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ce8 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 25 boot-framework.oat + 0x1e19034
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9cf0 rip = 0x0000000074d6c034
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 26 system@framework@boot-framework.art + 0x3f5b10
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9cf8 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 27 system@framework@boot.art + 0x1fd518
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d40 rip = 0x0000000070cac518
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 28 dalvik-main space (deleted) + 0x181700
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d60 rip = 0x0000000012d81700
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 29 dalvik-main space (deleted) + 0x25cb00
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d68 rip = 0x0000000012e5cb00
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 30 dalvik-main space (deleted) + 0x25cb00
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d70 rip = 0x0000000012e5cb00
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 31 system@framework@boot-framework.art + 0x3ad778
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d78 rip = 0x000000007164d778
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 32 boot-framework.oat + 0x1e160c4
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9db0 rip = 0x0000000074d690c4
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 33 system@framework@boot-framework.art + 0x39bd40
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9db8 rip = 0x000000007163bd40
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 34 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9dc8 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.161Z] 09:51:04 INFO - 35 dalvik-main space (deleted) + 0x5d700
[task 2019-11-25T09:51:04.161Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9dd0 rip = 0x0000000012c5d700

Flags: needinfo?(hikezoe.birchill)
Whiteboard: [retriggered][stockwell needs:work] → [retriggered][stockwell needswork:owner]

Landed now.

Flags: needinfo?(hikezoe.birchill)

Thank you!

For record, after bug 1564846 landed, this test still fails at 2) in comment 86.

From a failure log

942 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | Starting subtest helper_bug1280013.html
943 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | helper_bug1280013.html | Screen height must be at least 500 pixels for this test to work
944 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | helper_bug1280013.html | Main-thread scroll position is still at 0

Botond, this should be disabled as there are 133 total failures in the last 30 days and it doesn't seem like the failures will stop.

Can you please take a look?

Flags: needinfo?(botond)

According to OrangeFactor, the test has failed 8 times since Hiro's fix has landed, and all 8 failures are in the second place identified in comment 88.

The place in question is in the subtest helper_bug1280013.html, which I note is the first subtest. I made this Try push to try to determine if the issue is specific to that subtest, or something more general that just manifests in that subtest because it is the first:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e771cac19a9b6da2d3cda77fd0fcd656b716f75e&selectedJob=279637023

Great, thank you for working on this!

(In reply to Botond Ballo [:botond] from comment #96)

I made this Try push to try to determine if the issue is specific to that subtest, or something more general that just manifests in that subtest because it is the first:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e771cac19a9b6da2d3cda77fd0fcd656b716f75e&selectedJob=279637023

The Try push shows that out of 120 total test runs (60 opt and 60 debug), all 4 failures (2 opt and 2 debug) occurred in the new test containing helper_bug1280013.html. This suggests that the problem is specific to helper_bug1280013.html.

So, as a starting point, we can move helper_bug1280013.html out into its own test group. This way, if we need to disable it, we can disable just that one test and leave the others running.

Pushed by bballo@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/15b3e36e186a Split helper_bug1280013.html into its own test group. r=tnikkel
Regressions: 1601568

https://hg.mozilla.org/mozilla-central/rev/15b3e36e186a

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox73: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

Failures of this test group seem to be down. Will continue investigation of helper_bug1280013.html in bug 1601568.

Flags: needinfo?(botond)
Attachment #9106077 - Attachment is obsolete: true

This still shows up on beta, should the it be uplifted?

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282409811&repo=mozilla-beta&lineNumber=3943

Flags: needinfo?(hikezoe.birchill)

As you know, to fix this intermittent failure we need two changes in two different bugs, bug 1564846 and bug 1601568. Bug 1564846 is just only for test runner so that we can uplift it. Bug 1601568 changed logic a bit, it's hard to tell for me.

Anyway, the change in bug 1564846 should reduce the failure frequent. We should uplift it at least.

Flags: needinfo?(hikezoe.birchill)

James what do you think?

Failure rate on Beta is not high, 2 failure in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-16&endday=2019-12-23&tree=all&bug=1476893

Flags: needinfo?(snorp)

Given the low failure rate, and the fact that the fix in bug 1601568 touched some core scrolling code (==> not a super low-risk fix), I would recommend not uplifting that.

(I'll leave it to snorp to comment on uplifting bug 1564846.)

Thank you.

Yeah, no reason to uplift IMHO.

Flags: needinfo?(snorp)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: