https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Recent log failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237451395&repo=autoland&lineNumber=2008

[task 2019-04-01T23:25:57.237Z] 23:25:57 INFO - 375 INFO TEST-START | gfx/layers/apz/test/mochitest/test_group_touchevents.html
[task 2019-04-01T23:33:19.132Z] 23:33:19 INFO - wait for org.mozilla.fennec_aurora complete; top activity=org.mozilla.fennec_aurora
[task 2019-04-01T23:33:19.437Z] 23:33:19 INFO - org.mozilla.fennec_aurora unexpectedly found running. Killing...
[task 2019-04-01T23:33:19.437Z] 23:33:19 INFO - TEST-INFO | started process screentopng
[task 2019-04-01T23:33:19.776Z] 23:33:19 INFO - TEST-INFO | screentopng: exit 0
[task 2019-04-01T23:33:34.535Z] 23:33:34 WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_touchevents.html | application timed out after 370 seconds with no output
[task 2019-04-01T23:33:34.535Z] 23:33:34 INFO - remoteautomation.py | Application ran for: 0:16:27.506890
[task 2019-04-01T23:33:35.555Z] 23:33:35 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/I-_In94qQ7WE0GuHww2W2g/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-04-01T23:33:38.451Z] 23:33:38 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpetc0Vh/251ba815-3050-ae09-0979-554e073c0a2d.dmp /tmp/tmp1L_0iD
[task 2019-04-01T23:33:46.283Z] 23:33:46 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/251ba815-3050-ae09-0979-554e073c0a2d.dmp
[task 2019-04-01T23:33:46.284Z] 23:33:46 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/251ba815-3050-ae09-0979-554e073c0a2d.extra
[task 2019-04-01T23:33:46.285Z] 23:33:46 WARNING - PROCESS-CRASH | gfx/layers/apz/test/mochitest/test_group_touchevents.html | application crashed [@ libc.so + 0x1c3dc]
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - Crash dump filename: /tmp/tmpetc0Vh/251ba815-3050-ae09-0979-554e073c0a2d.dmp
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - Operating system: Android
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - 0.0.0 Linux 2.6.29-gea477bb #1 Wed Sep 26 11:04:45 PDT 2012 armv7l
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - CPU: arm
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - ARMv7 ARM Cortex-A8 features: swp,half,thumb,fastmult,vfpv2,edsp,neon,vfpv3
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - 1 CPU
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - GPU: UNKNOWN
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - Crash reason: SIGABRT
[task 2019-04-01T23:33:46.286Z] 23:33:46 INFO - Crash address: 0x7c6
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - Process uptime: not available
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - Thread 0 (crashed)
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - 0 libc.so + 0x1c3dc
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - r0 = 0xfffffffc r1 = 0xbe8fe4a8 r2 = 0x00000010 r3 = 0x051c67aa
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - r4 = 0x2a03f810 r5 = 0x00000000 r6 = 0x2a03f824 r7 = 0x000000fc
[task 2019-04-01T23:33:46.287Z] 23:33:46 INFO - r8 = 0x2a03f858 r9 = 0x00000014 r10 = 0x2a00d0a0 r12 = 0xbe8fe4a8
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - fp = 0xbe8fe61c sp = 0xbe8fe488 lr = 0x400b1fb5 pc = 0x400433dc
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - Found by: given as instruction pointer in context
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - 1 libutils.so + 0x1785c
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - sp = 0xbe8fe4a0 pc = 0x400b485e
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - Found by: stack scanning
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - 2 libbinder.so + 0x1bc2d
[task 2019-04-01T23:33:46.288Z] 23:33:46 INFO - sp = 0xbe8fe4d8 pc = 0x400f7c2f
[task 2019-04-01T23:33:46.289Z] 23:33:46 INFO - Found by: stack scanning
[task 2019-04-01T23:33:46.289Z] 23:33:46 INFO - 3 libbinder.so + 0x28dbe
[task 2019-04-01T23:33:46.289Z] 23:33:46 INFO - sp = 0xbe8fe4e4 pc = 0x40104dc0

This is starting to spike with the emulator version bump in bug 1556058, I'll take a look.

Seems to be hanging on this line - i.e. the promiseFocus wait is successful, but then the "all paints done" doesn't finish. It takes a lot of local --verify iterations to reproduce so debugging will take some time.

Seems like isMozAfterPaintPending is true but the paint event never fires. Or at least the paintListener function doesn't get triggered.

This bug also has not reached the disable-recommended phase; not sure why it's being disabled already. This is an important test for APZ test coverage on Android.

Botond, we proceeded to disable according to this discussion here: https://bugzilla.mozilla.org/show_bug.cgi?id=1553971#c21 We can backout or re-enable the test if it's important.

Yes, please back out the patch disabling the test. This is an important test and disabling it without consultation from the test owners is not really acceptable. Particularly since I was actively investigating the failure (see comment 17) and as Botond pointed out, the failure frequency had not reached the disable-recommended rate.

Backed out the patch disabling the test, changeset 1f905d44de1c:
https://hg.mozilla.org/integration/autoland/rev/8e8e47da345236e1335f1515781d485bb62ec2f1

As far as I can tell the test always times out waiting for a paint to happen, and shortly before that, there's this in the log:

07-09 16:30:26.366 11118 11175 I GeckoConsole: Renderer: Android Emulator OpenGL ES Translator (llvmpipe (LLVM 6.0, 256 bits))
07-09 16:30:26.366 11118 11175 I GeckoConsole: FBO Texture Target: TEXTURE_2D
07-09 16:30:26.368 11118 11175 E BufferQueueProducer: [SurfaceTexture-0-11118-18] connect: BufferQueue has been abandoned
07-09 16:30:26.368 11118 11175 E libEGL  : eglCreateWindowSurface: native_window_api_connect (win=0x7f48463d8010) failed (0xffffffed) (already connected to another API?)
07-09 16:30:26.368 11118 11175 E libEGL  : eglCreateWindowSurface:464 error 3003 (EGL_BAD_ALLOC)
07-09 16:30:26.368 11118 11175 I CompositorBridgeParent: Unable to renew compositor surface; remaining in paused state

which would explain why the paint never happens. It's not really clear to me why this happens, possibly a GeckoView bug, but it always seems to happen right after starting a new subtest. So maybe the code to launch a new subtest in a fresh window is cycling windows so fast that EGL or some other part of the pipeline gets confused about which surface is attached where. I'll see if inserting a small delay helps.

:snorp, does the log snippet in the last comment ring any bells? Do you think it might be a GV bug, or an emulator bug? Seems to happen intermittently when rapidly opening and closing new windows (tabs) in the org.mozilla.geckoview.test app.

It might be related, yes. It looks like we're not managing the lifetime of the SurfaceTexture backing the compositor for new windows in TestRunnerActivity. https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/TestRunnerActivity.java#141

We create a SurfaceTexture and associated Surface, but then nothing references the SurfaceTexture after that. If GC occurs, we could end up killing the SurfaceTexture, rendering the Surface useless.

Let's see if this helps: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d83caea76b88333b2bacdd0a60de5e7e2aa56741

Try is green.

Indeed. Let's put that patch on a new bug since it seems to help but may not fix all instances of this bug.

:snorp, could you land the patch using bug 1564846? I'm happy to review if you want

(In reply to Kartikaya Gupta (email:kats@mozilla.com) (away 17-Jul-2019 to Feb-2020) from comment #33)

:snorp, could you land the patch using bug 1564846? I'm happy to review if you want

Yup, done.

Thanks!

I'll leave this assigned to me for a few more days to keep an eye on the failure rate.

Looking pretty good so far, no new failures since snorp's patch landed. I'll unassign for now. I'm sure there will still be a trickle of failures that predated the emulator upgrade. If not then this bug should get closed by the husbandry bot eventually.

Unfortunately the patch in bug 1564846 was backed out for high failure rate of bug 1565072. https://bugzilla.mozilla.org/show_bug.cgi?id=1564846#c7

So even though bug 1564846 was backed out, the failure rate for this bug remains low. Not sure why. Maybe the failure moved to some other test in the interim.

Based on r/b: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Android%2C7.0%2Cx86-64%2Copt%2CMochitests%2Ctest-android-em-7.0-x86_64%2Fopt-geckoview-mochitest-e10s-4%2CM%284%29&tochange=1e655610cc11e0065e9c6d0865cffc6935c22d71&fromchange=ae8dbea0d35a5900b4194072953d3856bec1ecc9&selectedJob=271839335

it seems that bug 1579928 caused this failures.

Jaril, could you take a look?

Razvan, it seems unlikely that bug 1579928 could be the culprit. That patch only affects tests where firefox replay is enabled. This intermittent is also android specific which makes it further unlikely.

Hi Geoff, I continued retriggers here and could it be because Bug 1585119?

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitest&tochange=1e655610cc11e0065e9c6d0865cffc6935c22d71&fromchange=206c6ca088cbf938042044250682cf0eff47f0eb&selectedJob=272554618

Maybe? I did not change anything in gfx/layers/apz/ but my patch changed the number of test chunks and enabled tests in other directories. The test might be affected by other tests that run in the same task.

Recent failure:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=272509094&repo=mozilla-central&lineNumber=4110
https://taskcluster-artifacts.net/eybcbG-8ScmRv0_wf30lWw/2/public/test_info//logcat-emulator-5554.log

10-23 03:34:47.800  8745  8760 D GeckoViewLoadURIDelegate[C]: loadURI: uri=http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_basic_pan.html where=1 flags=0x0 tp=http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/test_group_touchevents.html
10-23 03:34:47.801  8677  8677 D GeckoSession: handleMessage GeckoView:OnLoadRequest uri=http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_basic_pan.html
10-23 03:34:47.803  8745  8760 D GeckoViewWebBrowserChrome[C]: shouldLoadURI http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_basic_pan.html
10-23 03:34:47.808  8745  8760 D GeckoViewContent[C]: handleEvent: pagehide
10-23 03:34:47.809  8745  8760 D GeckoViewAutofill: Clearing auto-fill
10-23 03:34:47.809  8677  8692 W GeckoEventDispatcher: No listener for GeckoView:ClearAutofill
10-23 03:34:47.810  8745  8760 I Gecko   : [Child 8745, Main Thread] WARNING: NS_ENSURE_TRUE(frame) failed: file /builds/worker/workspace/build/src/layout/base/nsPresContext.cpp, line 821
10-23 03:34:47.811  8677  8692 D GeckoViewNavigation: onLocationChange
10-23 03:34:47.812  8677  8677 D GeckoSession: handleMessage GeckoView:LocationChange uri=http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_basic_pan.html
10-23 03:34:47.815  8745  8760 D GeckoViewContent[C]: handleEvent: DOMTitleChanged
10-23 03:34:47.845  8677  8692 I GeckoDump: WaitUntilApzStable: flushed APZ repaints in parent proc, waiting for callback...
10-23 03:34:47.846  8745  8760 D GeckoViewContent[C]: handleEvent: DOMContentLoaded
10-23 03:34:47.847  8677  8692 I GeckoDump: WaitUntilApzStable: APZ flush done in parent proc
10-23 03:34:47.848  8745  8760 D GeckoViewContent[C]: handleEvent: pageshow
10-23 03:34:47.850  8745  8760 I GeckoDump: helper_basic_pan.html | WaitUntilApzStable: got apz-flush-done in child proc
10-23 03:34:47.857  8745  8760 I GeckoDump: helper_basic_pan.html | WaitUntilApzStable: done promiseFocus
10-23 03:34:47.857  8745  8760 I GeckoDump: helper_basic_pan.html | waiting for paint...
10-23 03:34:47.858  8745  8760 D GeckoViewContent[C]: handleEvent: MozFirstContentfulPaint
10-23 03:34:47.892  8677  8730 I Gecko   : Destroying context 0x74d3d781cb80 surface 0x0 on display 0x1
10-23 03:34:47.894  8677  8730 D EGL_emulation: eglMakeCurrent: 0x74d3d790a580: ver 2 0 (tinfo 0x74d3d78904a0)
10-23 03:34:48.071  8745  8760 I Gecko   : --DOMWINDOW == 37 (0x74d3b03efc00) [pid = 8745] [serial = 51] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/test_group_touchevents-2.html]
10-23 03:34:48.071  8745  8760 I Gecko   : --DOCSHELL 0x74d3af770800 == 5 [pid = 8745] [id = {10da8134-6aa3-420c-a229-7f0d940bab75}] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_bug1506497_touch_action_fixed_on_fixed.html]
10-23 03:34:48.071  8745  8760 I Gecko   : --DOMWINDOW == 36 (0x74d3b0578800) [pid = 8745] [serial = 64] [outer = 0x0] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 35 (0x74d3b04f0400) [pid = 8745] [serial = 54] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 34 (0x74d3b0328800) [pid = 8745] [serial = 57] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 33 (0x74d3b0484c00) [pid = 8745] [serial = 60] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 32 (0x74d3b056dc00) [pid = 8745] [serial = 63] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 31 (0x74d3b047d000) [pid = 8745] [serial = 76] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 30 (0x74d3ad379c00) [pid = 8745] [serial = 73] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 29 (0x74d3ad373000) [pid = 8745] [serial = 67] [outer = 0x0] [url = about:blank]
10-23 03:34:48.072  8745  8760 I Gecko   : --DOMWINDOW == 28 (0x74d3ad376800) [pid = 8745] [serial = 70] [outer = 0x0] [url = about:blank]
10-23 03:34:48.074  8745  8760 I Gecko   : --DOCSHELL 0x74d3ad3ad000 == 4 [pid = 8745] [id = {8a939ba6-0440-46e6-a86e-ec1470f5a7bf}] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_bug1473108.html]
10-23 03:34:48.074  8745  8760 I Gecko   : --DOCSHELL 0x74d3ad3a3000 == 3 [pid = 8745] [id = {8954350b-6ca6-4309-b86d-3b5071fca388}] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_bug1509575.html]
10-23 03:34:48.329  8745  8760 I Gecko   : --DOMWINDOW == 27 (0x74d3d1570880) [pid = 8745] [serial = 66] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_touch_action.html]
10-23 03:34:48.329  8745  8760 I Gecko   : --DOMWINDOW == 26 (0x74d3d156f5c0) [pid = 8745] [serial = 69] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_touch_action_complex.html]
10-23 03:34:48.329  8745  8760 I Gecko   : --DOMWINDOW == 25 (0x74d3d156f980) [pid = 8745] [serial = 72] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_touch_action_regions.html]
10-23 03:34:48.329  8745  8760 I Gecko   : --DOMWINDOW == 24 (0x74d3d1570a60) [pid = 8745] [serial = 75] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_touch_action_zero_opacity_bug1500864.html]
10-23 03:34:48.329  8745  8760 I Gecko   : --DOMWINDOW == 23 (0x74d3d1570c40) [pid = 8745] [serial = 80] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_bug1473108.html]
10-23 03:34:48.581  8677  8692 W GeckoEventDispatcher: No listener for GeckoView:StateUpdated
10-23 03:34:48.804  8677  8692 W GeckoEventDispatcher: No listener for GeckoView:StateUpdated
10-23 03:34:50.060  8677  8730 I Gecko   : Destroying context 0x74d3d7906200 surface 0x0 on display 0x1
10-23 03:34:50.184  8677  8692 I Gecko   : --DOCSHELL 0x74d3adfa0000 == 5 [pid = 8677] [id = {63b66bd0-fa75-418c-93b8-462904ab997f}] [url = chrome://geckoview/content/geckoview.xul]
10-23 03:34:50.184  8677  8692 I Gecko   : --DOCSHELL 0x74d3adf9c800 == 4 [pid = 8677] [id = {16d1648d-dfe7-4f37-a47c-2f88b5d004db}] [url = chrome://geckoview/content/geckoview.xul]
10-23 03:34:50.184  8677  8692 I Gecko   : nsWindow 0x74d3abb24400 destructor
10-23 03:34:50.184  8677  8692 I Gecko   : --DOCSHELL 0x74d3adf99800 == 3 [pid = 8677] [id = {96085e13-3bf4-484f-b7f4-bcac86a642ff}] [url = chrome://geckoview/content/geckoview.xul]
10-23 03:34:50.184  8677  8692 I Gecko   : nsWindow 0x74d3abb1c000 destructor
10-23 03:34:50.185  8677  8692 I Gecko   : nsWindow 0x74d3abb1a800 destructor
10-23 03:34:52.189  8745  8760 I Gecko   : --DOMWINDOW == 22 (0x74d3b03ad000) [pid = 8745] [serial = 74] [outer = 0x0] [url = about:blank]
10-23 03:34:52.189  8745  8760 I Gecko   : --DOMWINDOW == 21 (0x74d3ad379400) [pid = 8745] [serial = 68] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 20 (0x74d3af7d9800) [pid = 8745] [serial = 71] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 19 (0x74d3ad37c400) [pid = 8745] [serial = 84] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 18 (0x74d3b0483400) [pid = 8745] [serial = 65] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/test_group_touchevents-3.html]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 17 (0x74d3b0484400) [pid = 8745] [serial = 82] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 16 (0x74d3ad087000) [pid = 8745] [serial = 81] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 15 (0x74d3ad07bc00) [pid = 8745] [serial = 78] [outer = 0x0] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 14 (0x74d3b04f3400) [pid = 8745] [serial = 77] [outer = 0x0] [url = about:blank]
10-23 03:34:52.190  8745  8760 I Gecko   : --DOMWINDOW == 13 (0x74d3aee8c000) [pid = 8745] [serial = 87] [outer = 0x0] [url = about:blank]
10-23 03:34:52.230  8745  8760 I Gecko   : --DOMWINDOW == 12 (0x74d3d1570e20) [pid = 8745] [serial = 83] [outer = 0x0] [url = http://mochi.test:8888/tests/gfx/layers/apz/test/mochitest/helper_bug1509575.html]
10-23 03:34:52.546  8677  8730 I Gecko   : Destroying context 0x74d3d7906480 surface 0x0 on display 0x1
10-23 03:34:56.312  8745  8760 I Gecko   : --DOMWINDOW == 11 (0x74d3b03fcc00) [pid = 8745] [serial = 89] [outer = 0x0] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
10-23 03:34:56.312  8745  8760 I Gecko   : --DOMWINDOW == 10 (0x74d3aee8fc00) [pid = 8745] [serial = 85] [outer = 0x0] [url = about:blank]
10-23 03:34:56.312  8745  8760 I Gecko   : --DOMWINDOW == 9 (0x74d3b03ad800) [pid = 8745] [serial = 92] [outer = 0x0] [url = about:blank]

snorp - Any chance of getting this fixed, or should we disable?

I looked at one point and for some reason it's super racy with child process startup. We can probably just disable for now.

https://hg.mozilla.org/mozilla-central/rev/7f71e6babc36

To make tracking easier, let's close this and track re-enablement in a follow-up (filed bug 1591519 for it).

I do think this is an important one to re-enable for APZ test coverage.

https://hg.mozilla.org/releases/mozilla-beta/rev/1d82f36f1930

https://treeherder.mozilla.org/logviewer?job_id=327042601&repo=autoland&lineNumber=5573

:botond - You re-enabled this test in bug 1591519. Have a look at the recent failure?

I'm pretty swamped with higher-priority work right now, so unless this crosses the disable-recommended threshold, I'm going to have to put it off. We have dozens of low-frequency APZ intermittents caused by a variety of issues.

Also it would be better to keep this bug closed and file a new one for the new failure. It gets really confusing when bugs with patches landed are reopened long after the fact.