Filed by: archaeopteryx [at] coole-files.de https://treeherder.mozilla.org/logviewer.html#?job_id=159874738&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/Ti8COsHsQjOT0dh3Iin0RQ/runs/0/artifacts/public/logs/live_backing.log

When this happens, we get a raft of other failures afterward. It seems that something in this test breaks some part of Firefox in a way that is not recoverable. No outbound or inbound RTP for video seems to be happening. It does not seem to matter whether we're using fake video or not. Maybe we're deadlocking somewhere in the video path? 17843 06:30:19 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_basicScreenshare.html | Error in test execution: Error: Timeout checking for stats for track {17fa857c-b0cc-f44d-8305-633a3ddbf6dc} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest 18256 06:30:50 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_basicVideo.html | Error in test execution: Error: Timeout checking for stats for track {f4633628-ac47-2047-9417-dfd463a62345} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js 18660 06:31:21 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_basicVideoVerifyRtpHeaderExtensions.html | Error in test execution: Error: Timeout checking for stats for track {dd8f1393-21fd-964f-bedd-08a70ae867be} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/me 19058 06:31:53 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_basicWindowshare.html | Error in test execution: Error: Timeout checking for stats for track {15e9a964-64a9-bc47-9d6f-6be63333c08f} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest 20327 06:32:58 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html | Error in test execution: Error: Timeout for element _remote_{05d6cdaf-422c-7841-96d2-1e4b2f030e3c} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595:7 ... PC_ 20784 06:33:59 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d_noSSRC.html | Error in test execution: Error: Timeout for element _remote_{b7aa1e33-41b8-c44b-814d-996aeffe25c4} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595:7 20888 06:33:59 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_capturedVideo.html | seek.webm failed in playback (code=3). Stream should be OK. Continuing test. 22705 06:34:30 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_capturedVideo.html | Error in test execution: Error: Timeout checking for stats for track {29de5058-1d14-7845-a75a-8bb2f589bd2c} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc 24458 06:35:31 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_capturedVideo.html | Error in test execution: Error: Timeout for element _remote_{86bd4f5c-9bff-f841-a2b1-aeefb57ae0a0} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595:7 ... PC_REMOTE_WAI 24985 06:35:53 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_checkPacketDumpHook.html | Error in test execution: Error: Timeout waiting for rtp recv packet on level 1 timeout/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:572:63 ... promise callback*timeout@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:572:26 ... waitForRecvPacket@http://mochi.test:8888/tests/dom/media/tests/mochitest/test_peerConnection_checkPacketDumpHook.html:43:1 25805 06:36:26 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_constructedStream.html | Error in test execution: Error: Timeout checking for stats for track {91ffc767-2989-0f49-9007-6f3f6848c804} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochites 26546 06:36:58 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_forwarding_basicAudioVideoCombined.html | Error in test execution: Error: Timeout checking for stats for track {5ae36af0-15d6-1b49-923c-0c3cbdd86652} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/med 27588 06:37:29 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_forwarding_basicAudioVideoCombined.html | Error in test execution: Error: Timeout checking for stats for track {3e0e62c8-6d7c-cd4f-b28b-3f641d176b47} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/med 29358 06:38:44 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_multiple_captureStream_canvas_2d.html | Error in test execution: Error: Timeout for element _remote_{e171895f-20f3-fa43-a159-ee70cd379ac6} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595: 30521 06:39:19 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_offerRequiresReceiveVideo.html | Error in test execution: Error: Timeout checking for stats for track {0793a9fb-b890-1343-9001-b0559b15c6e5} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/ 31077 06:39:51 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_offerRequiresReceiveVideoAudio.html | Error in test execution: Error: Timeout checking for stats for track {58f6759c-8f76-ef4d-b5d9-26a9e869c1a4} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/t 33233 06:40:32 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_removeThenAddVideoTrack.html | Error in test execution: Error: Timeout checking for stats for track {9ca9c563-300c-4d45-ae40-abf342f689bc} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mo 33635 06:41:03 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_removeThenAddVideoTrackNoBundle.html | Error in test execution: Error: Timeout checking for stats for track {922df05f-ca48-2949-8da8-4b9536ec7ab5} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/ 34039 06:41:34 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_removeVideoTrack.html | Error in test execution: Error: Timeout checking for stats for track {b5bbd8ba-70a9-7a49-a2f7-2386b4ae6a37} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest 34998 06:46:43 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_renderAfterRenegotiation.html | Test timed out. 35551 06:47:16 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Error in test execution: Error: Timeout checking for stats for track {592a83e4-40f8-504c-9290-ad147f54f211} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc. 35990 06:47:47 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_replaceVideoThenRenegotiate.html | Error in test execution: Error: Timeout checking for stats for track {1eea32d9-bb5d-094d-8482-85be8c35384b} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/test 36623 06:48:18 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restartIce.html | Error in test execution: Error: Timeout checking for stats for track {37afde50-4ec1-4545-9d17-f37c86b9a024} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js 38219 06:48:54 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restartIceNoBundle.html | Error in test execution: Error: Timeout checking for stats for track {8627edf9-b888-d642-9035-320ace9ca77c} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochite 39080 06:49:25 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restartIceNoBundleNoRtcpMux.html | Error in test execution: Error: Timeout checking for stats for track {f225fd7a-3755-274a-aa09-a9daeef07d0b} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/test 39814 06:49:57 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restartIceNoRtcpMux.html | Error in test execution: Error: Timeout checking for stats for track {69ac3127-4145-8b4d-bedf-7382bad9d478} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochit 40237 06:50:28 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restrictBandwidthTargetBitrate.html | Error in test execution: Error: Timeout checking for stats for track {f5a28027-2a71-454a-842e-9f6178818418} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/t 40640 06:50:59 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_restrictBandwidthWithTias.html | Error in test execution: Error: Timeout checking for stats for track {7867b4f3-1a07-db44-b324-da972dcf4a49} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/ 41427 06:56:13 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_scaleResolution.html | Test timed out. 41961 06:56:46 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_sender_and_receiver_stats.html | Error in test execution: Error: Timeout checking for stats for track {d27694c4-38e1-d845-a76e-dfee55e74566} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/ 42543 06:57:18 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_setParameters.html | Error in test execution: Error: Timeout checking for stats for track {0bc606d7-15ce-0345-befd-f08375d172a2} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc 43109 06:57:50 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html | Error in test execution: Error: Timeout checking for stats for track {6f685fd9-d76b-2848-a8b5-998e97367ec7} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/ 43743 06:58:52 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_simulcastOffer.html | Error in test execution: Error: Timeout for element _remote_{ec892bb3-86af-3a48-9ab3-f6d029f04079} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595:7 ... PC_REMOTE_WA 44371 06:59:23 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_stats.html | Error in test execution: Error: Timeout checking for stats for track {7dbf858f-cfc3-4342-aa3f-60c0474de02b} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600 44791 06:59:55 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_syncSetDescription.html | Error in test execution: Error: Timeout checking for stats for track {56e32192-7dcd-4649-b4fa-602e8407bf8e} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochite 45799 07:00:27 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_trackDisabling.html | Error in test execution: Error: Timeout checking for stats for track {06b34df2-d518-3e4f-829d-777dc1654451} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/p 46317 07:00:58 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_trackDisabling_clones.html | Error in test execution: Error: Timeout checking for stats for track {15572941-f014-8247-9059-dbb9f2abce25} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/moch 50149 07:06:13 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_transceivers.html | Test timed out. 51623 07:06:48 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_twoAudioVideoStreams.html | Error in test execution: Error: Timeout checking for stats for track {265afc0a-be5e-8d47-8f22-4c6ae18b1e5f} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochi 52650 07:07:19 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_twoAudioVideoStreamsCombined.html | Error in test execution: Error: Timeout checking for stats for track {a1bed664-2794-6947-aabf-0cece22ad311} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tes 53197 07:07:51 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_twoVideoStreams.html | Error in test execution: Error: Timeout checking for stats for track {cc320bab-eb90-1941-99fa-d55b198a7127} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/ 53707 07:08:22 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_twoVideoTracksInOneStream.html | Error in test execution: Error: Timeout checking for stats for track {2cbaa23f-4687-5341-8f55-e5c84d641d75} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1600:58 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/ 54621 07:09:28 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_verifyVideoAfterRenegotiation.html | Error in test execution: Error: Timeout for element _remote_{da9cc4ad-9364-db43-9a5c-5e581f72ef18} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595:7 . 55003 07:09:59 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_videoCodecs.html | Error in test execution: Error: Timeout checking for stats for track {6d929b7f-1ac4-8044-8e2a-9f74a0bec4b8} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.j 55810 07:10:30 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_videoCodecs.html | Error in test execution: Error: Timeout checking for stats for track {c9b20606-b867-5442-84e6-3c3775d84bef} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.j 56604 07:11:01 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_videoCodecs.html | Error in test execution: Error: Timeout checking for stats for track {f7fa00c4-3b81-8a48-a84d-8220de7bd8bf} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.j 57403 07:11:32 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_videoCodecs.html | Error in test execution: Error: Timeout checking for stats for track {75e5398b-3dca-714a-8079-f38bf3167ef0} after at least30000ms waitForRtpFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1548:11 ... async*waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1601:49 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.j 57994 07:12:33 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_videoRenegotiationInactiveAnswer.html | Error in test execution: Error: Timeout for element _remote_{be0a88d3-3f05-1849-a73a-496d0057f43e} waitForMediaElementFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1495:13 ... waitForMediaFlow/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1599:22 ... waitForMediaFlow@http://mochi.test:8888/tests/dom/media/tests/mochitest/pc.js:1595

I'm going to try to figure out what push started this mess. Sadly, that's going to require lots of OS X retriggers. Sorry in advance.

First occurrence is here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0be3532802e3bad1ad91ee635a6b7abb902d3d27&selectedJob=159874738 Checking this push now: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=05f077d8f612a29db4495988e5b1fbafc8d12263

Seems to have started in this range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=48e318052aaa7cf995892f171dc697b2a0c9e3fc Now checking here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=48e318052aaa7cf995892f171dc697b2a0c9e3fc&selectedJob=159674546

Sorry, range should be: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=0be3532802e3bad1ad91ee635a6b7abb902d3d27&fromchange=05f077d8f612&selectedJob=159674546

Narrowed to range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=48e318052aaa7cf995892f171dc697b2a0c9e3fc&fromchange=05f077d8f612&selectedJob=164653064 Now trying: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2bda1bbb8d22eacccdb4ca5a997b32507a1f9f9a&selectedJob=159601268

Narrowed to range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=48e318052aaa7cf995892f171dc697b2a0c9e3fc&fromchange=2bda1bbb8d22eacccdb4ca5a997b32507a1f9f9a&selectedJob=159640370 Now trying: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=d3713e2970775bc22ac6c37d9339383c72632da3

Narrowed to range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=48e318052aaa7cf995892f171dc697b2a0c9e3fc&fromchange=d3713e2970775bc22ac6c37d9339383c72632da3&selectedJob=159656355 Now trying: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=1d3724ca38b1855d7348e53970d855cbf4df8faf&selectedJob=159656355

It is very likely that this started with this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=145ff65a890becba6c258e88c88916f9fc00ef42

Oh fun. I'm not gonna have time to look at this for a while, but if you do I'm happy to provide some log modules for narrowing it down. It would also be interesting to know if it fails when run standalone on try.

In the failure case, I see IProtocol::AllocShmem start failing when the basicScreenshare test runs, and it never succeeds again.

Also notable is that the first time it fails, it takes a minute to do so, whereas subsequent failures happen basically right away.

I think the reason this is failing is because the screenshare size limit got larger in bug 1299515, and the larger buffers required are frequently causing IPC Shmem on OS X to break. I am trying to test this hypothesis now.

It does not seem to be caused by larger buffer sizes. IPC Shmem is breaking though, and never recovers. Trying to figure out why...

IPC Shmem is breaking here: https://searchfox.org/mozilla-central/source/ipc/glue/SharedMemoryBasic_mach.mm#633 I'm not seeing any glaring difference in terms of how many Shmems CamerasParent is creating, nor how large these Shmems tend to be.

I'm going to check if we are failing to destroy Shmems, or not destroying them quickly enough.

When comparing the pre-regression code to the post-regression code, I do not see larger shmems allocated, or more bytes of shmem allocated at a time, or a larger number of shmems allocated at a time. This does not seem to be caused by any kind of increased shmem usage/load. Perhaps we are exceeding some other system resource?

I've tried reducing the IPC timeout (which is how these failures happen) from 60 seconds to 10 on the pre-regression changeset, and it still works reliably.

Have you been able to look at this recently, Byron?

I've been busy with the h264 related oranges lately, but now will have some time to look at this again.

I have been unable to reproduce this problem on a loaner. I wonder if only certain machines have this problem?

Hmm, so, bug 1299515 (the suspected regressor) did change so two gUM requests to capture the screen will request two captures at the backend that get shipped to the child over IPC, instead of one at the backend that is split up in the child. The test looks like it is using two separate gUM screen requests, so there should be higher churn of shmems than before. Byron, thoughts on the way forward? 1) Make the test one-way or use/implement something that either 2) shares these shmems across processes with some sort of ipc-based refcounts, or 3) does the split in the child instead of the parent, more like before. Note that for 3, we'd still not share images across multiple children, so that solution would result in more paths and complexity than before.

1) Would certainly be worth a try, if the alternative is disabling the test. 2) IPC-based refcounts strikes me as difficult, but we may already have something like this. 3) This seems ideal from a performance perspective, but I do not know how ugly the resulting code would be. We should probably verify that this is indeed a performance problem, perhaps by implementing 1.

Hmm, do we need to do the same for test_pc_basicWindowshare.html that comes directly after? https://treeherder.mozilla.org/#/jobs?repo=try&revision=4f329e25aa6c&selectedJob=180717840

Comment on attachment 8983400 [details] Bug 1434983: Make screenshare and windowshare test one-way, to work around limitations on OS X. https://reviewboard.mozilla.org/r/249314/#review255492

Pushed by bcampen@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0a06f4f9bd73 Make screenshare and windowshare test one-way, to work around limitations on OS X. r=pehrsons