Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=341017804&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OQ9lrENzR9S51j1Xynm6IQ/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OQ9lrENzR9S51j1Xynm6IQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2021-05-27T22:45:48.112Z] 22:45:48     INFO - TEST-START | dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayWithStun300.html
[task 2021-05-27T22:45:49.513Z] 22:45:49     INFO - GECKO(10656) | TEST DEVICES: No test device found in media.audio_loopback_dev, using fake audio streams.
[task 2021-05-27T22:45:49.513Z] 22:45:49     INFO - GECKO(10656) | TEST DEVICES: No test device found in media.video_loopback_dev, using fake video streams.
[task 2021-05-27T22:45:49.839Z] 22:45:49     INFO - TEST-INFO | started process screenshot
[task 2021-05-27T22:45:49.938Z] 22:45:49     INFO - TEST-INFO | screenshot: exit 0
[task 2021-05-27T22:45:49.938Z] 22:45:49     INFO - Buffered messages logged at 22:45:49
[task 2021-05-27T22:45:49.938Z] 22:45:49     INFO - TEST-PASS | dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayWithStun300.html | A valid string reason is expected 
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - TEST-PASS | dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayWithStun300.html | Reason cannot be empty 
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - websocket/process bridge open, starting ICE Server...
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - ICE Server: Traceback (most recent call last):
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - Buffered messages finished
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - TEST-UNEXPECTED-FAIL | dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayWithStun300.html | Error executing test: SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO - setupIceServerConfig/<@http://mochi.test:8888/tests/dom/media/webrtc/tests/mochitests/pc.js:2491:30
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO -  
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:417:16
[task 2021-05-27T22:45:49.939Z] 22:45:49     INFO -     runTestWhenReady@dom/media/webrtc/tests/mochitests/head.js:483:7
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - GECKO(10656) | MEMORY STAT | vsize 591MB | vsizeMaxContiguous 1293MB | residentFast 59MB | heapAllocated 4MB
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server:   File "iceserver\iceserver.py", line 25, in <module>
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server:     from twisted.internet.address import IPv6Address
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server: ImportError: cannot import name IPv6Address
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ProcessTerminated'>: A process has ended with a probable error condition: process ended with exit code 1.
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server: ]
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - ICE Server websocket closed
[task 2021-05-27T22:45:49.940Z] 22:45:49     INFO - TEST-OK | dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayWithStun300.html | took 1773ms

Is there something about the environment on the testers running TV tasks that is different than the other testers? Maybe the version of twisted we're using for python 2.7 is buggy somehow when importing IPv6Address?

They should be using the same workers as regular tests.. bizarre. We can add skip-if = verify to these tests to bypass if investigation hits a road block.

Besides twisted is installed at task runtime and in the logs I see twisted==10.2.0 installed successfully.

Set release status flags based on info from the regressing bug 857668

So there's definitely a python difference here. Both of these pushes were based on the same revision:

Works fine, was able to install iceserver's deps: https://treeherder.mozilla.org/jobs?repo=try&revision=4a1b5f5af399f0500a03a17a8ca5b723f520e06e

Busted, cannot install deps: https://treeherder.mozilla.org/jobs?repo=try&revision=49419f976423004f1299869a256f789f14fa0280

It seems that TV is using python2.7, whereas the regular mochitests are using python3. The most recent version of twisted in our pypi that can be used with python 2.7 is 10.2.0, which apparently sometimes fails to import twisted.IPv6Address. I do not know if there is a python2.7 friendly version of twisted without this bug. We probably want to be running TV on python3 though, right?

Good find! Yes, we should be using Python 3 with TV tasks. But there are still a handful of other suites still left to switch over (like xpcshell and reftest). We could likely switch TV over prior to these (as they are both Py3 compatible at least). Just a matter of someone finding the time to do it.

Given this, I definitely recommend using skip-if = verify to unblock yourself.

Is there a bug for getting TV jobs to use python3?

https://treeherder.mozilla.org/jobs?repo=try&revision=45a487ee47bdf54a0847f8986b30e87c2f7a3127

I had filed a JIRA ticket, but just filed https://bugzilla.mozilla.org/show_bug.cgi?id=1714406 as well.

https://hg.mozilla.org/mozilla-central/rev/67e1f353840d

The patch landed in nightly and beta is affected.
:bwc, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

If we see frequent failures on beta, we could uplift, but I don't think that is the case.