tests are running ok in our current 10.10 model, but are failing in 10.14.

here is a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a86266650e629f2384a1e979ebf0df8c7fc3415f&selectedJob=246408740

a log file:
https://taskcluster-artifacts.net/KaK_fe0WTkilSzh0AbR4MA/0/public/logs/live_backing.log

and a clip from the log file:
[task 2019-05-14T15:41:56.091Z] 0:41.11 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
[task 2019-05-14T15:41:56.092Z] 0:41.11 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_unicode_in_environment PASSED
[task 2019-05-14T15:41:56.092Z] 0:41.11
[task 2019-05-14T15:41:56.092Z] 0:41.11 =================================== FAILURES ===================================
[task 2019-05-14T15:41:56.092Z] 0:41.11 __________________ ProcTestMisc.test_process_timeout_no_kill ___________________
[task 2019-05-14T15:41:56.092Z] 0:41.11
[task 2019-05-14T15:41:56.092Z] 0:41.11 self = <test_misc.ProcTestMisc testMethod=test_process_timeout_no_kill>
[task 2019-05-14T15:41:56.092Z] 0:41.11
[task 2019-05-14T15:41:56.092Z] 0:41.11 def test_process_timeout_no_kill(self):
[task 2019-05-14T15:41:56.092Z] 0:41.11 """ Process is started, runs but we time out waiting on it
[task 2019-05-14T15:41:56.092Z] 0:41.11 to complete. Process should not be killed.
[task 2019-05-14T15:41:56.092Z] 0:41.11 """
[task 2019-05-14T15:41:56.092Z] 0:41.11 p = None
[task 2019-05-14T15:41:56.092Z] 0:41.11
[task 2019-05-14T15:41:56.092Z] 0:41.11 def timeout_handler():
[task 2019-05-14T15:41:56.092Z] 0:41.11 self.assertEqual(p.proc.poll(), None)
[task 2019-05-14T15:41:56.092Z] 0:41.11 p.kill()
[task 2019-05-14T15:41:56.092Z] 0:41.11 p = processhandler.ProcessHandler([self.python, self.proclaunch,
[task 2019-05-14T15:41:56.092Z] 0:41.11 "process_waittimeout.ini"],
[task 2019-05-14T15:41:56.092Z] 0:41.11 cwd=here,
[task 2019-05-14T15:41:56.092Z] 0:41.11 onTimeout=(timeout_handler,),
[task 2019-05-14T15:41:56.092Z] 0:41.11 kill_on_timeout=False)
[task 2019-05-14T15:41:56.092Z] 0:41.11 p.run(timeout=1)
[task 2019-05-14T15:41:56.092Z] 0:41.11 p.wait()
[task 2019-05-14T15:41:56.092Z] 0:41.11 > self.assertTrue(p.didTimeout)
[task 2019-05-14T15:41:56.092Z] 0:41.11 E AssertionError: False is not true
[task 2019-05-14T15:41:56.093Z] 0:41.11
[task 2019-05-14T15:41:56.093Z] 0:41.11 testing/mozbase/mozprocess/tests/test_misc.py:35: AssertionError
[task 2019-05-14T15:41:56.093Z] 0:41.11 ----------------------------- Captured stdout call -----------------------------
[task 2019-05-14T15:41:56.093Z] 0:41.11 Traceback (most recent call last):
[task 2019-05-14T15:41:56.093Z] 0:41.11 File "/Users/cltbld/tasks/task_1557847339/checkouts/gecko/testing/mozbase/mozprocess/tests/proclaunch.py", line 10, in <module>
[task 2019-05-14T15:41:56.093Z] 0:41.11 from six.moves import configparser
[task 2019-05-14T15:41:56.093Z] 0:41.11 ModuleNotFoundError: No module named 'six'
[task 2019-05-14T15:41:56.093Z] 0:41.11 ====================== 1 failed, 1 passed in 0.32 seconds ======================

I don't know what six.moves is for, it looks to be something related to formatting the output of stdout maybe? it is common on all of the failures in the log file.

:whimboo, I see you authored the patch to add these tests to the tree, did you see similar errors in the past? Do you have any ideas of how to fix these errors?

ModuleNotFoundError: No module named 'six'

mozprocess/setup.py doesn't explicitly list six as a dependency, but it does list mozinfo as a dependency; mozinfo/setup.py lists mozfile as a dependency; mozfile/setup.py lists six as a dependency:

    install_requires=["six >= 1.10.0"],

I have verified that running mozbase/setup_development.py installs six.

thanks :gbrown, I wonder if we are doing something odd with different python package locations (py2 vs py3) and osx. I also see that we are using virtualenv-clone which sounds awesome, maybe it needs some tweaks.

Six is getting installed correctly:

• ./mach python-test --python /usr/local/bin/python3 --subsuite mozbase
  [..]
  [task 2019-05-14T15:41:28.644Z] Processing ./third_party/python/six
  [task 2019-05-14T15:41:29.101Z] Installing collected packages: six
  [task 2019-05-14T15:41:29.102Z] Running setup.py install for six: started
  [task 2019-05-14T15:41:29.402Z] Running setup.py install for six: finished with status 'done'
  [task 2019-05-14T15:41:29.420Z] Successfully installed six-1.10.0

Is this for Python3 only or does it also affect Python27 jobs?

Beside that the following looks strange:

> [task 2019-05-14T15:41:14.585Z] + ./mach python-test --python /usr/local/bin/python3 --subsuite mozbase
[task 2019-05-14T15:41:16.283Z] New python executable in /Users/cltbld/tasks/task_1557847339/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/bin/python
[task 2019-05-14T15:41:18.499Z] Installing setuptools, pip, wheel...done.
[task 2019-05-14T15:41:21.008Z] running build_ext
[task 2019-05-14T15:41:21.008Z] building 'psutil._psutil_osx' extension
[task 2019-05-14T15:41:21.008Z] creating build
[task 2019-05-14T15:41:21.008Z] creating build/temp.macosx-10.14-intel-2.7
[task 2019-05-14T15:41:21.008Z] creating build/temp.macosx-10.14-intel-2.7/psutil
[task 2019-05-14T15:41:21.008Z] creating build/temp.macosx-10.14-intel-2.7/psutil/arch
[task 2019-05-14T15:41:21.008Z] creating build/temp.macosx-10.14-intel-2.7/psutil/arch/osx
[task 2019-05-14T15:41:21.008Z] cc -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_OSX=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c psutil/_psutil_common.c -o build/temp.macosx-10.14-intel-2.7/psutil/_psutil_common.o

We run with Python3 but the compilation of psutil uses the 2.7 framework?

Andrew might be a better person to answer that given that I wasn't helping with the Python3 conversation yet.

the python 2.7 tests run fine on the new OS images, it is just the python3 mozbase tests that have trouble.

Note that test spawns a subprocess which invokes the proclaunch binary (which is where the ImportError happens). So six is likely available in the host environment and this is an issue with the test. Also note that six is vendored in mozilla-central.

osx 10.14 py3 mozbase seems to be running fine currently on mozilla-central.