Richard Pappalardo [:rpapa][:rpappalardo] Reporter
	Description • 6 years ago

STAGE and PROD tests failing last several days with following error

09:04:31     from mozdevice import DeviceManagerADB, DroidADB
09:04:31 ImportError: cannot import name DeviceManagerADB

	Mark Hammond [:markh] [:mhammond]
	Comment 1 • 6 years ago

It looks like this slipped though the cracks in bug 1440714, but that bug should offer some clues how to fix it.

	Geoff Brown [:gbrown]
	Comment 2 • 6 years ago

Right, that bug deleted DeviceManagerADB from mozdevice and switched all in-tree consumers of mozdevice to use ADBDevice or similar. I'm not familiar with TPS test: Does this code live outside of mozilla-central? Or is it possibly running pre-bug-1440714 test code with a post-bug-1440714 mozdevice?? 

Happy to answer questions or help here if I can...

	Thom Chiovoloni [:tcsc] (ex-moco) Assignee
	Comment 3 • 6 years ago

This fell off my radar, patch incoming.

	Thom Chiovoloni [:tcsc] (ex-moco) Assignee
	Comment 4 • 6 years ago

Hm, there's some issue when I try to change the setup.py to list 7.0 or 7.0.1 as the dependency version. I think possibly the dependency lists in mozprofile's setup.py is too involved https://searchfox.org/mozilla-central/source/testing/mozbase/mozprofile/setup.py for my version of python? But I can't seem to get that updated, so maybe someone else can give a better guess as to what's happening

(In particular, I get `error: Setup script exited with error in mozrunner setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers` when I change `'mozrunner == 6.15'` to `'mozrunner == 7.0.1'`, or really anything that seems like it would resolve to that (> 6.15, etc)). 

@gbrown: the TPS python driver is in testing/tps, but it doesn't currently run in the same venv as everything else.

I'll upload a patch that can unstick e.g. kitcambridge who was trying to get TPS running, but not something we want to land.

	Geoff Brown [:gbrown]
	Comment 6 • 6 years ago

STR:

gbrown@mozpad:~/src/testing/tps$ python create_venv.py gbrown
gbrown@mozpad:~/src/testing/tps$ source gbrown/bin/activate
(gbrown)gbrown@mozpad:~/src/testing/tps$ runtps --help

Traceback (most recent call last):
  File "/home/gbrown/src/testing/tps/gbrown/bin/runtps", line 9, in <module>
    load_entry_point('tps==0.5', 'console_scripts', 'runtps')()
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/pkg_resources.py", line 356, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/pkg_resources.py", line 2431, in load_entry_point
    return ep.load()
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
    ['__name__'])
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/tps-0.5-py2.7.egg/tps/__init__.py", line 5, in <module>
    from . firefoxrunner import TPSFirefoxRunner
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/tps-0.5-py2.7.egg/tps/firefoxrunner.py", line 13, in <module>
    from mozrunner import FirefoxRunner
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/mozrunner-6.15-py2.7.egg/mozrunner/__init__.py", line 8, in <module>
    from .cli import *
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/mozrunner-6.15-py2.7.egg/mozrunner/cli.py", line 12, in <module>
    from .application import get_app_context
  File "/home/gbrown/src/testing/tps/gbrown/local/lib/python2.7/site-packages/mozrunner-6.15-py2.7.egg/mozrunner/application.py", line 12, in <module>
    from mozdevice import DeviceManagerADB, DroidADB
ImportError: cannot import name DeviceManagerADB

	Lina Butler [:lina]
	Comment 7 • 6 years ago

(In reply to Thom Chiovoloni [:tcsc] from comment #4)
> Hm, there's some issue when I try to change the setup.py to list 7.0 or
> 7.0.1 as the dependency version. I think possibly the dependency lists in
> mozprofile's setup.py is too involved
> https://searchfox.org/mozilla-central/source/testing/mozbase/mozprofile/
> setup.py for my version of python? But I can't seem to get that updated, so
> maybe someone else can give a better guess as to what's happening
> 
> (In particular, I get `error: Setup script exited with error in mozrunner
> setup command: 'install_requires' must be a string or list of strings
> containing valid project/version requirement specifiers` when I change
> `'mozrunner == 6.15'` to `'mozrunner == 7.0.1'`, or really anything that
> seems like it would resolve to that (> 6.15, etc)).

I get the same error if I try to manually install mozrunner in the venv. I thought pip might be out of date (`pip --version` reports `pip 1.5.6 from /Users/k/tps-venv/lib/python2.7/site-packages (python 2.7)`, and the current version is 10.0.1), running `pip install -U pip` in the venv, then `pip install mozrunner`, still gives me the same error.

	Geoff Brown [:gbrown]
	Comment 8 • 6 years ago

I would think a simple mozrunner version update is all that is required:

diff --git a/testing/tps/setup.py b/testing/tps/setup.py
--- a/testing/tps/setup.py
+++ b/testing/tps/setup.py
@@ -10,17 +10,17 @@ version = '0.5'
 
 deps = ['httplib2 == 0.9.2',
         'mozfile == 1.2',
         'mozhttpd == 0.7',
         'mozinfo == 0.10',
         'mozinstall == 1.15',
         'mozprocess == 0.26',
         'mozprofile == 1.0.0',
-        'mozrunner == 6.15',
+        'mozrunner == 7.0.1',
         'mozversion == 1.5',
        ]
 
 # we only support python 2.6+ right now
 assert sys.version_info[0] == 2
 assert sys.version_info[1] >= 6
 
 setup(name='tps',


But as :tcsc reported, that fails:

gbrown@mozpad:~/src/testing/tps$ python create_venv.py gbrown
Downloading virtualenv 1.11.6
Creating new virtual environment
New python executable in gbrown/bin/python
Installing setuptools, pip...done.
running install
running bdist_egg
running egg_info
writing requirements to tps.egg-info/requires.txt
writing tps.egg-info/PKG-INFO
writing top-level names to tps.egg-info/top_level.txt
writing dependency_links to tps.egg-info/dependency_links.txt
writing entry points to tps.egg-info/entry_points.txt
reading manifest file 'tps.egg-info/SOURCES.txt'
writing manifest file 'tps.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/tps
copying build/lib.linux-x86_64-2.7/tps/phase.py -> build/bdist.linux-x86_64/egg/tps
copying build/lib.linux-x86_64-2.7/tps/cli.py -> build/bdist.linux-x86_64/egg/tps
copying build/lib.linux-x86_64-2.7/tps/firefoxrunner.py -> build/bdist.linux-x86_64/egg/tps
copying build/lib.linux-x86_64-2.7/tps/__init__.py -> build/bdist.linux-x86_64/egg/tps
copying build/lib.linux-x86_64-2.7/tps/testrunner.py -> build/bdist.linux-x86_64/egg/tps
byte-compiling build/bdist.linux-x86_64/egg/tps/phase.py to phase.pyc
byte-compiling build/bdist.linux-x86_64/egg/tps/cli.py to cli.pyc
byte-compiling build/bdist.linux-x86_64/egg/tps/firefoxrunner.py to firefoxrunner.pyc
byte-compiling build/bdist.linux-x86_64/egg/tps/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/tps/testrunner.py to testrunner.pyc
installing package data to build/bdist.linux-x86_64/egg
running install_data
copying config/config.json.in -> build/bdist.linux-x86_64/egg/tps
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying tps.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/tps-0.5-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing tps-0.5-py2.7.egg
creating /home/gbrown/src/testing/tps/gbrown/lib/python2.7/site-packages/tps-0.5-py2.7.egg
Extracting tps-0.5-py2.7.egg to /home/gbrown/src/testing/tps/gbrown/lib/python2.7/site-packages
Adding tps 0.5 to easy-install.pth file
Installing runtps script to /home/gbrown/src/testing/tps/gbrown/bin

Installed /home/gbrown/src/testing/tps/gbrown/lib/python2.7/site-packages/tps-0.5-py2.7.egg
Processing dependencies for tps==0.5
Searching for mozversion==1.5
Reading https://pypi.python.org/simple/mozversion/
Best match: mozversion 1.5
Downloading https://files.pythonhosted.org/packages/5c/43/ff7bd757e82e8daa9c3ccd1ff17647f6b6801d154b155db535076023c700/mozversion-1.5.tar.gz#sha256=e9b11e4a46bf7a4a11469ea4589c75f3ba50b34b7801e7edf1a09147af8bf70f
Processing mozversion-1.5.tar.gz
Writing /tmp/easy_install-M_Mk_R/mozversion-1.5/setup.cfg
Running mozversion-1.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-M_Mk_R/mozversion-1.5/egg-dist-tmp-51io0l
Adding mozversion 1.5 to easy-install.pth file
Installing mozversion script to /home/gbrown/src/testing/tps/gbrown/bin

Installed /home/gbrown/src/testing/tps/gbrown/lib/python2.7/site-packages/mozversion-1.5-py2.7.egg
Searching for mozrunner==7.0.1
Reading https://pypi.python.org/simple/mozrunner/
Best match: mozrunner 7.0.1
Downloading https://files.pythonhosted.org/packages/be/d1/90b239e4a43f060e9772a4fba9adf6f8d7a76671771ad7eb239f00af4bd5/mozrunner-7.0.1.tar.gz#sha256=a4e3d532d1d7eb34b6332e0568612ebac3f56da849894005b3dbe68968e5d292
Processing mozrunner-7.0.1.tar.gz
Writing /tmp/easy_install-ijBA15/mozrunner-7.0.1/setup.cfg
Running mozrunner-7.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ijBA15/mozrunner-7.0.1/egg-dist-tmp-HtNzeI
error: Setup script exited with error in mozrunner setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
Traceback (most recent call last):
  File "create_venv.py", line 194, in <module>
    main()
  File "create_venv.py", line 160, in main
    os.path.join(here, 'setup.py'), 'install'])
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gbrown/bin/python', '/home/gbrown/src/testing/tps/setup.py', 'install']' returned non-zero exit status 1

	Geoff Brown [:gbrown]
	Comment 9 • 6 years ago

On the other hand, I can install mozrunner 7.0.1 manually:

gbrown@mozpad:~/xx$ virtualenv gb
gbrown@mozpad:~/xx$ source gb/bin/activate
(gb) gbrown@mozpad:~/xx$ pip install mozrunner==7.0.1
Collecting mozrunner==7.0.1
  Downloading https://files.pythonhosted.org/packages/be/d1/90b239e4a43f060e9772a4fba9adf6f8d7a76671771ad7eb239f00af4bd5/mozrunner-7.0.1.tar.gz
Collecting mozdevice>=1.* (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/49/ea/206bd3e105b8d52fe754c8609babd6548358a8daf766bea7847fe3f18f2c/mozdevice-1.0.0.tar.gz
Collecting mozfile==1.* (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/83/e2/ea5cbdecefd2fd824a836fc5bd16c254a903e1597e9708fab427b4024e0b/mozfile-1.2.tar.gz
Collecting mozinfo<1,>=0.7 (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/db/a3/ef600acad5177fc1dfe822f6645a9153a3a6c72213f061211ba600f935e0/mozinfo-0.10.tar.gz
Collecting mozlog==3.* (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/17/cc/027cef978ec8a630e0c6ff5d5d23a0a54dd35908a778011ae809a9cac10f/mozlog-3.7-py2-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 956kB/s 
Collecting mozprocess<1,>=0.23 (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/2f/f8/25c6593b16eda2dc1bea23a0ec92c59158c095981c5d8393ffd2379034bb/mozprocess-0.26-py2-none-any.whl
Collecting mozprofile<2,>=1.1.0 (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/02/10/ed5dec83cd3c7ae0106feeefd4dc403d5c51ff28f49266a176cdb3cc4bb9/mozprofile-1.1.0-py2-none-any.whl
Collecting six<2,>=1.10.0 (from mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting moznetwork>=0.24 (from mozdevice>=1.*->mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/d3/bf/69d003109e837f3e5a3d74f44407f3006fb3eb588968246b56ead8a26503/moznetwork-0.27.tar.gz
Collecting blessings>=1.3 (from mozlog==3.*->mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/81/bd/2629c84d08b9da0252ba8f139a8791b8b7416ed3b7537173e05c35588c2c/blessings-1.6.1-py2-none-any.whl
Collecting mozterm (from mozlog==3.*->mozrunner==7.0.1)
  Downloading https://files.pythonhosted.org/packages/f6/90/aff4ccdae22753369be0a14afbaf0f79f774daf4b24882b12ad540477933/mozterm-0.1.0-py2-none-any.whl
Building wheels for collected packages: mozrunner, mozdevice, mozfile, mozinfo, moznetwork
  Running setup.py bdist_wheel for mozrunner ... done
  Stored in directory: /home/gbrown/.cache/pip/wheels/af/b9/d9/4f23babe8d570cb8daab6c5da7576495fb6693953bc734130a
  Running setup.py bdist_wheel for mozdevice ... done
  Stored in directory: /home/gbrown/.cache/pip/wheels/7e/fb/56/eb1a0a9fa86ff3c244808dfb3e8a14c706ad8c218237460b9d
  Running setup.py bdist_wheel for mozfile ... done
  Stored in directory: /home/gbrown/.cache/pip/wheels/b8/87/92/aebf86f8305c60de92ed0694226a3049fd8b72b82d64e27c88
  Running setup.py bdist_wheel for mozinfo ... done
  Stored in directory: /home/gbrown/.cache/pip/wheels/7c/74/af/c04213ac535a149f5ebe9c1d0f216d7eaa14d2f4847c9c8d74
  Running setup.py bdist_wheel for moznetwork ... done
  Stored in directory: /home/gbrown/.cache/pip/wheels/c3/f6/ed/ac0af1e6be733e088b2f70845130a48fb16cbb0ccb5f89db28
Successfully built mozrunner mozdevice mozfile mozinfo moznetwork
Installing collected packages: mozfile, blessings, mozterm, six, mozlog, mozinfo, moznetwork, mozprocess, mozdevice, mozprofile, mozrunner
Successfully installed blessings-1.6.1 mozdevice-1.0.0 mozfile-1.2 mozinfo-0.10 mozlog-3.7 moznetwork-0.27 mozprocess-0.26 mozprofile-1.1.0 mozrunner-7.0.1 mozterm-0.1.0 six-1.11.0

	Lina Butler [:lina]
	Comment 10 • 6 years ago

Hmm, would you mind posting your Python and pip versions? I get this if I try to manually install mozrunner locally:

> (tps-venv)k@KC-MBA ~/tps-venv> pip install mozrunner==7.0.1
> Collecting mozrunner==7.0.1
>   Using cached https://files.pythonhosted.org/packages/be/d1/90b239e4a43f060e9772a4fba9adf6f8d7a76671771ad7eb239f00af4bd5/mozrunner-7.0.1.tar.gz
>     Complete output from command python setup.py egg_info:
>     error in mozrunner setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
>     
>     ----------------------------------------
> Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/8w/nf4nw9vn45b5802570r63hqw0000gn/T/pip-install-cWpQr3/mozrunner/
> (tps-venv)k@KC-MBA ~/tps-venv> pip --version
> pip 10.0.1 from /Users/k/tps-venv/lib/python2.7/site-packages/pip (python 2.7)
> (tps-venv)k@KC-MBA ~/tps-venv> python --version
> Python 2.7.14

	Geoff Brown [:gbrown]
	Comment 11 • 6 years ago

(gb) gbrown@mozpad:~/xx$ pip --version
pip 10.0.1 from /home/gbrown/xx/gb/local/lib/python2.7/site-packages/pip (python 2.7)
(gb) gbrown@mozpad:~/xx$ python --version
Python 2.7.12

	Geoff Brown [:gbrown]
	Comment 12 • 6 years ago

Aha, this works for me! Does it work for you?

Updating VERSION_VIRTUALENV resolved "must be a string or list of strings" but still complained about a conflict in mozprofile versions -- easily addressed.

	Lina Butler [:lina]
	Comment 13 • 6 years ago

Comment on attachment 8973033 [details] [diff] [review]
update virtualenv version + update mozrunner and mozprofile versions

runtps works for me locally now! \o/ Thanks, gbrown!

	Thom Chiovoloni [:tcsc] (ex-moco) Assignee
	Comment 14 • 6 years ago

Comment on attachment 8973033 [details] [diff] [review]
update virtualenv version + update mozrunner and mozprofile versions

Review of attachment 8973033 [details] [diff] [review]:
-----------------------------------------------------------------

It's probably going to continue to be problematic that we pin all of these but this seems fine for now.

	Pulsebot
	Comment 15 • 6 years ago

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8f5a675993ab
Update tps package versions; r=tcsc

	Stefan Hindli [:stefan_hindli]
	Comment 16 • 6 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/8f5a675993ab