It would be nice to have a one "click" test runner to run the talos tests. Since I'm the only one that runs the tests, my current methodology cd tests for i in *.py; do python $i; done is probably mostly fine, but it would be easy to write a script that did the equivalent and could be extended. Ideally manifestparser and moztest would be used, but itd be better not to incur these depdencies just for testing. A `python setup.py test` approach could also be used.

There are several ways of doing this none of which is particularly hard. One such way is using something like https://github.com/mozilla/mozbase/blob/master/test.py for guidance

If nobody is going for it, I would like to give it a try. Any suggested bedtime reading that I should be aware of? "Ideally manifestparser and moztest would be used, but itd be better not to incur these depdencies just for testing." So that means, not using manifestparser and moztest?

(In reply to nebelhom from comment #2) > If nobody is going for it, I would like to give it a try. Any suggested > bedtime reading that I should be aware of? mozbase's test.py serves a similar need: https://github.com/mozilla/mozbase/blob/master/test.py nose is also very popular these days: http://nose.readthedocs.org/en/latest/ . damned if i know how to wired it up to setup.py, but maybe mdas or jgriffin in #ateam could help. Ideally it would run with `python setup.py test` > "Ideally manifestparser and moztest would be used, but itd be better not to > incur these depdencies just for testing." > > So that means, not using manifestparser and moztest? It depends. If they're test-only and it can work from `python setup.py test` then fine. Otherwise, not so much

Could you just try this for me please. It is a very simple change and I am scared that it is "too" simple. Essentially added test_suite='tests' to setup(...) The missing piece to the puzzle was adding an empty __init__.py to the tests folder. This works fine with me although the test_configuration.py always fails, but that is unrelated. Hopefully, this is what we were looking for....

Comment on attachment 663204 [details] [diff] [review] First Patch to this bug Sorry, but we shouldn't do this. This makes `tests` a top level module -- in other words, it will override 'tests' in the top-level python namespace.

I know it is not what you asked for, but it works and could make things easier for you for now. I will still speak to other folks, but so far I couldn't find a way around the import issue. I used the site module in this case to import the files in the tests directory, but when used to together with setup.py test, it throws me an AttributeError AttributeError: 'module' object has no attribute 'test_oauth' test_oauth is found in ./talos/lib/python2.7/site-packages/oauth2-1.5.211-py2.7.egg/tests I "could" try and add all necessary directory paths to PYTHONPATH manually, but that is sounds very silly to me. I'll be back if I find anymore out...

Comment on attachment 663391 [details] [diff] [review] seperate test.py from setup.py This won't really work. We don't want to have to add new tests into the test runner each time we add a new test (or remove a test, etc). Instead, we should gather tests using something like https://github.com/mozilla/mozbase/blob/master/test.py#L22 . An additional function will be needed to gather the files from the tests directory: test_files = [os.path.join(tests_dir, i) for i in os.listdir(tests_dir) if i.endswith('.py')] +tests_dir = os.path.join(os.getcwd(), 'tests') You'll want the directory of the file, not the directory of the user: tests_dir = os.path.join(os.path.dirname(os.path.abspath(__here__)), 'tests')

After speaking to carljm, I was able to connect the test_suite to setup.py using runtests.py. I am sure that it still needs work, but I hope it is a step in the right direction. I just wanted to mention for completion's sake (maybe it is useful some time later) regarding imports, carljm mentioned the following: " i think the "proper" solution to avoid polluting the top-level namespace is to put the tests inside the existing talos namespace talos.tests which involves moving them in the source tree, dunno if that's a problem " Anyhow, hope this is better.

Comment on attachment 664121 [details] [diff] [review] setup.py adjustment with runtests.py works like a charm. thanks!

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=97a8a925b721

looks pretty green: http://hg.mozilla.org/build/talos/rev/30e7bcdcb54f

Try run for 97a8a925b721 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=97a8a925b721 Results (out of 72 total builds): exception: 5 success: 66 failure: 1 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jhammel@mozilla.com-97a8a925b721