We'd like to be able to re-use most of our python test harness stack when running gaia-integration tests in tbpl. To do this we should be able to create (or modify an existing) host that: 1. sends a profile path over a socket 2. waits to receive a test to run 3. runs test 4. sends back result 5. go to 2 and repeat The python side would be responsible for starting b2gdesktop (or device), and doing any necessary cleanup between tests.

Wouldn't we go back to step 1 and send over a new python path? eg https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L68

I assumed we built one profile per test run, but I guess individual tests require specialized profiles?

I started working on the transport piece the last day or so. Though the focus of it is this use case for now, I want the transport to be re-usable by any future test harnesses which might want: A) bindings across multiple languages, or B) the ability to easily distribute tests across threads, processes or machines (or combinations of The service is: https://github.com/ahal/corredor The JS client for it is: https://github.com/ahal/corredor-js-client Right now the prototype simulates sending and running tests. To try it out locally you need zmq: libzmq - apt-get install libzmq3-dev, yum install zeromq3-devel, brew install zmq python bindings - pip install pyzmq node bindings - npm install zmq Then clone both repos and in separate terminals (and either order), run: python test_program.py node test_worker.js The python side sends tests to the js side one at a time and receives the result. It also receives all stdout and stderr from the js side via a separate socket.

(In reply to Andrew Halberstadt [:ahal] from comment #2) > I assumed we built one profile per test run, but I guess individual tests > require specialized profiles? Yep!

I have a very hacky prototype almost working. I'm still not sure where everything will go, but so far I've created new branches on the related projects: https://github.com/ahal/gaia/tree/python_service https://github.com/ahal/marionette-b2gdesktop-host/tree/python_service https://github.com/ahal/mocha-tbpl-reporter/tree/python_service I hope to have tests running sometime early next week. A couple things to note.. I am a complete node newb (and my javascript skills in general are quite rusty). So my hope is for you to use my hacky code as an example only and to help me package every thing properly (to get this working locally I created a jumble of symlinks). Also feedback on the transport piece (e.g corredor-js-client) is more than welcome! It will still likely change a lot before this is all done. Speaking of which, I guess I'll need to register it on npm at some point?

So I have one test running successfully, but I'm running into some trouble trying to get the next one to start. This is output from an *unmodified* setup: START RUNNER baz ✓ is running and listening STOP RUNNER START RUNNER STOP RUNNER Music player tests START RUNNER baz Audio channels tests ✓ Interrupted by a higher priority channel (14717ms) STOP RUNNER START RUNNER STOP RUNNER Awesomescreen Gareth, do you know what's up with the extra "START/STOP" call after each suite? I put those debug statements here: https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L68 That in-between call is tripping up the ordering of events across the socket, and I'm having difficulty understanding its purpose or where it's coming from.

I discovered it's not the ordering that was causing me problems, but I'm still curious about the question.

Interesting... I don't think this is happening in marionette-js-runner proper, but I am meaning to play around with your code later this week. I'll probably have a good diagnosis for you then.

That output is from marionette-js-runner proper (I just added a few console.logs). I think what is happening is that the callback from stop() always calls start() again, even if it's the last test in the suite. Then suiteTeardown calls stop() one last time.

This brings me to another question though. I noticed that each suite creates a new Host instance. Do we want to support multiple Host's all connected to the same socket endpoint at the same time? If yes, then I'll need to change the transport a bit as I currently assume a 1-1 connection (this shouldn't be a big deal). Other than that, I have tests running on my local machine. My goal is to have things ready for you to look at by the end of the week. I still need to handle output, results and make things less hacky.

(In reply to Andrew Halberstadt [:ahal] from comment #10) > This brings me to another question though. I noticed that each suite creates > a new Host instance. Do we want to support multiple Host's all connected to > the same socket endpoint at the same time? I don't have a strong use case for this right now. Not to say that there's no reason to do this... just that I don't [personally] have a user story :). > If yes, then I'll need to change > the transport a bit as I currently assume a 1-1 connection (this shouldn't > be a big deal). Other than that, I have tests running on my local machine. Sweet! > My goal is to have things ready for you to look at by the end of the week. I > still need to handle output, results and make things less hacky. Cool I'm starting to read now.

Hey if you need a reviewer for some of this stuff I am happy to get to know the codebase

(In reply to Andrew Halberstadt [:ahal] from comment #3) > Right now the prototype simulates sending and running tests. To try it out > locally you need zmq: > libzmq - apt-get install libzmq3-dev, yum install zeromq3-devel, brew > install zmq > python bindings - pip install pyzmq > node bindings - npm install zmq I think we ask gaia developers to install a bunch of stuff locally and it will be important for us to be able to run this locally without too much friction. Is there any way that we can package zmq in a way that's less global?

Also for language bindings, you should add package.json and requirements.txt so that you can specify binding versions and install local to project

So for the python/node bindings we can just have pip/npm install them as packages. For the library we can add it to the Firefox OS build prereqs and the mach bootstrap command. Alternatively, the python zmq binding will attempt to compile and install the library from source if it's not present, we could just let it attempt this. The way I envision it whatever entry point we use (e.g a mach command) would be responsible for making sure this is all set up. It can install the python stuff in a virtualenv to avoid global packages (not sure if node has something similar or not). Also I filed bug 994764 as a top level tracking bug for all this. I was thinking we could use this specifically for the js host part.

(In reply to Andrew Halberstadt [:ahal] from comment #15) > The way I envision it whatever entry point we use (e.g a mach command) would > be responsible for making sure this is all set up. It can install the python > stuff in a virtualenv to avoid global packages (not sure if node has > something similar or not). That sounds good. In node/npm require will find things installed in node_modules before it looks at globally installed packages. > Also I filed bug 994764 as a top level tracking bug for all this. I was > thinking we could use this specifically for the js host part. Sure sounds good.

corredor-js-client/strategy.js:19 this.master.socket.actionMap[action] = callback; ^ TypeError: Cannot set property 'test_start' of undefined at PingPong.registerAction (/Users/gareth/Documents/corredor-js-client/strategy.js:19:40) at Object.<anonymous> (/Users/gareth/Documents/corredor-js-client/test_worker.js:17:11) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Function.Module.runMain (module.js:497:10) at startup (node.js:119:16) at node.js:902:3 yikes

Oh, I changed things around a bit since comment 3. I haven't been using test_worker.js anymore since I've got actual tests running now :). I just forgot to delete it. To try running tests locally you can: 1) use the 'python_service' branch of gaia, marionette-b2gdesktop-host and mocha-tbpl-reporter on my github. (I just linked them all to ~/.node_modules since I had no idea how to get them pulled in via the Makefile) 2) install corredor (preferably in a virtualenv): $ cd path/to/corredor $ python setup.py develop 3) Run the python runner: $ cd path/to/gaia $ python tests/python/runner-service/runner.py Sorry, as you can see I'm still trying to sort all this out :).

Link: https://github.com/ahal/marionette-socket-host Gareth, could you also give this a quick look? It's the same idea as the last time. I copied marionette-b2gdesktop-host and changed around what was needed.

Hey :ahal. I have a few thoughts... (1) https://github.com/ahal/marionette-socket-host/blob/master/package.json#L15 lock down dependencies so we don't get burned by breaking, upstream api changes (2) promises over callbacks in corredor since that's what much of gaia is moving toward (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise supported in node >= 0.11 and there's a polyfill for node <0.11 here https://github.com/jakearchibald/es6-promise) (3) If you want developers to run these when and if tests break, you'd have much better luck selling them on it if they didn't have to install zeromq. Maybe linux container something? (4) Stuff got copied over from b2g desktop host that's not being used (eg https://github.com/ahal/marionette-socket-host/blob/master/index.js#L40) and some comments (eg https://github.com/ahal/marionette-socket-host/blob/master/index.js#L11) are outdated (5) The debug statements could be much more useful (6) Small nit https://github.com/ahal/marionette-socket-host/blob/master/test/index.js should be named somethingorother_test.js or test_somethingorother.js. (7) https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L68 is a subject in the scope of that function block that "shadows" this one https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L21 (8) https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L57 seems like you call done synchronously here I have many more nits, but I hope that can get you started in the right direction. Let me know if you have any questions and thanks for the help!

Comment on attachment 8453383 [details] Host for sending start/stop commands over a socket Reflag if you'd like another review pass later!

(In reply to Gareth Aye [:gaye] from comment #21) > Hey :ahal. I have a few thoughts... Thanks, this is helpful! > (1) > https://github.com/ahal/marionette-socket-host/blob/master/package.json#L15 > lock down dependencies so we don't get burned by breaking, upstream api > changes Even "debug"? mozilla-b2g/gaia-node-modules doesn't peg it either, so I think requiring a specific version here might cause problems in automation? > (2) promises over callbacks in corredor since that's what much of gaia is > moving toward > (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/ > Global_Objects/Promise supported in node >= 0.11 and there's a polyfill for > node <0.11 here https://github.com/jakearchibald/es6-promise) This seems like a good idea. I'll file a follow up for it though. My immediate goal is to get what I have now landed by next week so others can start playing around with it. > (3) If you want developers to run these when and if tests break, you'd have > much better luck selling them on it if they didn't have to install zeromq. > Maybe linux container something? Agreed. Pyzmq will attempt to download the source and compile it for you. Barring that, I can also get whatever will be running this (e.g mach) to do the necessary setup. Again though, I'll do this in a follow-up. > (8) > https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L57 > seems like you call done synchronously here Isn't child_process.fork synchronous? At least I don't see any events that might signal when the fork is ready.. Either way, in this case it doesn't matter. ZeroMQ makes no assumptions about the ordering of calls to bind/connect. It will buffer messages in the event there is nothing on the other end and deliver them as soon as something connects. > I have many more nits, but I hope that can get you started in the right > direction. Let me know if you have any questions and thanks for the help! Cool, I pushed the fixes to my repo minus some comments I replied to here.

Comment on attachment 8453383 [details] Host for sending start/stop commands over a socket If you have other nits, I'd love to hear them.

I transferred this to mozilla-b2g: https://github.com/mozilla-b2g/marionette-socket-host Need someone to add the chefs list for commit access. Will wait for that and additional review comments before resolving this.

I'm declaring this done. We can file new bugs for further fixes. Final repo is here: https://github.com/mozilla-b2g/marionette-socket-host