Right now eideticker shells out to talos like this: It would be great if talos exposed a nice API for running tests and automatically generating an in-memory configuration, for use-cases like this. There's nothing stopping me from importing talos directly now, but it would look pretty weird. I expect working on this bug would involve the following: 1. Providing a clean API to generate a configuration in-memory. 2. Providing an API to run a configuration. Fixing this bug in an elegant way might involve moving the logic to rewrite remote manifest files into the part of talos which runs a test. It doesn't really make sense as it is, but a user-facing API definitely shouldn't have such surprising behaviour IMO.

(In reply to William Lachance (:wlach) from comment #0) > Right now eideticker shells out to talos like this: I meant to include a link there, but didn't. Here's one: https://github.com/mozilla/eideticker/blob/master/bin/runtest.py#L118

I'm interested in working on this bug. But I'm a bit confused about where in the above link eideticker is using the talos api. Also do you guys have any ideas on other places in talos I should look at that might help for this bug? Thanks!

Hey Fahim, this bug was originally filed when it looked like we wanted to use Talos with Eideticker. This is not currently the case. Nonetheless, it looks like things will be much more reasonable to use Talos programmatically when :jhammel's unified configuration work lands. Here's an example of what it will enable: conf = PerfConfigurator()({'config': 'i', 'want': 1}); run_tests.run_tests(conf.config) I'd say the best thing is probably to build on Jeff's work here. I'm going to mark this bug as a duplicate.