Aki Sasaki (not active) Assignee
	Description • 13 years ago

This should get complexity out of buildbotcustom and allow the a-team,
developers, and releng to all use the same scripts.

By "port" I mean wrap run_tests.py in mozharness, for now.

	Jeff Hammel
	Comment 1 • 13 years ago

I'm not sure if this makes sense to do until bug 688604 is landed so that we can use the mozhttpd server.  If not, there is no good way of using a webserver that doesn't require root privileges since, at least on linux, you're not allowed to open port 80

	Aki Sasaki (not active) Assignee
	Comment 2 • 13 years ago

a) i have a talos zipfile that lets me use --develop without being blocked
b) we can install mozhttpd directly
c) we can start a webserver and test

I don't think this blocks.  At most it blocks the ability to start a webserver automatically without a patched talos; there's plenty of other work that can happen here by using one of the workarounds above.

	Jeff Hammel
	Comment 3 • 13 years ago

So it seems like the first step is to replace the existing buildbotcustom logic with a mozharness script.  Is this an accurate assessment?

	Aki Sasaki (not active) Assignee
	Comment 4 • 13 years ago

Correct. They'll probably run in parallel until we can prove desktop talosrunner is stable, but the goal is to replace it. A secondary goal is to allow a user to run the same script locally.

	Jeff Hammel
	Comment 5 • 13 years ago

So I had incorrectly perceived the goal here to be getting something working for developers vs something to replace the existing buildbotcustom setup.  Here's what I have now, which should mostly work except for the webserver part, just for posterity.

	Jeff Hammel
	Comment 6 • 13 years ago

So this is another WIP slated towards buildbot as a consumer.  It errs our at

10:27:23     INFO -  utils.talosError: 'Found processes still running: firefox. Please close them before running talos.'

I don't have any firefox processes running so something is awry

	Jeff Hammel
	Comment 7 • 13 years ago

(In reply to Jeff Hammel [:jhammel] from comment #6)
> Created attachment 572882 [details]
> WIP 2
> 
> So this is another WIP slated towards buildbot as a consumer.  It errs our at
> 
> 10:27:23     INFO -  utils.talosError: 'Found processes still running:
> firefox. Please close them before running talos.'
> 
> I don't have any firefox processes running so something is awry

Ah, this is bug 700722.  This should probably be fixed before this script is usable.

	Aki Sasaki (not active) Assignee
	Comment 8 • 13 years ago

Thanks, Jeff!

Looks like installing talos installs pyyaml by default, but I'm a bit worried about production, which can't get to pypi.  Listing pyyaml explicitly might help us there.

For user-friendliness making this script and the other mozharness talos- and unittest- scripts "feel" the same in terms of actions and config naming would be good, but I'm fine leaving that for later.

It does sound like --develop and the process issue will make running this not so user-friendly, so getting those fixed properly would be good. If we want/need to roll something out before those are resolved, I think we can work around them.

	Jeff Hammel
	Comment 9 • 13 years ago

Once https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py lands we should unify the options a bit, maybe have a common base class

	Jeff Hammel
	Comment 11 • 13 years ago

This is currently blocked by bug 701573.  The virtualenv-ed talos does not shell out to the correct python process, which is the fault of Talos.  I will fix that first.

	Aki Sasaki (not active) Assignee
	Comment 12 • 13 years ago

(In reply to Jeff Hammel [:jhammel] from comment #11)
> This is currently blocked by bug 701573.  The virtualenv-ed talos does not
> shell out to the correct python process, which is the fault of Talos.  I
> will fix that first.

I'm working around this by setting PATH in the run_tests.py call (where I need to change it to use os.pathsep).
I'd love a real fix.
https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py#L340

	Aki Sasaki (not active) Assignee
	Comment 14 • 13 years ago

Comment on attachment 573963 [details] [diff] [review]
basic mozharness for talos

>+    def _fetch(self, url, dest):
>+        """fetch a static resource from the web"""
>+        # TODO: could be upstreamed
>+        f = file(dest, 'w')
>+        resource = urllib2.urlopen(url)
>+        f.write(resource.read())
>+        resource.close()
>+        f.close()

self.download_file() ?
http://hg.mozilla.org/build/mozharness/file/cbcc9a5d5549/mozharness/base/script.py#l147

>+    def _set_talos_dir(self):
>+        """XXX this is required as talos must be run out of its directory"""
>+        python = self.query_python_path()
>+        self.talos_dir = self.get_output_from_command([python, '-c', 'import os, talos; print os.path.dirname(os.path.abspath(talos.__file__))'])
>+        if not self.talos_dir:
>+            self.fatal("Talos directory could not be found")
>+        self.info('Talos directory: %s' % self.talos_dir)

I'm more of a "check talos out from hg and know where the path is" type person rather than "install talos from pip and figure out where it's installed" type person.

However, if this works, I'm not opposed.

>+    def install_pageloader(self):
>+        """install pageloader"""
>+        if not hasattr(self, 'talos_dir'):
>+            self._set_talos_dir()
>+        dest = os.path.join(self.talos_dir, 'page_load_test', 'pageloader.xpi')
>+        self._fetch(self.config['pageloader_url'], dest)

I solved this by checking out hg.m.o/build/pageloader into the talos directory.  Tmtowtdi, but this may be part of what we want to standardize at some point.

https://github.com/escapewindow/mozharness/blob/talosrunner/configs/users/aki/tablet1.py#L42
https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py#L217

>+    def generate_config(self):
>+        """generate talos configuration"""

There may be more options we want here, but this is a good start.

There may be additional work that needs doing before this script is both fully featured for everything we do in production, and we have parity between the two talos scripts.  But other than possibly s,_fetch,download_file, I don't see anything glaringly wrong here.

Thanks for the script!

	Jeff Hammel
	Comment 15 • 13 years ago

(In reply to Aki Sasaki [:aki] from comment #14)
> Comment on attachment 573963 [details] [diff] [review] [diff] [details] [review]
> basic mozharness for talos
> 
> >+    def _fetch(self, url, dest):
> >+        """fetch a static resource from the web"""
> >+        # TODO: could be upstreamed
> >+        f = file(dest, 'w')
> >+        resource = urllib2.urlopen(url)
> >+        f.write(resource.read())
> >+        resource.close()
> >+        f.close()
> 
> self.download_file() ?
> http://hg.mozilla.org/build/mozharness/file/cbcc9a5d5549/mozharness/base/
> script.py#l147

Ah, I missed that somehow.  Thanks.  I'll change before I check in.

> >+    def _set_talos_dir(self):
> >+        """XXX this is required as talos must be run out of its directory"""
> >+        python = self.query_python_path()
> >+        self.talos_dir = self.get_output_from_command([python, '-c', 'import os, talos; print os.path.dirname(os.path.abspath(talos.__file__))'])
> >+        if not self.talos_dir:
> >+            self.fatal("Talos directory could not be found")
> >+        self.info('Talos directory: %s' % self.talos_dir)
> 
> I'm more of a "check talos out from hg and know where the path is" type
> person rather than "install talos from pip and figure out where it's
> installed" type person.

I would like to see a method that is "check out from hg/download from tarball and install via `setup.py`.  Would that be worth ticketing? I'm more inclined to use a direct .tar.gz/.zip download versus hg to avoid another dependency but agree that the way I've done things is a bit magic.  

Also note that there are also upstream talos issues here.  Talos should not require to be run from a specific directory, but it does.  Talos should also separate the software pieces from the data pieces but it does not.  We'll have to fix these too, probably this quarter.  I just haven't gotten around to figuring out what all needs to be done there.

> However, if this works, I'm not opposed.
> 
> >+    def install_pageloader(self):
> >+        """install pageloader"""
> >+        if not hasattr(self, 'talos_dir'):
> >+            self._set_talos_dir()
> >+        dest = os.path.join(self.talos_dir, 'page_load_test', 'pageloader.xpi')
> >+        self._fetch(self.config['pageloader_url'], dest)
> 
> I solved this by checking out hg.m.o/build/pageloader into the talos
> directory.  Tmtowtdi, but this may be part of what we want to standardize at
> some point.
> 
> https://github.com/escapewindow/mozharness/blob/talosrunner/configs/users/
> aki/tablet1.py#L42
> https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/
> device_talosrunner.py#L217
> 
> >+    def generate_config(self):
> >+        """generate talos configuration"""
> 
> There may be more options we want here, but this is a good start.
> 
> There may be additional work that needs doing before this script is both
> fully featured for everything we do in production, and we have parity
> between the two talos scripts.  But other than possibly
> s,_fetch,download_file, I don't see anything glaringly wrong here.

Indeed, will fix the _fetch -> download_file issue and push

> Thanks for the script!

	Jeff Hammel
	Comment 16 • 13 years ago

pushed: http://hg.mozilla.org/build/mozharness/rev/1d1144817b2a

	Jeff Hammel
	Comment 17 • 13 years ago

see also https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py

	Aki Sasaki (not active) Assignee
	Comment 18 • 13 years ago

(In reply to Jeff Hammel [:jhammel] from comment #15)
> I would like to see a method that is "check out from hg/download from
> tarball and install via `setup.py`.  Would that be worth ticketing?

bug 697462 ?

> I'm more
> inclined to use a direct .tar.gz/.zip download versus hg to avoid another
> dependency but agree that the way I've done things is a bit magic.  

The thing about the hg.m.o zip file is it's only tip, which means any checkin will get picked up immediately.  We may want that behavior, but we may also want to lock a particular revision to a branch.

We do have talos tarballs already, so this isn't something that needs solving immediately.

> Also note that there are also upstream talos issues here.  Talos should not
> require to be run from a specific directory, but it does.  Talos should also
> separate the software pieces from the data pieces but it does not.  We'll
> have to fix these too, probably this quarter.  I just haven't gotten around
> to figuring out what all needs to be done there.

I'd love to see those things fixed, great to hear!

We may want followup bugs for a) --download support so this is user-at-desktop friendly, and b) to make device_talosrunner and this script share more backend methods.

	Jeff Hammel
	Comment 19 • 13 years ago

Reopened due to the reopening of https://bugzilla.mozilla.org/show_bug.cgi?id=700722 . The code is still valid, it will just no longer work because of the downstream talos snafu.  Unless there is a burning reason, I'd rather no backout the script

	Aki Sasaki (not active) Assignee
	Comment 20 • 13 years ago

I'm fine with keeping the script checked in, and adding patches to it as needed; nothing in production is calling this script.

	Jeff Hammel
	Comment 21 • 12 years ago

re comment 19, bug 700722 is now fixed; closing