Closed Bug 707218 Opened 13 years ago Closed 13 years ago

write a script to create a talos.zip file with the dependencies from mozbase

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: k0scist, Unassigned)

References

Details

(Whiteboard: [mozbase][mozharness+talos])

Attachments

(2 files, 1 obsolete file)

unbitrot and update patch
(deleted), patch
jmaher
: review+
bear
: feedback+
Details | Diff | Splinter Review
compress the zipfile
(deleted), patch
jmaher
: review+
Details | Diff | Splinter Review 
Talos currently has two upstream dependencies from mozbase:

- mozdevice: devicemanagerADB.py, devicemanager.py, devicemanagerSUT.py
- mozhttpd.py, which in the mozbase directory now has a directory structure: https://github.com/mozilla/mozbase/tree/master/mozhttpd/mozhttpd

We also want to include mozinfo.py soon: https://bugzilla.mozilla.org/show_bug.cgi?id=699129

None of these three packages have any dependencies and may be used standalone

Currently, we version these separately for talos and take fixes as needed.  Instead we should use the upstream dependencies.  For the python package (e.g. `python setup.py develop`) case, we can add these to setup.py and have setuptools pull them down.  

For the talos.zip case, we can write a script that puts these files in the the talos directory and not have to change how talos imports things.  This is a good interim solution until we have a infrastructure pypi server that we can use to install these dependencies for buildbot.  The script can just point at these few files, put them in the appropriate place in the talos directory and zip it appropriately (and cleanup).

Note that this only works because none of these dependencies have dependencies.  For e.g. mozprocess and mozrunner, we will have to be more clever or be transitioned to a package index infrastructure.  But its at least something to build on.
Note that we should update the instructions on how to deploy Talos once this script is written
pushed mozdevice to pypi as we'll need this to test/use the setup.py route: http://pypi.python.org/pypi/mozdevice
I've tested the resulting talos.zip with tsvg using the instructions at https://wiki.mozilla.org/Mobile/Fennec/Android#talos and things work fine.  Preliminary testing on staging (minus the down fedora slave) seem fine, though I'll run it again as I've made changes since my initial changes. I will also test the python setup.py develop method using ts and tsvg.  

Are there other tests that should be run before I put this up for review?  I will also work on updating deployment and setup instructions for this.

mozinfo.py isn't currently used.  I put it there such that Bug 699129 is more easily actionable.  However, I can remove if not desired at this time.
Attachment #579517 - Flags: feedback?(jmaher)
I've tested this locally in a clean virtualenv (although with some things, like pageloader, already in talos).  python setup.py works fine and I was able to run ts.  Is there anything else I should test with this code path?
Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

putting up for review
Attachment #579517 - Flags: review?(jmaher)
Attachment #579517 - Flags: feedback?(jmaher)
Attachment #579517 - Flags: feedback?(bear)
Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

i'll have to make changes to the foopy environment (we symlink into the talos directory structure) and also to our helper tools - none of this is worthy of blocking this fine refactoring IMO
Attachment #579517 - Flags: feedback?(bear) → feedback+
I have created https://developer.mozilla.org/en/Python/Virtualenv , https://developer.mozilla.org/en/Python#section_4, and updated https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code so that the documentation comes closer to reflecting reality.  More updates will need to be done wrt how talos.zip now works on landing.
Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

Review of attachment 579517 [details] [diff] [review]:
-----------------------------------------------------------------

Just some general questions and nits, otherwise this looks great.

::: create_talos_zip.py
@@ +41,5 @@
> +        except:
> +            print url
> +            raise
> +        f = file(filename, 'w')
> +        f.write(contents)

what if the file exists and is locked?  can we put some try/catch around here?

@@ +73,5 @@
> +    # make the talos.zip file
> +    zip = zipfile.ZipFile('talos.zip', mode='w')
> +    talosdir = os.path.abspath(os.path.join(here, 'talos'))
> +    for dirpath, dirnames, filenames in os.walk(talosdir):
> +        filenames = [i for i in filenames if not i.endswith('.pyc')]

.pyc is in .hgignore, why filter it here?
Attachment #579517 - Flags: review?(jmaher) → review+
(In reply to Joel Maher (:jmaher) from comment #9)
> Comment on attachment 579517 [details] [diff] [review]
> script to create talos zips and import changes and additions to .hgignore
> file
> 
> Review of attachment 579517 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Just some general questions and nits, otherwise this looks great.
> 
> ::: create_talos_zip.py
> @@ +41,5 @@
> > +        except:
> > +            print url
> > +            raise
> > +        f = file(filename, 'w')
> > +        f.write(contents)

> what if the file exists and is locked?  can we put some try/catch around
> here?

Sure, I can add that.
 
> @@ +73,5 @@
> > +    # make the talos.zip file
> > +    zip = zipfile.ZipFile('talos.zip', mode='w')
> > +    talosdir = os.path.abspath(os.path.join(here, 'talos'))
> > +    for dirpath, dirnames, filenames in os.walk(talosdir):
> > +        filenames = [i for i in filenames if not i.endswith('.pyc')]
 
> .pyc is in .hgignore, why filter it here?

Oops, that was for an earlier incarnation where I didn't filter through .hgrc.  I'll remove.

Should I retest with these changes?
(In reply to Jeff Hammel [:jhammel] from comment #10)
> (In reply to Joel Maher (:jmaher) from comment #9)
> > Comment on attachment 579517 [details] [diff] [review]
> > script to create talos zips and import changes and additions to .hgignore
> > file
> > 
> > Review of attachment 579517 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > Just some general questions and nits, otherwise this looks great.
> > 
> > ::: create_talos_zip.py
> > @@ +41,5 @@
> > > +        except:
> > > +            print url
> > > +            raise
> > > +        f = file(filename, 'w')
> > > +        f.write(contents)
> 
> > what if the file exists and is locked?  can we put some try/catch around
> > here?
> 
> Sure, I can add that.

Actually, wait;  I'm not sure what is being asked here.  If the file is locked and you try to write to it, shouldn't it just err out (as it would as stands) vs trying to raise?
Also, I don't currently create a pageloader.xpi inside of page_load_test.  This leads to the following:

(tmp)│python PerfConfigurator.py -e `which firefox` -a tsvg
(tmp)│ls
20111208_1416_config.yml  ffprocess_win32.py    places_generated_med
addon.config              ffprofile_unix.py     places_generated_min
base_profile              ffprofile_win32.py    post_file.py
bcontroller.py            ffsetup.py            remote.config
breakpad                  getInfo.html          remotePerfConfigurator.py
cmanager_linux.py         __init__.py           run_tests.py
cmanager_mac.py           mobile.config         sample.config
cmanager_remote.py        mobile_profile        scripts
cmanager_win32.py         mozdevice             startup_test
cycles.config             mozhttpd.py           ttest.py
ffprocess_linux.py        mozinfo.py            utils.py
ffprocess_mac.py          page_load_test        xperf.config
ffprocess.py              PerfConfigurator.py   xtalos
ffprocess_remote.py       places_generated_max
(tmp)│python run_tests.py 
(tmp)│python run_tests.py 20111208_1416_config.yml 
RETURN:s: qm-pxp01
RETURN:id:20111208031134
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/6785d3003414">rev:6785d3003414</a>
qm-pxp01: 
		Started Thu, 08 Dec 2011 14:16:35
Running test tsvg: 
		Started Thu, 08 Dec 2011 14:16:35
Failed tsvg: 
		Stopped Thu, 08 Dec 2011 14:16:35
FAIL: Busted: tsvg
FAIL: Found processes still running: firefox, plugin-container. Please close them before running talos.
Traceback (most recent call last):
  File "run_tests.py", line 591, in <module>
    main()
  File "run_tests.py", line 588, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 534, in test_file
    raise e
utils.talosError: 'Found processes still running: firefox, plugin-container. Please close them before running talos.'
(tmp)│python run_tests.py 20111208_1416_config.yml 
RETURN:s: qm-pxp01
RETURN:id:20111208031134
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/6785d3003414">rev:6785d3003414</a>
qm-pxp01: 
		Started Thu, 08 Dec 2011 14:16:49
Running test tsvg: 
		Started Thu, 08 Dec 2011 14:16:49
Traceback (most recent call last):
  File "run_tests.py", line 591, in <module>
    main()
  File "run_tests.py", line 588, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 520, in test_file
    browser_dump, counter_dump, print_format = mytest.runTest(browser_config, test)
  File "/home/jhammel/tmp/talos/ttest.py", line 238, in runTest
    browser_config['bundles'][bundlename])
  File "/home/jhammel/tmp/talos/ffsetup.py", line 276, in InstallBundleInBrowser
    zip_extractall(zipfile.ZipFile(bundle_path), destpath)
  File "/usr/lib/python2.7/zipfile.py", line 697, in __init__
    self.fp = open(file, modeDict[mode])
IOError: [Errno 2] No such file or directory: 'page_load_test/pageloader.xpi'

I'm going to assume we want this here? And add it???  Not sure what is desired here.
pageloader is not included in talos.zip proper, so please don't add it in.  

Regarding a file writing error, lets just catch the error and exit cleanly.
(In reply to Joel Maher (:jmaher) from comment #13)
> pageloader is not included in talos.zip proper, so please don't add it in.  

Okay, thanks.

> Regarding a file writing error, lets just catch the error and exit cleanly.

From http://k0s.org/mozilla/hg/talos-patches/file/tip/create-zips-zip#l72 :

        try:
            f = file(filename, 'w')
            f.write(contents)
            f.close()
        except:
            print filename # the file may not be writable
            raise
:bear, can you check to see if http://k0s.org/mozilla/talos/bug-707218/talos.zip works with the build infrastructure? It is generated with the latest version of the patch and checked locally.  If there are any problems please let me know.  Otherwise, with your go-ahead, I'd like to check in.
Blocks: 713055
:bear, any ETA for this?
Whiteboard: [mozbase] → [mozbase][mozharness+talos]
I ran this thru a number of staging runs on the tegra side and found no problems
So this will need to be unbitrotted and corrected to include mozinfo and kill the sys.path addition in run_tests.py.  In fact, mozinfo.py will be mirrored twice for talos.zip
The following can be patched successfully:

│patch -p1 < ~/create-talos-zip.diff 
patching file .hgignore
patching file create_talos_zip.py
patching file talos/PerfConfigurator.py
patching file talos/devicemanager.py
patching file talos/devicemanagerADB.py
patching file talos/devicemanagerSUT.py
patching file talos/ffprocess_remote.py
patching file talos/mozhttpd.py
patching file talos/run_tests.py
Hunk #1 succeeded at 464 (offset 10 lines).

I will unbitrot the others by hand
Attached patch unbitrot and update patch (deleted) — — Splinter Review 
This unbitrots the patch and updates it. Not sure what all testing we want to do.  I am staging it now.  Not sure which local tests I should run on my tablet?  Not sure if :bear wants another round of tegra testing?  Please comment.  test_install.py still works...that is something.  I am happy to add more tests if there is a need.  The staged zip is at http://k0s.org/mozilla/talos/tip/zips/talos.zip
Attachment #579517 - Attachment is obsolete: true
Attachment #584664 - Flags: review?(jmaher)
Attachment #584664 - Flags: feedback?(bear)
Comment on attachment 584664 [details] [diff] [review]
unbitrot and update patch

Review of attachment 584664 [details] [diff] [review]:
-----------------------------------------------------------------

it all looks good but I'm really wanting to see this run thru the try environment as that is a much better indicator than my tegra environment 

let me know if you need an assist on that - I can convince someone in releng to lend some brain cells to this
Attachment #584664 - Flags: feedback?(bear) → feedback+
Comment on attachment 584664 [details] [diff] [review]
unbitrot and update patch

Review of attachment 584664 [details] [diff] [review]:
-----------------------------------------------------------------

overall call it a r+

::: .hgignore
@@ +10,5 @@
> +mozhttpd.py
> +talos.zip
> +mobile_tp4
> +remoteapp.ini
> +svg.manifest.develop
\ No newline at end of file

can this be *.develop?
Attachment #584664 - Flags: review?(jmaher) → review+
(In reply to Joel Maher (:jmaher) from comment #24)
> Comment on attachment 584664 [details] [diff] [review]
> unbitrot and update patch
> 
> Review of attachment 584664 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> overall call it a r+
> 
> ::: .hgignore
> @@ +10,5 @@
> > +mozhttpd.py
> > +talos.zip
> > +mobile_tp4
> > +remoteapp.ini
> > +svg.manifest.develop
> \ No newline at end of file
> 
> can this be *.develop?

WFM
Staging with a corrected .hgignore file (not that that is important, but I cleaned it up for checkin. See http://k0s.org/mozilla/hg/talos-patches/rev/5287c5f15d5a
All passing on staging and local tests.  Things seem to pass on try, minus a seemingly unrelated failure (https://tbpl.mozilla.org/php/getParsedLog.php?id=8218022&tree=Try&full=1) . Pushed: http://hg.mozilla.org/build/talos/rev/8031d427ef14

I will add instructions to https://wiki.mozilla.org/Buildbot/Talos wrt instructions on how to generate a talos.zip file now.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Note that currently we pull the latest files from mozbase master: http://hg.mozilla.org/build/talos/file/8031d427ef14/create_talos_zip.py#l18 . This is fine as long as we stay in sync with them, but we should be double-careful that this is the case, and/or we should think about pegging to a version.
Try run for b58b094081b0 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b58b094081b0
Results (out of 79 total builds):
    success: 78
    failure: 1
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jhammel@mozilla.com-b58b094081b0
So I did not hitherto realize that the default for python's zipfile.ZipFile object is to not compress: http://docs.python.org/library/zipfile.html#zipfile-objects. This causes the talos.zip to be about 19M versus about 5.5M . I will post a follow-up patch
Attached patch compress the zipfile (deleted) — — Splinter Review 
This restores the talos.zip size to 5.9M
Attachment #584873 - Flags: review?(jmaher)
(In reply to Jeff Hammel [:jhammel] from comment #31)
> Created attachment 584873 [details] [diff] [review]
> compress the zipfile
> 
> This restores the talos.zip size to 5.9M

testing on try: https://tbpl.mozilla.org/?tree=Try&rev=1c11ce7c9f25
Comment on attachment 584873 [details] [diff] [review]
compress the zipfile

good find!
Attachment #584873 - Flags: review?(jmaher) → review+
Try run for 1c11ce7c9f25 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1c11ce7c9f25
Results (out of 79 total builds):
    success: 77
    failure: 2
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jhammel@mozilla.com-1c11ce7c9f25
pan opt again burning on Android xul (timeout exceeded) and failure in tzoom:

getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed tzoom: 
		Stopped Thu, 29 Dec 2011 17:11:12
FAIL: Busted: tzoom
FAIL: unrecognized output format
Traceback (most recent call last):
  File "run_tests.py", line 603, in <module>
    main()
  File "run_tests.py", line 600, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 546, in test_file
    raise e
utils.talosError: 'unrecognized output format'
program finished with exit code 1

Not sure what that is about. From https://tbpl.mozilla.org/php/getParsedLog.php?id=8223480&tree=Try

I highly doubt its related to this fix.

Okay to push?
yes, tpan is a common failure these days :(
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: