516984 - Run mozmill tests on buildbot

Reporter

Description

•

15 years ago

We should be running mozmill tests on our regular depend builds. Clint, any need for them to be run on debug builds as well?

Chris AtLee [:catlee]

Reporter

•

15 years ago

Latest version of mozmill still has issues with hangs post running. I think we need to set up testdev with their own automation systems so they can work on debugging these issues. Al mentioned that there is already, or will be soon, a bug for that.

Assignee: catlee → nobody

Component: Release Engineering → Release Engineering: Future

Whiteboard: Q4 goal

Chris Cooper [:coop] (he/him)

Comment 3

•

15 years ago

Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Component: Release Engineering: Future → Release Engineering

Chris AtLee [:catlee]

Reporter

Comment 4

•

15 years ago

Attached patch WIP (obsolete) (deleted) — Details — Splinter Review

Jeff Hammel

Updated

•

15 years ago

Assignee: nobody → jhammel

Jeff Hammel

Comment 5

•

15 years ago

Attached patch port of attachment 433163 to the code today, March 24, 2010; not yet tested (obsolete) (deleted) — Details — Splinter Review

Attachment #433163 - Attachment is obsolete: true

Jeff Hammel

Comment 6

•

15 years ago

Attached patch new version of mozmill + buildbot patch, unclean (obsolete) (deleted) — Details — Splinter Review

functional cleanup of the patch to integrate mozmill and buildbot. All of the steps actually run, though mozmill does not work successfully yet on the nightly builds (see https://wiki.mozilla.org/Auto-tools/Projects/MozmillBuildbot#Outstanding_Issues ) virtualenv is used to install mozmill. Currently, the pypi version is used (http://pypi.python.org/pypi/mozmill ; v1.4) at the time of this writing. The patch includes details that aren't important to the work, so should be cleaned up. The previous method of installing mozmill is still present, but commented out An alternative is to not use virtualenv. To be determined.

Attachment #434790 - Attachment is obsolete: true

Jeff Hammel

Comment 7

•

15 years ago

(In reply to comment #6) > Created an attachment (id=437934) [details] > new version of mozmill + buildbot patch, unclean > > functional cleanup of the patch to integrate mozmill and buildbot. All of the > steps actually run, though mozmill does not work successfully yet on the > nightly builds (see > https://wiki.mozilla.org/Auto-tools/Projects/MozmillBuildbot#Outstanding_Issues > ) > ... Issue moved here: Running mozmill does not work as expected. This is shown in the buildbot step as well as reproducible from the command line following the build. > /home/jhammel/mozilla/src/buildbot/slave/full/mozmill/bin/mozmill --showall -b firefox/firefox -t mozmill-tests/firefox --showall (firefox-bin:13024): GLib-WARNING **: g_set_prgname() called multiple times (firefox-bin:13024): GLib-WARNING **: g_set_prgname() called multiple times Traceback (most recent call last): File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/bin/mozmill", line 8, in <module> load_entry_point('mozmill==1.4', 'console_scripts', 'mozmill')() File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/mozmill-1.4-py2.6.egg/mozmill/__init__.py", line 541, in cli CLI().run() File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/mozmill-1.4-py2.6.egg/mozmill/__init__.py", line 518, in run self._run() File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/mozmill-1.4-py2.6.egg/mozmill/__init__.py", line 468, in _run self.mozmill.start(runner=runner, profile=runner.profile) File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/mozmill-1.4-py2.6.egg/mozmill/__init__.py", line 162, in start self.create_network() File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/mozmill-1.4-py2.6.egg/mozmill/__init__.py", line 142, in create_network self.jsbridge_port) File "/home/jhammel/mozilla/src/buildbot/slave/full/mozmill/lib/python2.6/site-packages/jsbridge-2.3.4-py2.6.egg/jsbridge/__init__.py", line 72, in wait_and_create_network raise Exception("Sorry, cannot connect to jsbridge extension") Exception: Sorry, cannot connect to jsbridge extension The command (<tt>mozmill -b /home/jhammel/bin/firefox -t /home/jhammel/mozilla/src/buildbot/slave/full/build/mozmill-tests/firefox --showall</tt>) works on Firefox 3.6.2 but not on on 3.7.a{4,5}pre. -- This is due to the maxVersion tags in the extesions/install.rdf file being set to 3.7a1pre in the versions of both mozmill (1.4) and jsbridge (2.3.4) on pypi at this date (thanks, Clint!). Changing these causes the test to run correctly. So the patch should work when next week's upcoming release of these packages are posted on pypi, though the patch should be cleaned up first.

Jeff Hammel

Comment 8

•

15 years ago

Comment 20

•

15 years ago

(In reply to comment #19) > I've > only skimmed bug 500142 and am not sure when to do make this switch or how > heavy of a switch this is (sorry, I'm new here). I'll do a brief summary of what Thunderbird does and why. 1) We have two make targets built into the build system: "mozmill" and "mozmill-one". Both of these mean that any developer can run all Thunderbird mozmill tests (or just one) by typing one or two commands (as long as they have mozmill installed, although we're also thinking about revising how we manage that). Defined at the bottom of this file: http://mxr.mozilla.org/comm-central/source/mail/build.mk The output of "make mozmill" is very similar to xpcshell-tests. Debug mode has additional output, and there are summaries at the end. Our wrapper around mozmill manages this and profile directories automatically: http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtest.py http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtestlist.py Having the wrapper produce the similar log means that we get buildbot and tinderbox parsing of the logs for free. 2) Buildbot then runs mozmill tests in exactly the same way as xpcshell-tests: http://hg.mozilla.org/build/buildbotcustom/file/59119d3afe97/process/factory.py#l4501 This means no extra/special code in buildbot, which in turn means that developers just have to remember/work with one or two commands rather than many if they are trying to resolve issues. This also means we hook into the crash-stack reporting "free-of-charge". The only complication in the Firefox case is the separate repository, but IMO, there's no reason you couldn't have a third "mozmill-update" command that gets/updates the mozmill test directory.

Jeff Hammel

•

15 years ago

(In reply to comment #21) > It was decided not to use virtualenv to install the mozmill software. > The solution proposed is to use virtualenv and e.g. pip to create binary > environments for each slave platform which are then uploaded to a > mozilla URL upon packaging mozmill (and presumedly any python package > needed for testing, see below). Are there meeting notes available to get more information why this decision has been made? Is it just to not automatically use the new version once it is available?

Henrik Skupin [:whimboo][⌚️UTC+2]

Updated

•

15 years ago

Summary: Run mozmill tests → Run mozmill tests on buildbot

Jeff Hammel

Comment 25

•

15 years ago

Attached patch port of patch to work with buildbotcustom as of today [April 19, 2010] (obsolete) (deleted) — Details — Splinter Review

i updated the original patch to work with today's buildbotcustom. it doesn't work OOTB (to be expected, as neither did the original patch), but it is syntatically sound. I also made one logic change of removing self.super_class and just calling ShellCommandReportTimeout.<method>(...) vs self.super_class.<method>(...) the one place it was used.

Attachment #434790 - Attachment is obsolete: true

Chris AtLee [:catlee]

Reporter

Comment 26

•

15 years ago

(In reply to comment #25) > Created an attachment (id=439959) [details] > port of patch to work with buildbotcustom as of today [April 19, 2010] > > i updated the original patch to work with today's buildbotcustom. it doesn't > work OOTB (to be expected, as neither did the original patch), but it is > syntatically sound. > > I also made one logic change of removing self.super_class and just calling > ShellCommandReportTimeout.<method>(...) vs self.super_class.<method>(...) the > one place it was used. We need to use the self.super_class idiom so that buildbot reconfigs don't break.

Henrik Skupin [:whimboo][⌚️UTC+2]

Comment 27

•

15 years ago

Chris, can you or Ben please give a reply to my comment 24? Thanks.

Mark Banner (:standard8)

Comment 28

•

15 years ago

(In reply to comment #21) > It was decided not to use virtualenv to install the mozmill software. > The solution proposed is to use virtualenv and e.g. pip to create binary > environments for each slave platform which are then uploaded to a > mozilla URL upon packaging mozmill (and presumedly any python package > needed for testing, see below). So attempting to put this simpler, I think you're saying: - Build (or someone) creates a virtualenv binary of the mozmill software - Mozmill sofware gets deployed onto the slaves by pip. What happens when if we get the situation that one branch needs to use one version of mozmill, and another branch wants to use a different version? Is it possible for developers to run/install these without having pip setups? I'd also like to know why you're not heading down the route that Thunderbird has for the unit tests themselves that really does make it easier for developers to test, and also allow buildbot code re-use and greater inter-op with other unit tests.

Jeff Hammel

Comment 29

•

15 years ago

Attached patch patch that works with today's buildbotcustom without any functionality change (obsolete) (deleted) — Details — Splinter Review

Fix that removes self.super_class functionality change to get around python reloading. See: * https://bugzilla.mozilla.org/show_bug.cgi?id=464079 * https://bugzilla.mozilla.org/show_bug.cgi?id=460077 * http://atlee.ca/blog/2008/11/21/python-reload-danger-here-be-dragons/

Attachment #439959 - Attachment is obsolete: true

Jeff Hammel

Comment 30

•

15 years ago

(In reply to comment #28) > (In reply to comment #21) > > It was decided not to use virtualenv to install the mozmill software. > > The solution proposed is to use virtualenv and e.g. pip to create binary > > environments for each slave platform which are then uploaded to a > > mozilla URL upon packaging mozmill (and presumedly any python package > > needed for testing, see below). > > So attempting to put this simpler, I think you're saying: > > - Build (or someone) creates a virtualenv binary of the mozmill software > - Mozmill sofware gets deployed onto the slaves by pip. There are details to be worked out but what was discussed was - a virtualenv is created on each platform - pip (or other) is used to install appropriate versions of mozmill and it's dependencies - this virtualenv is tarred up and uploaded to some part of the mozilla.org URL space - the slaves retrieve the appropriate tarball and unpack it to run the contained python software > What happens when if we get the situation that one branch needs to use one > version of mozmill, and another branch wants to use a different version? in generateBranchObjects, currently the hardcoded string '1.4.1' is used regardless of branch. This could be made branch-dependent if necessary. > Is it possible for developers to run/install these without having pip setups? I'm not sure I understand who the target audience is. Buildbot developers? End developers? Since this approach is tailored very specifically to the buildbot setup, the tarballs are useless outside of this context. The slaves will not need to install pip (see above). Of course, other developers will be able to run/install mozmill in the usual way, paying no attention to these tarballs. > I'd also like to know why you're not heading down the route that Thunderbird > has for the unit tests themselves that really does make it easier for > developers to test, and also allow buildbot code re-use and greater inter-op > with other unit tests. I will look at this and assess this approach and present the alternatives here. I'm not sure if it is my decision to make, though again am for any sort of unification of process for the better.

cmtalbert

Comment 31

•

15 years ago

(In reply to comment #28) > (In reply to comment #21) > Is it possible for developers to run/install these without having pip setups? Yes, we'd want to solve that problem too to make life easier too, but first we need to get some kind of consensus on the approach here. I'm a bit baffled as to what the reasons are for some of the implicit assumptions that seem to be being made. > I'd also like to know why you're not heading down the route that Thunderbird > has for the unit tests themselves that really does make it easier for > developers to test, and also allow buildbot code re-use and greater inter-op > with other unit tests. So, let's talk about those implicit assumptions that I mentioned, and I think that the number one reason we're in this predicament is the answer to your quandary. Let's get these on the table so we're all on the same page, I'm not saying these aren't justified, but I don't think everyone shares the same mental map for the problem and that's causing us to get tangled up here. So, let's deal the cards; I see two basic assumptions in the works here: Implicit Assumption 1: * We can't install Mozmill tools on buildbot slaves. Thunderbird has taken this approach (or seems to in the code you linked us to in comment 20). For Firefox with 800 or so slaves, that's a big headache to keep everything in sync and updated. Implicit Assumption 2: * We don't want virtualenv on the slaves either. I really don't understand this one. If some of the bugs that Jeff found are fixed (like the PYTHONHOME thing) I don't know why this wouldn't be a good approach because it seems it could be generally useful for future python based projects. So, I see three possible ways to make this work: 1. Install virtualenv as part of the buildplatform on the slaves, use it for Mozmill. * This has a good incremental approach: We only need to download the virtualenv.py file and can use that without adding virtualenv to the reference image and we can get this effort off the ground quickly. * But this seems to be blocked by Implicit Assumption 2. 2. Package mozmill/virtualenv/anything else as tarballs and download them * This seems like a winner on the surface. But, when you look at what has to be packaged, we start seeing that packing up an entire virtualenv could pull in a bunch of binary components and then we'd have to start packaging these for each OS, which is going to be a much bigger maintenance burden (more error prone) than option 1. * If we just tarball the site-packages directories we might be able to avoid the per-platform binary issue, but that seems pretty hacky. 3. Use the Thunderbird method and install the mozmill tools onto the buildslaves. * This runs right into Implicit Assumption 1. Chris, Ben and John, are there other issues you see with these approaches than the ones I called out? Are there other things to consider? Let's get them all on the table so we can move forward and get this long-standing item crossed off the list. I'll volunteer to set up a call this week so we can talk through everything if that'd be easier.

Robert Kaiser

Comment 32

•

15 years ago

(In reply to comment #31) > Implicit Assumption 1: > * We can't install Mozmill tools on buildbot slaves. Thunderbird has taken > this approach (or seems to in the code you linked us to in comment 20). For > Firefox with 800 or so slaves, that's a big headache to keep everything in sync > and updated. Actually, we have puppet and OPSI which can automatically install software packages on all slaves. It's some work the releng team needs to go through to set up and test things for a new package to be deployed, but it's surely doable in a reasonable manner across all slaves.

Mark Banner (:standard8)

Comment 33

•

•

Comment 65

•

15 years ago

So, I agree with most of Clint's suggestion in comment 60. If virtualenv.py and the mozmill code is contained within mozilla-central in some form or other, that makes things really simple for the buildbot automation. All that needs to happen is for 'make packagetests' to create a relocatable virtualenv that gets packaged up with the rest of the tests. This package gets uploaded to FTP alongside the build, and then downloaded by the test machines. They merely have to run mozmill from the virtualenv to run the tests. The actual format of mozmill in the source tree doesn't matter that much to us...My $0.02 is that it's cleaner to have just the source there instead of tarballs. My original need for tarballs would go away because the exact versions are in the source tree, and would be packaged up with other tests.

Jeff Hammel

Comment 66

•

15 years ago

an initial patch integrating mozmill into m-c is here: http://k0s.org/mozilla/hg/mozilla-central-patches/file/629b462bdb1e/bug-516984 This currently only works with `make package-tests`. Working on extending.

cmtalbert

Comment 67

•

15 years ago

(In reply to comment #65) > So, I agree with most of Clint's suggestion in comment 60. > > If virtualenv.py and the mozmill code is contained within mozilla-central in > some form or other, that makes things really simple for the buildbot > automation. All that needs to happen is for 'make packagetests' to create a > relocatable virtualenv that gets packaged up with the rest of the tests. This > package gets uploaded to FTP alongside the build, and then downloaded by the > test machines. They merely have to run mozmill from the virtualenv to run the > tests. > > The actual format of mozmill in the source tree doesn't matter that much to > us...My $0.02 is that it's cleaner to have just the source there instead of > tarballs. My original need for tarballs would go away because the exact > versions are in the source tree, and would be packaged up with other tests. Sounds like a great way to move forward. We'll work toward that. Thanks Chris!

Jeff Hammel

Comment 68

•

15 years ago

I've completed a first pass of integrating mozmill and the mozilla-central build system. See patches in http://k0s.org/mozilla/hg/mozilla-central-patches/ http://k0s.org/mozilla/hg/mozilla-central-patches/file/8bfc082197c6/bug-516984 The patches are kinda hard to read due to the binary data, so if anyone wants to see any of the particular files I've patched, lemme know and I'll post them. The functionality introduced: - the source code to mozmill and its deps + virtualenv have been added to m-c - `make package-tests` and `make mozmill` from MOZ_OBJDIR now do what I hope they're supposed to do - I added a bare-bones runmozmill.py.in that gets interpolated to runmozmill.py ; The only thing I need to interpolate is the path to the browser, so maybe this is overkill? This script may certainly be extended if we want/need more of a front-end. I haven't fully grokked the build system so there are a number of things which are probably not preferable. Currently, mozmill is only packaged (via `make package-tests`) to $(DIST)/test-package-stage/mozmill . `make mozmill` depends on `stage-mozmill` in testing/testuite-targets.mk . In other words, mozmill is run from $(DIST)/test-package-stage/mozmill. This feels odd and it isn't how the other tests work, but I don't really grok how and where things are supposed to go. I'm not really sure of the intentions behind where the build system puts tests. One thing I thought of is to have a meta-target with a variable, let's call it $(DEST), and then have two other targets for packaging and in-situ testing (or whatever the intent is) which set $(DEST) to different values (one of them $(DIST)/test-package/stage/mozmill, the other one ???) and invoke the meta-target with these two values. Though, again, as best I can tell this isn't how other tests do it, though maybe mozmill+virtualenv is a different enough to warrant separate treatment? Or is the intention for what I am calling in-situ test running to keep the code in the src directory? In this case, maybe using `python setup.py install` for `make package-tests` and `python setup.py develop` for the other style of tests (making a virtualenv in both cases). The code itself (mozmill, mozrunner, jsbridge, etc) could continue to live in tarballs or be unpacked (the latter avoids a little logic, but probably not enough to be a factor). Again, I'm not sure if I have my head wrapped around how this should work, so outside of saying "worksforme" probably the best thing I can do is explain how I've been doing things and hope to defer to those that know the system better. There is a stub in MOZ_OBJDIR/testing/mozmill with just the rules for stage-package. I suppose this is the way things should work? Relatedly, perhaps, is getting the path to the binary. This is done in different ways throughout the makefiles. (I can audit, if desired, but I'm not sure if the intention differs or not). http://mxr.mozilla.org/mozilla-central/source/build/automation-build.mk has the most thorough treatment, and maybe I should just include this. If it were purely my code and I were doing this, I would break off the browser-finding part of that file into its own file were I to go down this route. For now, I did something simpler: BROWSER_PATH := $(shell $(PYTHON) -c 'import os; print os.path.abspath("$(DIST)/bin/$(MOZ_APP_NAME)$(BIN_SUFFIX)")') This worksforme. The shell call using python to find the absolute path is necessary as, while BROWSER_PATH is correct where it is interpolated from (MOZ_OBJDIR/testing/mozmill/Makefile), it is not where it is intepolated to (MOZ_OBJDIR/dist/test-package-stage/mozmill/runmozmill.py) . Alternatively, if the target in $(DIST)/test-package-stage is the only one we care about, I can just interpret relative to the file (../../../$(MOZ_APP_NAME)$(BIN_SUFFIX) or the like).

Jeff Hammel

Comment 69

•

•

15 years ago

•

15 years ago

Attached patch build modifications to mozilla-central (obsolete) (deleted) — Details — Splinter Review

only have mozmill in packaged tests; changes as per https://bugzilla.mozilla.org/show_bug.cgi?id=516984#c90

Attachment #444120 - Attachment is obsolete: true

Attachment #447580 - Flags: review?(ted.mielczarek)

(not currently active) Ted Mielczarek

Comment 92

•

15 years ago

(In reply to comment #89) > > What's with the extra set of enclosing parentheses? (They don't actually do > > anything, do they?) > > Again, this is cargo-culted. I don't believe they do anything in make > except act as delimeters. Should I get rid of them? I'm not sure if > I need to suppress echoing with the `@` either. They're not interpreted by make at all, make just passes that all down to the shell, so it's just like typing (foo) at the shell, which is equivalent to 'foo' by itself. Get rid of them. @ echo suppressing is a matter of choice. It's obviously useful for @echo foo, but if you're running actual commands you might want to drop it, since it makes it easier to see what commands are failing in build logs. I think I answered all the other questions via IRC.

(not currently active) Ted Mielczarek

Comment 93

•

15 years ago

Comment on attachment 447580 [details] [diff] [review] build modifications to mozilla-central >diff --git a/testing/mozmill/Makefile.in b/testing/mozmill/Makefile.in >new file mode 100644 >--- /dev/null >+++ b/testing/mozmill/Makefile.in >+# Harness packages from the srcdir; >+# python packages to be installed IN INSTALLATION ORDER I asked you to clarify this, but you asked me a question in return and I didn't respond. I think you should add something like "packages later in the list can depend only on packages earlier in the list." >diff --git a/toolkit/toolkit-tiers.mk b/toolkit/toolkit-tiers.mk >--- a/toolkit/toolkit-tiers.mk >+++ b/toolkit/toolkit-tiers.mk >@@ -254,12 +254,18 @@ endif > ifdef MOZ_LEAKY > tier_platform_dirs += tools/leaky > endif > > ifdef MOZ_MAPINFO > tier_platform_dirs += tools/codesighs > endif > >+# test harnesses >+ >+TIERS += testharness >+ Get rid of this, mochitest is already in tier_platform, so it doesn't seem necessary. > ifdef ENABLE_TESTS > tier_platform_dirs += testing/mochitest >+tier_testharness_dirs += testing/xpcshell >+tier_testharness_dirs += testing/mozmill and make these both tier_platform_dirs. r=me with those changes (+ the minor tweaks from my previous comment)

Attachment #447580 - Flags: review?(ted.mielczarek) → review+

Jeff Hammel

Comment 94

•

15 years ago

(In reply to comment #92) > (In reply to comment #89) > > > What's with the extra set of enclosing parentheses? (They don't actually do > > > anything, do they?) > > > > Again, this is cargo-culted. I don't believe they do anything in make > > except act as delimeters. Should I get rid of them? I'm not sure if > > I need to suppress echoing with the `@` either. > > They're not interpreted by make at all, make just passes that all down to the > shell, so it's just like typing (foo) at the shell, which is equivalent to > 'foo' by itself. Get rid of them. I stole this from http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/Makefile.in which seems to have the same issues. I wasn't sure if enclosing parens were passed to shell or if they had a special meaning to make. Thanks for the clarification. > @ echo suppressing is a matter of choice. It's obviously useful for @echo foo, > but if you're running actual commands you might want to drop it, since it makes > it easier to see what commands are failing in build logs. Done so. > I think I answered all the other questions via IRC. Fixed: http://k0s.org/mozilla/hg/mozilla-central-patches/rev/10e0fad404ac

Jeff Hammel

Comment 95

•

15 years ago

(In reply to comment #93) > (From update of attachment 447580 [details] [diff] [review]) > >diff --git a/testing/mozmill/Makefile.in b/testing/mozmill/Makefile.in > >new file mode 100644 > >--- /dev/null > >+++ b/testing/mozmill/Makefile.in > >+# Harness packages from the srcdir; > >+# python packages to be installed IN INSTALLATION ORDER > > I asked you to clarify this, but you asked me a question in return and I didn't > respond. I think you should add something like "packages later in the list can > depend only on packages earlier in the list." Done, http://k0s.org/mozilla/hg/mozilla-central-patches/rev/93a870432e8c > > >diff --git a/toolkit/toolkit-tiers.mk b/toolkit/toolkit-tiers.mk > >--- a/toolkit/toolkit-tiers.mk > >+++ b/toolkit/toolkit-tiers.mk > >@@ -254,12 +254,18 @@ endif > > ifdef MOZ_LEAKY > > tier_platform_dirs += tools/leaky > > endif > > > > ifdef MOZ_MAPINFO > > tier_platform_dirs += tools/codesighs > > endif > > > >+# test harnesses > >+ > >+TIERS += testharness > >+ > > Get rid of this, mochitest is already in tier_platform, so it doesn't seem > necessary. > > > ifdef ENABLE_TESTS > > tier_platform_dirs += testing/mochitest > >+tier_testharness_dirs += testing/xpcshell > >+tier_testharness_dirs += testing/mozmill > > and make these both tier_platform_dirs. Done, http://k0s.org/mozilla/hg/mozilla-central-patches/rev/78631196458e > r=me with those changes (+ the minor tweaks from my previous comment) Thanks! Will upload the new patch and obselete the old one

Jeff Hammel

Comment 96

•

15 years ago

Attached patch build modifications to mozilla-central (obsolete) (deleted) — Details — Splinter Review

update of the patch with :ted 's recommendations

Attachment #447580 - Attachment is obsolete: true

Jeff Hammel

Updated

•

15 years ago

Blocks: 568642

Jeff Hammel

Comment 97

•

15 years ago

(In reply to comment #90) > The scope of this bug is limited to getting mozmill working on > buildbot. Therefore, the for getting mozmill into mozilla-central > (the first step of this bug, the second being adding the code to > buildbot to run the mozmill tests), it is only necessary to get > mozmill working from the packaged tests. Currently, the state of the > patch implements functionality unnecessary to this. `make mozmill` > provides a way to run the tests, but it is dependent on `make > package-tests` step. > > Making `make mozmill` work correctly, and in a way consistent with the > other tests, will be moved to a separate bug. This bug will be > updated with a link to this other bug once filed. > > Currently, the patch to the build system (ignoring the addition of > mozmill + dependencies, virtualenv, and the tests from their canonical > sources) consists of the following files: > > Makefile.in > build/app-path.mk > testing/mozmill/Makefile.in > testing/mozmill/README.txt > testing/mozmill/runmozmill.py.in > testing/testsuite-targets.mk > toolkit/toolkit-tiers.mk > > For the packaged tests, runmozmill.py.in is not required at all. > Since app-path.mk is only interpolated for this file, it can probably > go away too (although, ideally, this should be abstracted from > automation-build.mk and the latter consume it, but it is not needed > for this purpose). `testing/mozmill/Makefile.in` should be updated to > account for these changes, and the mozmill target may be removed from > testsuite-targets.mk. The rest can remain. > > I'll make these changes and upload a revised (truncated) patch and > follow up with a bug tracking running mozmill from the object > directory as a distinct issue. The additional step necessary to run `make mozmill` has been noted as bug 568642

cmtalbert

Comment 98

•

15 years ago

Comment on attachment 447815 [details] [diff] [review] build modifications to mozilla-central Jhammel forgot to carry Ted's r+ forward.

Attachment #447815 - Flags: review+

(not currently active) Ted Mielczarek

Comment 99

•

15 years ago

(In reply to comment #94) > I stole this from > http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/Makefile.in > which seems to have the same issues. I wasn't sure if enclosing parens were > passed to shell or if they had a special meaning to make. Thanks for the > clarification. Right, they're used there because those lines are of the form: (cd foo && tar cf - *) | (cd bar && tar xf -) Where we're tarring the contents of one directory to stdout, then back from stdin to another directory.

Jeff Hammel

Comment 100

•

15 years ago

Attached patch build modifications to mozilla-central (obsolete) (deleted) — Details — Splinter Review

removes extraneous trailing slash which fails on windows builds (linux collapses multiple slashes)

Attachment #447815 - Attachment is obsolete: true

Attachment #447878 - Flags: review+

Jeff Hammel

Comment 101

•

15 years ago

Pushing to the try server, there is some issue: """ if test -d obj-firefox/ppc/dist/test-package-stage -a \ -d obj-firefox/i386/dist/test-package-stage; then \ cp obj-firefox/ppc/dist/test-package-stage/mochitest/automation.py \ obj-firefox/i386/dist/test-package-stage/mochitest/; \ cp obj-firefox/ppc/dist/test-package-stage/reftest/automation.py \ obj-firefox/i386/dist/test-package-stage/reftest/; \ /builds/slave/tryserver-macosx/build/build/macosx/universal/unify \ --unify-with-sort "all-test-dirs\.list$" \ obj-firefox/ppc/dist/test-package-stage \ obj-firefox/i386/dist/test-package-stage \ obj-firefox/ppc/dist/universal/test-package-stage; fi /builds/slave/tryserver-macosx/build/build/macosx/universal/unify: copyIfIdentical: files differ: obj-firefox/ppc/dist/test-package-stage/mozmill/bin/activate, obj-firefox/i386/dist/test-package-stage/mozmill/bin/activate make[1]: *** [postflight_all] Error 1 make: *** [build] Error 2 program finished with exit code 2 """ I'm not exactly exactly sure what this error really means. Obviously the files differ, but why is that a show-stopper?

(not currently active) Ted Mielczarek

Comment 102

•

15 years ago

For OS X, we build a universal binary. This means that we also build a universal test package to go along with it. the 'unify' script is what we use to build those universal packages. It has some pretty simple rules: 1) Mach-O binaries get lipo'ed into universal binaries 2) Other plain files must be identical (because we are only going to ship one copy) 3) zip files are identical if they have the same list of files with the same sizes and checksums 4) Directories are processed recursively 5) As a special case, filenames matching a pattern passed to the script as --unify-with-sort are considered equal if their contents are the same after sorting the lines alphabetically.

Jeff Hammel

Comment 103

•

15 years ago

(In reply to comment #102) > For OS X, we build a universal binary. This means that we also build a > universal test package to go along with it. the 'unify' script is what we use > to build those universal packages. It has some pretty simple rules: > 1) Mach-O binaries get lipo'ed into universal binaries > 2) Other plain files must be identical (because we are only going to ship one > copy) > 3) zip files are identical if they have the same list of files with the same > sizes and checksums > 4) Directories are processed recursively > 5) As a special case, filenames matching a pattern passed to the script as > --unify-with-sort are considered equal if their contents are the same after > sorting the lines alphabetically. Thanks for the info, Ted. Since the `activate` script is generated according to the given version of python, it does not surprise me that the two would be different for different platforms. In light of this, I think a better solution is to not generated the virtualenv on stage-package and to push this off to buildbot (or the developer) but to have a script to do this that will be run by e.g. buildbot. I think this actually solves a few conceptual problems that I've only recently become aware of: - virtualenv symlinks the .pyc files and not just the .py files from system. This is probably bad if the python interpreters don't match. The above approach ensures the python versions will match - currently, there is only the stage-package target in the Makefile.in . If make mozmill is to work in the standard way (see https://bugzilla.mozilla.org/show_bug.cgi?id=568642), this should make it easy to abstract this into three dummy targets: a master (e.g. install-mozmill) and two sub-targets (stage-package and a new one) that just set a variable and call install-mozmill to that directory If we go start using virtualenv for more stuff than mozmill, it should probably live somewhere else (e.g. in build), but for now it can live where it's at. Patch forthcoming

Jeff Hammel

Comment 104

•

15 years ago

Attached patch build modifications to mozilla-central, does not install the packages, just copies them (deleted) — Details — Splinter Review

version that does not actually create a virtualenv, just packages the software needed to create one and adds a courtesy install script. Notably, this script is pretty generic and has little/nothing to do with mozmill specifically. This should solve the previous issues of files differening

Attachment #447878 - Attachment is obsolete: true

Attachment #448104 - Flags: review?(ted.mielczarek)

Jeff Hammel

Comment 105

•

15 years ago

(In reply to comment #104) > Created an attachment (id=448104) [details] > build modifications to mozilla-central, does not install the packages, just > copies them > > version that does not actually create a virtualenv, just packages the software > needed to create one and adds a courtesy install script. Notably, this script > is pretty generic and has little/nothing to do with mozmill specifically. > > This should solve the previous issues of files differening Pushed to try. While the patch doesn't actually do much (all it does is include and package the tests), things seem to pass. There is burning on maemo and some oranges, but they look unrelated as best i can tell from the logs. Downloading the packaged tests, mozmill is there and it installs and runs fine.

(not currently active) Ted Mielczarek

Comment 106

•

Comment 113

•

15 years ago

•

14 years ago

Attached patch run mozmill via buildbot from packaged tests (obsolete) (deleted) — Details — Splinter Review

includes a larger timeout and corrects paths for the restart tests

Jeff Hammel

Updated

•

14 years ago

No longer depends on: 574043

John Ford [:jhford] CET/CEST Berlin Time

Comment 125

•

Attachment #461392 - Flags: review?(jhford)

Attachment #461282 - Flags: review?(jhford)

bhearsum@mozilla.com (:bhearsum)

Updated

•

14 years ago

Blocks: 588398

Jeff Hammel

Comment 130

•

14 years ago

Just a status update: the main blocker for this bug is bug 557336. One of the patches there, or variation thereon, is tentatively scheduled to land next week during a releng downtime. Once that is done, a new patch can be easily constructed here (if necessary, but likely) and then hopefully we can move to landing this in buildbotcustom.

Jeff Hammel

Comment 131

•

14 years ago

Comment on attachment 461392 [details] [diff] [review] run mozmill via buildbot inheriting from MozillaPackagedTest factory out of date patch. will post a new one soon

Attachment #461392 - Flags: review?(jhford)

Jeff Hammel

Comment 132

•

14 years ago

Attached patch mozmill + buildbot patch (obsolete) (deleted) — Details — Splinter Review

This successfully runs the needed mozmill commands via buildbot. the mozmill run is not successful due to https://bugzilla.mozilla.org/show_bug.cgi?id=585086 but this should be landed such that mozmill may be turned on when the software and the tests are updated and reliably green

Attachment #472674 - Flags: review?(jhford)

John Ford [:jhford] CET/CEST Berlin Time

Comment 133

•

•

14 years ago

Comment on attachment 461285 [details] buildbot master.cfg used for testing, for reference My new master.cfg is somewhat different. Can upload if needed

Attachment #461285 - Attachment is obsolete: true

John Ford [:jhford] CET/CEST Berlin Time

Updated

•

14 years ago

Attachment #461285 - Attachment mime type: application/octet-stream → text/plain

Jeff Hammel

Updated

•

14 years ago

Attachment #461392 - Attachment is obsolete: true

Jeff Hammel

Comment 140

•

14 years ago

•

14 years ago

OS: Linux → All

Hardware: x86 → All

John Ford [:jhford] CET/CEST Berlin Time

Comment 150

•

14 years ago

That's really good to hear!

Jeff Hammel

Comment 151

•

14 years ago

(In reply to comment #143) > Comment on attachment 473287 [details] [diff] [review] <snip/> > Looking at > > >+ self.addStep(unittest_steps.MozmillStep( > >+ name="run_mozmill", > <snip> > >+ self.addStep(unittest_steps.MozmillStep( > >+ name="run_mozmill_restart", > > I wonder what the approximate amount of time that the two sets take to > complete? For the full suite of tests, it is currently 6m49s for the regular tests and 3m45s for the restart tests. Only a subset of these tests will initially be landed on m-c so the runs will initially be even shorter. > If they are both particularly long it might be worthwhile to split > them into two separate builders. To do that we would have to be able to create > factories that have either plain tests or restart tests. So I see three options for this: 1. Keep things as they are. The easiest option, if this isn't going to be a blocker. 2. Have a __init__ argument for the factory that tells which test suite to run. 3. Have a subclass for the mozmill-restart tests whose only difference is to run the restart tests instead of the normal mozmill tests. I'd prefer 1., since it is easiest and we can do course correction if the ~10min (or less) run time will be a concern. Any objections?

Jeff Hammel

Comment 152

•

14 years ago

John Ford [:jhford] CET/CEST Berlin Time

Comment 161

•

14 years ago

Comment on attachment 475672 [details] [diff] [review] mozmill in UnittestPackagedBuildFactory >@@ -6492,17 +6492,17 @@ class UnittestPackagedBuildFactory(Mozil > thisChunk=None, chunkByDir=None, **kwargs): > platform = platform.split('-')[0] > self.test_suites = test_suites > self.totalChunks = totalChunks > self.thisChunk = thisChunk > self.chunkByDir = chunkByDir > self.env = MozillaEnvironments['%s-unittest' % platform].copy() > self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(platform) >- self.env.update(env) >+ self.env.update(env or {}) I am not sure if this is the right bug to change this behaviour in. If you'd like to make the current behaviour more obvious, lets assert env. >+ elif suite == 'mozmill': >+ >+ # install mozmill into its virtualenv >+ self.addStep(ShellCommand( >+ name='install mozmill', >+ command=['python', >+ 'mozmill/installmozmill.py'], >+ flunkOnFailure=True, >+ haltOnFailure=True, >+ )) >+ >+ env = {'PYTHONHOME': '.'} Is this so that the slave's environment doesn't break mozmill? Can we instead use python -E in the MozmillStep? >+ >+ # run the mozmill tests >+ self.addStep(unittest_steps.MozillaPackagedMozmillTests( >+ name="run_mozmill", >+ tests_dir='tests/firefox', >+ binary=WithProperties('%(basedir)s/build/%(exepath)s'), >+ platform=self.platform, >+ env=env, >+ workdir='build/mozmill', >+ timeout=10*60, >+ flunkOnFailure=True >+ )) >+ self.addStep(unittest_steps.MozillaPackagedMozmillTests( >+ name="run_mozmill_restart", >+ tests_dir='tests/firefox/restartTests', >+ binary=WithProperties('%(basedir)s/build/%(exepath)s'), platform=self.platform, Assuming that this is a typo but please fix for the next patch >+ env=env, >+ workdir='build/mozmill', >+ timeout=5*60, >+ flunkOnFailure=True >+ )) I don't think it is critical, but it would be nice if the workdir parameter used the default of 'build'. This would be consistent with how we have the other tests set up. We could do this by passing in a mozmill subdirectory and changing paths as appropriate in the step. We could avoid using absolute paths. Not going to r- on this though. As you mentioned today, the restart tests need to use mozmill-restart instead of mozmill. >+def summarizeMozmillLog(name, log): >+ log = log.getText() >+ m = re.search("^Passed: (\d+) :: Failed (\d+) :: Skipped (\d+)", log, re.M) >+ if not m: >+ return "TinderboxPrint: %s %s\n" % (name, summaryText(0, 0)) >+ else: >+ return "TinderboxPrint: %s %s\n" % (name, summaryText( >+ int(m.group(1)), >+ int(m.group(2)), >+ )) >+ This is not valid for the current output of the harness. The TinderboxPrint should be similar to: TinderboxPrint: harness-name pass_count/fail_count/skip_count You might find http://hg.mozilla.org/build/buildbotcustom/file/tip/steps/unittest.py#l43 useful >+class MozillaPackagedMozmillTests(ShellCommandReportTimeout): >+ warnOnFailure=True >+ warnOnWarnings=True >+ >+ # XXX the log parsing will need to be redone when Mozmill 1.5 >+ # lands in mozilla-central >+ # see https://bugzilla.mozilla.org/show_bug.cgi?id=585086 >+ # Mozmill 1.5 conforms close to standard Mozilla logging conventions >+ >+ def __init__(self, tests_dir, binary, platform, restart=False, **kwargs): >+ """ >+ run mozmill tests; >+ should be run from the mozmill virtualenv directory >+ - tests_dir : path to the tests to run >+ - binary : path to the binary to be used >+ - platform : system platform >+ - restart : whether to run the restart tests >+ """ >+ >+ self.super_class = ShellCommandReportTimeout >+ ShellCommandReportTimeout.__init__(self, **kwargs) >+ >+ self.addFactoryArguments(tests_dir=tests_dir, >+ binary=binary, >+ platform=platform, >+ restart=restart) If it is realistic that someone would ever use a script other than 'mozmill' or 'mozmill-restart', could we instead pass in a mozmill_script_name variable? If it defaulted to 'mozmill', we could override for the restart tests >+ >+ # set platform-dependent python scripts >+ # relative to the created virtualenv >+ if platform == "win32": >+ python = r'Scripts\python' >+ if restart: >+ mozmill = r'Scripts\mozmill-restart-script.py' >+ else: >+ mozmill = r'Scripts\mozmill-script.py' >+ else: >+ python = 'bin/python' >+ if restart: >+ mozmill = 'bin/mozmill-restart' >+ else: >+ mozmill = 'bin/mozmill' >+ >+ >+ self.command = [ >+ python, >+ mozmill, >+ '--showall', >+ '-b', binary, >+ '-t', tests_dir, >+ ] >+ >+ def createSummary(self, log): >+ self.addCompleteLog('summary', summarizeMozmillLog(self.name, log)) >+ >+ def evaluateCommand(self, cmd): >+ superResult = self.super_class.evaluateCommand(self, cmd) >+ if superResult != SUCCESS: >+ return superResult >+ >+ # If we find "Test-Failed", then return WARNINGS >+ if "Test-Failed" in cmd.logs['stdio'].getText(): >+ return WARNINGS You mention moving towards standard logging output. Lots of other frameworks print out TEST-UNEXPECTED-PASS and TEST-UNEXPECTED-FAIL. Maybe we can add a check for TEST-UNEXPECTED This patch isn't ready for production, but it is looking good. I will run it through staging again when the patches have landed on mozilla-central.

Attachment #475672 - Flags: review?(jhford) → review-

Jeff Hammel

Comment 162

•

14 years ago

(In reply to comment #161) > Comment on attachment 475672 [details] [diff] [review] > mozmill in UnittestPackagedBuildFactory > > >@@ -6492,17 +6492,17 @@ class UnittestPackagedBuildFactory(Mozil > > thisChunk=None, chunkByDir=None, **kwargs): > > platform = platform.split('-')[0] > > self.test_suites = test_suites > > self.totalChunks = totalChunks > > self.thisChunk = thisChunk > > self.chunkByDir = chunkByDir > > self.env = MozillaEnvironments['%s-unittest' % platform].copy() > > self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(platform) > >- self.env.update(env) > >+ self.env.update(env or {}) > > I am not sure if this is the right bug to change this behaviour in. If you'd > like to make the current behaviour more obvious, lets assert env. Should I file this as a separate bug then? Passing env=None and then doing a dictionary update that will fail on the default value sure seems like undesired behaviour. It is not important for this bug, though it does make testing more difficult > > >+ elif suite == 'mozmill': > >+ > >+ # install mozmill into its virtualenv > >+ self.addStep(ShellCommand( > >+ name='install mozmill', > >+ command=['python', > >+ 'mozmill/installmozmill.py'], > >+ flunkOnFailure=True, > >+ haltOnFailure=True, > >+ )) > >+ > >+ env = {'PYTHONHOME': '.'} > > Is this so that the slave's environment doesn't break mozmill? Can we instead > use python -E in the MozmillStep? The only reason I didn't use python -E is I wasn't sure if any of the other python variables were defined or depended on on the slaves. If they aren't, we can certainly use python -E. Are they? I'll address the other issues tomorrow.

John Ford [:jhford] CET/CEST Berlin Time

Comment 163

•

14 years ago

(In reply to comment #162) > (In reply to comment #161) > > Comment on attachment 475672 [details] [diff] [review] [details] > > mozmill in UnittestPackagedBuildFactory > > > > >@@ -6492,17 +6492,17 @@ class UnittestPackagedBuildFactory(Mozil > > > thisChunk=None, chunkByDir=None, **kwargs): > > > platform = platform.split('-')[0] > > > self.test_suites = test_suites > > > self.totalChunks = totalChunks > > > self.thisChunk = thisChunk > > > self.chunkByDir = chunkByDir > > > self.env = MozillaEnvironments['%s-unittest' % platform].copy() > > > self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(platform) > > >- self.env.update(env) > > >+ self.env.update(env or {}) > > > > I am not sure if this is the right bug to change this behaviour in. If you'd > > like to make the current behaviour more obvious, lets assert env. > > Should I file this as a separate bug then? Passing env=None and then doing a > dictionary update that will fail on the default value sure seems like undesired > behaviour. It is not important for this bug, though it does make testing more > difficult Instead of changing this behaviour, lets make it an explicit |assert env, 'UnittestPackagedBuildFactory needs an environment'|. This requires no extra testing. > > Is this so that the slave's environment doesn't break mozmill? Can we instead > > use python -E in the MozmillStep? > > The only reason I didn't use python -E is I wasn't sure if any of the other > python variables were defined or depended on on the slaves. If they aren't, we > can certainly use python -E. Are they? > > I'll address the other issues tomorrow. ShellCommands work at the low level by running an os.exec* function. This shouldn't have any impact on the running buildbot. If mozmill works with python -E, lets use it and remove a potential source of future errors :) It seems like there were two blocking issues for this landing. For the next round of testing, I'd like to do it in our staging environment on all the platforms we want to deploy to. For this, can we make sure that there is a build available for windows, linux, linux64, mac 32bit and mac64bit. It doesn't matter to me if they are mozilla-central tinderbox/nightly/try builds or one-off testing builds like the one Clint did today.

Jeff Hammel

Comment 164

•

14 years ago

(In reply to comment #163) > (In reply to comment #162) > > (In reply to comment #161) > > > Comment on attachment 475672 [details] [diff] [review] [details] [details] > > > mozmill in UnittestPackagedBuildFactory > > > > > > >@@ -6492,17 +6492,17 @@ class UnittestPackagedBuildFactory(Mozil > > > > thisChunk=None, chunkByDir=None, **kwargs): > > > > platform = platform.split('-')[0] > > > > self.test_suites = test_suites > > > > self.totalChunks = totalChunks > > > > self.thisChunk = thisChunk > > > > self.chunkByDir = chunkByDir > > > > self.env = MozillaEnvironments['%s-unittest' % platform].copy() > > > > self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(platform) > > > >- self.env.update(env) > > > >+ self.env.update(env or {}) > > > > > > I am not sure if this is the right bug to change this behaviour in. If you'd > > > like to make the current behaviour more obvious, lets assert env. > > > > Should I file this as a separate bug then? Passing env=None and then doing a > > dictionary update that will fail on the default value sure seems like undesired > > behaviour. It is not important for this bug, though it does make testing more > > difficult > > Instead of changing this behaviour, lets make it an explicit |assert env, > 'UnittestPackagedBuildFactory needs an environment'|. This requires no extra > testing. As you point out, this really has nothing to do with this bug. Maybe we should just leave as is and ticket separately if desired. > > > > Is this so that the slave's environment doesn't break mozmill? Can we instead > > > use python -E in the MozmillStep? > > > > The only reason I didn't use python -E is I wasn't sure if any of the other > > python variables were defined or depended on on the slaves. If they aren't, we > > can certainly use python -E. Are they? > > > > I'll address the other issues tomorrow. > > ShellCommands work at the low level by running an os.exec* function. This > shouldn't have any impact on the running buildbot. If mozmill works with > python -E, lets use it and remove a potential source of future errors :) Yeah, mozmill should work fine with it. Will do. > It seems like there were two blocking issues for this landing. For the next > round of testing, I'd like to do it in our staging environment on all the > platforms we want to deploy to. For this, can we make sure that there is a > build available for windows, linux, linux64, mac 32bit and mac64bit. It > doesn't matter to me if they are mozilla-central tinderbox/nightly/try builds > or one-off testing builds like the one Clint did today.

Jeff Hammel

Comment 165

•

14 years ago

Attached patch mozmill + buildbot patch (obsolete) (deleted) — Details — Splinter Review

•

14 years ago

Comment on attachment 476361 [details] [diff] [review] mozmill + buildbot patch As discussed "^INFO Passed: (\d+).*\sINFO Failed: (\d+)\sINFO Skipped: (\d+)" should really be "^INFO Passed: (\d+).*\sINFO Failed: (\d+).*\sINFO Skipped: (\d+)" I can do that as a part of landing the patch. I will add this to the next bug.

Attachment #476361 - Flags: review?(jhford) → review+

John Ford [:jhford] CET/CEST Berlin Time

Comment 173

•

14 years ago

Attached patch buildbot-configs v1 (deleted) — Details — Splinter Review

add the mozmill-all test suite

Attachment #476361 - Attachment is obsolete: true

Attachment #477196 - Flags: review?(aki)

John Ford [:jhford] CET/CEST Berlin Time

Comment 174

•

14 years ago

Comment on attachment 476361 [details] [diff] [review] mozmill + buildbot patch whoops

Attachment #476361 - Attachment is obsolete: false

Aki Sasaki (not active)

Updated

•

14 years ago

Attachment #477196 - Flags: review?(aki) → review+

John Ford [:jhford] CET/CEST Berlin Time

Updated

•

14 years ago

Blocks: releng-downtime

John Ford [:jhford] CET/CEST Berlin Time

Comment 175

•

14 years ago

Further testing has shown that tests aren't working on windows because of the mozmill installation. The current plan is for this to be fixed an landed before the downtime on Thursday.

Jeff Hammel

Comment 176

•

14 years ago

Attached patch fix for installing mozmill on windows (deleted) — Details — Splinter Review

there is an error for windows because windows uses different paths for everything....not sure why, but hopefully this fixes

Attachment #477279 - Flags: review?(ctalbert)

Jeff Hammel

Comment 177

•

14 years ago

(In reply to comment #175) > Further testing has shown that tests aren't working on windows because of the > mozmill installation. > > The current plan is for this to be fixed an landed before the downtime on > Thursday. The patch in https://bug516984.bugzilla.mozilla.org/attachment.cgi?id=477279 should fix, I think

cmtalbert

Updated

•

14 years ago

Attachment #477279 - Flags: review?(ctalbert) → review+

cmtalbert

Comment 178

•

14 years ago

Comment 206

•

14 years ago

results from staging on Releng systems. All mozilla-central builds: 10.5.8 debug all:24/1/0 restart:2/0/0 10.5.8 opt all:25/0/0 restart:2/0/0 10.6.2 debug all:24/1/0 restart:2/0/0 10.6.2 opt all:25/0/0 restart:2/0/0 f12-64 debug all:24/1/0 restart:2/0/0 f12-64 opt all:24/1/0 restart:2/0/0 f12-32 opt all:25/0/0 restart:2/0/0 f12-32 debug all:24/1/0 restart:2/0/0 The windows slave that I am using is in a bad state, so I don't have results for it. I will get this fixed asap to get real results Notes: -Fedora 12 shows 'T-FAIL' instead of the pass/fail/known even if tests pass -All failures are mozmill/tests/firefox/testPrivateBrowsing/testStartStopPBMode.js | testStopPrivateBrowsingMode This looks like great progress, but not quite ready to land. [1]TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/mozilla-central_leopard-o-debug_test-mozmill-all/build/mozmill/tests/firefox/testPrivateBrowsing/testStartStopPBMode.js | testStopPrivateBrowsingMode [2]TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/mozilla-central_snowleopard-debug_test-mozmill-all/build/mozmill/tests/firefox/testPrivateBrowsing/testStartStopPBMode.js | testStopPrivateBrowsingMode [3]TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/mozilla-central_fedora64_test-mozmill-all/build/mozmill/tests/firefox/testPrivateBrowsing/testStartStopPBMode.js | testStopPrivateBrowsingMode [4]TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/mozilla-central_fedora64-debug_test-mozmill-all/build/mozmill/tests/firefox/testPrivateBrowsing/testStartStopPBMode.js | testStopPrivateBrowsingMode

John Ford [:jhford] CET/CEST Berlin Time

Comment 207

•

14 years ago

win7 opt all:25/0/0 restart:2/0/0 I noticed the same T-FAIL behaviour for the summary even though it passed

Jeff Hammel

Comment 208

•

14 years ago

•

Closed: 14 years ago

Resolution: --- → FIXED

John Ford [:jhford] CET/CEST Berlin Time

•

14 years ago

Attachment #489065 - Flags: review?(aki)

Aki Sasaki (not active)

Comment 217

•

14 years ago

Comment on attachment 489065 [details] [diff] [review] follow up to address indentation We're going to need a mini postmortem on this to figure out how this regressed.

Attachment #489065 - Flags: review?(aki) → review+

John Ford [:jhford] CET/CEST Berlin Time

Comment 218

•

14 years ago

Comment on attachment 489065 [details] [diff] [review] follow up to address indentation http://hg.mozilla.org/build/buildbotcustom/rev/5a7cdaecd158

John Ford [:jhford] CET/CEST Berlin Time

Updated

•

14 years ago

Flags: needs-reconfig+

John Ford [:jhford] CET/CEST Berlin Time

Comment 219

•

14 years ago

landed, waiting to see passing tests not have T-FAIL

John Ford [:jhford] CET/CEST Berlin Time

Comment 220

•

14 years ago

this is working now.

•

11 years ago

Assignee: jhammel → nobody

Nobody; OK to take it and work on it

Assignee

Updated

•

8 years ago

Product: Testing → Testing Graveyard

WIP 15 years ago Chris AtLee [:catlee] (deleted), patch		Details \| Diff \| Splinter Review
port of attachment 433163 to the code today, March 24, 2010; not yet tested 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
new version of mozmill + buildbot patch, unclean 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
clean version of mozmill+buildbot integration using virtualenv (semi-tested) 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
port of patch to work with buildbotcustom as of today [April 19, 2010] 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
patch that works with today's buildbotcustom without any functionality change 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
patch to mozilla-central without python packages 15 years ago Jeff Hammel (deleted), patch	ted : review-	Details \| Diff \| Splinter Review
build modifications to mozilla-central 15 years ago Jeff Hammel (deleted), patch	ted : review+	Details \| Diff \| Splinter Review
build modifications to mozilla-central 15 years ago Jeff Hammel (deleted), patch	cmtalbert : review+	Details \| Diff \| Splinter Review
build modifications to mozilla-central 15 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
build modifications to mozilla-central, does not install the packages, just copies them 15 years ago Jeff Hammel (deleted), patch	ted : review+	Details \| Diff \| Splinter Review
copy the environment instead of popping PYTHONHOME as the latter isn't portable 15 years ago Jeff Hammel (deleted), patch	ted : review+	Details \| Diff \| Splinter Review
run mozmill via buildbot from packaged tests 15 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
run mozmill via buildbot from packaged tests 14 years ago Jeff Hammel (deleted), patch	jhford : review-	Details \| Diff \| Splinter Review
run mozmill via buildbot inheriting from MozillaPackagedTest factory 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
run mozmill via buildbot inheriting from MozillaPackagedTest factory 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
buildbot master.cfg used for testing, for reference 14 years ago Jeff Hammel (deleted), text/plain		Details
run mozmill via buildbot inheriting from MozillaPackagedTest factory 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
mozmill + buildbot patch 14 years ago Jeff Hammel (deleted), patch	jhford : review-	Details \| Diff \| Splinter Review
mozmill + buildbot patch 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
mozmill + buildbot patch 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
mozmill in UnittestPackagedBuildFactory 14 years ago Jeff Hammel (deleted), patch	jhford : review-	Details \| Diff \| Splinter Review
mozmill + buildbot patch 14 years ago Jeff Hammel (deleted), patch	jhford : review+	Details \| Diff \| Splinter Review
buildbot-configs v1 14 years ago John Ford [:jhford] CET/CEST Berlin Time (deleted), patch	mozilla : review+	Details \| Diff \| Splinter Review
fix for installing mozmill on windows 14 years ago Jeff Hammel (deleted), patch	cmtalbert : review+	Details \| Diff \| Splinter Review
Add a restart test 14 years ago cmtalbert (deleted), patch	whimboo : review+	Details \| Diff \| Splinter Review
mozmill + buildbot patch 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
buildbot configs example patch 14 years ago Jeff Hammel (deleted), patch		Details \| Diff \| Splinter Review
mozmill buildbotcustom patch 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
mozmill buildbotcustom patch 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
mozmill buildbotcustom patch 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
mozmill+buildbot; fix whitespace issue 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
mozmill+buildbot; fix whitespace issue 14 years ago Jeff Hammel (deleted), patch	k0scist : review+	Details \| Diff \| Splinter Review
buildbot-configs v1 14 years ago John Ford [:jhford] CET/CEST Berlin Time (deleted), patch	armenzg : review+	Details \| Diff \| Splinter Review
follow up to address indentation 14 years ago John Ford [:jhford] CET/CEST Berlin Time (deleted), patch	mozilla : review+	Details \| Diff \| Splinter Review