l10n-linux-tbox, bm-xserve12 and l10n-win32-tbox generate L10n nightly repackages sometime around 9 am PDT every day and tinderbox gets called with --interval 1800 We would like to this work under generic build slaves machines in production-1.9-master. This requires initially to be able to do this in staging-1.9-master by updating tinder-config.pl in the "l10n" branch of it. NOTE: These machines also do Tb-Trunk-L10n, therefore, these machines won't be able to be shut down even if we move firefox l10n repackages to buildbot

Comment on attachment 329587 [details] [diff] [review] linux/tinder-config.pl ("l10n" branch) >Index: mozconfig >=================================================================== >RCS file: /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/mozconfig,v >retrieving revision 1.2.6.8 ... > mk_add_options MOZ_CO_LOCALES=all >-mk_add_options LOCALES_CVSROOT=:ext:ffxbld@cvs.mozilla.org:/l10n >+mk_add_options LOCALES_CVSROOT=cltbld@cvs.mozilla.org:/l10n You should use a CONFIG here too, i.e # CONFIG: mk_add_options LOCALES_CVSROOT=%l10nCvsroot% >Index: tinder-config.pl >=================================================================== >RCS file: /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v >retrieving revision 1.8.2.16 ... >-$moz_cvsroot = ":ext:ffxbld\@cvs.mozilla.org:/cvsroot"; >+$moz_cvsroot = 'cltbld@cvs.mozilla.org:/cvsroot'; Why not use: # CONFIG: $moz_cvsroot = '%mozillaCvsroot%'; > %WGetFiles = ( > "http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/firefox-%version%.en-US.mac.dmg" => Did you decide to continue to d/l the production en-US builds ? I think we should have a chat to sync me >+# CONFIG: $ssh_server = "%sshUser%"; s/server/user/ > $ssh_user = "ffxbld"; >+# CONFIG: $ssh_key = "'$ENV{HOME}/.ssh/%sshUser%_dsa'"; > $ssh_key = "'$ENV{HOME}/.ssh/ffxbld_dsa'"; There's no cltbld_dsa file on the staging machines so this will fail when setting sshUser = cltbld for a staging config. * Why only mac config files here ? * Which bootstrap.cfg will be used with this ? We should probably have a chat to get me up to speed with your progress over the last two weeks.

It fixes all that you mention in the previous patch It has all 3 platforms and the bootstrap.cfg file is this: http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-nightly-staging-bootstrap.cfg Once it works there, the next one would be: http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-nightly-bootstrap.cfg I would really love to have that meeting

Nick, I have fixed the %version% substitution problem by not touching it and using $BaseUrlPath to determine the appropriate ftpServer for staging purposes I also wget to /build/tinderbox/Fx-Trunk-l10n (which is %l10n_buildDir%) instead of %l10n_buildDir%/%l10n_plafotm% because of the kernel version problem BTW, the kernel version problem does not affect normal builds, just the wget feature in l10n This patch helps us do repackages for nightlies under our release automation system - the same way we do l10n repackages for releases The log of it http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Staging/1216315503.1216318159.7817.gz&fulltext=1 I will be now be working on running it in another way that will do repackage-push-announce per locale instead of repackaging all locales, pushing all of them and then announcing them all I will also make sure that windows and mac will be fine with the patch I gave you

Comment on attachment 329720 [details] [diff] [review] add linux and windows slaves to staging and production 1.9 >Index: automation/production-1.9/master.cfg >=================================================================== >RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v >retrieving revision 1.31 >diff -u -8 -p -r1.31 master.cfg >--- automation/production-1.9/master.cfg 2 Jul 2008 12:01:35 -0000 1.31 >+++ automation/production-1.9/master.cfg 15 Jul 2008 20:35:09 -0000 >@@ -12,17 +12,19 @@ c = BuildmasterConfig = {} > # tuple of bot-name and bot-password. These correspond to values given to the > # buildslave's mktap invocation. > > from buildbot.buildslave import BuildSlave > > c['slaves'] = [ > BuildSlave("production-1.9-master",""), > BuildSlave("fx-linux-1.9-slave2",""), >+ BuildSlave("fx-linux-1.9-slave4",""), > BuildSlave("fx-win32-1.9-slave2", ""), >+ BuildSlave("fx-win32-1.9-slave4",""), > BuildSlave("fx-linux64-1.9-slave1", ""), > BuildSlave("fx-mac-1.9-slave2", ""), > ] > > # 'slavePortnum' defines the TCP port to listen on. This must match the value > # configured into the buildslaves (with their --master option) > c['slavePortnum'] = 9989 > >Index: automation/staging-1.9/master.cfg >=================================================================== >RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v >retrieving revision 1.40 >diff -u -8 -p -r1.40 master.cfg >--- automation/staging-1.9/master.cfg 3 Jun 2008 11:49:07 -0000 1.40 >+++ automation/staging-1.9/master.cfg 15 Jul 2008 20:35:09 -0000 >@@ -12,17 +12,19 @@ c = BuildmasterConfig = {} > # tuple of bot-name and bot-password. These correspond to values given to the > # buildslave's mktap invocation. > > from buildbot.buildslave import BuildSlave > > c['slaves'] = [ > BuildSlave("staging-1.9-master",""), > BuildSlave("fx-linux-1.9-slave1",""), >+ BuildSlave("fx-linux-1.9-slave3",""), > BuildSlave("fx-win32-1.9-slave1", ""), >+ BuildSlave("fx-win32-1.9-slave3",""), > BuildSlave("mini-test", ""), > BuildSlave("fx-mac-1.9-slave1", ""), > ] > > # 'slavePortnum' defines the TCP port to listen on. This must match the value > # configured into the buildslaves (with their --master option) > c['slavePortnum'] = 9989 > Are you going to do anything with them...? Let's add them to the list when they're attached to a builder.

Comment on attachment 330072 [details] [diff] [review] tinder-config.pl for all 3 platforms in the "l10n" branch As per talk in IRC, I will be pulling from the real ftp instead of the staging ftp. I will modify the patch accordingly

They will WGet from the official ftp server instead of adding the extra code to allow grabbing it from the staging ftp server I would like this patch in even if we move away from Repack.pm and we do repackages according to attachment 330636 [details] [diff] [review] to have the configuration the same way we have it in the "release_l10n" branch

Comment on attachment 330666 [details] [diff] [review] staging1.9/master.cfg - It shows what it looks like repackages without tinderbox client or bootstrap >+import subprocess >+ >+def getLocales(): >+ """ >+ It returns a list with all locales from all-locales in the repository >+ """ >+ from twisted.python import log >+ tuple = subprocess.Popen( >+ ['cvs', '-q', '-d', ':ext:stgbld@cvs.mozilla.org:/cvsroot', >+ 'co', '-p', 'mozilla/browser/locales/all-locales'], >+ stdout=subprocess.PIPE).communicate() >+ #communicate() returns a tuple >+ #the output of cvs has a \n at the end >+ locales = tuple[0].split('\n') >+ if locales[-1]=='':#I don't think that I need this 2 lines >+ locales = locales[0:-1] #get rid of the last one >+ return locales >+ nthomas, bhearshum - where do you think I should put this functionality? >+zipin = 'firefox-3.0.2pre.en-US.linux-i686.tar.bz2' >+latest_en_US = 'http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/firefox-3.0.2pre.en-US.linux-i686.tar.bz2' >+ >+ >+L10nBuildFactory.addStep(ShellCommand( >+ command=["wget", "-nv", "--output-document", >+ zipin, latest_en_US], >+ workdir="mozilla/dist/" >+ )) >+ The only thing that we need in here is to know which *version* (3.0.2pre - http://mxr.mozilla.org/mozilla/source/browser/config/version.txt) I could do something like what I do with getLocales() I think in moz2 we have if conditions to determine the platform we are using and I could have a look to learn how to determine which file type I should be wgetting It would be good if there was a make target something like: "make wget-%locale" and by giving en-US, it would download the appropriate binary. This way we wouldn't have to figure it out in the chosen automation tool, in this case, buildbot

The current version in RCS (CVS or hg) is not necessarily the version of the latest nightly, our l10n trees burn reliably on each version bump for not finding the latest build of a release that wasn't built yet. This is a chicken and egg problem, you need the source to get the version, you need the version to get the binary, you need the binary to get the source stamp. IMHO, this should be solved in the automation.

I want to bring to the attention one thing The configuration that I attached it iterates in the list of locales and adds a couple of steps per locale there is. the problem with this is that you can't add anything like haltOnFailure since it would stop the whole process. Why do I bring this to our attention? From what I learned in Axel's work that a build request is created per locale. Why would this be important? Because we could stop the creation of a locale if it doesn't pass the compare-locales test. Axel does it makes sense?

You can use it like this: export FTP_PATH=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0 make wget-en-US and it will be downloaded to $(abs_dist) and there is where make installers-FOO looks for

The same as before but for CVS instead of HG

I'd r- attachment 330832 [details] [diff] [review] and attachment 330833 [details] [diff] [review]. make echo-variable-PACKAGE does most of this right away. And the patches don't cover that windows needs both the PACKAGE and installer. Regarding your question in comment 13, I'll whack up a quick blog post with my thoughts.

(In reply to comment #16) > I'd r- attachment 330832 [details] [diff] [review] and attachment 330833 [details] [diff] [review]. > > make echo-variable-PACKAGE > > does most of this right away. > I tried it and it works if all I want is to echo the filename, but I want the build system to know how to download the binaries for me without having to call tinderbox or a custom class of buildbot to be able to do a wget

Either way you shouldn't replicate the file name logic but just use $(PACKAGE) instead. I'm not sure if wget is actually part of our build requirements, btw. And I don't know if a wget is really all that interesting if you need to pass in the right ftp path after all. And I don't see great value in coding up our ftp infrastructure in the build system. Btw, 0.7.9 has a SetProperty step, with which you could just get the output of make echo-variable-PACKAGE and feed it to a build property, to be used in a buildbot download step.

(In reply to comment #18) > Either way you shouldn't replicate the file name logic but just use $(PACKAGE) > instead. > Right - I will modify to reuse it > I'm not sure if wget is actually part of our build requirements, btw. And I > don't know if a wget is really all that interesting if you need to pass in the > right ftp path after all. And I don't see great value in coding up our ftp > infrastructure in the build system. > The ftp info cannot be extracted as you mention from our build system and it would not make sense to store it in it I think that being able to do something like: command=["make","wget-en-US"] and env={FTP_PATH:"http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0"} platform independent is IMHO an advantage and we don't have to rely on buildbot > Btw, 0.7.9 has a SetProperty step, with which you could just get the output of > make echo-variable-PACKAGE and feed it to a build property, to be used in a > buildbot download step. > That is a great step and would be good to have it now (aside of this specific problem) but we currently don't

Axel, what is the echo variable that gives us the installer for windows? I have been looking for a while

toolkit/mozapps/installer/windows/nsis/makensis.mk hardcodes that to $(DIST)/install/sea/$(PKG_BASENAME).installer.exe I guess you could make it a variable there, and use that from browser/locales/Makefile.in. You should test that thoroughly as we're using PACKAGE twice, once with AB_CD = $(MOZ_UI_LOCALE) and once with AB_CD = $*. Probably should be good enough to make sure it's a foo=bar var instead of foo:=bar, but you should test that.

(In reply to comment #21) > toolkit/mozapps/installer/windows/nsis/makensis.mk hardcodes that to > $(DIST)/install/sea/$(PKG_BASENAME).installer.exe > > I guess you could make it a variable there, and use that from > browser/locales/Makefile.in. > What do you think of calling it like PACKAGE_INSTALLER or WIN32_INSTALLER/ > You should test that thoroughly as we're using > PACKAGE twice, once with AB_CD = $(MOZ_UI_LOCALE) and once with AB_CD = $*. > Probably should be good enough to make sure it's a foo=bar var instead of > foo:=bar, but you should test that. > I am not sure what you mean but I will make sure it works in all 3 platforms. My make gnu skills are limited :)

You should check in with Ted, Benjamin or Rob Strong on the variable name. And if it should include install/sea/ or not. I'd be in favor of including it.

Axel, I have tried to set PACKAGE_INSTALLER in the file you mentioned but it did not capture it I am going to use the .installer.exe way Ted, I will add the hg patch afterwards

I just wanted you to have a look to know where I am This works so far but there are some things missing - Upload the windows installer - Upload the xpi's with the right name, instead of firefox-3.0.2pre.af.xpi (I can't recall the name) have af.xpi (which is what I have seen in ftp) - Enable update for locales - ** I am going to need as much help as possible - I know nothing ** - getLocales() - I bet it won't detect new locales added unless the configuration is read (buildbot restart) - am I right? - What is the behaviour if I use "for locale in getLocales()", is it the same? * More testing * - getLocales() should accept different cvsroot (staging and production wise) - I would like to try having a DependentL10n/SchedulerL10n scheduler with the subclassed Build class that worked for me. This allow us to see how the parallelization will work but with just one slave Once we have the nightly L10n repackages having the right configuration under buildbot. 1) What do we do with *dependent L10n repackages*? Shall we do dependent repackages every 2 hours and push them to tinderbox-builds? Could we stop calling it "dependent" builds and remove objdir and source but push it to tinderbox-builds? Shall we use a poller and just repackage those locales that have changed? 2) Whatever I make work for nightly builds should work for releases What else is done after a repackage to be called a release? Updating? Branding as well. Pulling a TAG instead of locale's tip

This is the HG patch

This would be the classes that will be used to make paralell builds. This is not final since I have to fix a bug regarding multiple builders

Nick, how does it look? Do you know what am I doing when we put it in connection to the previous patch? As I mentioned before I have to fix the multiple builders scenario This patch is obviously not final but it looks a lot on how it should be * remove the remove step * remove the step to apply my patch MENTAL NOTE: Add mozconfig-l10n to the patch

Comment on attachment 331330 [details] [diff] [review] browser/locales/Makefile.in - It downloads en-US packages and the installer for windows in dist/install/sea - comments added You don't need to submit separate patches for hg and cvs if they contain the same code.

Comment on attachment 331331 [details] [diff] [review] browser/locales/Makefile.in (HG) - It downloads en-US packages and the installer for windows in dist/install/sea - comments added +ifndef FTP_PATH I think you should make this variable more descriptive, but I can't seem to come up with a better name for it. +wget-en-US:+ @echo "Downloading $(FTP_PATH)/$(PACKAGE) to $(_ABS_DIST)/$(PACKAGE)" That echo should be on the next line. + @wget -nv --output-document $(_ABS_DIST)/$(PACKAGE) $(FTP_PATH)/$(PACKAGE) I don't think you should use wget here directly. You should add a check for it in configure, and then error out in this makefile target if wget is not installed. It's pretty simple, just something like this: http://mxr.mozilla.org/mozilla-central/source/configure.in#5785 You'll just need an AC_CHECK_PROGS and AC_SUBST, and then you'll need to add the variable to config/autoconf.mk.in. r- for that, but fix that and you should get r+.

We have a queue per builder and now the queues are contained in the L10nHelper In staging I have 3 builders, one of them with 3 slaves and it seems to work so far

BTW, the patch does not add __init__.py to the folder l10n as I wanted - I used cvsdo add as with the other files

I have changed the URL to something more meaningful since what we want is the URL prefix to the latest en-US binaries The configure part seems pretty straight forward - is it right?

I attached the old patch

Comment on attachment 333045 [details] [diff] [review] browser/locales/Makefile.in & configure.in - It checks that wget is available and helps us determine which en-US installers to install +if test -z "$WGET"; then + AC_MSG_ERROR([no wget found in \$PATH]) +fi Remove this part from configure, we shouldn't make it a configure error to not have wget. You should use an ifdef in the wget-en-US rule in the makefile, something like: wget-en-US: ifdef WGET ... (all the stuff you have currently) else $(error Wget not installed) endif You'll also need to modify config/autoconf.mk.in, and add a WGET = @WGET@ line for this to work properly.

Ted, do you know why do I get this message? "make: WGET@: Command not found" I added the patch to the staging repo and I can pass through make wget-en-US in dir /Users/cltbld/trunk-automation/macosx_l10n_nightly/mozilla/browser/locales (timeout 1200 secs) watching logfiles {} argv: ['make', 'wget-en-US'] environment: CVS_RSH=ssh EN_US_BINARY_URL=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0 HOME=/Users/cltbld LOGNAME=cltbld PATH=/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin PWD=/Users/cltbld PYTHONHOME=/tools/python PYTHONPATH=/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages/:/tools/zope-interface/lib/python2.5/site-packages/ SECURITYSESSIONID=485d10 SHELL=/bin/bash SHLVL=1 TBOX_CLIENT_CVS_DIR=/builds/tinderbox/mozilla/tools TERM=xterm-color TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=133 USER=cltbld _=/tools/buildbot/bin/buildbot __CF_USER_TEXT_ENCODING=0x1F6:0:0 closing stdin using PTY: False make: WGET@: Command not found make: *** [wget-en-US] Error 127 program finished with exit code 2

I have added a new target that organizes the binaries and xpi files into the same structure as what we offer in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/

Comment on attachment 332979 [details] [diff] [review] buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes Can you post the master.cfg you're using? Not for review, just so I can see how this stuff fits together.

Comment on attachment 333664 [details] [diff] [review] wget the en-US binary and prepare L10n repackages to be uploaded +ifeq (Linux, $(OS_ARCH)) +XPI_DESTINATION = linux-xpi +endif +ifeq (Darwin, $(OS_ARCH)) You can structure this like: ifeq (Linux, $(OS_ARCH)) XPI_DESTINATION = linux-xpi else ifeq (Darwin, $(OS_ARCH)) ... endif You should also put in a final: else XPI_DESTINATION = $(error Unsupported platform) endif just in case someone tries to run this on some other platform. +# Move the windows' installer nit: get rid of the apostrophe there. The Makefile bits look fine to me, but I don't really know anything about the l10n repackaging. Do you think Axel would like to take a look at this? r=me with those changes

This is the current master.cfg I am working on reporting to Tinderbox properly as discussed in the email

Fixed what ted mentioned and added "win32." before the word installer.exe Axel this patch is working so far for me, you can see the calls to this targets in the last master.cfg that I uploaded

This allow us to report to locale specific tinderbox pages which are generated like this: - MyTboxPage-locale therefore: tree = WithProperties("MyTboxPage-%(locale)s")

Comment on attachment 333999 [details] [diff] [review] tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree" >+ assert tree is type(tree) is str \ >+ or isinstance(tree, WithProperties), \ >+ "tree must be a string or a WithProperties instance" tree is type(tree) is str? >- text += "%s tree: %s\n" % (t, self.tree) >+ if type(self.tree) is str: >+ # use the exact string given >+ text += "%s tree: %s\n" % (t, self.tree) >+ elif isinstance(self.tree, WithProperties): >+ # interpolate the WithProperties instance, use that >+ text += "%s tree: %s\n" % (t, self.tree.render(build)) >+ else: >+ raise Exception("tree is an unhandled value") >+ I'd prefer isinstance(self.tree, basestring) in general.

I have added Axel's suggestion and fixed a problem in the assert line I have applied this patch only to staging-1.9-master: [root@staging-trunk-automation status]# pwd /tools/buildbot/lib/python2.5/site-packages/buildbot/status [root@staging-trunk-automation status]# patch -p4 < /home/cltbld/tboxnotifier.445454.diff

Comment on attachment 334010 [details] [diff] [review] tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree" Looks fine to me...

Comment on attachment 333972 [details] [diff] [review] It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly This bug is getting really hard to read, you should fork some items out to independent bugs. r=me with one change, wget-en-US: ifndef WGET $(error Wget not installed) endif at the start and then no else clause, that works just as fine, and removes the nested ifs which are hard to read.

(In reply to comment #49) > (From update of attachment 333972 [details] [diff] [review]) > This bug is getting really hard to read, you should fork some items out to > independent bugs. > seconded.

Comment on attachment 332979 [details] [diff] [review] buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes Nick is going to be busy to review this

(In reply to comment #50) > (In reply to comment #49) > > (From update of attachment 333972 [details] [diff] [review] [details]) > > This bug is getting really hard to read, you should fork some items out to > > independent bugs. > > > > seconded. > Will do

Comment on attachment 332979 [details] [diff] [review] buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes I have moved this patch to bug 451056

There are now 3 not obsolete patches in this bug. * The master.cfg file (which I will move to another bug when it is completely done) * The wget patch which I will fix the comments of Axel * The tinderbox.py modification which can land whenever someone can do that From now on, I will keep this bug for the comments of wget and move the few upcoming master configurations to another bug

Comment on attachment 333963 [details] [diff] [review] staging-1.9/master.cfg - It repackages locales as individual build objects and pushes them to the staging ftp A patch has been added to bug 451088

Fixed as per Axel's comments Can please someone check this in?

I removed one line more than needed - It is ready now

Comment on attachment 334010 [details] [diff] [review] tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree" checkin-needed for this one

Comment on attachment 334340 [details] [diff] [review] It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly this patch has been moved to bug 451466 where I ask for checkin-needed NOTE: From now on as I have been told I will file smaller bugs. I am sorry this bug has turned into this monster

(In reply to comment #58) > (From update of attachment 334010 [details] [diff] [review]) > checkin-needed for this one Can someone please check this patch in to CVS HEAD?

I suggest opening a new bug on the tinderbox changes, with proper deps. Is there an upstream bug for this?

(In reply to comment #61) > I suggest opening a new bug on the tinderbox changes, with proper deps. Is > there an upstream bug for this? Patch has been moved to Bug 453229 No more patches in this bug and I will attach no more to this one