This bug will contain the patches that change the master.cfg for staging1.9 and production1.9 to run L10n repackages in parallel

Comment on attachment 334327 [details] [diff] [review] staging-1.9 configuration to run L10n repackages >+L10nBuildFactory.addStep(ShellCommand( >+ command=["cvs", "-q", "-z3", "-d", cvsStagingRoot, "co", >+ "mozilla/client.mk", >+ "mozilla/browser/locales", >+ "mozilla/nsprpub", >+ # We need this for windows >+ "mozilla/other-licenses/7zstub/firefox"], >+ descriptionDone="client.mk checked out", > This is not appropriate because it does more than checking out client.mk which is misleading >+L10nBuildFactory.addStep(ShellCommand, >+ command=["make", "-f", "client.mk", "l10n-checkout"], > I should write a make target that will just check out the code needed to repackage or get the missing that I added in the previous comment (mozilla/nsprpub, mozilla/other-licenses) Axel, is the l10n-checkout target the target to checkout the necessary code to run a compare-locale step? >+#Since we are wgetting from ftp.m.o but using code from staging >+L10nBuildFactory.addStep(ShellCommand( >+ command=["cvs", "-q", "-z3", "-d", cvsroot, "co", >+ "mozilla/browser/config/version.txt"], >+ descriptionDone="version.txt", > This step will go away for the production configuration >+L10nBuildFactory.addStep(ShellCommand, >+ command=["make", "-f", "client.mk", "configure"], > I would really love to have another configure file to run that would do the minimum verification for just l10n-repackages. Just for the sake of improving the performance even it is few seconds You can look at a log: http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1219095408.1219095453.17772.gz&fulltext=1 and see that you have a lot of messages like: "can't read blah blah" >+L10nBuildFactory.addStep(ShellCommand, >+ name = "upload locale", >+# This will upload everything on dist/upload, I assume that the first >+# step run in this build is "rm -rf mozilla/dist/upload" >+ command=['sh','-c','scp -r * '+ftpserver+":"+uploadPath], >+ description="uploading packages", >+ descriptionDone="uploaded packages", >+ workdir="mozilla/dist/upload" >+) > maybe another make target for this?

This configurations has an extra element which is an extra TinderboxMailNotifier for the Mozilla-l10n page This patch also adds as the previous one, it adds the "mozconfig-l10n" file

Comment on attachment 334375 [details] [diff] [review] master.cfg for production 1.9 as the previous one let's wait until staging configs are checked in before tackling this

Comment on attachment 334327 [details] [diff] [review] staging-1.9 configuration to run L10n repackages r-. There are different reasons for this, first and foremost, this bug should depend on a bug to actually implement buildbotcustom.l10n. I don't think we need the nsprpub checkout. Other extra parts should probably go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout. The locks are wrong. This bug should depend on the make targets, too. The blocking bugs are likely somewhere in the dependency tree, but we should have the dependencies between the bugs set up right in order to attach this beast.

FWIW, I think this is a bad idea in general, as I've noted on IRC a couple times: doing l10n repacks in parallel means you have to replicate work across a lot of builds. It looks like, from http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current repacks take about 30 minutes for all locales. Why are we trying to improve on this number, when it will significantly slow down the rest of the system?

(In reply to comment #5) > FWIW, I think this is a bad idea in general, as I've noted on IRC a couple > times: doing l10n repacks in parallel means you have to replicate work across a > lot of builds. It looks like, from > http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current > repacks take about 30 minutes for all locales. Why are we trying to improve on > this number, when it will significantly slow down the rest of the system? One main reason: 1) Windows repackages take too long - 4hours and 20 minutes -> WINNT 5.2 l10n-win32-tbox Depend Fx-Moz1.9.0-l10n- Started 06:02, finished 10:22

This time will increase in the long run since we keep on adding more and more locales to our system. Being scalable will help us. It sucks to have logic being repeated, but it allow us to throw more slaves in if we want to improve the performance In staging, I have been running this code and with 3 windows slaves the execution time was less than an hour and a half when the work being done by a single slave was more than 3 hours and a half

Do we know why? I have found that in my windows slave VM, a repack without running compare-locales takes 6 minutes per locale. Almost all of this time (60%) is spent in the upload step. Does this match up with your timings?

(In reply to comment #8) > Do we know why? I have found that in my windows slave VM, a repack without > running compare-locales takes 6 minutes per locale. Almost all of this time > (60%) is spent in the upload step. Does this match up with your timings? I am not sure why it is taking that long but my timings where under 3 minutes on windows and around 30 secs on the xserve but I can't remember right now. BTW, I used "make -f client.mk l10n-checkout" to checkout less code This is an old blog post that I did in some early staging runs: http://armenzg.blogspot.com/2008/08/l10n-parallelization-results.html There are things that could be improved at a later point like checking out less code and have a shorter configure step

(In reply to comment #4) > There are different reasons for this, first and foremost, this bug should > depend on a bug to actually implement buildbotcustom.l10n. > I have been fixing the list of dependent bugs bug 451056 for the buildbotcsutom.l10n implementation > I don't think we need the nsprpub checkout. Other extra parts should probably > go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout. > I will play with this again I will file a bug about nsprpub when I find why was the reason that I was checking out and find a way of not having to check it out > The locks are wrong. True > > This bug should depend on the make targets, too. > I have added the dependence to the three make targets > The blocking bugs are likely somewhere in the dependency tree, but we should > have the dependencies between the bugs set up right in order to attach this > beast. :( I wish I had a better leash for this creature Added dependency as mentioned before

So here's my take on the timings: We currently have a smaller list of languages on central, i.e. 46 instead of 63. Then you're not running compare-locales right now. I assume your machine is beafier, too, but when we have round about 2 minutes per locale, 60 locales, we get 2 hours, and growing. We're currently facing a few issues that make the parallization cost not per slave, but per build, but those can be fixed. Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around. I wonder if we can make the wget target only fetch the file if it's newer, too. The l10n checkout is not that bad, as soon as you limit the check-outs to the locale you're building, that's scaling pretty well.

(In reply to comment #11) > Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around. What do you mean? > I wonder if we can make the wget target only fetch the file if it's newer, too. I have thought of this and I think it can be done

(In reply to comment #12) > (In reply to comment #11) > > Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around. > What do you mean? Sry, that was mercurial specific. I think the corresponding ticket on bb is http://buildbot.net/trac/ticket/277.

Comment on attachment 334327 [details] [diff] [review] staging-1.9 configuration to run L10n repackages I really don't have time to review this stuff right now - sorry. Assuming this patch is still valid, I'm passing to coop.

Comment on attachment 334327 [details] [diff] [review] staging-1.9 configuration to run L10n repackages I think this review is probably out-of-date by now. Most of the dependency issues that Axel brought up seem to have been filed at least, if not yet fixed. As Axel noted, the locks are wrong. Looks like you've cut-n-pasted the linux_lock for all platforms.

As coop mentions this master.cfg has not been updated and most of the issues that Axel raised are being landed by coop This master.cfg has been in use during the first 2 weeks of August with correct results but a cleaner master.cfg patch will be posted as soon as I have the system running again in staging-1.9

This configuration file is working right now in staging-1.9 correctly This is ready to be reviewed and commited when review process goes trough It should be reporting to MozillaTest and MozillaTest-%locale% as the emails pasted at the bottom show but for an unknown reason is not showing in http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest While rerunning all of this, we should be filing as Axel mentions a bug for "mozilla/other-licenses/7zstub/firefox" to be included in l10n-checkout which should be an easy fix and another to find out why do I need to check out "nsprpub" to make-installers (if I remove the line it won't generate the installers) ----------------------------------------------------- tinderbox: tree: MozillaTest tinderbox: builddate: 1222047002 tinderbox: status: busted tinderbox: build: win32_l10n_nightly tinderbox: errorparser: unix tinderbox: binaryurl: tinderbox: logcompression: bzip2 tinderbox: logencoding: base64 tinderbox: END QlpoOTFBWSZTWepo/UkAAGNfgAAQQIGAEAwAQAB/L98wIACJCSo9EAepoGgARUTIE9TCaGjNGK9r Cgy0YN0x3GgKpY6whBYBMHcCYbhmmDeOQzJJV4f7Bs8u7nslToMem449N0woRgFFyeQhqkGVPgaX hnN2ItiUAkzeQwcWSimZDcYXr74uexPJUh+LuSKcKEh1NH6kgA== ----------------------------------------------------- tinderbox: tree: MozillaTest-ar tinderbox: builddate: 1222047002 tinderbox: status: busted tinderbox: build: win32_l10n_nightly tinderbox: errorparser: unix tinderbox: binaryurl: http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/ tinderbox: logcompression: bzip2 tinderbox: logencoding: base64 tinderbox: END QlpoOTFBWSZTWaqUo5sAAKLfgAAQQIP8MAQBQAD/59/wMADmEJRTSeKeQIaMaTajTQ0GMmmQMmhk GRpgRgkkmEmmjQZGmmI0HqaZb4Kq2iKY0AyZqUmzhzBgak35zhfqY74cSNkYnuzo26VY5se22ZzM 5VSpFLqktQ68KEgFfpBe7H0BAMfp9DHkQhsStqHM8TTTFTs0pmDlHeey7knjbGpXTmNk88jXA41D UibbDeHzAnaELBEuwWo6L8rXPL5BnQDnm2C63L8V3hSMEChEA14pTBqyMqUOWS3EEXUczORGRLRl FuHdp5bkaSOINei14QUqW/JCTw/i7kinChIVUpRzYA==

You can now see the results of the l10n repackages in parallel on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

This patch as not been tried in production-1.9 but it is basically the same as what is being run in staging-1.9

Comment on attachment 339721 [details] [diff] [review] [checked in] staging-1.9 configuration to run L10n repackages > ####### BUILDERS > > cvsroot = ":ext:stgbld@cvs.mozilla.org:/cvsroot" >+cvsStagingRoot = 'staging-1.9-master:/builds/cvsmirror/cvsroot' >+cvsl10nroot = 'staging-1.9-master:/builds/cvsmirror/l10n' > cvsmodule = "mozilla/tools/release" > ftpserver = "fx-linux-1.9-slave1" > automation_tag = "HEAD" >+uploadPath = "/home/ftp/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/" >+enUS_binaryURL = 'http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0' Shouldn't this be pointing at firefox/experimental/nightly/ for staging builds?

Comment on attachment 339802 [details] [diff] [review] production-1.9 configuration to run L10n repackages Every third l10n build or so on the MozillaTest tree seems to be red right now. Is that expected?

(In reply to comment #21) > (From update of attachment 339802 [details] [diff] [review]) > Every third l10n build or so on the MozillaTest tree seems to be red right now. > Is that expected? > Yes, it is expected. Not all l10n repos are in the staging cvs mirror and not every locale succeeds since they might have missing entities or files

Hey coop, I didn't know what was the best way to write this patch Once we decide to do the full switch I will write a patch to don't point to experimental and to MozillaStaging I left a keyword #CHANGE to know exactly which lines to modify next time; is that fine?

Comment on attachment 339855 [details] [diff] [review] [checked in] production-1.9 configuration that will report to MozillaStaging and push to experimental (In reply to comment #23) > I left a keyword #CHANGE to know exactly which lines to modify next time; is > that fine? If you and bhearsum have already had the conversation about where these builds should be directed to, that's fine by me.

Armen: I've checked in the production master.cfg patch. Are we really still waiting for a review on the staging patch?

(In reply to comment #25) > Armen: I've checked in the production master.cfg patch. Are we really still > waiting for a review on the staging patch? > No, go ahead whenever you can

This patch fixes some missed typos, correct the experimental ftp path, use correct make target and report to MozillaTest instead of Mozilla-l10n