Build Team, i've noticed a few locales for windows in the http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk-l10n/ folder are unable to load. The download works, and the installation is successful, but when you try to open firefox.exe, a window pops up saying: "Error launching browser window: no XBL binding for browser" Then it shuts down and app is never able to be opened. This breaks for the NL and es-ES locale. However, it works fine for en-US, fr-FR, and de. Is this a known issue? Thanks, Tony

Comments from Nick Thomas via email: ----------- Hi Tony, Looks like the two builds that don't work are dated Oct 03, while the ones that work are dated today. A bunch of string & packaging changes in the SSL UI landed between 04:43 and 04:56 on the 3rd, http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=PhoenixTinderbox&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=kaie%kuix.de&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2007-10-02&maxdate=&cvsroot=%2Fcvsroot That's just too late for the en-US nightly (first build after 4am), but before the l10n nightly starts (first build after 9am, using the en-US build from firefox/latest-trunk/). The XBL binding error seems to be associated with chrome errors, so I suspect the packaging changes break l10n builds until the localisers catch up with the new strings. Anywhere close Axel ? :-) Cheers, Nick

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=1191409200&maxdate=1191431254&cvsroot=%2Fcvsroot shows the check-ins between 4 am (windows start time) and 10:07 am (l10n start time). That has both en-US string additions and removals, and thus inheritently breaks any l10n build. To fix this, we'd need to actually know the exact build time of the en-US build, to pull the right source. Tony, this shouldn't be confidential in any way, could you uncheck the restrictions on this bug?

Moving the email thread on this topic to here. ------------- If I understand correctly, the problem is that we're not pulling the same timestamp for "mozilla" and "l10n" sources that was used for the en-US build that we are repackaging? Couldn't we just extract the build ID (which contains the build time) from the application.ini after downloading and unpacking the latest en-US nightly? If that doesn't work for some reason, we could get it from Tinderbox server instead. Both Tinderbox (for the test-only-tinderbox code) and Buildbot (TinderboxPoller) have ways of parsing the quickparse.txt to get things like the last successful build time for the en-US build. That will not give you any info about the latest clobber build though, parsing the json.js would probably be better for that (e.g. porting TinderboxPoller over to json instead of quickparse, as the available json data is a superset of what's available in quickparse). Pulling it into Tinderbox would not be too hard either, there are many JSON modules for Perl. rhelmer ------------------------- Embarrass yourself ahead :-) Yes, and we really need to know the timestamp for the nightly. Last clobber may be the same thing, I don't know. At least for linux, we have a few non-clobber builds between the nightly and the l10n builds. The good news is, I think I found the last bad bug in my l10n scripts earlier today, it's currently baking again. The remaining problem is, we can't really give good feedback either way, as we don't know if we're supposed to work against the nightly or against trunk at a particular point in time. It'd be so nice if we could just l10n-merge and be done with it. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=399014 for that. Axel -------------------------- I think we'd want to pull mozilla/ with the same timestamp as the last clobber, but l10n from the tip. From the build logs, it seems we pull the en-US build from latest-$branch/ each time a l10n cycle starts. nick thomas -------------------------- hi; Ideally, I think we would want to build l10n with the same timestamp as the en-US build *and* do the actual l10n build at the same time as the en-US build. This is obviously gated on the l10n-merge tool which doesnt exist yet, but I believe would really detangle some stuff and make l10n life much better... so wanted to keep that scenario on people's radar! ;-) $0.02. Tony is right: we should work this discussion in the bug, so others can follow along and/or contribute. Its all important and good discussions. I'll go get some food and practice what I preach... tc John.

tchung asked me to commment in this bug. I just tried to run the zh_CN build on Mac and received the same error.

changing title and adding bsmedberg, who might be able to contribute...

Is there a bug here other than "the localization isn't updated"? We typically don't worry about broken localizations until we hit string-slushyfreeze. I don't think it's worth the effort to try and synchronize checkout dates between the main tree and the l10n tree just to solve a nightly problem.

Solving the nightly problem is a good case, IMHO. Anyway, it's not dead simple, and the world didn't fall over so far. Fixing this would have a higher priority if we had l10n-merge, our pet ghost in l10n-land these days. Adjust topic once again, to make it clear that this is a build automation problem.

I agree with axel; setting priority and dependent bug.

I'm not sure that I agree with the dependency here. Might be that we lack a bug to actually use l10n-merge for the build, at which point that'd be blocked by both this bug and the l10n-merge one.

(In reply to comment #2) > To fix this, we'd need to actually know the exact build time of the en-US > build, to pull the right source. I have my buildbot setup and I am going to trigger l10n repackages just after an en-US build *BUT* the slaves would be grabbing different time stamp than the one the en-US build checked out Let's forget for a moment about builds and repackages, what is the easiest *manual* way to check out the same source twice? I would like to take this bug

The best way to get the same source twice is actually via specifying a check-out date, aka MOZ_CO_DATE. Here's my stab at getting that from a nightly build, as far as that has that for now.

(In reply to comment #11) > Created an attachment (id=324662) [details] > stab at getting the MOZ_CO_DATE from application.ini > > The best way to get the same source twice is actually via specifying a > check-out date, aka MOZ_CO_DATE. > > Here's my stab at getting that from a nightly build, as far as that has that > for now. > From your code, I think that your attempt it is a better approach than what is being run in the l10n machines but I will be aiming on touching the mechanism behind Bootstrap -o Build to create a file with the date of the checked out code, upload it to a master and distribute it to all slaves that will be doing the l10n repackages. As it was said before exact time stamp for mozilla/ and tip for l10n/. Unless it is desired to reuse that timestamp for each l10n/locale as well For example: linux -> BuildID=2008061104 Log associated - http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1213182540.1213185439.27497.gz&fulltext=1 but the timestamp is: |checkout start: Wed Jun 11 04:10:39 PDT 2008 |cvs -q -z 3 co -D "06/11/2008 11:09 +0000" The source checked out could be anytime from 11:00 to 11:59 +0000 self.command = '''python << EOF +from ConfigParser import ConfigParser +cp=ConfigParser() +cp.read('firefox/application.ini') +print cp.get('App','BuildID') +EOF +''' + cvscodate = "%s/%s/%s %s:00 +0000" % (month,day,year,hour) + self.setProperty('MOZ_CO_DATE', cvscodate) NOTE: Do not read this next line - just testing if the next line will become a hyperlink comment 11

This can allow using this after instantiating the factory depBuildFactory.addStep(ShellCommand, command=['echo',depBuildFactory.cvscodate]) The bad thing is that this value cannot be passed to: depBuildFactory.addStep(ShellCommand, description='Build', workdir='build', command=['perl', './release', '-o', 'Build'], timeout=36000, haltOnFailure=1, env={'CVS_RSH': 'ssh'}, ) I don't see an easy solution here, unless whatever gets done inside of the previous step gets move to buildbot. If I add my steps to do the l10n repackages after that step, they will probably be using a cvscodate previous to what the en-US will be using (maybe 0, 1 or 2 minutes)

Comment on attachment 324841 [details] [diff] [review] it patches builbotcustom/process/factory.py to have a "cvscodate" property Forget the review, it makes no sense, that is the factory. The checkout date value would be from the time that the master was turned on if I am not mistaken

Adding this allows me to do this: armenFactory.addStep(GenerateCvsCoDate) armenFactory.addStep(ShellCommand, command = ["cvs","-d",cvsroot,"co","mozilla/client.mk"], workdir = '.') armenFactory.addStep(ShellCommand, command = ['make','-f','client.mk','checkout'], workdir = 'mozilla', env={'MOZ_CO_DATE':WithProperties("%(cvscodate)s"), 'MOZ_CO_PROJECT':'browser'}) #Other steps to create an en-US build armenFactory.addStep(ShellCommand, command = ['make','-f','client.mk','l10n-checkout'], workdir = 'mozilla-l10n', env={'MOZ_CO_DATE':WithProperties("%(cvscodate)s"), 'MOZ_CO_PROJECT':'browser'})

This should allow us to set a cvs checkout time to the en-US build. You can also not use and it should still work the way it has been doing so far I know that we generate nightlies on production 1.8 that generate builds that go live |armenFactory = BootstrapFactory( | cvsroot=cvsroot, | cvsmodule=cvsmodule, | automation_tag=automation_tag, | logdir='/builds/logs', | bootstrap_config='configs/fx-moz18-nightly-staging-bootstrap.cfg', |) |#This step adds a cvscodate property to the build that can be reused |armenFactory.addStep(GenerateCvsCoDate) |armenFactory.addStep(ShellCommand, | description='Build', | workdir='build', | command=['perl', './release', '-o', 'Build'], | timeout=36000, | haltOnFailure=1, | env={'MOZ_CO_DATE':WithProperties("%(cvscodate)s"), <--- reused here | 'TBOX_CLIENT_CVS_DIR':'/builds/tinderbox/mozilla/tools'}, |)

Do we want this fixed as well? I believe we should bring the precision to the second my $time_str = POSIX::strftime("%m/%d/%Y %H:%M +0000", gmtime($start_time)); to my $time_str = POSIX::strftime("%m/%d/%Y %H:%M:%S +0000", gmtime($start_time));

Comment on attachment 325145 [details] [diff] [review] misc.py & build-seamonkey-util.pl - We can specify a MOZ_CO_DATE to the buildbot step "perl ./release -o Build" >Index: tools/buildbotcustom/steps/misc.py >=================================================================== >RCS file: /cvsroot/mozilla/tools/buildbotcustom/steps/misc.py,v >retrieving revision 1.4 >diff -u -8 -p -r1.4 misc.py >--- tools/buildbotcustom/steps/misc.py 7 Apr 2008 13:48:03 -0000 1.4 >+++ tools/buildbotcustom/steps/misc.py 15 Jun 2008 06:34:50 -0000 Thanks for putting this here :). >+class GenerateCvsCoDate(ShellCommand): This can just be a subclass of BuildStep, more below. >+ def __init__(self, date = None): If you're going to make it an optional argument, please add a docstring for it, and require it to be a datetime object, not a string. >+ if date: >+ self.date = date >+ else: >+ self.date = Popen(['date','+%m/%d/%Y %H:%M:%S +0000'], >+ stdout=PIPE).communicate()[0][:-1] If you're going to generate this yourself, I believe that the build start time, as already defined by Buildbot is the correct way to do this. You'll have to grab it inside of start(), though. Here's an example: http://lxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/status/tinderbox.py#139 >Index: tools/tinderbox/build-seamonkey-util.pl >=================================================================== >RCS file: /cvsroot/mozilla/tools/tinderbox/build-seamonkey-util.pl,v >retrieving revision 1.389 >diff -u -8 -p -r1.389 build-seamonkey-util.pl >--- tools/tinderbox/build-seamonkey-util.pl 12 May 2008 18:05:14 -0000 1.389 >+++ tools/tinderbox/build-seamonkey-util.pl 15 Jun 2008 06:34:52 -0000 >@@ -957,16 +957,19 @@ sub BuildIt { > > my $cvsco = ''; > > # Note: Pull-by-date works on a branch, but cvs stat won't show > # you this. Thanks to cls for figuring this out. > if ($Settings::UseTimeStamp) { > $start_time = adjust_start_time($start_time); > my $time_str = POSIX::strftime("%m/%d/%Y %H:%M +0000", gmtime($start_time)); >+ if (defined($ENV{MOZ_CO_DATE})) { >+ $time_str = $ENV{MOZ_CO_DATE}; >+ } Seems fine. None of the existing Tinderboxen _should_ have this set, so nothing should break...going to have coop take a look too, as he's more familiar with Tinderbox than I.

Comment on attachment 325145 [details] [diff] [review] misc.py & build-seamonkey-util.pl - We can specify a MOZ_CO_DATE to the buildbot step "perl ./release -o Build" >Index: tools/buildbotcustom/steps/misc.py <...> >+class GenerateCvsCoDate(ShellCommand): >+ haltOnFailure = 1 >+ >+ def __init__(self, date = None): >+ """ >+ It just adds the current date as a build property called "cvscodate" >+ You can specify another one if pleased >+ """ >+ if date: >+ self.date = date >+ else: >+ self.date = Popen(['date','+%m/%d/%Y %H:%M:%S +0000'], >+ stdout=PIPE).communicate()[0][:-1] >+ self.command = ['date'] I think that this .... >+ ShellCommand.__init__(self) >+ >+ def start(self): >+ self.setProperty('cvscodate', self.date) >+ ShellCommand.start(self) ... and this get you to run the command twice, once one the master and once on the slave. But as Ben said, just use the buildbot BuildStatus. Follow dummy.Dummy, and use self.step_status.parent.getTimes()[0].

Comment on attachment 325145 [details] [diff] [review] misc.py & build-seamonkey-util.pl - We can specify a MOZ_CO_DATE to the buildbot step "perl ./release -o Build" > my $time_str = POSIX::strftime("%m/%d/%Y %H:%M +0000", gmtime($start_time)); >+ if (defined($ENV{MOZ_CO_DATE})) { >+ $time_str = $ENV{MOZ_CO_DATE}; >+ } If MOZ_CO_DATE is defined, you end up setting $time_str twice. Why not: my $time_str; if (defined($ENV{MOZ_CO_DATE})) { $time_str = $ENV{MOZ_CO_DATE}; } else { $time_str = POSIX::strftime("%m/%d/%Y %H:%M +0000", gmtime($start_time)); }

I forgot to mention in my comment, I'd call the build property MOZ_CO_DATE instead of cvscodate. As that's what it is, right?

Comment on attachment 325321 [details] [diff] [review] misc.py - inherithing from BuildStpe instead of ShellCommand + docString Does this work? I'd expect that you had to do some kwargs magic in __init__. And we shouldn't need to call out to Popen date, but really use the start date of the build for this, like Ben said. Do you have a example in the code for calling finish() directly from start()? I'm not sure if that's working right, you might have to do a reactor.callLater(0,...)

Comment on attachment 325321 [details] [diff] [review] misc.py - inherithing from BuildStpe instead of ShellCommand + docString I tried to do what you asked me to do about builddate = int(build.getTimes()[0]), but I had too much trouble. I have also changed it so a user can specify a format instead of the default one. BTW, in the waterfall does not show any name when done on the box associated it might be that there is no "stdio" associated (I have tried to set descriptionDone but it seems to don't work)

Too much trouble means exactly what? I bet that the problem with descriptionDone has something to do with calling finished() from start() directly, without getting the reactor in the middle.

(In reply to comment #22) > I forgot to mention in my comment, I'd call the build property MOZ_CO_DATE > instead of cvscodate. As that's what it is, right? > It is (In reply to comment #25) > (From update of attachment 325321 [details] [diff] [review]) > Does this work? I'd expect that you had to do some kwargs magic in __init__. > It does completely work from the first patch that I did -- I do not know how to use kwargs... Does it get passed from a finished step to another one?? I use setProperty so I can do things like this: "env={'MOZ_CO_DATE':WithProperties("%(cvscodate)s")}" in any other step > And we shouldn't need to call out to Popen date, but really use the start date > of the build for this, like Ben said. I will try again. I had some trouble realizing how to use self.step_status. I just saw know self.step_status.parent.getTimes()[0]. and dummy.Dummy which is an useful example > > Do you have a example in the code for calling finish() directly from start()? > I'm not sure if that's working right, you might have to do a > reactor.callLater(0,...) > Not sure what you mean, but maybe you can answer I do my next patch

(In reply to comment #19) > >+ if date: > >+ self.date = date > >+ else: > >+ self.date = Popen(['date','+%m/%d/%Y %H:%M:%S +0000'], > >+ stdout=PIPE).communicate()[0][:-1] > > If you're going to generate this yourself, I believe that the build start time, > as already defined by Buildbot is the correct way to do this. You'll have to > grab it inside of start(), though. Here's an example: > http://lxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/status/tinderbox.py#139 > I tried to go this way but I did not know how to do it. It might also require some twisting of the date format given by buildbot since the format has to be accepted by cvs and maybe some verification that build-seamonkey.util.pl might do (not sure) I basically have to give at a later point a value to MOZ_CO_DATE to be used inside of "release -o Build" and again by checking out the preparatory code for l10n repackages

Comment on attachment 325329 [details] [diff] [review] [checked in] build-seamonkey-util.pm - It sets only once the variable Landed for Armen. Checking in build-seamonkey-util.pl; /cvsroot/mozilla/tools/tinderbox/build-seamonkey-util.pl,v <-- build-seamonkey-util.pl new revision: 1.390; previous revision: 1.389 done

(In reply to comment #30) > (In reply to comment #19) > > >+ if date: > > >+ self.date = date > > >+ else: > > >+ self.date = Popen(['date','+%m/%d/%Y %H:%M:%S +0000'], > > >+ stdout=PIPE).communicate()[0][:-1] > > > > If you're going to generate this yourself, I believe that the build start time, > > as already defined by Buildbot is the correct way to do this. You'll have to > > grab it inside of start(), though. Here's an example: > > http://lxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/status/tinderbox.py#139 > > > I tried to go this way but I did not know how to do it. It might also require > some twisting of the date format given by buildbot since the format has to be > accepted by cvs and maybe some verification that build-seamonkey.util.pl might > do (not sure) > This is the magic you're looking for: int(self.step_status.build.getTimes()[0]) (from here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/steps/transfer.py#158) In a BuildStep, this will get you the start time, in unix time, as an integer. You can then use time.strftime() to translate it into a better format. Please adjust your patch to use this.

Comment on attachment 325321 [details] [diff] [review] misc.py - inherithing from BuildStpe instead of ShellCommand + docString I have attached the next patch on Bug 439778 since I need it to be able to synchronize en-US builds

Just to update a little bit this bug since many dependancies have changed or resolved. Axel was able to accomplish this a little while ago by using a couple of new make targets in browser/locales/Makefile.in What these targets do is: 1) unpack the binary that was wgetted 2) set a property with the en-US revision that is on that is stored on that binary 3) update the en-US tree (mozilla-central or mozilla-1.9.1) to that revision All that is left for this bug to be fixed is to add these steps to the RepackFactory NOTE: The mentioned targets did not make it to CVS http://hg.mozilla.org/users/axel_mozilla.com/tooling/file/tip/buildbot-configs/l10n/master.cfg#l182 177 hg_f.addStep(ShellCommand, 178 workdir = WithProperties('%(tree)s/%(app)s/%(app)s/locales'), 179 command = ['make', 'wget-en-US', 180 WithProperties('EN_US_BINARY_URL=%(en_us_binary)s')], 181 haltOnFailure = True) 182 hg_f.addStep(ShellCommand, 183 workdir = WithProperties('%(tree)s/%(app)s/%(app)s/locales'), 184 command = ['make', 'unpack'], 185 haltOnFailure = True) 186 hg_f.addStep(SetProperty, 187 workdir = WithProperties('%(tree)s/%(app)s/%(app)s/locales'), 188 command = ['make', 'ident'], 189 property = 'binary_revision', 190 haltOnFailure = True) 191 hg_f.addStep(ShellCommand, 192 workdir = WithProperties('%(tree)s/mozilla-central'), 193 command = ['hg', 'update', 194 '-r', WithProperties('%(binary_revision)s')], 195 haltOnFailure = True)

I guess the trunk patch is slowly stable enough to land on CVS, but it'd actually require a new patch, which carries some risk. We should have a good understanding on the differences between development and release builds on both cvs and hg before trying to port this over, as mar generation and pretty-names have been sensitive to the hg patch.

returning back to the pool. I want to concentrate on the repack on change bug

Armen will be picking this up again after he's done with bug 464163.

This is code that's converting the output of make ident to build properties, and uses that to update the en-US source to the one given by the repackaged build. This step should end up before the compare-locales (with merge) step. I tested with with: - starting a mozilla2-staging master - using the converter function for both fx32x and fennec10x on my local setup, testing that both a plain ident and a multiple repo ident work. Coop, can you run this through staging?

(In reply to comment #38) > Coop, can you run this through staging? Sorry for the delay. I'm going to start this in staging today.

(In reply to comment #39) > Sorry for the delay. I'm going to start this in staging today. This is running on staging-master right now.

Comment on attachment 366924 [details] [diff] [review] add a utility function to convert make ident output to build props, and use those to update en-US to the right source >+ self.addStep(ShellCommand, >+ command=['hg', 'update', '-R', >+ WithProperties('%(fx_revision)s')], Had to change this to '-r' to get it to run. (e.g. http://staging-master.build.mozilla.org:8010/builders/Linux%20mozilla-1.9.1%20l10n/builds/8022/steps/shell_18/logs/stdio)

This is an updated patch that uses -r to specify a revision. I removed the hard-coded 'browser', too, and used self.appName as everywhere else in the factory.

http://hg.mozilla.org/build/buildbotcustom/rev/cbcff91de3e6, marking FIXED.

Congrats! This is great to ensure the quality of the nightly l10n builds!

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.