Can't forget these.

Passes checkconfig. Far far from done.

Does this look like the right approach?

I have to think about how we keep configurations with a mobileRepo and without. If without, how do we sendchange without kicking off both deskop+mobile releases?

Now working out of http://hg.mozilla.org/users/asasaki_mozilla.com/mobile-0.8-configs/ and http://hg.mozilla.org/users/asasaki_mozilla.com/mobile-0.8-custom/ because I foresee a lot of bitrotting in the future.

We chatted on IRC about the current work for a bit. Overall, I think these are looking very good and I'm happy with most of it. There's a few things I'd like to see addressed though: - Get rid of enable_mobile_releases flag in favour of ACTIVE_MOBILE_RELEASE_BRANCHES - Move sourceBundle* to Factorys, if possible. If bug 510770 will enable us to get that logic out of Buildbot completely, I'm OK with these bits for now, provided there's a follow-up bug for us to get around to eventually. - I like the factoring you've done in release.py, but I'd like to see the rest of the boiler plate go away. Specifically, the non-mobile specific parts of generate* all the way down to the "builders = []" line. - In the final patches, delete lines in the mobile release configs/generate function rather than commenting them out. On a somewhat related note, the mobile-only mozilla-2.1 branch made us realize that we need to key desktop/mobile for an entire branch off of len(platforms) and len(mobile_platforms). Not directly related to this bug, but it came up in the course of the discussion. Great work so far, Aki!

(In reply to comment #6) > - Get rid of enable_mobile_releases flag in favour of > ACTIVE_MOBILE_RELEASE_BRANCHES > - Move sourceBundle* to Factorys, if possible. If bug 510770 will enable us to > get that logic out of Buildbot completely, I'm OK with these bits for now, > provided there's a follow-up bug for us to get around to eventually. Done and done in http://hg.mozilla.org/users/asasaki_mozilla.com/mobile-0.8-configs/ and http://hg.mozilla.org/users/asasaki_mozilla.com/mobile-0.8-custom/ , though still untested (passes test-masters) > - I like the factoring you've done in release.py, but I'd like to see the rest > of the boiler plate go away. Specifically, the non-mobile specific parts of > generate* all the way down to the "builders = []" line. The remaining functions in generate*ReleaseBranchObjects() differ between mobile and desktop and are called multiple times, so factoring them out as is will make calling them error prone. I'd like these more generic as well, but I'm not sure this pass is the time to do it. > - In the final patches, delete lines in the mobile release configs/generate > function rather than commenting them out. Yup. I'm not anywhere near done yet. > On a somewhat related note, the mobile-only mozilla-2.1 branch made us realize > that we need to key desktop/mobile for an entire branch off of len(platforms) > and len(mobile_platforms). Not directly related to this bug, but it came up in > the course of the discussion. I forget what this is about =P I'm going to finish up generateMobileReleaseBranchObjects(), either reinstate AndroidReleaseBuildFactory & MaemoReleaseBuildFactory or port those fully to mozharness, then do some staging runs and clean up cruft.

Copying mobile mozilla-beta release mozconfigs from their old 0.7 location to their new 0.8 location. Also remove objdir as that's being set in MercurialBuildFactory now. John Ford did this for nightlies: http://hg.mozilla.org/build/buildbot-configs/rev/47d5646788ea .

Comment on attachment 536784 [details] [diff] [review] mozilla-beta release mozconfigs for mobile This made it to production today.

There were 2 ways to go. 1) Extend existing generateReleaseBranchObjects 2) Implement generateMobileReleaseBranchObjects The deadline for 0.8 mobile release automation is very close, so I decided to go with 2 to prevent Firefox release automation brokenness. In the future we may want to merge them together or move common pieces to release_common. Configs: * Added ACTIVE_MOBILE_RELEASE_BRANCHES list, we can add ENABLE_MOBILE_RELEASES to not import release/mobile_release modules. * release-fennec-mozilla-beta.py is a placeholder for now, almost nothing inside it has been verified * Ignore staging_builder_master_sm01_localconfig.py and staging_release-firefox-mozilla-2.0.py personal modifications * setup-master.py hasn't been checked yet buildbotcustom: * Ported all schedulers. Nothing changed for now. In the future I plan to use Trigerrable schedulers for builds to use the same build ids. * I used UrlPoller instead of FtpPoller for android verification * tons of unused (commented) variables * some of the functions moved to release_common and wrapped * builders are also ported. I used different category and branch for mobile, so we can use a separate sendchange and waterfall view for mobile release automation. * Haven't touched factories yet. I wrapped MaemoReleaseBuildFactory, MaemoReleaseBuildFactory, and MaemoReleaseBuildFactory just to pass checkconfig. * I think we can reuse Firefox tagging and source factories for Fennec as well.

Comment on attachment 537144 [details] [diff] [review] configs I know the config is just a stub for now, but do you actually need base_enUS_binaryURL? You should be able to calculate that in the runtime l10n scripts. The glue in builder_master.cfg looks sane enough to me.

Comment on attachment 537145 [details] [diff] [review] buildbotcustom Review of attachment 537145 [details] [diff] [review]: ----------------------------------------------------------------- If we had more time, I'd have lots to say about integration with the desktop stuff, but taking the time constraints into consideration, this looks pretty good. ::: process/mobile_release.py @@ +354,5 @@ > + reallyShort(builderPrefix('mobile_source'))} > + }) > + > + envJava = builder_env.copy() > + envJava['PATH'] = '/tools/jdk6/bin:%s' % envJava.get( What's this for? Seems like it should be something that goes in config.py rather than here ::: process/release_common.py @@ +9,5 @@ > + > +from release.paths import makeCandidatesDir > + > + > +def common_builderPrefix(branch, builder_name, platform=None): I'm not really sold on this "common_" prefix on functions, since they're in a file called "release_common". How about "from process import release_common" or "from process.release_common import builderPrefix as common_builderPrefix" in mobile_release.py instead?

Comment on attachment 537144 [details] [diff] [review] configs (In reply to comment #12) > There were 2 ways to go. > > 1) Extend existing generateReleaseBranchObjects > 2) Implement generateMobileReleaseBranchObjects > > The deadline for 0.8 mobile release automation is very close, so I decided > to go with 2 to prevent Firefox release automation brokenness. In the future > we may want to merge them together or move common pieces to release_common. Makes sense. > * Added ACTIVE_MOBILE_RELEASE_BRANCHES list, we can add > ENABLE_MOBILE_RELEASES to not import release/mobile_release modules. I'm hoping setting ACTIVE_MOBILE_RELEASE_BRANCHES to [] is enough there. (Not sure if you're worried about something else?) >+ for product, sourceRepoKey, releaseBranch, releaseObjGenerator in \ >+ [('firefox', 'mozilla', b, generateReleaseBranchObjects) for b in ACTIVE_RELEASE_BRANCHES] + \ >+ [('fennec', 'mobile', b, generateMobileReleaseBranchObjects) for b in ACTIVE_MOBILE_RELEASE_BRANCHES]: Are we splitting mozilla and mobile so we can join them but keep them separate at some later point? We probably want to revisit this in version 2, when we consolidate; not important now. For the short term, I think we can enable desktop mozilla-beta on one master, and mobile mozilla-beta on another. Is that what you're thinking? Otherwise we may get duplicate schedulers on mozilla-beta. >--- a/mozilla/production_builder_master_pm03_localconfig.py >+++ b/mozilla/production_builder_master_pm03_localconfig.py >@@ -17,6 +17,7 @@ ACTIVE_BRANCHES = ['shadow-central', 'mozilla-1.9.1', 'mozilla-1.9.2', 'mozilla- > ] + ACTIVE_PROJECT_BRANCHES > ACTIVE_PROJECTS = PROJECTS.keys() > ACTIVE_RELEASE_BRANCHES = ['mozilla-1.9.2', 'mozilla-beta'] >+ACTIVE_MOBILE_RELEASE_BRANCHES = ['mobile-beta',] Is this a leftover, or are you abstracting mozilla-beta -> mobile-beta for mobile? We've done a lot of fake-naming for mobile; I'm hoping we don't need to this time around, but I haven't dug fully into this yet. > * release-fennec-mozilla-beta.py is a placeholder for now, almost nothing > inside it has been verified Ok. But I'll comment on it anyway :) >+ 'mobile': { >+ 'name': 'mozilla-beta', >+ 'path': 'releases/mozilla-beta', >+ 'revision': '23d449276096', >+ 'relbranch': None, >+ 'bumpFiles': { >+# 'browser/config/version.txt': { >+# 'version': releaseConfig['appVersion'], >+# 'nextVersion': releaseConfig['nextAppVersion'] >+# }, We'll want to add mobile/confvars.sh to bumpFiles. One thing that's been worrying me a bit -- we'll be releasing off the same repo, and we've been using the same changeset and a very similar 'go' time. The relbranches we're using are using timestamps down to the hour. I think for the first 0.8 mobile beta, we may have to wait until the time changes to up to an hour after the Firefox release starts, so that we can create a second relbranch. Either that, or use the same relbranch somehow. >+releaseConfig['l10nPlatforms'] = ('linux-maemo5-gtk',) >+releaseConfig['shippedLocalesPath'] = 'mobile/locales/shipped-locales' We don't use shipped locales; all that data is in the json.

Comment on attachment 537145 [details] [diff] [review] buildbotcustom (In reply to comment #12) > buildbotcustom: > * Ported all schedulers. Nothing changed for now. In the future I plan to > use Trigerrable schedulers for builds to use the same build ids. > * I used UrlPoller instead of FtpPoller for android verification > * tons of unused (commented) variables > * some of the functions moved to release_common and wrapped > * builders are also ported. I used different category and branch for mobile, > so we can use a separate sendchange and waterfall view for mobile release > automation. Ok. I think separate masters might work, but other than not particularly liking the 'mobile-beta' abstraction, this sounds ok. > * I think we can reuse Firefox tagging and source factories for Fennec as > well. I think so too. >+from buildbotcustom.process.factory import ReleaseBuildFactory >+class MaemoReleaseBuildFactory(ReleaseBuildFactory): >+ def __init__(self, *args, **kwargs): >+ pass >+class AndroidReleaseBuildFactory(ReleaseBuildFactory): >+ def __init__(self, *args, **kwargs): >+ pass >+class MaemoReleaseRepackFactory(ReleaseBuildFactory): >+ def __init__(self, *args, **kwargs): >+ pass I think we'll be using ReleaseBuildFactory for all mobile release builds now. However, if subclassing them makes things easier, cleaner, or faster, that's fine with me. For emails, do you want them all to still say [release] or do you want to have something like [mobile release] ? The former would be better for consolidated releases, but the latter might be better for split releases. >+ unix_slaves = branchConfig['platforms'].get('linux', {}).get('slaves', >+ []) + \ >+ branchConfig['platforms'].get('linux64', {}).get('slaves', >+ []) + \ >+ branchConfig['platforms'].get('macosx', {}).get('slaves', >+ []) + \ >+ branchConfig['platforms'].get('macosx64', {}).get('slaves', []) >+ all_slaves = unix_slaves + \ >+ branchConfig['platforms'].get('win32', {}).get('slaves', []) + \ >+ branchConfig['platforms'].get('win64', {}).get('slaves', []) This probably needs changing to match the mobile platform list, but you probably know that :) >+ if 'signedPlatforms' in releaseConfig.keys(): >+ signedPlatforms = releaseConfig['signedPlatforms'] >+ else: >+ signedPlatforms = ('android-r7',) android-r7 is now linux-android. Which presents another small problem -- it's 'linux-android' internally, but I want it to be 'android' on all externally-facing interfaces... that's currently stored in pf['stage_platform']. So we may have to do a little dance to figure out if we're talking about the internal platform name or the external platform name. >+ # Detect platform from builder name by tokenizing by '_', and matching >+ # the first token after the prefix This might be problematic that way; hopefully not. I think a lot of the standalone functions don't have the full release config though, which may give us headaches. Really hoping not. >+ change_source.append(UrlPoller( >+ branch=builderPrefix('android_signature_verification'), >+ url='%s/android-r7/en-US/' % makeCandidatesDir( this'll be '%s/android/en-US'. >+ change_source.append(UrlPoller( >+ branch=builderPrefix('android_signature_verification'), >+ url='%s/android-r7/multi/' % makeCandidatesDir( '%s/android/multi/'. We might want to put this inside of an 'if enable_multilocale' but I don't think it'll hurt if we keep it. Also, with rapid releases I'm hoping we don't have another non-localized release. >+ android_signature_verification_scheduler = Scheduler( >+ name=builderPrefix('android_signature_verification_scheduler'), >+ branch=builderPrefix('android_signature_verification'), >+ treeStableTimer=None, >+ builderNames=[builderPrefix('android_signature_verification')], >+ ) >+ schedulers.append(android_signature_verification_scheduler) We can do this inside of sign_android.sh, as a check before even attempting to upload. I'm fine with this approach too, though. >+ tag_scheduler = Scheduler( >+ name=builderPrefix('mobile_tag'), >+ branch=sourceRepoInfo['path'], >+ builderNames=[builderPrefix('mobile_tag')], >+ treeStableTimer=None, >+ fileIsImportant=lambda c: not isHgPollerTriggered(c, branchConfig['hgurl']) >+ ) As mentioned in the configs feedback, we're going to have to be careful with how the tagging (and staging repo setup) builders work with each other; they can easily break each other. For this pass, I think we've got some workarounds, like waiting up to an hour before starting the mobile release after the desktop release finishes tagging. >+ if releaseConfig['doPartnerRepacks'] and \ >+ platform in releaseConfig['partnerRepackPlatforms']: >+ partner_scheduler = Dependent( >+ name=builderPrefix('mobile_partner_repack', platform), >+ upstream=build_scheduler, >+ builderNames=[builderPrefix('mobile_partner_repack', platform)] >+ ) We need to do partner repacks at some point, but we only need them by 6.0 final. If we ignore them in the rush to 6.0b1, that's fine with me. >+ if platform in releaseConfig['l10nPlatforms']: >+ l10nPlatform = platform >+ if l10nPlatform.startswith('maemo'): >+ l10nPlatform = 'maemo' a) it'll be linux-maemo5-gtk, so it won't startswith('maemo') anymore, and b) our l10nPlatforms will be linux-mobile, macosx-mobile, and win32-mobile now. This is because Maemo single locale repacks are broken, we have much more reliable desktop-platform repacks, and we may or may not be discontinuing Maemo support at some point. >+ for platform in releaseConfig['enUSDesktopPlatforms']: >+ if platform in releaseConfig['l10nDesktopPlatforms']: Ah, but those will be in l10nDesktopPlatforms. This split actually isn't really required. Ben asked me to split desktop and non-desktop mobile release builders in case we ever have the same platform-name in both. We haven't needed it to date. I don't mind keeping it, but if it's easier to put them in the same loop, that's fine with me. >+ if platform.startswith('linux-maemo'): Ah, good :) >+ if releaseConfig['enableMultiLocale']: >+ multiLocale = branchConfig['enable_multi_locale'] >+ else: >+ multiLocale = False In 0.8, Maemo multilocale will need to use mozharness. I hacked that into 0.7 release configs by specifying an android multilocale config file. For 0.8 releases we probably want to a) determine the config file name programmatically (BRANCH-PLATFORM-release.json ?) and b) find the multilocale script via some config. linux-android: mozharness/scripts/multil10n.py linux-maemo5-gtk: mozharness/scripts/maemo_multi_locale_build.py >+ elif platform.startswith('linux-android'): >+ mozconfig = 'mobile/android/%s/release' % \ >+ releaseConfig['mozconfigDir'] >+ releaseWorkDir = 'build-release' # pf['base_workdir'] + '-release' >+ previousCandidateDir = '%s-candidates/build%s/%s' % \ >+ (releaseConfig['oldVersion'], >+ releaseConfig['oldBuildNumber'], >+ platform) >+ currentCandidateDir = '%s-candidates/build%s/%s' % \ >+ (releaseConfig['version'], >+ releaseConfig['buildNumber'], >+ platform) >+ updatePlatform = 'Android_arm-eabi-gcc3' >+ ausPreviousUploadDir = \ >+ "%s/%s/%s/%%(previous_buildid)s/en-US/beta-cck-test" % \ >+ (releaseConfig['ausBaseUploadDir'], >+ releaseConfig['oldVersion'], updatePlatform) >+ ausFullUploadDir = '%s/%s/%s/%%(buildid)s/en-US/beta-cck-test' % \ >+ (releaseConfig['ausBaseUploadDir'], >+ releaseConfig['version'], updatePlatform) The beta-cck-test hardcode probably needs to go into the config file :) I think we currently build Android release snippets but don't do anything with them. This is because a) we update via Android Market b) the generated snippets are for the unsigned Android apk, so the hash breaks when we sign, and we need to edit manually I don't think we'll need to have working Android release snippets for 6.0b1. Having said that, Android partner repacks may require Android release snippets in the not distant future. Again, if we miss this for 6.0b1 but get in place by 6.0 final, that's probably fine. >+ build_factory = AndroidReleaseBuildFactory( <snip> >+ multiLocale=multiLocale, >+ mozharnessConfig=releaseConfig['androidMozharnessConfig'], >+ productName=releaseConfig['productName'], >+ ) I hardcoded the mozharness_repo_path and tag in the 0.7 android build factory; in 0.8 land we probably want to pass those in here. >+ if platform in releaseConfig['l10nPlatforms']: >+ if platform.startswith('maemo'): >+ # releaseBuildDir = pf['base_builddir'] + '-l10n-release' >+ repack_factory = MaemoReleaseRepackFactory( As mentioned above, let's kill MaemoReleaseRepackFactory :) Let's add ScriptFactory's for desktop l10n repacks instead. >+ packageGlobList = [] >+ if platform == 'linux-mobile': >+ packageGlobList = ['-r', 'dist/*.tar.bz2', >+ 'dist/*.zip'] >+ elif platform == 'macosx-mobile': >+ packageGlobList = ['-r', 'dist/*.dmg'] >+ elif platform == 'win32-mobile': >+ packageGlobList = ['-r', 'dist/*.zip'] packageGlobLists are old cruft -- remove these lines. |make upload| does all the dirty work for us now. >+ build_factory = ReleaseMobileDesktopBuildFactory( I *think* ReleaseBuildFactory will work for us. I mainly looked at the common functions for areas where the platform/stage_platform difference will trip us up. I *think* we can send stage_platform (android and maemo5-gtk rather than linux-android and linux-maemo5-gtk) to all or most of them, which is a relief.

First patch that should adapt ReleaseBuildFactory to be mobile-friendly. -product name to brand name relation -we current take brand name to be capitalized product name -product name would need to be in a release config -we could move product name to platform variable -prettynames -tested with mobile-on-desktop linux and it seems to work properly -make package and make package-tests resulted in -linux-x86_64/en-US/fennec-5.tar.bz2 -linux-x86_64/en-US/fennec-5.tests.zip -linux-x86_64/en-US/fennec-5.txt -created generateMars variable to decide whether mars are generated -possible filesystem collisions between maemo5 gtk and qt -post_upload.py as well as packager.mk issues. -need to make sure that unittestBranch is set properly, as it is outside of the factory

Comment on attachment 537848 [details] [diff] [review] ReleaseBuildFactory patch >diff --git a/process/factory.py b/process/factory.py >--- a/process/factory.py >+++ b/process/factory.py >@@ -2537,7 +2537,7 @@ class CCNightlyBuildFactory(CCMercurialB > class ReleaseBuildFactory(MercurialBuildFactory): > def __init__(self, env, version, buildNumber, brandName=None, > unittestMasters=None, unittestBranch=None, talosMasters=None, >- **kwargs): >+ generateMars=True, **kwargs): We'll want to add android snippet capability at some point. >+ # XXX this is likely to be invalid for mobile, should we remove this >+ # case and assert brandName? > self.brandName = kwargs['productName'].capitalize() Not sure what we use this for. > self.addStep(ShellCommand, > name='echo_buildID', > command=['bash', '-c', > WithProperties('echo buildID=%(buildid)s > ' + \ >- '%s_info.txt' % self.platform)], >+ '%s_info.txt' % self.complete_platform)], This should be stage_platform. >@@ -2586,28 +2591,34 @@ class ReleaseBuildFactory(MercurialBuild > uploadEnv.update({'UPLOAD_HOST': self.stageServer, > 'UPLOAD_USER': self.stageUsername, > 'UPLOAD_TO_TEMP': '1', >- 'UPLOAD_EXTRA_FILES': '%s_info.txt' % self.platform}) >+ 'UPLOAD_EXTRA_FILES': '%s_info.txt' % self.complete_platform}) Same here. >+ # XXX there could be filesystem name collisions with >+ # maemo5 gtk and qt here. Good to keep in mind, but I doubt maemo5-qt will ever be a release platform. > uploadEnv['POST_UPLOAD_CMD'] = postUploadCmdPrefix( >- product=self.productName, >+ product=self.stageProduct, > version=self.version, > buildNumber=str(self.buildNumber), > to_candidates=True, > as_list=False) Do we need to add: * stage platform * nightly_dir = 'candidates' for mobile ? >+ # XXX need to ensure that whatever generates unittestBranch >+ # uses complete_platform We don't have unit tests for Fennec releases yet, but will someday.

(In reply to comment #18) > Comment on attachment 537848 [details] [diff] [review] [review] > ReleaseBuildFactory patch > > >diff --git a/process/factory.py b/process/factory.py > >--- a/process/factory.py > >+++ b/process/factory.py > >@@ -2537,7 +2537,7 @@ class CCNightlyBuildFactory(CCMercurialB > > class ReleaseBuildFactory(MercurialBuildFactory): > > def __init__(self, env, version, buildNumber, brandName=None, > > unittestMasters=None, unittestBranch=None, talosMasters=None, > >- **kwargs): > >+ generateMars=True, **kwargs): > > We'll want to add android snippet capability at some point. > > >+ # XXX this is likely to be invalid for mobile, should we remove this > >+ # case and assert brandName? > > self.brandName = kwargs['productName'].capitalize() > > Not sure what we use this for. I dont see it used in ReleaseBuildFactory, but it is used in other release related factories > >- '%s_info.txt' % self.platform)], > >+ '%s_info.txt' % self.complete_platform)], > > This should be stage_platform. > <snip> > Same here. sounds good. > Good to keep in mind, but I doubt maemo5-qt will ever be a release platform. Not sure why, but I thought it was. Cool! > Do we need to add: > > * stage platform no, that is taken care of as part of by pretty name packaging, it turns out. A platform subdirectory is created in the dist dir which is uploaded. we should figure out what android builds set that to. > * nightly_dir = 'candidates' for mobile ? I would guess that is something we want to be standardized between desktop and mobile. If not, we can key adding that to the kwargs on the stage_product.

Comment on attachment 537929 [details] [diff] [review] ReleaseBuildFactory patch v2 >+ # XXX there could be filesystem name collisions with >+ # maemo5 gtk and qt here. >+ uploadArgs = { >+ 'product': self.stageProduct, >+ 'version': self.version, >+ 'buildNumber': str(self.buildNumber), >+ 'to_candidates': True, >+ 'as_list': False} >+ # XXX need to decide if we want this change for all >+ # release builds or just mobile >+ if 'mobile' in stageProduct: >+ uploadArgs['nightly_dir'] = 'candidates' Looks like you forgot to use uploadArgs in postUploadCmdPrefix: > uploadEnv['POST_UPLOAD_CMD'] = postUploadCmdPrefix( >- product=self.productName, >+ product=self.stageProduct, Other hunks look good

(In reply to comment #13) > Comment on attachment 537144 [details] [diff] [review] [review] > configs > > I know the config is just a stub for now, but do you actually need > base_enUS_binaryURL? You should be able to calculate that in the runtime > l10n scripts. The glue in builder_master.cfg looks sane enough to me. Fixed. (In reply to comment #14) > Comment on attachment 537145 [details] [diff] [review] [review] > buildbotcustom > > + envJava = builder_env.copy() > > + envJava['PATH'] = '/tools/jdk6/bin:%s' % envJava.get( > > What's this for? Seems like it should be something that goes in config.py > rather than here It was ported from 0.7 as is. Instead of refactoring it I left it as is. It will be removed because android verification will be moved to tools/release/signing/Makefile. > I'm not really sold on this "common_" prefix on functions, since they're in > a file called "release_common". How about "from process import > release_common" or "from process.release_common import builderPrefix as > common_builderPrefix" in mobile_release.py instead? Function renamed to getSomething to prevent name collision. (In reply to comment #15) > Comment on attachment 537144 [details] [diff] [review] [review] > configs > > >+ for product, sourceRepoKey, releaseBranch, releaseObjGenerator in \ > >+ [('firefox', 'mozilla', b, generateReleaseBranchObjects) for b in ACTIVE_RELEASE_BRANCHES] + \ > >+ [('fennec', 'mobile', b, generateMobileReleaseBranchObjects) for b in ACTIVE_MOBILE_RELEASE_BRANCHES]: > > Are we splitting mozilla and mobile so we can join them but keep them > separate at some later point? Splitting firefox and fennec objects are better, IMHO, at least at this phase. We may have firefox and mobile release builders on different masters, so it will be easier to reconfig them. > We probably want to revisit this in version 2, when we consolidate; not > important now. ++ > For the short term, I think we can enable desktop mozilla-beta on one > master, and mobile mozilla-beta on another. Is that what you're thinking? Yes. > Otherwise we may get duplicate schedulers on mozilla-beta. They have different names, so there shouldn't be any collision even if we have the builders on the same master. > >+ACTIVE_MOBILE_RELEASE_BRANCHES = ['mobile-beta',] > > Is this a leftover, or are you abstracting mozilla-beta -> mobile-beta for > mobile? We've done a lot of fake-naming for mobile; I'm hoping we don't > need to this time around, but I haven't dug fully into this yet. It's ACTIVE_MOBILE_RELEASE_BRANCHES = ['mozilla-beta',] now. My fault. > We'll want to add mobile/confvars.sh to bumpFiles. Fixed. > One thing that's been worrying me a bit -- we'll be releasing off the same > repo, and we've been using the same changeset and a very similar 'go' time. > > The relbranches we're using are using timestamps down to the hour. > > I think for the first 0.8 mobile beta, we may have to wait until the time > changes to up to an hour after the Firefox release starts, so that we can > create a second relbranch. Either that, or use the same relbranch somehow. Using the same relbranch may introduce some problems. Let's think the following scenario. 0) Firefox and Fennec use the same changeset on default. 1) Firefox tagging generates relbranch, bumps files on relbranch, commits (what creates another changeset on relbranch with properly set bumped versions), and tags this *new* version. 2) Firefox gets a fix on relbranch and the corresponding changeset (biuld2). 3) For some reasons Fennec is needed to be released off initial changeset (not the build2 one). In this case release automation can't use initial changeset for Fennec without creating new relbranch (or an anon branch) because mobile/confvars.sh is needed to be bumped. So we have some ways to go: 1) Easiest and safest: wait ~1h and a new relbranch for mobile will be generated automatically. Devs need to land build2 fixes on this branch besides Firefox relbranch. 2) Use minutes in relbranch name. Wait 1m. Devs need to land build2 fixes on this branch besides Firefox relbranch. 3) Use FENNEC_* relbranch. Devs need to land build2 fixes on this branch besides Firefox relbranch. 4) Use Firefox relbranch. In this case we won't be able to create Fennec build1 after Firefox build2 using changesets before build2 without creating another head (relbranch). > >+releaseConfig['shippedLocalesPath'] = 'mobile/locales/shipped-locales' > > We don't use shipped locales; all that data is in the json. Removed. (In reply to comment #16) > Comment on attachment 537145 [details] [diff] [review] [review] > buildbotcustom > > I think we'll be using ReleaseBuildFactory for all mobile release builds now. Here I need some help with setting proper factories. I used 0.7 factories as is, but I know that there are a lot of changes have landed recently, so we need completely review the factories used for mobile releases. > > For emails, do you want them all to still say [release] or do you want to > have something like [mobile release] ? The former would be better for > consolidated releases, but the latter might be better for split releases. I've changed it to [mobile release] in releaseConfig for now. > >+ unix_slaves = branchConfig['platforms'].get('linux', {}).get('slaves', > >+ []) + \ > >+ branchConfig['platforms'].get('linux64', {}).get('slaves', > >+ []) + \ > >+ branchConfig['platforms'].get('macosx', {}).get('slaves', > >+ []) + \ > >+ branchConfig['platforms'].get('macosx64', {}).get('slaves', []) > >+ all_slaves = unix_slaves + \ > >+ branchConfig['platforms'].get('win32', {}).get('slaves', []) + \ > >+ branchConfig['platforms'].get('win64', {}).get('slaves', []) > > This probably needs changing to match the mobile platform list, but you > probably know that :) Fixed. > >+ if 'signedPlatforms' in releaseConfig.keys(): > >+ signedPlatforms = releaseConfig['signedPlatforms'] > >+ else: > >+ signedPlatforms = ('android-r7',) > > android-r7 is now linux-android. Fixed. > Which presents another small problem -- it's 'linux-android' internally, but > I want it to be 'android' on all externally-facing interfaces... that's > currently stored in pf['stage_platform']. > > So we may have to do a little dance to figure out if we're talking about the > internal platform name or the external platform name. I planned to cope with this on-the-fly, when we have real factories set. > >+ # Detect platform from builder name by tokenizing by '_', and matching > >+ # the first token after the prefix > > This might be problematic that way; hopefully not. > I think a lot of the standalone functions don't have the full release config > though, which may give us headaches. Really hoping not. The same here. > >+ change_source.append(UrlPoller( > >+ branch=builderPrefix('android_signature_verification'), > >+ url='%s/android-r7/en-US/' % makeCandidatesDir( > > this'll be '%s/android/en-US'. > > >+ change_source.append(UrlPoller( > >+ branch=builderPrefix('android_signature_verification'), > >+ url='%s/android-r7/multi/' % makeCandidatesDir( > > '%s/android/multi/'. We might want to put this inside of an 'if > enable_multilocale' but I don't think it'll hurt if we keep it. Also, with > rapid releases I'm hoping we don't have another non-localized release. Pollers are used for android_signature_verification. Hope it will be removed soon. > >+ android_signature_verification_scheduler = Scheduler( > >+ name=builderPrefix('android_signature_verification_scheduler'), > >+ branch=builderPrefix('android_signature_verification'), > >+ treeStableTimer=None, > >+ builderNames=[builderPrefix('android_signature_verification')], > >+ ) > >+ schedulers.append(android_signature_verification_scheduler) > > We can do this inside of sign_android.sh, as a check before even attempting > to upload. I'm fine with this approach too, though. Removing is easy. :) > >+ tag_scheduler = Scheduler( > >+ name=builderPrefix('mobile_tag'), > >+ branch=sourceRepoInfo['path'], > >+ builderNames=[builderPrefix('mobile_tag')], > >+ treeStableTimer=None, > >+ fileIsImportant=lambda c: not isHgPollerTriggered(c, branchConfig['hgurl']) > >+ ) > > As mentioned in the configs feedback, we're going to have to be careful with > how the tagging (and staging repo setup) builders work with each other; they > can easily break each other. > > For this pass, I think we've got some workarounds, like waiting up to an > hour before starting the mobile release after the desktop release finishes > tagging. ++ > a) it'll be linux-maemo5-gtk, so it won't startswith('maemo') anymore, and Fixed. > b) our l10nPlatforms will be linux-mobile, macosx-mobile, and win32-mobile > now. This is because Maemo single locale repacks are broken, we have much > more reliable desktop-platform repacks, and we may or may not be > discontinuing Maemo support at some point. I planned to fix this while testing it in staging. > >+ for platform in releaseConfig['enUSDesktopPlatforms']: > >+ if platform in releaseConfig['l10nDesktopPlatforms']: > > Ah, but those will be in l10nDesktopPlatforms. > > This split actually isn't really required. Ben asked me to split desktop > and non-desktop mobile release builders in case we ever have the same > platform-name in both. We haven't needed it to date. Let's set up the factories, then we can fix this as well. 0.7 uses different factories here, so this was the only reason for another loop. > The beta-cck-test hardcode probably needs to go into the config file :) I hope it will be gone after switching to the new factories. > packageGlobLists are old cruft -- remove these lines. The same here. Conclusion. At this point I have 2 stoppers. 1) Need to properly assign and setup factories. Help is appreciated (read needed :) ). 2) Can't really play with this because of HG madness. See bug 661828 and bug 662760.

I am going to push my patch to my user repo at http://hg.mozilla.org/users/jford_mozilla.com/buildbotcustom

this includes attachment 537981 [details] [diff] [review] and changes to it that I needed to make to get it to checkconfig. This patch instantiates ReleaseBuildFactory as needed.

This time, I ran hg diff in the correct repository!

Comment on attachment 538111 [details] [diff] [review] buidlbotcustom with factory work >--- a/process/factory.py >+++ b/process/factory.py >@@ -2608,7 +2608,7 @@ class ReleaseBuildFactory(MercurialBuild > 'as_list': False} > # XXX need to decide if we want this change for all > # release builds or just mobile >- if 'mobile' in stageProduct: >+ if 'mobile' in self.stageProduct: > uploadArgs['nightly_dir'] = 'candidates' > uploadEnv['POST_UPLOAD_CMD'] = postUploadCmdPrefix(**uploadArgs) BTW, this hunk doesn't apply. Looks like it's for attachment 537929 [details] [diff] [review].

This should fix bumpFile regexps. Otherwise tag-release.py has no idea how to bump mobile/confvars.sh.

Comment on attachment 538240 [details] [diff] [review] Fix bumpFile regexps Review of attachment 538240 [details] [diff] [review]: -----------------------------------------------------------------

Comment on attachment 538286 [details] [diff] [review] buidlbotcustom with factory work v2 The call to ReleaseBuildFactory.__init__ should also have stageProduct=pf['stage_product'] in the kwargs

Comment on attachment 538240 [details] [diff] [review] Fix bumpFile regexps http://hg.mozilla.org/build/tools/rev/51318fe29cea

Quick and dirty locale->revision extractor from json files used for mobile. It worked fine, but I'm not happy with .endswith('.json') approach. Ideas are welcome.

My current configs with some local changes.

Current buildbotcustom with jford's changes incorporated.

Summary. * Tagging worked as expected. Need to figure out how to separate firefox and fennec sendchanges. * Source builder fails. It tries to create firefox*tar.bz2 tarball. * Builds fail. To be investigated. * Need to add l10n repacks * Need to test partner repacks

Comment on attachment 538302 [details] [diff] [review] tools Review of attachment 538302 [details] [diff] [review]: ----------------------------------------------------------------- ::: lib/python/release/l10n.py @@ +37,5 @@ > if not l10nRepoPath.endswith('/'): > l10nRepoPath = l10nRepoPath + '/' > repositories = {} > + if fileName.endswith('.json'): > + for locale, data in json.load(open(fileName)).iteritems(): How about using try/except instead of if/else, and catching ValueError?

One of the problems we have here is the fact that the repos of firefox and fennec are the same, so sendchanges will be sent to the masters will contain the same branch name. Separating fennec and firefox masters won't help, because we use one scheduler master so the sendchange will be broadcasted to all masters. We could add another parameter to the sendchange and check it in fileIsImportant function of schedulers. Checking this parameter in the builder won't work if we want to disable (assign a dummy builder) tagging. Another (I think, more elegant) approach will be setting a "fake" branch for the scheduler and use it in the sendchange. In our case scheduler parameter will be branch=builderPrefix('mobile') and sendchange will look like: buildbot sendchange ... --branch release-mozilla-beta-mobile ... This works fine except Mercurial step, which tries to reuse this fake branch to checkout the sources: hg clone --verbose --noupdate http://hg.mozilla.org/release-mozilla-beta-mobile build The attached patch should resolve this problem. Not tested yet. Ideas are welcome as usual. :)

Comment on attachment 538475 [details] [diff] [review] use repourl for Mercurial This didn't work. I had to set branchType='inrepo' to make Mercurial source step happy, but even in this case it tries to update to the branch passed via the scheduler, which is release-mozilla-beta-mobile in our case...

(In reply to comment #40) > Created attachment 538475 [details] [diff] [review] [review] > use repourl for Mercurial > > One of the problems we have here is the fact that the repos of firefox and > fennec are the same, so sendchanges will be sent to the masters will contain > the same branch name. Separating fennec and firefox masters won't help, > because we use one scheduler master so the sendchange will be broadcasted to > all masters. > > We could add another parameter to the sendchange and check it in > fileIsImportant function of schedulers. Checking this parameter in the > builder won't work if we want to disable (assign a dummy builder) tagging. > > Another (I think, more elegant) approach will be setting a "fake" branch for > the scheduler and use it in the sendchange. In our case scheduler parameter > will be branch=builderPrefix('mobile') and sendchange will look like: > > buildbot sendchange ... --branch release-mozilla-beta-mobile ... > > This works fine except Mercurial step, which tries to reuse this fake branch > to checkout the sources: > > hg clone --verbose --noupdate > http://hg.mozilla.org/release-mozilla-beta-mobile build > > The attached patch should resolve this problem. Not tested yet. > > Ideas are welcome as usual. :) What if we created a new change property (say, 'products') that is set to a list of products (e.g. firefox, mobile) that the sendchange is intended for. For the build process, we could have two schedulers, one for each product enabled in the release config. Each of these schedulers could have a change filter that only says a change is important if its product is matches the schedulers's assigned product. The downside is that we still have two schedulers, but this is no worse than before. We also get the advantage of having one shared code path, that has two schedulers created in the same place.

I liked the idea. In this case we can start both Firefox and Fennec releases by one senchange, just pass "-p products:firefox,fennec". Something like this? def change_contains_product(change, productName): products = change.properties.getProperty("products") if products: if productName in products.split(','): return True return False tag_scheduler = Scheduler( name=builderPrefix('mobile_tag'), branch=sourceRepoInfo['path'], builderNames=[builderPrefix('mobile_tag')], treeStableTimer=None, fileIsImportant=lambda c: not isHgPollerTriggered( c, branchConfig['hgurl']) and change_contains_product(c, 'fennec') )

We can even drop isHgPollerTriggered in this case.

* Ignore local changes * I'm not happy with "if releaseConfig['productName'] == 'firefox'" * android gpg verification isn't integrated yet, shouldn't be hard * It passes checkconfig, looks good on the waterfall. However I haven't tried to start a staging release yet. I think, we need to decide which approach is better, integrated into release.p or separated.

Comment on attachment 539537 [details] [diff] [review] tools >diff --git a/lib/python/release/l10n.py b/lib/python/release/l10n.py >index da75121..ab1ffa3 100644 >--- a/lib/python/release/l10n.py >+++ b/lib/python/release/l10n.py >@@ -1,10 +1,14 @@ > from urllib2 import urlopen > from urlparse import urljoin >+try: >+ import simplejson as json >+except ImportError: >+ import json Hm, we may need to get attachment 539351 [details] [diff] [review] and this patch to play nicely together. I put code in release/platforms, but that doesn't necessarily obsolete this code. > ftp_platform_map = {'win32': 'win32', 'win64': 'win64', 'macosx': 'mac', > 'linux': 'linux-i686', 'linux64': 'linux-x86_64', >- 'macosx64': 'mac', 'android': 'android-r7'} >+ 'macosx64': 'mac', 'android': 'android-r7', >+ 'linux-android': 'android', >+ 'linux-maemo5-gtk': 'maemo5-gtk', >+ 'linux-mobile': 'linux-i686', >+ 'macosx-mobile': 'macosx-i686', >+ 'win32-mobile': 'win32-i686', >+ } Instead of using buildbot2ftp, I used stage_platform for this. This means {linux,macosx,win32}-mobile will show up as linux, macosx, win32. If using the buildbot2ftp function is easier, I'm not opposed. I suppose we could use stage_platform and override with buildbot2ftp if it returns something other than None. >+ if config.get('relbranchPrefix'): >+ generatedRelbranch = generateRelbranchName( >+ config['milestone'], prefix=config['relbranchPrefix']) It's awesome you can tag both products simultaneously.

Comment on attachment 539538 [details] [diff] [review] configs (integrated) > +releaseConfig['androidMozharnessConfig'] = 'multi_locale/beta_release_android.json' We'll need to create a beta_release_android.json, and we'll need a maemo5MozharnessConfig as well. Would that be best if it were programmatic? release_mozilla-beta_linux-android.json and release_mozilla-beta_linux-maemo5-gtk.json? Otherwise, I think I'm pretty happy with this. I hear that Catlee wanted some changes, though.

Comment on attachment 539539 [details] [diff] [review] buildbotcustom (integrated) > assert platform in ('linux', 'linux64', 'win32', 'win64', > 'macosx', 'macosx64', 'osx', 'osx64', >- 'maemo', 'android-r7') >+ 'maemo', 'android-r7', 'linux-android', 'linux-maemo5-gtk') Pretty sure we can get rid of 'maemo' and 'android-r7' here. Not a blocker. > # '-c' is for "release to candidates dir" > postUploadCmd = 'post_upload.py -p %s -v %s -n %s -c' % \ > (productName, version, buildNumber) >+ if productName == 'fennec': >+ postUploadCmd = 'post_upload.py -p mobile --nightly-dir candidates -v %s -n %s -c' % \ >+ (version, buildNumber) For post_upload, I've been using --upload-to-mobile-candidates-dir as the --upload-to-candidates-dir has Firefox desktop specific logic hardcoded. If we use that, we need --builddir (either platform name or platform/locale) passed in. If not, we may need to revisit post_upload.py to make --upload-to-candidates-dir mobile friendly. > def builderPrefix(s, platform=None): > if platform: > return "release-%s-%s_%s" % (sourceRepoInfo['name'], platform, s) > else: > return "release-%s-%s" % (sourceRepoInfo['name'], s) Will this work? (Wondering if firefox/fennec builder names will conflict). > if releaseConfig.get('enable_repo_setup'): >+ # FIXME: shouldn't be run twice Hm, maybe this is where doing one pass of generateReleaseBuilders() per branch would help. We could create a repo setup builder per branch, and make sure that all desktop- and mobile-required repos were uniquely identified in the list of repos. Same with tagging, though running two factories in parallel seems to work fine there. I don't see a need to do this in the first pass as long as we clean it up in a second pass; others may disagree. >- name=builderPrefix('release_downloader'), >+ name=builderPrefix( >+ '%s_release_downloader' % releaseConfig['productName']), Aha, this is how builderPrefix doesn't conflict. > if releaseConfig['doPartnerRepacks']: >+ # TODO: revisit this scheduler > for platform in releaseConfig['l10nPlatforms']: > partner_scheduler = Scheduler( > name=builderPrefix('partner_repacks', platform), > treeStableTimer=0, > branch=builderPrefix('post_%s_l10n' % platform), > builderNames=[builderPrefix('partner_repack', platform)], > ) > schedulers.append(partner_scheduler) For desktop it looks like we do partner repacks per l10nPlatform; for mobile we will need linux-maemo5-gtk, and potentially linux-android later. Maybe partnerRepackPlatforms ? We would be triggering after the build rather than the l10n repack, however, so maybe that's a flag. > if not releaseConfig.get('skip_release_download'): >+ # TODO: mobile relase downloader should know about >+ # "mobile" instead of productName and >+ # "candidates" instead of "nightly" > release_downloader_factory = ScriptFactory( > scriptRepo=tools_repo, > extra_args=[branchConfigFile], > scriptName='scripts/staging/release_downloader.sh', > ) Hm, if release download is for update generation, we don't have to get that done now (we currently enable updates at build time, but we either don't generate snippets or we don't use them, I forget.) If it's for testing repacks etc. without having to build first, then n/m. It would be nice to see a lot of the removed builders ported to work with Fennec, but that's completely out of scope for this bug.

Comment on attachment 539537 [details] [diff] [review] tools Review of attachment 539537 [details] [diff] [review]: ----------------------------------------------------------------- ::: scripts/release/tag-release.py @@ +209,5 @@ > # We generate this upfront to ensure that it's consistent throughout all > # repositories that use it. However, in cases where a relbranch is provided > # for all repositories, it will not be used > generatedRelbranch = generateRelbranchName(config['milestone']) > + if config.get('relbranchPrefix'): Sucks that we need this, but I don't see a way around it until we combine the tagging processes!

> > if releaseConfig['doPartnerRepacks']: > >+ # TODO: revisit this scheduler > > for platform in releaseConfig['l10nPlatforms']: > > partner_scheduler = Scheduler( > > name=builderPrefix('partner_repacks', platform), > > treeStableTimer=0, > > branch=builderPrefix('post_%s_l10n' % platform), > > builderNames=[builderPrefix('partner_repack', platform)], > > ) > > schedulers.append(partner_scheduler) > > For desktop it looks like we do partner repacks per l10nPlatform; for mobile > we will need linux-maemo5-gtk, and potentially linux-android later. > Maybe partnerRepackPlatforms ? We would be triggering after the build > rather than the l10n repack, however, so maybe that's a flag. partnerRepackPlatforms seems like a good approach to me. Not sure what to do about the triggering though....we definitely don't want to artificially block on l10n...

(In reply to comment #46) > I liked the idea. In this case we can start both Firefox and Fennec releases > by one senchange, just pass "-p products:firefox,fennec". > > Something like this? > > def change_contains_product(change, productName): > products = change.properties.getProperty("products") > if products: > if productName in products.split(','): > return True > return False > > tag_scheduler = Scheduler( > name=builderPrefix('mobile_tag'), > branch=sourceRepoInfo['path'], > builderNames=[builderPrefix('mobile_tag')], > treeStableTimer=None, > fileIsImportant=lambda c: not isHgPollerTriggered( > c, branchConfig['hgurl']) and change_contains_product(c, > 'fennec') > ) This is quite clever, I like it.

Comment on attachment 539539 [details] [diff] [review] buildbotcustom (integrated) Review of attachment 539539 [details] [diff] [review]: ----------------------------------------------------------------- Seeing all of the if productName == 'firefox' makes me wonder if full integration is the right thing. Overall, I don't think it's _too_ bad, but there's a lot of hacky or otherwise nasty stuff anywhere the release processes between the two differ. ::: l10n.py @@ +89,5 @@ > self.baseTag = baseTag > self.localesURL = localesURL if localesURL else "%s%s/raw-file/%s/%s" \ > + % (repo, branch, self.baseTag, localesFile) > + # TODO: check "revision" variable > + # % (repo, branch, revision or self.baseTag, localesFile) Are you reverting to the original code here, in the final version? ::: process/factory.py @@ +4120,5 @@ > # '-c' is for "release to candidates dir" > postUploadCmd = 'post_upload.py -p %s -v %s -n %s -c' % \ > (productName, version, buildNumber) > + if productName == 'fennec': > + postUploadCmd = 'post_upload.py -p mobile --nightly-dir candidates -v %s -n %s -c' % \ I haven't fully thought this through, but could we have post_upload.py be smart enough to use "candidates" as the nightly directory, for --release-to-candidates + product == mobile? If so, you could add another argument to the factory, "stage_product" or something, and pass productName for Firefox, and "mobile" for Fennec. ::: process/release.py @@ +287,3 @@ > )) > > + if releaseConfig['productName'] == 'firefox': I'm wondering if we can key this off of 'win' in signedPlatforms instead. Maybe that's a worse assumption, though =\. @@ +302,5 @@ > name=builderPrefix('repo_setup'), > branch=sourceRepoInfo['path'], > treeStableTimer=None, > builderNames=[builderPrefix('repo_setup')], > fileIsImportant=lambda c: not isHgPollerTriggered(c, Shouldn't you be checking changeContainsProduct here, too? Otherwise you'll fire tag for both desktop and mobile releases in staging. @@ +307,5 @@ > branchConfig['hgurl']) > ) > schedulers.append(repo_setup_scheduler) > + # FIXME: tag_scheduler should be triggered by repo_setup > + tag_scheduler = Triggerable( It's nice to see some transition towards Triggerable happening. The sooner we get there, the sooner we can have multi-master releases :). @@ +342,3 @@ > > + if releaseConfig['buildNumber'] == 1 \ > + and releaseConfig['productName'] == 'firefox': Do we not have Bouncer entries for mobile? If not, I'd like to see this controlled by an explicit parameter. Eg, if release.get('createBouncerEntries') or something similar. @@ +389,5 @@ > builderNames=[builderPrefix('partner_repack', platform)], > ) > schedulers.append(partner_scheduler) > > + if releaseConfig['productName'] == 'firefox': This makes me wonder: Why don't we run l10n verify for mobile? It seems to me that if we're running it on Desktop, we should run it on mobile, too. That definitely should be a separate bug/issue, though. This is OK for the purposes of the initial port. @@ +399,5 @@ > + builderNames=[builderPrefix('l10n_verification', platform)] > + ) > + schedulers.append(l10n_verify_scheduler) > + > + updates_scheduler = Scheduler( This can be keyed off something else, too. Either patcherConfig or verifyConfigs.key() should be fine @@ +421,2 @@ > > + check_permissions_scheduler = Dependent( Like l10n verify, maybe we should be running similar things for this & virus scan. Totally fine to hide with a productName check for now, though. @@ +982,2 @@ > > + if releaseConfig['productName'] == 'firefox' and \ Same thing here w.r.t. checking a different variable.

Comment on attachment 539538 [details] [diff] [review] configs (integrated) Review of attachment 539538 [details] [diff] [review]: -----------------------------------------------------------------

(In reply to comment #58) > Seeing all of the if productName == 'firefox' makes me wonder if full > integration is the right thing. Overall, I don't think it's _too_ bad, but > there's a lot of hacky or otherwise nasty stuff anywhere the release > processes between the two differ. It's my understanding that Rail's doing the |if productName| portions to get this shoehorned in quickly, and that later we can either key off more relevant config settings or make existing Firefox desktop release builders work for mobile (e.g. bouncer submitter).

(In reply to comment #60) > (In reply to comment #58) > > Seeing all of the if productName == 'firefox' makes me wonder if full > > integration is the right thing. Overall, I don't think it's _too_ bad, but > > there's a lot of hacky or otherwise nasty stuff anywhere the release > > processes between the two differ. > > It's my understanding that Rail's doing the |if productName| portions to get > this shoehorned in quickly, and that later we can either key off more > relevant config settings or make existing Firefox desktop release builders > work for mobile (e.g. bouncer submitter). I don't want to block on enabling bouncer submission or other such things for mobile, but it's really no more to add a new flag in obvious cases, is it?

In the future we should use *_RELEASE tags for mozilla-beta as well.

Comment on attachment 543751 [details] [diff] [review] tools Review of attachment 543751 [details] [diff] [review]: ----------------------------------------------------------------- ::: lib/python/release/download.py @@ +37,5 @@ > files = makeReleaseRepackUrls(productName, brandName, version, appVersion, > platform) > > env = {} > + for fileName, remoteFile in files.iteritems(): s/file/fileName/ to not overload internal file function ::: lib/python/release/l10n.py @@ +29,5 @@ > > def getCommonLocales(a, b): > return [locale for locale in a if locale in b] > > +def getL10nRepositories(fileName, l10nRepoPath, relbranch=None): the same here @@ +39,5 @@ > l10nRepoPath = l10nRepoPath + '/' > repositories = {} > + file_handle = open(fileName) > + try: > + for locale, data in json.load(file_handle).iteritems(): Try to load a JSON file first, then regular l10n changesets file.

Comment on attachment 543753 [details] [diff] [review] configs Review of attachment 543753 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/production_builder_master_pm01_localconfig.py @@ +19,5 @@ > ] + ACTIVE_PROJECT_BRANCHES > ACTIVE_PROJECTS = PROJECTS.keys() > ACTIVE_RELEASE_BRANCHES = ['mozilla-1.9.1', 'mozilla-2.0', 'mozilla-release',] > +ACTIVE_MOBILE_RELEASE_BRANCHES = [] > +ACTIVE_MOBILE_RELEASE_BRANCHES = ['mozilla-beta'] Ignore the first added line, it will be removed from the final patch. ::: mozilla/production_builder_master_pm03_localconfig.py @@ +16,5 @@ > 'mozilla-2.0', 'mozilla-beta', 'mozilla-aurora', 'mozilla-release', > ] + ACTIVE_PROJECT_BRANCHES > ACTIVE_PROJECTS = PROJECTS.keys() > ACTIVE_RELEASE_BRANCHES = ['mozilla-1.9.2', 'mozilla-beta'] > +ACTIVE_MOBILE_RELEASE_BRANCHES = ['mozilla-release',] We can run fennec mozilla-beta on the same master, up to you.

Comment on attachment 543758 [details] [diff] [review] buildbotcustom Review of attachment 543758 [details] [diff] [review]: ----------------------------------------------------------------- ::: bin/log_uploader.py @@ -291,4 @@ > to_try=False, > who=None, > revision=None, > - builddir=None, No need to pass default value. ::: changes/ftppoller.py @@ +75,4 @@ > c = changes.Change(who = url, > comments = "success", > files = [], > + properties={'who': url}, Need this to run android signature verification for 2 different files using 1 builder. ::: process/factory.py @@ +161,5 @@ > cmd.append("--release-to-candidates-dir") > + if to_mobile_candidates: > + cmd.append("--release-to-mobile-candidates-dir") > + if nightly_dir: > + cmd.append("--nightly-dir=%s" % nightly_dir) sync with the same function from tools/lib/python :/ @@ +190,4 @@ > retval['symbolsUrl'] = m > elif m.endswith("tests.tar.bz2") or m.endswith("tests.zip"): > retval['testsUrl'] = m > + elif m.endswith('apk') and 'unsigned-unaligned' in m: don't ignore unsigned/android/*/*.apk, only unsigned-unaligned @@ +2570,5 @@ > env = env.copy() > # Make sure MOZ_PKG_PRETTYNAMES is on and override MOZ_PKG_VERSION > # The latter is only strictly necessary for RCs. > + if usePrettyNames: > + env['MOZ_PKG_PRETTYNAMES'] = '1' Otherwise it breaks Feenec build. @@ +2588,5 @@ > + command=['make', '-C', > + '%s/tools/update-packaging' % self.mozillaObjdir], > + env=self.env, > + haltOnFailure=True > + ) the same here @@ +2622,5 @@ > + builddir = '%s/%s' % (self.stagePlatform, postUploadBuildDir) > + uploadArgs['builddir'] = builddir > + uploadArgs['to_mobile_candidates'] = True > + uploadArgs['nightly_dir'] = 'candidates' > + uploadArgs['product'] = 'mobile' Fennec should upload using --to-mobile-candidates, --product mobile, --nightly-dir candidates and --builddir set to {android,maemo5-gtk,linux,macosx,win32}/{en-US,multi} ... @@ +2636,2 @@ > name='make_upload', > + command=['make', 'upload'] + upload_vars, ... and run `make upload AB_CD=multi` for multi @@ +2638,2 @@ > env=uploadEnv, > + workdir=objdir, WithProperites is important for scratchbox builds, but not for win builds @@ +4148,4 @@ > self.env['MOZ_OBJDIR'] = self.objdir > self.env['MOZ_PKG_PRETTYNAMES'] = '1' > self.env['MOZ_PKG_VERSION'] = version > + self.env['MOZ_PKG_APPNAME'] = productName Need to set it here to prevent `make source` generating firefox-* files. ::: process/release.py @@ +146,5 @@ > + 'stage_platform', buildbot2ftp(bare_platform)) > + if 'xulrunner' in platform: > + platformDir = '' > + if bare_platform in signedPlatforms: > + platformDir = 'unsigned/%s' % platformDir This fixes Bug 635512 - "xulrunner build available" email contains a broken link. @@ +343,3 @@ > fileIsImportant=lambda c: not isHgPollerTriggered(c, > branchConfig['hgurl']) > ) Repo setup doesn't work if you want to run Firefox and Fennec staging releases simultaneously. I set enable_repo_setup to False and run it manually in this case. @@ +417,5 @@ > )) > > if releaseConfig['doPartnerRepacks']: > + for platform in releaseConfig.get('partnerRepackPlatforms', > + releaseConfig['l10nPlatforms']): We use partnerRepackPlatforms for Fennec and l10nPlatforms for Firefox to distinguish partner repack platforms. If we want, I can make Firefox use partnerRepackPlatforms as well. @@ +648,5 @@ > + 'builddir': builderPrefix( > + '%s_tag' % releaseConfig['productName']), > + 'slavebuilddir': reallyShort( > + builderPrefix('%s_tag' % releaseConfig['productName'])), > + 'release_config': releaseConfigFile, I added release_config property, so we don't really need to pass it anymore. releaseConfigFile is the same thing. @@ +768,5 @@ > if not releaseConfig.get('skip_build'): > + if platform in releaseConfig['l10nPlatforms']: > + triggeredSchedulers = [builderPrefix('%s_repack' % platform)] > + else: > + triggeredSchedulers = None Need this to not trigger repacks if l10n is disabled. @@ +842,4 @@ > )) > > if platform in releaseConfig['l10nPlatforms']: > + if not releaseConfig.get('disableStandaloneRepacks'): Standalone repacks don't work for Fennec. @@ +972,5 @@ > 'properties': {'slavebuilddir': > reallyShort(builderPrefix('xulrunner_%s_build' % platform))} > }) > + notify_builders.append( > + builderPrefix('xulrunner_%s_build' % platform)) Xulrunner links are not broken anymore, can send emails to r-d. @@ +1015,5 @@ > + 'env': builder_env, > + 'properties': {'slavebuilddir': > + reallyShort(builderPrefix( > + 'partner_repack', platform))} > + }) I just fixed indentation here. @@ +1419,5 @@ > + envJava['PATH'] = '/tools/jdk6/bin:%s' % envJava.get( > + 'PATH', ':'.join(('/opt/local/bin', '/tools/python/bin', > + '/tools/buildbot/bin', '/usr/kerberos/bin', > + '/usr/local/bin', '/bin', '/usr/bin', > + '/home/cltbld/bin'))) this builder will be removed once the verification is the part of singing process, so I didn't moved the env to config.py. @@ +1455,5 @@ > extraRecipients=[recipient], > branches=[sourceRepoInfo['path']], > messageFormatter=createReleaseChangeMessage, > + changeIsImportant=lambda c: \ > + changeContainsProduct(c, releaseConfig['productName']) Don't send duplicates. @@ +1466,5 @@ > + fromaddr="release@mozilla.com", > + relayhost="mail.build.mozilla.org", > + sendToInterestedUsers=False, > + extraRecipients=[recipient], > + branches=[builderPrefix('post_signing')], post_signing is Firefox specific so far. May need to change this in the future if we have post_signing steps for Fennec. @@ +1488,4 @@ > sendToInterestedUsers=False, > extraRecipients=releaseConfig['AllRecipients'], > mode='all', > + builders=[b['name'] for b in builders + test_builders], Be more specific to prevent duplicates.

Overall comments. * I tested the patches for Firefox 5.0b6 build1, Fennec 5.0b6 build1 and build2, Firefox 5.0 build1 and build2, Fennec 5.0 build1 and build2. * Tested waterfall builders: mozilla-beta diff: http://people.mozilla.org/~raliiev/mobile_automation/builders_beta.diff mozilla-release diff: http://people.mozilla.org/~raliiev/mobile_automation/builders_release.diff * Tested binaries: Firefox diff: http://people.mozilla.org/~raliiev/mobile_automation/files.diff. Expected differences in contrib and files generated by signing. Fennec diff: http://people.mozilla.org/~raliiev/mobile_automation/fennec.files.diff ** some expected diffs (en-US files moved to en-US, maemo5 repacks removed, added desktop repacks) ** missing fennec-5.0.en-US.linux-gnueabi-arm.crashreporter-symbols-full.zip (investigating, probably not needed) ** android_info.txt renamed to linux_info.txt (investigating)

Comment on attachment 543751 [details] [diff] [review] tools Review of attachment 543751 [details] [diff] [review]: ----------------------------------------------------------------- What happened to the platforms.py changes here?

(In reply to comment #72) > What happened to the platforms.py changes here? I used stage_platofrm for mobile platforms instead, something like this: ftpPlatform = branchConfig['platforms'].get(p, {}).get('stage_platform', buildbot2ftp(p)) I can sync the platfoms.py, if it's necessary.

Comment on attachment 543751 [details] [diff] [review] tools >+ # Fennec files are uploaded to mobile/candidates instead of >+ # fennec/nightly >+ if product == 'fennec': >+ product = 'mobile' >+ if not nightlyDir: >+ nightlyDir = 'candidates' Are we always going to be passing in 'fennec' rather than 'mobile' ? If so, this is good. If not, we'll fail to set the nightlyDir to 'candidates' if we set product to 'mobile'. Other than that, this looks good.

(In reply to comment #74) > Are we always going to be passing in 'fennec' rather than 'mobile' ? This is for case when we pass default values from release config. If you are passing 'mobile', you'll probably pass 'candidates' as well.

Comment on attachment 543753 [details] [diff] [review] configs >diff --git a/mozilla/production_builder_master_pm01_localconfig.py b/mozilla/production_builder_master_pm01_localconfig.py >--- a/mozilla/production_builder_master_pm01_localconfig.py >+++ b/mozilla/production_builder_master_pm01_localconfig.py >@@ -14,16 +14,18 @@ c['manhole'] = manhole.PasswordManhole(" > import config > reload(config) > from config import BRANCHES, SLAVES, PROJECTS, ACTIVE_PROJECT_BRANCHES > ACTIVE_BRANCHES = ['shadow-central', 'mozilla-1.9.2', 'mozilla-central', > 'mozilla-2.0', 'mozilla-beta', 'mozilla-aurora', 'mozilla-release', > ] + ACTIVE_PROJECT_BRANCHES > ACTIVE_PROJECTS = PROJECTS.keys() > ACTIVE_RELEASE_BRANCHES = ['mozilla-1.9.1', 'mozilla-2.0', 'mozilla-release',] >+ACTIVE_MOBILE_RELEASE_BRANCHES = [] >+ACTIVE_MOBILE_RELEASE_BRANCHES = ['mozilla-beta'] We may need to find different masters to put these on since Catlee's retiring pm01-pm03 in bug 663195. >diff --git a/mozilla/release-fennec-mozilla-beta.py b/mozilla/release-fennec-mozilla-beta.py >new file mode 100644 >--- /dev/null >+++ b/mozilla/release-fennec-mozilla-beta.py >@@ -0,0 +1,111 @@ >+releaseConfig = {} >+ >+# Release Notification >+releaseConfig['AllRecipients'] = ['release@mozilla.com',] >+releaseConfig['PassRecipients'] = ['release-drivers@mozilla.org',] >+releaseConfig['releaseTemplates'] = 'release_templates' >+releaseConfig['messagePrefix'] = '[release] ' Are we going to keep [release] ? In a way, this will be good to integrate mobile and desktop more tightly. Since it says "Fennec" in the subject we're good. >+releaseConfig['enUSPlatforms'] = ('linux-maemo5-gtk', 'linux-android', >+ 'linux-mobile', 'macosx-mobile', >+ 'win32-mobile') >+releaseConfig['signedPlatforms'] = ('linux-android',) >+releaseConfig['unittestPlatforms'] = releaseConfig['enUSPlatforms'] >+releaseConfig['talosTestPlatforms'] = () >+releaseConfig['enableUnittests'] = True We currently only have the capability to test linux-maemo5-gtk, linux-android, and linux-mobile. Pretty sure only linux-android and linux-mobile will work via sendchange. And we haven't really had any release automated tests, yet, so whether they'll work out of the box is questionable. Not sure if this matters... we may be able to ignore this for now. Just noting ftr. >+releaseConfig['mozharness_config'] = { >+ 'linux-maemo5-gtk': 'multi_locale/mozilla-beta_linux-maemo5-gtk.json', >+ 'linux-android': 'multi_locale/mozilla-beta_linux-android.json', >+} As noted in email, these are the nightly multilocale configs. We've got a split-repo 6.0 android config that needs to be massaged to work for mozilla-beta and 0.8 configs; I think we need to add a 6.0 maemo config. Let me know if you need help with these, but it sounded like you had a handle on it last night. >+++ b/mozilla/release-fennec-mozilla-release.py We'll want to add partner repacks for mozilla-release linux-maemo5-gtk, but that's not a blocker for 6.0b2. If you're still having issues with partner-repacks.py and want me to look, I can do that. >diff --git a/mozilla/release_templates/fennec_tag_change b/mozilla/release_templates/fennec_tag_change >new file mode 120000 >--- /dev/null >+++ b/mozilla/release_templates/fennec_tag_change >@@ -0,0 +1,1 @@ >+tag_change >\ No newline at end of file Are these all softlinks? If so, are they intentionally all the same as the original file? Overall, this looks good. The mozharness release multilocale config changes are required for 6.0b2; I'd also like to know about the release_templates. Those are my only areas of concern, so I was waffling on whether to plus or minus this patch.

(In reply to comment #76) > >+releaseConfig['mozharness_config'] = { > >+ 'linux-maemo5-gtk': 'multi_locale/mozilla-beta_linux-maemo5-gtk.json', > >+ 'linux-android': 'multi_locale/mozilla-beta_linux-android.json', > >+} > > As noted in email, these are the nightly multilocale configs. > We've got a split-repo 6.0 android config that needs to be massaged to work > for mozilla-beta and 0.8 configs; I think we need to add a 6.0 maemo config. > > Let me know if you need help with these, but it sounded like you had a > handle on it last night. Yeah, it would be better to keep configs separated. The patch adds production and staging configs for mozilla-beta and mozilla-release based releases for android and maemo5-gtk (8 files). They will be used in buildbot-confis.

(In reply to comment #76) > Comment on attachment 543753 [details] [diff] [review] [review] > We may need to find different masters to put these on since Catlee's > retiring pm01-pm03 in bug 663195. Yup. Will be available in the next patch. > Are we going to keep [release] ? > In a way, this will be good to integrate mobile and desktop more tightly. > Since it says "Fennec" in the subject we're good. +1 > We've got a split-repo 6.0 android config that needs to be massaged to work > for mozilla-beta and 0.8 configs; I think we need to add a 6.0 maemo config. Will be fixed in the next patch. > If you're still having issues with partner-repacks.py and want me to look, I > can do that. Partner repacks factory (actually script) didn't work out of the box. Haven't dug into it yet. > >diff --git a/mozilla/release_templates/fennec_tag_change b/mozilla/release_templates/fennec_tag_change > >new file mode 120000 > >--- /dev/null > >+++ b/mozilla/release_templates/fennec_tag_change > >@@ -0,0 +1,1 @@ > >+tag_change > >\ No newline at end of file > > Are these all softlinks? > If so, are they intentionally all the same as the original file? Yes and yes. They point to a shared template. The final emails are rendered properly, using Firefox and Fennec. > Overall, this looks good. > The mozharness release multilocale config changes are required for 6.0b2; > I'd also like to know about the release_templates. > Those are my only areas of concern, so I was waffling on whether to plus or > minus this patch. Nice. I'll update my final patches (with some additions) and submit them in a bit.

Interdiff comments: > - signed_apk_url = '%s%s/%s-%s.%s.eabi-arm.apk' % \ > + signed_apk_url = '%s%s/%s/%s-%s.%s.eabi-arm.apk' % \ Added locale (en-US or multi) to the URL. > + mozconfigBranch=releaseTag, ReleaseBuildFactory was updating configs to default "production" branch instead of the release tag. > - WithProperties('--apk=%(who)')] > + WithProperties('--apk=%(who)s')] Booooo. :)

1) makeReleaseRepackUrls used to use appVersion to generate URLs. It works for releases when version == appVersion, but not for betas. For example in 6.0b1 version is 6.0b1, appVersion is 6.0 (it used to alter some files while tagging), and binaries contain 6.0b1. 2) release_sanity.py: added --products parameter. sendchange doesn't send release_config parameter anymore. No special checks for mobile.

> We may need to find different masters to put these on since Catlee's retiring pm01-pm03 in bug 663195. Done. Also see mozilla/build_localconfig.py changes. * I've added l10n-changesets_mobile-{beta,release}.json as placeholders by copying from mozilla2/l10n-changesets_mobile-6.0.json. * Some minor changes in staging configs. * beta and release fennec configs use the updated mozharness configs.

Since the previous one has not been reviewed yet ( :P ), I replace it with a new one which fixes partner repacks.

Comment on attachment 544764 [details] [diff] [review] multi_l10n mozharness release configs This doesn't really affect anything since we don't run those steps, but the mozconfigs should s,nightly,release, . r=me with that. Also, we're going to have to bump these every single release to bump versions. I think we'll want to abstract this out and pass in the version via commandline. Not part of this bug; I added that to bug 664476 comment 1.

Probably the last piece to be changed - puppet templates for BuildSlaves.py. Can be landed any time (read ASAP).

Comment on attachment 545348 [details] [diff] [review] puppet-manifests http://hg.mozilla.org/build/puppet-manifests/rev/04e9c2ea7594

Comment on attachment 544764 [details] [diff] [review] multi_l10n mozharness release configs http://hg.mozilla.org/build/mozharness/rev/ec548c970ea4

Comment on attachment 544768 [details] [diff] [review] tools http://hg.mozilla.org/build/tools/rev/a39d94b5bb3e

Comment on attachment 544776 [details] [diff] [review] configs http://hg.mozilla.org/build/buildbot-configs/rev/65f29a2b52b4

Comment on attachment 544804 [details] [diff] [review] buildbotcustom http://hg.mozilla.org/build/buildbotcustom/rev/407c31d4c137

Comment on attachment 545652 [details] [diff] [review] Enable mobile releases in preproduction http://hg.mozilla.org/build/buildbot-configs/rev/fc3f2968bbb7

Pushed http://hg.mozilla.org/build/buildbot-configs/rev/0631f9380cd2 (r=aki on IRC).

I created symlinks to release-fennec-mozilla-{beta,release}.py in master_dirs and reconfigured the following masters: buildbot-master07.build.sjc1.mozilla.com:/builds/buildbot/build1 buildbot-master08.build.sjc1.mozilla.com:/builds/buildbot/build1 buildbot-master12.build.scl1.mozilla.com:/builds/buildbot/build1 buildbot-master13.build.scl1.mozilla.com:/builds/buildbot/build1 buildbot-master5.build.mozilla.org:/builds/buildbot/build_master6

Just want to be super duper sure that changeContainsProduct doesn't fail.

Comment on attachment 545900 [details] [diff] [review] safer changeContainsProduct http://hg.mozilla.org/build/buildbotcustom/rev/ef21ee3bf41f

Landed on production and reconfigured.