bhearsum@mozilla.com (:bhearsum) Reporter
	Description • 15 years ago

If the Try server master was simply another block in config.py we could have a master3.cfg alongside that setups up the /try branch _exactly_ the same as any other branch. This would be a great way of keeping it in line with the rest of production.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 1 • 15 years ago

Had this running in it's own buildbot master instance on try staging.  Builds work fine with the exception of symbol uploads and graph server posts, which makes sense at the moment.

The Mailer doesn't work yet, and the packaged builds aren't triggered with this config yet either. 

Will continue to work on this, but posting the patch in the interim as proof of concept.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 2 • 15 years ago

Comment on attachment 405089 [details] [diff] [review]
WIP - Adds a master3.cfg and 'try' as a branch in config.py

This is a great start. Some drive by comments, hopefully they'll shortcut some problems.

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>+BRANCHES['try']['brand_name'] = 'Minefield'

Not sure what this is used for, but may be problematic if people build m-1.9.2 or m-1.9.1 on Try.

>+BRANCHES['try']['platforms']['linux']['mozconfig'] = 'linux/tryserver/nightly'

Could we use something more appropriate than 'nightly' ? Maybe 'build' or 'optimized'

>+BRANCHES['try']['platforms']['linux-debug']['builds_before_reboot'] = 5

Ben is setting/has set this to 1 on another bug. You could change it closer to review time if that's a pain for debugging.

>+BRANCHES['try']['enable_codecoverage'] = True
>+BRANCHES['try']['enable_shark'] = True
>+BRANCHES['try']['create_snippet'] = True

Should all be False. The last one enables nightly updates, you may be to ditch a bunch of the lines that follow (aus2_base_upload_dir, the update_platform's) by disabling that.

>+    'SYMBOL_SERVER_USER': 'ffxbld',
>+    'SYMBOL_SERVER_SSH_KEY': "/home/cltbld/.ssh/ffxbld_dsa",

Need to use the trybld account here, which is id_dsa on the linux slaves.

>diff --git a/mozilla2-staging/linux/try-staging b/mozilla2-staging/linux/try-staging
>new file mode 120000
>--- /dev/null
>+++ b/mozilla2-staging/linux/try-staging
>@@ -0,0 +1,1 @@
>+mozilla-central

I thought we had a bunch of try specific mozconfig options that aren't used in production builds. Did you look into that ? It would be great if that could go away too. Yay for proper try staging setup.

>diff --git a/mozilla2-staging/master3.cfg b/mozilla2-staging/master3.cfg
>+L10N_SLAVES = {

You may be able to kill all the l10n references in this file.

>+########## Mobile
>+
>+import mobile_master
>+reload(mobile_master)

Is this for the Maemo and WinME builds ? Looks like you get all the branches for Mobile with this.

	Chris AtLee [:catlee]
	Comment 3 • 15 years ago

Is turning on PGO for windows try builds going to be part of this?

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 8 • 15 years ago

Comment on attachment 417552 [details] [diff] [review]
[WIP] buildbotcustom changes for try as branch of config

I think we can do this without hardcoding "try server" stuff. See comments below:

>+    # Try Server notifier
>+    branchObjects['status'].append(MailNotifier(
>+        fromaddr="tryserver@build.mozilla.org",
>+        mode="all",
>+        sendToInterestedUsers=True,
>+        lookup=MercurialEmailLookup(),
>+        customMesg=lambda attrs: buildTryCompleteMessage(attrs,
>+          '/'.join([PACKAGE_URL, PACKAGE_DIR]), TINDERBOX_TREE),
>+        subject="Try Server: %(result)s on %(builder)s",
>+        relayhost="mail.build.mozilla.org",
>+        extraHeaders={'In-Reply-To': WithProperties('%(identifier)s')},
>+        builders=tryBuilders
>+    ))

Something like....:

if 'enable_mail_notifier' in config and config['enable_mail_notifier']:
    # add it

'builders' is built up per branch, so there's no need to build up tryBuilders on its own.

Also, note that PACKAGE_URL and PACKAGE_DIR and TINDERBOX_TREE are not defined here. They probably need to be defined as stage_server, tinderbox_tree, and maybe others in config.py though - so you might get to use them instead.

I don't think 'identifier' is going to exist anymore, either - since we aren't using TryBuildFactory.


>+    for builder in tryBuilders:
>+        try_scheduler=NoMergeScheduler(
>+            name=builder,
>+            branch=config['repo_path'],
>+            treeStableTimer=3,
>+            builderNames=[builder],
>+        )
>+        branchObjects['schedulers'].append(try_scheduler)
>+

Similar thing here, add an 'enable_merging' config flag and test for it.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 9 • 15 years ago

Comment on attachment 417551 [details] [diff] [review]
[WIP] buildbot config changes for try as branch in config.py

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>--- a/mozilla2-staging/config.py
>+++ b/mozilla2-staging/config.py
>@@ -8,16 +8,19 @@ from buildbotcustom.env import MozillaEn
> # big dict
> MAC_MINIS = ['moz2-darwin9-slave%02i' % x for x in range(1,38)]
> XSERVES   = ['bm-xserve%02i' % x for x in [7,9,11,12,16,17,18,19,21,22]]
> SLAVES = {
>     'linux': ['moz2-linux-slave%02i' % x for x in range(1,51)],
>     'linux64': ['moz2-linux64-slave%02i' % x for x in range(1,13)],
>     'win32':  ['win32-slave%02i' % x for x in range(1,60)],
>     'macosx': MAC_MINIS + XSERVES,
>+    'try-win32': ['try-win32-slave%02i' % x for x in range(1,20)],
>+    'try-linux': ['try-linux-slave%02i' % x for x in range(1,20)],
>+    'try-macosx': ['try-mac-slave%02i' % x for x in range(1,20)],
> }

Note that win32 slaves are 'try-w32-slave' not 'try-win32-slave'

>+######## tryserver
>+# This is a path, relative to HGURL, where the repository is located
>+# HGURL + repo_path should be a valid repository
>+BRANCHES['try']['repo_path'] = 'users/lsblakk_mozilla.com/try-staging'
>+BRANCHES['try']['config_repo_path'] = 'users/lsblakk_mozilla.com/buildbot-configs'
>+BRANCHES['try']['config_subdir'] = 'tryserver'
>+BRANCHES['try']['major_version'] = '1.9.2'
>+BRANCHES['try']['product_name'] = 'firefox'
>+BRANCHES['try']['app_name'] = 'browser'
>+BRANCHES['try']['start_hour'] = [3] 
>+BRANCHES['try']['start_minute'] = [2]

We need to flip off the 'nightly' builders for try - and you probably won't need those when we do.




>+BRANCHES['try']['platforms']['linux']['upload_symbols'] = True
>+BRANCHES['try']['platforms']['win32']['upload_symbols'] = True
>+BRANCHES['try']['platforms']['wince']['upload_symbols'] = False
>+BRANCHES['try']['platforms']['macosx']['upload_symbols'] = True

Dunno if uploadsymbols will work for all of these - but give it a try.


>+BRANCHES['try']['platforms']['win32']['env'] = {'CVS_RSH': 'ssh',
>+    'CVS_RSH': 'ssh',
>+    'MOZ_OBJDIR': 'obj-firefox',

Not OBJDIR?

>+    'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,

Note that SYMBOL_SERVER_PATH is different for try.

>+ACTIVE_BRANCHES = ['try']
>+L10N_SLAVES = {
>+    'linux': SLAVES['linux'],
>+    'win32': SLAVES['win32'],
>+    'macosx': MAC_MINIS + XSERVES,
>+}

I don't think we need L10N_SLAVES here....

>+
>+from buildbotcustom.scheduler import NoMergeScheduler

I don't think we need this Scheduler in here.

>+from buildbotcustom.changes.hgpoller import HgPoller
>+
>+mozillaTryConfig = BRANCHES['try']
>+
>+c['change_source'].append(HgPoller(
>+    hgURL=mozillaTryConfig['hgurl'],
>+    branch=mozillaTryConfig['repo_path'],
>+    pushlogUrlOverride='%s/%s/pushlog' % (mozillaTryConfig['hgurl'],
>+                                          mozillaTryConfig['repo_path']),
>+    pollInterval=1*60
>+))
>+

And I'm pretty sure generateBranchObjects will set this up for us. We only need it for mobile on pm02 because it's a weird case AFAIK.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 10 • 15 years ago

We will hopefully get this for free, but lets make sure to fix bug 534753 here.

	Aki Sasaki (not active)
	Comment 11 • 15 years ago

Curious: do you have winmo/maemo builders in your setup?
If not, would it be easy to keep them there, or is bug 523946 blocking?

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 12 • 15 years ago

Alice put this in graphs-staging already.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 13 • 15 years ago

I'd like to get this up in try-master running in parallel and steal at least a linux slave in order to confirm that the builds upload and the email is sent properly.  

At this juncture, I've run this in try-staging and created the patches to add in the mobile as well as changed how 'identifier' is set in buildbotcustom/status/generators.py so that it just uses the changeset instead of attaching 'try-' before the changeset like the current tryserver does. This probably needs to be tweaked but it's hard to tie it all together without the ability to upload the builds and then pass along the uploaded build's location in the email.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 16 • 15 years ago

Comment on attachment 421359 [details] [diff] [review]
[WIP] buildbotcustom changes for try as branch of config

>diff --git a/misc.py b/misc.py
>--- a/misc.py
>+++ b/misc.py
>@@ -1,37 +1,37 @@
> from urlparse import urljoin
>-try:
>-    import json
>-except:
>-    import simplejson as json
>-import collections

Don't know why you're removing this - it's used in a few places in this file.

>-def get_locales_from_json(jsonFile, l10nRepoPath, relbranch):
>-    if not l10nRepoPath.endswith('/'):
>-        l10nRepoPath = l10nRepoPath + '/'
>-
>-    l10nRepositories = {}
>-    platformLocales = collections.defaultdict(dict)
>-
>-    file = open(jsonFile)
>-    localesJson = json.load(file)
>-    for locale in localesJson.keys():
>-        revision = localesJson[locale]['revision']
>-        if revision == 'FIXME':
>-            raise Exception('Found FIXME in %s for locale "%s"' % \
>-                           (jsonFile, locale))
>-        localeUrl = urljoin(l10nRepoPath, locale)
>-        l10nRepositories[localeUrl] = {
>-            'revision': revision,
>-            'relbranchOverride': relbranch,
>-            'bumpFiles': []
>-        }
>-        for platform in localesJson[locale]['platforms']:
>-            platformLocales[platform][locale] = localesJson[locale]['platforms']
>-
>-    return (l10nRepositories, platformLocales)
> 

hmm?



>+    # Try Server notifier
>+    if config['enable_mail_notifier']:
>+        packageUrl = config['package_url']
>+        packageDir = config['package_dir']

It's a bit icky to have this here, given that's it unlikely to be used by anything but try - but I think we can live with it.

>+    if config['enable_merging']:
>+        branchObjects['schedulers'].append(MozScheduler(
>+            name=name,
>+            branch=config['repo_path'],
>+            treeStableTimer=3*60,
>+            idleTimeout=config.get('idle_timeout', None),
>+            builderNames=builders + unittestBuilders + debugBuilders,
>+            fileIsImportant=lambda c: isHgPollerTriggered(c, config['hgurl'])
>+        ))
>+    else:
>+        branchObjects['schedulers'].append(NoMergeScheduler(
>+            name=name,
>+            branch=config['repo_path'],
>+            treeStableTimer=0,
>+            builderNames=builders + unittestBuilders + debugBuilders,
>+            fileIsImportant=lambda c: isHgPollerTriggered(c, config['hgurl'])
>+        ))

This block needs a little bit of work...it claims to turn merging on and off, but it does more than that - MozScheduler is also used to trigger periodic builds. Since we're using our own Scheduler anyways, I think the better solution is to given MozScheduler a mergeBuilds option.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 17 • 15 years ago

Comment on attachment 421357 [details] [diff] [review]
[WIP] buildbot config changes for try as branch in config.py

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>+# This is a path, relative to HGURL, where the repository is located
>+# HGURL + repo_path should be a valid repository
>+BRANCHES['try']['repo_path'] = 'users/lsblakk_mozilla.com/try-staging'
>+BRANCHES['try']['config_repo_path'] = 'users/lsblakk_mozilla.com/buildbot-configs'

Probably want to point this at production versions when we land this.

>+BRANCHES['try']['config_subdir'] = 'tryserver'

I think you want mozilla2-staging here.

>+BRANCHES['try']['package_url']  = 'http://build.mozilla.org/tryserver-builds'
>+BRANCHES['try']['package_dir']  = '%(identifier)s'

Is identifier still valid now that we're not using TryBuildFactory?


>+BRANCHES['try']['platforms']['linux']['enable_unittests'] = True
>+BRANCHES['try']['platforms']['macosx']['enable_unittests'] = True
>+BRANCHES['try']['platforms']['win32']['enable_unittests'] = True

Hmmmm, this is going to get interesting. We need these builders on for 1.9.1 and 1.9.2 pushes, but not m-c or m-c based branches.

>+BRANCHES['try']['platforms']['linux-debug']['enable_unittests'] = True
>+BRANCHES['try']['platforms']['macosx-debug']['enable_unittests'] = True
>+BRANCHES['try']['platforms']['win32-debug']['enable_unittests'] = True
>+BRANCHES['try']['platforms']['linux']['enable_opt_unittests'] = True
>+BRANCHES['try']['platforms']['macosx']['enable_opt_unittests'] = True
>+BRANCHES['try']['platforms']['win32']['enable_opt_unittests'] = True

...and a lot of these are going to fail every time for 1.9.1/1.9.2 pushes.

>+BRANCHES['try']['aus2_base_upload_dir'] = '/opt/aus2/build/0/Firefox/try'

Just fake these, please. We won't be doing updates on try.

>+BRANCHES['try']['platforms']['linux']['update_platform'] = 'Linux_x86-gcc3'
>+# We're actually using gcc4, but Firefox hardcodes gcc3
>+BRANCHES['try']['platforms']['win32']['update_platform'] = 'WINNT_x86-msvc'
>+BRANCHES['try']['platforms']['wince']['update_platform'] = 'WINCE_arm-msvc'
>+BRANCHES['try']['platforms']['macosx']['update_platform'] = 'Darwin_Universal-gcc3'
>+# If True, 'make buildsymbols' and 'make uploadsymbols' will be run
>+# SYMBOL_SERVER_* variables are setup in the environment section below
>+BRANCHES['try']['platforms']['linux']['upload_symbols'] = False
>+BRANCHES['try']['platforms']['win32']['upload_symbols'] = True
>+BRANCHES['try']['platforms']['wince']['upload_symbols'] = False
>+BRANCHES['try']['platforms']['macosx']['upload_symbols'] = False
>+BRANCHES['try']['tinderbox_tree'] = 'MozillaTest'
>+BRANCHES['try']['platforms']['linux']['slaves'] = SLAVES['try-linux']
>+BRANCHES['try']['platforms']['win32']['slaves'] = SLAVES['try-win32']
>+BRANCHES['try']['platforms']['wince']['slaves'] = SLAVES['try-win32']
>+BRANCHES['try']['platforms']['macosx']['slaves'] = SLAVES['try-macosx']
>+BRANCHES['try']['platforms']['linux-debug']['slaves'] = SLAVES['try-linux']
>+BRANCHES['try']['platforms']['win32-debug']['slaves'] = SLAVES['try-win32']
>+BRANCHES['try']['platforms']['macosx-debug']['slaves'] = SLAVES['try-macosx']
>+# This is used in a bunch of places where something needs to be run from
>+# the objdir. This is necessary because of universal builds on Mac
>+# creating subdirectories inside of the objdir.
>+BRANCHES['try']['platforms']['linux']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['win32']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['wince']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['macosx']['platform_objdir'] = '%s/ppc' % OBJDIR
>+BRANCHES['try']['platforms']['linux-debug']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['macosx-debug']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['win32-debug']['platform_objdir'] = OBJDIR
>+BRANCHES['try']['platforms']['linux']['env'] = {
>+    'MOZ_OBJDIR': OBJDIR,
>+    'TINDERBOX_OUTPUT': '1',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+}
>+BRANCHES['try']['platforms']['win32']['env'] = {'CVS_RSH': 'ssh',
>+    'CVS_RSH': 'ssh',
>+    'MOZ_OBJDIR': OBJDIR,
>+    'TINDERBOX_OUTPUT': '1',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+    'SYMBOL_SERVER_HOST': 'build.mozilla.org',
>+    'SYMBOL_SERVER_USER': 'trybld',
>+    'SYMBOL_SERVER_PATH': '/symbols/windows',
>+    'SYMBOL_SERVER_SSH_KEY': '$ENV{HOME}/.ssh/id_dsa',
>+    'NO_FAIL_ON_TEST_ERRORS': '1' 
>+}
>+BRANCHES['try']['platforms']['wince']['env'] = MozillaEnvironments['winmo-arm'].copy()
>+BRANCHES['try']['platforms']['wince']['env'].update(
>+    BRANCHES['try']['platforms']['win32']['env'])
>+BRANCHES['try']['platforms']['macosx']['env'] = {
>+    'MOZ_OBJDIR': OBJDIR,
>+    'TINDERBOX_OUTPUT': '1',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+    'CHOWN_ROOT': '~/bin/chown_root',
>+    'CHOWN_REVERT': '~/bin/chown_revert',
>+}
>+BRANCHES['try']['platforms']['linux-debug']['env'] = {
>+    'MOZ_OBJDIR': OBJDIR,
>+    'DISPLAY': ':2',
>+    'LD_LIBRARY_PATH': '%s/dist/bin' % OBJDIR,
>+    'XPCOM_DEBUG_BREAK': 'stack-and-abort',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+}
>+BRANCHES['try']['platforms']['macosx-debug']['env'] = {
>+    'MOZ_OBJDIR': OBJDIR,
>+    'XPCOM_DEBUG_BREAK': 'stack-and-abort',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+}
>+BRANCHES['try']['platforms']['win32-debug']['env'] = {
>+    'MOZ_OBJDIR': OBJDIR,
>+    'XPCOM_DEBUG_BREAK': 'stack-and-abort',
>+    'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+}
>diff --git a/mozilla2-staging/linux/mozilla-central/build b/mozilla2-staging/linux/mozilla-central/build
>new file mode 120000
>--- /dev/null
>+++ b/mozilla2-staging/linux/mozilla-central/build
>@@ -0,0 +1,1 @@
>+nightly
>\ No newline at end of file
>diff --git a/mozilla2-staging/linux/try-staging b/mozilla2-staging/linux/try-staging

Just use 'try' here - staging is implied inside of 'mozilla2-staging' and it also matches the same of the branch, like the other dirs.. Same thing for production.



>+ACTIVE_BRANCHES = ['try']
>+# We don't need l10n slaves, but until it's not called in master-main.cfg we need them here
>+L10N_SLAVES = {
>+    'linux': SLAVES['linux'],
>+    'win32': SLAVES['win32'],
>+    'macosx': MAC_MINIS + XSERVES,
>+}

Despite not being used, i think it's better to put in the try slaves here.


>+c['slavePortnum'] = 9011

Why not 9010?

>+
>+from buildbot.status.html import WebStatus
>+c['status'].append(WebStatus(
>+    http_port=8011,

same thing here


What's the plan for Mobile builds here?

Good progress here, still some things to address there.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 18 • 15 years ago

this is running in sm02 at the moment and I am waiting for some emails on completed builds.  i got one build to go through all the way sucessfully but needed to see the packageUrl and then go back in to factor that into the email notifier and status/generator.py

I should know in the morning if the packageUrl is emailed to me properly, and see if the packaged unittests go off too.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 19 • 15 years ago

So now I've got the MozScheduler firing off a NoMergeScheduler if enable_merging is set to False.  I pushed 3 changes and that triggered 3 separate sendchanges but I'm not 100% sure I'm doing this right.  Again, it's running in staging-master02 right now and I'll know more in the morning.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 20 • 15 years ago

Comment on attachment 422888 [details] [diff] [review]
[WIP] buildbotcustom changes for try as branch of config

>diff --git a/misc.py b/misc.py

>     def fireTimer(self):
>-        Scheduler.fireTimer(self)
>-        self.setIdleTimer()
>+        if mergeBuilds:
>+            Scheduler.fireTimer(self)
>+            self.setIdleTimer()
>+        else:
>+            NoMergeScheduler.fireTimer(self)
> 

This isn't how you want to this. Rather than calling out, just pull in the code from NoMergeScheduler.fireTimer(). Also, you want to reset the timer regardless of merging.

So, it ends up looking like this:
def fireTimer(self):
    if mergeBuilds:
        Scheduler.fireTimer(self)
    else:
        for change in self.importantChanges:
        ...
    self.setIdleTimer()

Note that the timer stuff from NoMergeScheduler has been dropped.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 21 • 15 years ago

This patch adds support for the tryserver-builds dirs in post_upload.py. TryBuildFactory needs to be adjusted to use 'make upload', and have a POST_UPLOAD_CMD something like:
post_upload.py --release-to-tryserver-builds --revision %(got_revision)s --who %(who)s

revision and who probably need to come in through WithProperties().

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 22 • 15 years ago

work in progress for upload command & using the TryBuildFactory

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 23 • 15 years ago

updated for bitrot with lorentz branch being added, also adding 'enable_try' for factory test in misc.py

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 24 • 15 years ago

i notice that there are no mobile builds as a part of this currently, is there any plan for having mobile builds as a part of try-as-branch?

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 25 • 15 years ago

(In reply to comment #24)
> i notice that there are no mobile builds as a part of this currently, is there
> any plan for having mobile builds as a part of try-as-branch?
Yes, mobile builds should be on try-server-as-a-branch. After all, the idea is to have try-as-branch be on par with everything on mozilla-central at least. 

Also, by being similar to other project branches, going forward, it should be easier to add new buildOS/unittests/talos on try-master-as-a-branch at same time as we add to mozilla-central.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 32 • 15 years ago

updated due to bug 508674, in testing right now

	Chris AtLee [:catlee]
	Comment 33 • 15 years ago

Comment on attachment 425072 [details] [diff] [review]
[tested] try support in post_upload.py

This looks pretty good.  We've got a good set of staging / production settings now though.  Could you reorder them so that all the staging configs are together in one block and commented out?

Something like,

# Staging configs start here.  Uncomment when working on staging
# TRYSERVER_DIR = "..."
# TRYSERVER_URL_PATH = "..."
# CANDIDATES_URL_PATH = "..."
etc.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 34 • 15 years ago

Re-organized the staging/production blocks

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 35 • 15 years ago

Status update: Running some final tests in staging - trying it with the actual try repo instead of my own copy, double checking the no-merge and adding linux 64, bringing it all up to date with tip. Should have patches up for review (for staging) first thing in the morning and then can put together the production patches to get this running in parallel with try by week's end.

Still need to do patches for Talos integration with this too.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 38 • 15 years ago

I'm leaving this running overnight in staging (emailing to me only) responding to the real push-to-try, my only concern at this point is 

http://staging-master.build.mozilla.org:8011/builders/Linux%20tryserver%20build/builds/24

shouldn't have any merges, so I'm not sure why this happened.  Will look into it in the morning.

	Chris AtLee [:catlee]
	Comment 40 • 15 years ago

assuming that once this is done we'll either have the JS tests for free, or that they'll be trivial to enable.  adding dependency here just for tracking.

	Chris AtLee [:catlee]
	Comment 41 • 15 years ago

Comment on attachment 430215 [details] [diff] [review]
[tested] buildbotcustom changes for try as branch of config

>+            
>+            extra_args = {}
>+            if config.get('enable_try'):
>+                factory_class = TryUnittestBuildFactory
>+                extra_args['tinderboxBuildsDir'] = tinderboxBuildsDir
>+            else:
>+               factory_class = UnittestBuildFactory

Indentation is a bit funny here

>+class TryBuildFactory(MercurialBuildFactory):
[snip]
>+        #uploadDir = '/'.join([self.stageBasePath, self.packageDir])
>+        uploadEnv = self.env.copy()
>+        uploadEnv.update({
>+            'UPLOAD_HOST': self.stageServer,
>+            'UPLOAD_USER': self.stageUsername,
>+            'UPLOAD_TO_TEMP': '1',
>+            #'UPLOAD_PATH': WithProperties(uploadDir)

Please remove the commented lines if you don't need them any more

>+class TryUnittestBuildFactory(MozillaBuildFactory):
[snip]

TryUnittestBuildFactory really should inherit from UnittestBuildFactory, otherwise we risk the two getting out of sync again.  This probably means refactoring UnittestBuildFactory a bit so that you can do the try-specific stuff(just uploading by the looks of it) in the sub class.  Let me know if you need a hand with this piece.

Other than that, looks good!

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 42 • 15 years ago

I will post sql insert (and remove for the existing builders in staging-grapheserver)

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 44 • 15 years ago

I tested this on staging to make sure that nothing is broken with the current staging configs.  This is all the stuff that will not affect try as it is right now, the next patch will affect factory.py and generators.py in ways that would break the current try set up.  This patch could go in now, to staging, to help with avoiding bitrot as I clean up other issues like MailNotifier handling %(got_revision)s and looking into potential merge issues.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 45 • 15 years ago

This is the part two of the bigger patch - I can carry it around with me as I continue getting everything working on staging and then we can check it in when it's time to flip the switch on try-as-branch.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 46 • 15 years ago

Comment on attachment 426011 [details] [diff] [review]
[tested] try support in post_upload.py

http://hg.mozilla.org/build/tools/rev/09acf9a0e0d7

I updated staging-stage as well.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 47 • 15 years ago

(In reply to comment #46)
> (From update of attachment 426011 [details] [diff] [review])
> http://hg.mozilla.org/build/tools/rev/09acf9a0e0d7
> 
> I updated staging-stage as well.

Nthomas just updated stage.m.o with the new post_upload.py

	Chris AtLee [:catlee]
	Comment 48 • 15 years ago

Comment on attachment 430460 [details] [diff] [review]
[tested] Buildbotcustom changes that do not affect current try or staging

A few style nits, but other than that looks good to me.

>     # schedulers
>     # this one gets triggered by the HG Poller
>+    mergeBuilds=True
>+    if config.get('enable_merging'):
>+        mergeBuilds=config['enable_merging']

This could be written as
mergeBuilds = config.get('enable_merging', True)

>diff --git a/scheduler.py b/scheduler.py
>--- a/scheduler.py
>+++ b/scheduler.py
>@@ -20,21 +20,22 @@ class MozScheduler(Scheduler):
>     attention to a single branch. You you can provide a C{fileIsImportant}
>     function which will evaluate each Change to decide whether or not it
>     should trigger a new build. If no builds have occurred after
>     C{idleTimeout} seconds, then a build will be started. Set
>     C{idleTimeout} to None to disable this.
>     """
> 
>     fileIsImportant = None
>+    mergeBuilds=True

put spaces around '=' please

>     compare_attrs = ('name', 'treeStableTimer', 'builderNames', 'branch',
>                      'fileIsImportant', 'properties', 'idleTimeout')
>     
>     def __init__(self, name, branch, treeStableTimer, builderNames,
>-                 idleTimeout=None, fileIsImportant=None, properties={}):
>+                 idleTimeout=None, fileIsImportant=None, mergeBuilds=True, properties={}):

this line is too long now, can you bump something onto the next line?

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 49 • 15 years ago

Comment on attachment 430460 [details] [diff] [review]
[tested] Buildbotcustom changes that do not affect current try or staging

http://hg.mozilla.org/build/buildbotcustom/rev/d0f3663c255b

(with nits addressed)

	Aki Sasaki (not active)
	Comment 50 • 15 years ago

Comment on attachment 430460 [details] [diff] [review]
[tested] Buildbotcustom changes that do not affect current try or staging

Looks like this is causing:

Traceback (most recent call last):
  File "/tools/buildbot-clean/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/scripts/runner.py", line 924, in doCheckConfig
    ConfigLoader(configFile)
  File "/tools/buildbot-clean/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile)
  File "/tools/buildbot-clean/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/master.py", line 529, in loadConfig
    exec f in localDict
  File "master.cfg", line 28, in <module>
    branchObjects = generateBranchObjects(BRANCHES[branch], branch)
  File "/tools/buildbotcustom/buildbotcustom/misc.py", line 512, in generateBranchObjects
    mergeBuilds=mergeBuilds,
UnboundLocalError: local variable 'mergeBuilds' referenced before assignment
make: *** [checkconfig] Error 1

working around this for now.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 51 • 15 years ago

This replaces the got_revision property with "unknown" if it's not set yet so that emails can still be sent if a build fails before clone is complete.

	Chris AtLee [:catlee]
	Comment 52 • 15 years ago

Comment on attachment 431772 [details] [diff] [review]
Fix for MailNotifier balking at %(got_revision)s in extraHeaders

Very cool, I had forgotten you could do that!

	Chris AtLee [:catlee]
	Comment 53 • 15 years ago

Comment on attachment 430462 [details] [diff] [review]
[tested] Buildbotcustom changes that affect current try or staging

>-    if 'unit test' in builder:
>-        task = 'unit test'
>+    if 'test' in builder:
>+        task = 'test'
>     elif 'talos' in builder:
>         task = 'talos'
>     else:
>         task = 'build'

What do the debug builds show up as here?  I think they have "test" in their name, so would they show up as a "test"?

>     url = packageDir % locals()
> 
>     if attrs['result'] == 'success':
>         text = """\
>-Your Try Server %(task)s (%(identifier)s) was successfully completed on \
>+Your Try Server %(task)s (%(got_revision)s) was successfully completed on \
> %(platform)s.  """ % locals()
>     elif attrs['result'] == 'warnings':
>         text = """\
>-Your Try Server %(task)s (%(identifier)s) completed with warnings on \
>+Your Try Server %(task)s (%(got_revision)s) completed with warnings on \
> %(platform)s.  """ % locals()
>     else:
>         text = """\
>-Your Try Server %(task)s (%(identifier)s) failed to complete on \
>+Your Try Server %(task)s (%(got_revision)s) failed to complete on \
> %(platform)s.\n\n""" % locals()

I think the builder name should be included somewhere in the message as well.

> 
>     if attrs['result'] in ('success', 'warnings') and packageDir:
>         text += "It should be available for download at %(url)s\n\n" % locals()
> 
>-    if task == 'unit test':
>-        text += "Summary of unittest results:\n"
>+    if attrs['result'] in ('failure', 'warnings') and 'unknown' not in who:
>+        text += "While some of the build steps haven't succeeded, your build package might have been \
>+        successfully created and could be available for download at %(url)s\n\n" % locals()
>+
>+    if task == 'test':
>+        text += "Summary of test results:\n"
>         for log in attrs['logs']:
>             if 'summary' not in log[0]:
>                 continue
>             summary = ''.join(log[2]).replace('TinderboxPrint: ', '')
>             summary = summary.replace('<br/>', ': ')
>             text += '%s\n' % summary
>         text += '\n'
>

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 54 • 15 years ago

Comment on attachment 431772 [details] [diff] [review]
Fix for MailNotifier balking at %(got_revision)s in extraHeaders

http://hg.mozilla.org/build/buildbotcustom/rev/1a8968c1ce08

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 55 • 15 years ago

If the config sets enable_merging to false then the schedulers are created with NoMergeScheduler (as needed by try) instead of the current MozScheduler.

Tested in staging with both true/false to make sure proper scheduler is created.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 56 • 15 years ago

updated to reflect current configs.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 57 • 15 years ago

Comment on attachment 432209 [details] [diff] [review]
Fixes merging issue for try builds - now using NoMergeScheduler in misc.py

http://hg.mozilla.org/build/buildbotcustom/rev/5e5c12c1e83a

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 58 • 15 years ago

now with permanent port 9012/8012 for living on staging-master.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 59 • 15 years ago

Comment on attachment 432234 [details] [diff] [review]
[tested] buildbot staging config changes for try as branch

http://hg.mozilla.org/build/buildbot-configs/rev/e3ef7bcb39e8

	Chris AtLee [:catlee]
	Comment 60 • 15 years ago

Comment on attachment 430462 [details] [diff] [review]
[tested] Buildbotcustom changes that affect current try or staging

Still a valid patch?  Re-r? if so.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 62 • 15 years ago

Should have done this in the original patch, it's currently changed locally on try-staging.

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 63 • 15 years ago

What left to do here?

	Nick Thomas [:nthomas] (UTC+12)
	Comment 64 • 15 years ago

I added a cron job to staging-stage to clean up the builds coming off sm04 (new try staging). They'll have a 4 day lifetime to match ones coming off sm01 thru 03. Hopefully everyone is asking themselves questions like "If I'm sending lots more data to machine <x> is it ready for that ? And what do I need to do to prevent it running out space ?".

In a similar vein, it looks like a successful try-as-a-branch run will use about 1400MB of space, eg sdwilsh@shawnwilsher.com-3c689de8ee14:
 157	tryserver-linux
 174	tryserver-linux-debug
 157	tryserver-linux-unittest
 419	tryserver-macosx
 126	tryserver-macosx-debug
 137	tryserver-macosx-unittest
  82	tryserver-win32
 102	tryserver-win32-debug
  67	tryserver-win32-unittest
1417	total
a lot of which is the symbols we want for crashes on unit/talos. That's a lot more than the 100MB/push we generate now, so we'll need to think ahead about space on build.m.o before try-as-a-branch goes into production.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 65 • 15 years ago

Status update:

This is running in staging now, as a permanent staging-master.  What's left to test out before rolling this into production (without a web interface) is:

* bug 555988 to get the Linux/Mac slaves in line with the other builders for accurate build and test results
* Fixing the path for logs so that codesize.log steps stop failing
* Testing bug 541364 for custom mozconfig
* Enabling mobile-try for this new setup

The merging issues have been fixed and will be in the updated patch coming soon.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 66 • 15 years ago

this is what's running in try-staging right now.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 67 • 15 years ago

this one includes 10.6.2

	Armen [:armenzg]
	Comment 70 • 15 years ago

I am also including a naming fix for 10.6 debug.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 71 • 15 years ago

Comment on attachment 443186 [details] [diff] [review]
small fix


>         'macosx64-debug': {
>-            'base_name': 'OS X 10.6 %(branch)s leak test',
>+            'base_name': 'OS X 10.6.2 %(branch)s leak test',
>             'mozconfig': 'macosx64/%(branch)s/debug',

This will be in my up to date config patch which is being tested in staging right now, I'd like the nits like this to all be in one patch (i had to change it for regular macosx64 too)

>     'mozilla-1.9.1': { 'platforms': { 'linux': {}, 'linux-debug': {}, 
>-                                      'linux64': {}, 'linux64-debug': {}, 
>                                       'macosx': {}, 'macosx-debug': {}, 
>                                       'win32': {}, 'win32-debug': {}, 
>                                     },
>                      },
>     'mozilla-1.9.2': { 'platforms': { 'linux': {}, 'linux-debug': {}, 
>                                       'linux64': {}, 'linux64-debug': {}, 
>                                       'macosx': {}, 'macosx-debug': {}, 
>+                                      'macosx64': {}, 'macosx64-debug': {}, 
>                                       'win32': {}, 'win32-debug': {}, 
>                                     },

This looks like what you're really trying to do - so maybe this is the patch that can go in a separate bug?

> BRANCHES['tryserver']['repo_path'] = 'try'
>-BRANCHES['tryserver']['stage_base_path'] = '/home/ftp/pub/tryserver-builds'
>+BRANCHES['tryserver']['stage_base_path'] = '/home/ftp/pub/mozilla.org/tryserver-builds'
> BRANCHES['tryserver']['start_hour'] = [3]

Same as above - this fix will be in my large, up to date config patch - which is being finalized in staging right now.  It will make more sense as part of that patch than as a little fix.

	Armen [:armenzg]
	Comment 72 • 15 years ago

Comment on attachment 443186 [details] [diff] [review]
small fix

OK you have it all covered.

WRT to:
-            'base_name': 'OS X 10.6 %(branch)s leak test',
+            'base_name': 'OS X 10.6.2 %(branch)s leak test',
I gave you the wrong patch. That should go in another bug.

Thanks for looking at it.

	alice nodelman [:alice] [:anode]
	Comment 73 • 15 years ago

Comment on attachment 442223 [details] [diff] [review]
add tryserver to graph server data.sql

Don't we want to get rid of machines called '10.6.2' and have them be just '10.6'?

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 74 • 15 years ago

(In reply to comment #72)
> (From update of attachment 443186 [details] [diff] [review])
> OK you have it all covered.
> 
> WRT to:
> -            'base_name': 'OS X 10.6 %(branch)s leak test',
> +            'base_name': 'OS X 10.6.2 %(branch)s leak test',
> I gave you the wrong patch. That should go in another bug.
> 
> Thanks for looking at it.

Armen, please don't go putting that in another bug/patch - it will bitrot my tryserver config patch and since you aren't testing tryserver stuff, I'm certain you do not need this change.

Alice - we don't need to call it 10.6.2, but the graphserver entries are stated that way.  It's more of a PITA to change that.  If there's a strong sense that it's worth moving it to 10.6 I'll put together SQL fixes for that.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 75 • 15 years ago

now with handling for setting email from hg author and changeset cropped to 12 chars because the long hash was too crazy for directory name consistency.  basically it's more buildbot 0.8.0-proof and allows you to force build off the buildbot waterfall now.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 76 • 15 years ago

now with added uploadSymbols set so we can get symbol server for try.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 77 • 15 years ago

updated with handling for uploadSymbols so we can set that in the try configs

	Chris AtLee [:catlee]
	Comment 78 • 15 years ago

Comment on attachment 443677 [details] [diff] [review]
[tested] up to date buildbotcustom patch that affects current production try

Looks good, just one nit:

> 
>+    def addSourceSteps(self):
>+        self.addStep(Mercurial,
>+         name='hg_update',
>+         mode='clobber',
>+         baseURL='http://%s/' % self.hgHost,
>+         defaultBranch=self.repoPath,
>+         timeout=60*60, # 1 hour
>+        )
>+        self.addStep(SetProperty,
>+         name = 'set_got_revision',
>+         command=['hg', 'parent', '--template={node}'],
>+         extract_fn = short_hash
>+        )
>+        if self.buildRevision:
>+            self.addStep(ShellCommand,
>+             name='hg_update',
>+             command=['hg', 'up', '-C', '-r', self.buildRevision],
>+             haltOnFailure=True
>+            )
>+            self.addStep(SetProperty,
>+             name='set_got_revision',
>+             command=['hg', 'parent', '--template={node}'],
>+             extract_fn = short_hash
>+            )

You just need one 'set_got_revision' step here.  Delete the one inside the 'if self.buildRevision' block, and move the other one down.  That way it's always called after we're done cloning and updating to whatever revision.

r+ with that change

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 79 • 15 years ago

Comment on attachment 443677 [details] [diff] [review]
[tested] up to date buildbotcustom patch that affects current production try

http://hg.mozilla.org/build/buildbotcustom/rev/da6f3180a3de

Took out the second set_got_revision as per review nit.

	Aki Sasaki (not active)
	Comment 81 • 15 years ago

Comment on attachment 444761 [details] [diff] [review]
moving try-slaves to build network in production-puppet list

assuming these are logically laid out, this looks good.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 82 • 15 years ago

Comment on attachment 444761 [details] [diff] [review]
moving try-slaves to build network in production-puppet list

http://hg.mozilla.org/build/puppet-manifests/rev/8484e5b2a09d

	Nick Thomas [:nthomas] (UTC+12)
	Comment 84 • 15 years ago

Comment on attachment 445215 [details] [diff] [review]
fix for post_upload.py tryserver production path/dir

Don't think this is going to work.

>diff --git a/stage/post_upload.py b/stage/post_upload.py
>+TRYSERVER_DIR = "/tryserver-builds/%(who)s-%(revision)s/%(builddir)s"

This is used here:

def ReleaseToTryserverBuilds(options, upload_dir, files):
    tryserverBuildsPath = TRYSERVER_DIR % {'who': options.who, ...
    ...
    for f in files:
        CopyFileToDir(f, upload_dir, tryserverBuildsPath)

and tryserverBuildsPath is not an absolute path.

>+TRYSERVER_URL_PATH = "http://stage.mozilla.org/pub/mozilla.org/tryserver-builds/%(who)s-%(revision)s/%(builddir)s"

stage is ok for unit/talos sendchanges, but lets report ftp.m.o in emails to the person who pushed.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 85 • 15 years ago

Comment on attachment 445215 [details] [diff] [review]
fix for post_upload.py tryserver production path/dir

>diff --git a/stage/post_upload.py b/stage/post_upload.py
>+TRYSERVER_URL_PATH = "http://stage.mozilla.org/pub/mozilla.org/tryserver-builds/%(who)s-%(revision)s/%(builddir)s"

Did you mean to change this from /pub/mozilla.org/firefox/tryserver-builds ? That's fine by me, but let me know if you want the directory moved on stage.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 86 • 15 years ago

(In reply to comment #84)
> and tryserverBuildsPath is not an absolute path.

Sorry, it is an absolute path but it's bogus.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 87 • 15 years ago

yes, we want /pub/mozilla.org/firefox/tryserver-builds and also to direct the users to ftp.m.o for downloads.

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 88 • 15 years ago

corrected the paths and added replacement for %(product)s

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 89 • 15 years ago

Comment on attachment 445227 [details] [diff] [review]
fix for post_upload.py tryserver production path/dir

http://hg.mozilla.org/build/tools/rev/1656868d765c

	Lukas Blakk [:lsblakk] use ?needinfo Assignee
	Comment 90 • 15 years ago

Until we figure out how to use graphserver (if at all) for try.