Please see the above URL. If we can build nightly builds with Shark enabled (plus symbols) it will allow a wide variety of Mozilla + web developers to very easily profile Firefox performance by just downloading a build and running shark. I'd like to get this done asap.

--enable-dtrace as well.

Reassigning to Build&Release

What are the runtime effects of a shark-enabled build? Does that mean the build won't run on 10.4 or computers where a new-enough Shark isn't present? Same thing for dtrace? Is this a request that the normal nightly builds be produced this way, or a special separate nightly build?

(In reply to comment #3) > What are the runtime effects of a shark-enabled build? None, if you don't use Shark, I think. > Does that mean the build > won't run on 10.4 or computers where a new-enough Shark isn't present? Same > thing for dtrace? Regarding Shark, I'm not sure. It may just be that the headers require a new copy of CHUD, since they move around. Dtrace would require 10.5. > Is this a request that the normal nightly builds be produced > this way, or a special separate nightly build? > AIUI, it would be a special nightly, since it would need symbols.

I'm thinking about making a builder on staging-1.9-master to get this working (and copy it to production-1.9-master once we do). We can use a custom slave if necessary but would be nice if we could use fx-1.9-mac-slave* IMHO, and just provide a shark-specific mozconfig in the builder settings. I think we should just have the builder do the build directly and not integrate shark into Tinderbox. What do you guys think?

Any idea if this will work with our current Mac (Intel, OS X 10.5) reference platform?: http://wiki.mozilla.org/ReferencePlatforms/Mac

It will need a new CHUD framework installed for CHUD.h. http://developer.apple.com/tools/download/

(In reply to comment #7) > It will need a new CHUD framework installed for CHUD.h. > > http://developer.apple.com/tools/download/ Thanks. I think we should just add this to the standard ref platform, if there are no objections (assuming we get it working in the staging environment first of course!).

(In reply to comment #4) > AIUI, it would be a special nightly, since it would need symbols. What kind of symbols does it need? Full stabs symbols make for a ~500Mb libxul on Mac. I think DWARF is smaller, but probably still big. Also, our packaging system strips binaries by default, but I think there's a way around that.

(In reply to comment #5) > I think we should just have the builder do the build directly and not integrate > shark into Tinderbox. What do you guys think? Agreed.

> I think we should just have the builder do the build directly and not integrate > shark into Tinderbox. What do you guys think? > I favour the Buildbot approach, too.

(In reply to comment #11) > > I think we should just have the builder do the build directly and not integrate > > shark into Tinderbox. What do you guys think? > > > I favour the Buildbot approach, too. +1. We also discussed this in this morning's Build&Release meeting. This same approach should be good for the mobile & moz2 builds too. I'd like to see if we can use this same logic afterwards for some of the automation on trunk, and maybe 1.8? Even if we do this "direct to buildbot" for only some steps on trunk, 1.8, I think it would be worth doing.

ben, what do you think of this approach? This particular patch is totally untested, but based on my dev buildbot. ted/rsayre, did you want to run "buildsymbols" and upload these somewhere, or should we leave the binaries unstripped? Not clear to me from comments so far.

(In reply to comment #13) > > ted/rsayre, did you want to run "buildsymbols" and upload these somewhere, or > should we leave the binaries unstripped? Not clear to me from comments so far. The binaries should be unstripped.

ted tells me on irc that we do want a packaged, unstripped build, and bsmedberg tells us the way to win: make package PKG_SKIP_STRIP=1 Things I know we need still: 1) good way to upload the build to stage (could use your advice on this one bhearsum, should we use a custom class like the tryserver?) Also curious if you think it'd be better to use the custom classes for everything or just do it more directly like this. 2) latest CHUD tool in ref platform and deployed 3) good way to auto-update mozconfig (maybe check out mozilla/tools/tinderbox-configs on master as a step, like we do for bootstrap-configs?) Might as well do this on the slave instead of downloading from the master then, imho.

Comment on attachment 298611 [details] [diff] [review] nightly shark runs on staging take 2 This looks fine. > 1) good way to upload the build to stage (could use your advice on this one > bhearsum, should we use a custom class like the tryserver?) Also curious if you > think it'd be better to use the custom classes for everything or just do it > more directly like this. The package will always be in the same location with the same file name, right? If that's the case, then I recommend using a ShellCommand. The try server does a bunch of wacky renaming and grabs information from build properties, so a custom step seemed easier at the time. I don't think it's necessary in this case and would probably be unwieldy. > 3) good way to auto-update mozconfig (maybe check out > mozilla/tools/tinderbox-configs on master as a step, like we do for > bootstrap-configs?) Might as well do this on the slave instead of downloading > from the master then, imho. Sounds like a fine idea to me. If the slave's builddir is getting clobbered on every run you'll have to checkout+rename the file rather than keeping a standard 'tinderbox-configs' directory around, though.

Ok, this checks out the mozconfig on the slave on each run (and each run is a clobber). I added a really dumb upload command; the thing I don't like about it is that it would just overwrite the same binary every night, instead of creating a date-stamped and latest dir like tinderbox does (see e.g. http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/textframe/). I'm going to go ahead and start testing this version on staging, but maybe we should just put together a nice upload class that does the datestamp-dir/latest-dir thing? We'll need it elsewhere (mobile and moz2 f.e.). Also I think we really should be getting the date from the scheduler, and passing that along to both client.mk and this hypothetical upload class, what do you thing?

Comment on attachment 298814 [details] [diff] [review] simple upload, tinderbox-configs checkout Found a couple issues testing; I'll post a working/tested patch shortly.

The "actually works" edition. Produces a ~200MB DMG file, so maybe we should just clobber the file in-place every night - I think we want a fancy upload class for other things, but let's not block on it for this bug. Note I had to do some funky things like create the root of the objdir, and checkout 'mozilla/build/macosx/universal/mozconfig' because the checked-in mozconfig includes it. Also, commented out the "-r shark" because that branch does not exist in CVS. "make package PKG_SKIP_STRIP=1" is run in {objdir}/universal/i386. I don't know if this gives a universal binary or just intel (or which we want for shark builds), maybe someone can enlighten me?

Comment on attachment 298873 [details] [diff] [review] [checked in] nightly shark runs on staging take 4 This looks fine. Do you really want it built at 10am, though? I would've assumed it'd be done earlier in the morning.

(In reply to comment #20) > (From update of attachment 298873 [details] [diff] [review]) > This looks fine. Do you really want it built at 10am, though? I would've > assumed it'd be done earlier in the morning. Since this is for staging, trying to get it out of the way of the automation :) Left to do here: * branch mozilla/tools/tinderbox-configs/firefox/macosx/mozconfig ("shark") * get latest CHUD tools into ref platform * switch this on in production

Comment on attachment 298873 [details] [diff] [review] [checked in] nightly shark runs on staging take 4 Checking in master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v <-- master.cfg new revision: 1.20; previous revision: 1.19 done

Comment on attachment 298974 [details] [diff] [review] shark mozconfig should have --enable-dtrace as well.

Comment on attachment 299868 [details] [diff] [review] [checked in] add --enable-dtrace, for shark branch /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/mozconfig,v <-- mozconfig new revision: 1.16.2.1; previous revision: 1.16 done

Just talked to sayrer and bhearsum. Since we're tightening down for the 3.0b3 release, we're going to upgrade the build machines and turn this on immediately following the release.

Since we don't have any Leopard build machines, had to turn off --enable-dtrace. The staging box (with the latest CHUD tools) then produced this: http://people.mozilla.org/~rhelmer/tmp/shark/firefox-3.0b4pre.en-US.mac.dmg

* put slavelocks around all steps * add dedicated shark_scheduler * make sure that destination dir exists before uploading * merge production config little cleanup to make these configs more alike and easier to merge: * remove old, commented old "clobber" scheduler from staging

This seems to work fine on staging. I've added CHUD to the ref platform docs and repository: http://wiki.mozilla.org/ReferencePlatforms/Mac I think all that is left is: * get buildbot configs reviewed/landed * install CHUD tools on production boxes * ??? * profit

Comment on attachment 302963 [details] [diff] [review] [checked in] working staging config, merge to production >Index: production-1.9/master.cfg >+ftpserver = "fx-linux-1.9-slave1" Is this supposed to be fx-linux-1.9-slave2? (Or maybe stage.mozilla.org, since this is the production config?) Other than that, things look fine.

(In reply to comment #31) > (From update of attachment 302963 [details] [diff] [review]) > >Index: production-1.9/master.cfg > >+ftpserver = "fx-linux-1.9-slave1" > > Is this supposed to be fx-linux-1.9-slave2? (Or maybe stage.mozilla.org, since > this is the production config?) > > > Other than that, things look fine. Thanks, should be stage :)

Comment on attachment 302963 [details] [diff] [review] [checked in] working staging config, merge to production r+ with that change.

Comment on attachment 302963 [details] [diff] [review] [checked in] working staging config, merge to production Landed with ftpserver set to "stage.mozilla.org" for production: Checking in production-1.9/master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v <-- master.cfg new revision: 1.11; previous revision: 1.10 done Checking in staging-1.9/master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v <-- master.cfg new revision: 1.24; previous revision: 1.23 done

Ok, I just forced the first production shark build: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/shark/ Every night at 1 am, a clobber build will be triggered and it will upload to this location (replacing the previous file it it's the same Firefox version). Can someone take a look at this build and let me know if it's ok?

Comment on attachment 303187 [details] [diff] [review] [checked in] send log to MozillaExperimental, set binaryURL Looks fine, I'd forgotten that the binaryURL option existed though ;).

Comment on attachment 303187 [details] [diff] [review] [checked in] send log to MozillaExperimental, set binaryURL Thanks, I'll force a build now to make sure this works ok. Checking in production-1.9/master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v <-- master.cfg new revision: 1.12; previous revision: 1.11 done

Ok, you can see the shark builds here (macosx_shark_build): http://tinderbox.mozilla.org/MozillaExperimental/

The builds in shark-10.5 haven't been updated since Mar 5 -- is that expected? My impression is that those are the better ones to use, since we only really tested this stuff out on Leopard, but I could be wrong. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/shark-10.5/

(In reply to comment #40) > The builds in shark-10.5 haven't been updated since Mar 5 -- is that expected? > My impression is that those are the better ones to use, since we only really > tested this stuff out on Leopard, but I could be wrong. > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/shark-10.5/ > Per comment 28, there's only 10.4 happening automatically right now, as we didn't have any 10.5 machines at the time. I'm pretty sure the 10.5 was a one-off test done by Nick (is that right Nick?)

(In reply to comment #41) > Per comment 28, there's only 10.4 happening automatically right now, as we > didn't have any 10.5 machines at the time. I'm pretty sure the 10.5 was a > one-off test done by Nick (is that right Nick?) That's right. I was working up an image for a 10.5 tinderbox and wanted to check there were no problems with doing a Shark build. We subsequently decided not to switch to 10.5 for building Firefox 3.