Busted from the merges. Work around for bug 1195546.

I've tried to limit the scope of this right down, so that it doesn't break anything else. It should fix Firefox & T'bird ci builds, as well as releases.

Comment on attachment 8681016 [details] [diff] [review] [buildbotcustom] Set MOZ_CURRENT_PROJECT for make package in opt mac builds https://hg.mozilla.org/build/buildbotcustom/rev/09ad19193503 https://hg.mozilla.org/build/buildbotcustom/rev/ef70aa76c86d

Explanation of what's going on - the patch is setting MOZ_CURRENT_PROJECT=i386 in the env for the 'make package' steps of mac universal builds. We're calling that in %(basedir)s/build/obj-firefox/i386. This is based on glandium's recommendation that we either do that or unset MOZ_OBJDIR.

We got past make package on a mozilla-beta build. Jobs are running on other branches to make sure we haven't busted them.

We need to apply the same treatment the make package calls with pretty names, to resolve the orange status of the builds. There's one for firefox & thunderbird on m-b, but three for Thunderbird builds on comm-central and comm-aurora. The extra TB errors, in 'make package-tests MOZ_PKG_PRETTYNAMES=1' and 'make -C objdir-tb/i386/tools/update-packaging MOZ_PKG_PRETTYNAMES=1', may be from something else given how long Thunderbird has been broken. Firefox on m-r and m-esr38 seems to be fine. I won't have time to work on this, could someone else pick it up ?

(In reply to Nick Thomas [:nthomas] from comment #5) > We need to apply the same treatment the make package calls with pretty > names, to resolve the orange status of the builds. There's one for firefox & > thunderbird on m-b, but three for Thunderbird builds on comm-central and > comm-aurora. The extra TB errors, in 'make package-tests > MOZ_PKG_PRETTYNAMES=1' and 'make -C objdir-tb/i386/tools/update-packaging > MOZ_PKG_PRETTYNAMES=1', may be from something else given how long > Thunderbird has been broken. > > Firefox on m-r and m-esr38 seems to be fine. I won't have time to work on > this, could someone else pick it up ? Does the patch in bug 1214403 address some of these issues?

r+ from jlund on irc

comment 7 is the attempted fix for comment 5. which got us from an orange build to actually green[1]. however, we are, as philor pointed out, still failing the xpcshell and mochi-other tests. The suspected culprit is still the build as signs point to wrong arch (i386) [1] https://treeherder.mozilla.org/logviewer.html#?job_id=617621&repo=mozilla-beta

xpcshell failure - https://treeherder.mozilla.org/logviewer.html#?job_id=617938&repo=mozilla-beta mochi-other failure - https://treeherder.mozilla.org/logviewer.html#?job_id=617926&repo=mozilla-beta glandium, ted - any ideas re: comment 8? we are likely going to find the culprit in a 'mach/mozharness' only change that is being ignored on beta as beta uses 'client.mk build' directly through buildbot (skipping mach/mozharness wrappers). e.g. https://hg.mozilla.org/releases/mozilla-beta/rev/9126c5fcc788#l1.23

The third thing wrong is https://treeherder.mozilla.org/logviewer.html#?job_id=617949&repo=mozilla-beta - mochitest-2 failing EME and H264 tests, which could easily either be other things from the wrong arch, or using ctypes and being the same wrong arch thing.

philor and I both discovered bug 1201224 at much the same time. Previously we were unifying i386 and x86_64 architectures into universal test binaries. Now we're packaging tests in <objdir>/i386 but running the tests on hosts that default to 64bit, and that doesn't go so well. I've landed this r=bustage https://hg.mozilla.org/build/buildbotcustom/rev/9797e9b039e9 https://hg.mozilla.org/build/buildbotcustom/rev/a615dd77dfe3 but would appreciate retrospective review. It also removes 'make package-tests MOZ_PKG_PRETTYNAMES=1' steps, since we don't run tests on release builds and those are the only things to use pretty names.

Comment on attachment 8681721 [details] [diff] [review] [buildbotcustom] Use 64bit part of objdir for test packaging This turned out to be a bad idea because we don't end up with any test archives uploaded. Here's the major steps and working directory they get called in: build - %(basedir)s/build buildsymbols - %(basedir)s/build/obj-firefox/i386 package-tests - should %(basedir)s/build/obj-firefox/x86_64 (with this patch, was i386) package - %(basedir)s/build/obj-firefox/i386 python build/config/printconfigsetting.py build/obj-firefox/i386/dist/bin/application.ini x3 - . upload - %(basedir)s/build/obj-firefox/i386 package MOZ_PKG_PRETTYNAMES=1 - %(basedir)s/build/obj-firefox/i386 check - %(basedir)s/build/obj-firefox/i386 Looking at running all the i386 tasks on x86_64 instead for m-b, pretty sure that's what m-c/m-a are doing.

Comment on attachment 8681721 [details] [diff] [review] [buildbotcustom] Use 64bit part of objdir for test packaging Backed this out: https://hg.mozilla.org/build/buildbotcustom/rev/6f01af020d4e https://hg.mozilla.org/build/buildbotcustom/rev/8f4da13e9ffe ... and landed a config change instead: https://hg.mozilla.org/build/buildbot-configs/rev/615851873863 https://hg.mozilla.org/build/buildbot-configs/rev/9442d0cd2501 NB - this is a very specific patch for mozilla-beta. I looked at changing the platform default, then setting it back for items_before(..., 42) (ie m-r, m-esr38) but the dump_master.py diff was alarmingly large. Don't have time to digest that or clobber a lot of things right now. If this sticks, then we'll need revisit. m-r will need it at the next merge at the very least. And it's kinda sucky to have old branches still using i386 too, but l10n.

Retrigger running on m-b 38f98cdc8d49, with clobber via cloberrer. Also retriggering m-esr38 13e1af34b11e because the buildbotcustom patch introduced the media errors we're seeing on beta.

mozilla-beta went green and philor has reopened that. On esr38 he's clobbered and pushed a whitespace change to get new builds, fingers crossed that comes up green on the media tests too. To do: * check state of esr38 * check the Thunderbird branches * make sure we apply the buildbot-configs change on release at the next merge. The net effect of the changes here is * on mozilla-beta, the working directory is build/obj-firefox/x84_64 instead of build/obj-firefox/i386 when calling make with targets buildsymbols, package-tests, package, 3 printconfigsetting.py calls, upload, package with pretty names, and check (https://hg.mozilla.org/build/buildbot-configs/rev/615851873863) * on all branches still using buildbot factories for compilation (m-b, m-r, m-esr38, c-c, c-a, c-b, c-esr38, maybe a project branch or too) set MOZ_CURRENT_PROJECT to x86_64 or i386 for make sdk, package-tests, package, and any pretty name equivalents (hg diff -r fa3d45a6ec1e:8f4da13e9ffe process/factory.py)

(In reply to Nick Thomas [:nthomas] from comment #15) > To do: > * check state of esr38 Looks like philor has also retriggered a mac opt build on mozilla-release (no clobber there), so that needs checking too. > * check the Thunderbird branches I've retriggered on mozilla-aurora, since that only has this bug in play. The other branches are more complicated. > * make sure we apply the buildbot-configs change on release at the next > merge.

I asked this of catlee on IRC: what would we have to do to make the beta/etc build environments match aurora/nightly? This sort of thing is going to be come more commonplace as we make larger build system changes, and it's pretty hard for us to know that we'll break it in advance.

The biggest change and risk is to release automation, which is still using buildbot. That would have to be moved also. It's arguable that having beta/release use completely different build system invocations is a larger risk than changing release automation to use mozharness/mach though.

dump_master_diff: http://people.mozilla.org/~jlund/doUpload_fix_for_RBF-dump_master_diff.patch this will also affect other branches like esr but I didn't have that in my master_config.json['release_branches']

same as before but with comment fix: diff --git a/process/factory.py b/process/factory.py index a8377a2..136f593 100644 --- a/process/factory.py +++ b/process/factory.py @@ -2636,7 +2636,7 @@ class ReleaseBuildFactory(MercurialBuildFactory): if self.enableUpdatePackaging: update_env = self.env.copy() if self.complete_platform == 'macosx64': - # make package is broken in universal (opt) builds) if + # automation make targets are broken in universal (opt) builds) if # MOZ_OBJDIR is set but you're calling in a specific arch, bug 1195546 update_env.update({'MOZ_CURRENT_PROJECT': os.path.basename(self.objdir)}) self.addStep(MockCommand( @@ -2672,7 +2672,7 @@ class ReleaseBuildFactory(MercurialBuildFactory): 'UPLOAD_TO_TEMP': '1', 'UPLOAD_EXTRA_FILES': ' '.join(self.UPLOAD_EXTRA_FILES)}) if self.complete_platform == 'macosx64': - # make package is broken in universal (opt) builds) if + # automation make targets are broken in universal (opt) builds) if # MOZ_OBJDIR is set but you're calling in a specific arch, bug 1195546 uploadEnv.update({'MOZ_CURRENT_PROJECT': os.path.basename(self.objdir)}) if self.stageSshKey:

Comment on attachment 8682134 [details] [diff] [review] 151102_bug_1220018_doUpload_fix_for_RBF_moz_current_project-2.patch I think we shouldn't take this actually. The CI builds aren't setting this so lets keep release the same.

> To do: > * check state of esr38 m-r and esr38 look good to me. sheriffs feel free to correct me. > * check the Thunderbird branches fallen: given changes outlined in comment 15, how are continuous integration comm- trees holding up? > * make sure we apply the buildbot-configs change on release at the next > merge. > the buildbot-config patch in question is: https://hg.mozilla.org/build/buildbot-configs/rev/615851873863 I've filed and blocked this against next uplift: Bug 1220979 if tb looks good and there is no objection, I'm going to close this bug since we are not changing ReleaseBuildFactory make targets (comment 21). 1220979 will track work required for next uplift

m-r is... probably okay. A clobber and multiple retriggers to get past the way that our cache invalidation for retriggers is nonexistent got me green m2 and m-oth, and no green xpcshell. The odds are fairly good that https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=a420b49d3b37 will wind up green.

m-r is okay.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/615851873863

Thanks philor. I think we're all done here.

clearing needinfo