Marcia Knous [:marcia] Reporter
	Description • 8 years ago

[Tracking Requested - why for this release]: Need symbols for Mac during the run up to final release

+++ This bug was initially created as a clone of Bug #1301751 +++

Beta 8 Mac builds seem to be suffering from the same issue seen in the cloned bug. See 

http://bit.ly/2eFhoe9

	(not currently active) Ted Mielczarek
	Comment 1 • 8 years ago

From my comment on that bug:


I looked at the 50.0b8 OS X build log:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=1815769

and I didn't see any evidence of this bug happening. There are no llvm-dsymutil crash stacks.

There are a bunch of warnings like:
09:29:55     INFO -  warning: no debug symbols in executable (-arch x86_64)

	Marcia Knous [:marcia] Reporter
	Comment 2 • 8 years ago

Ted: Who would be a good person to help us investigate this?

	(not currently active) Ted Mielczarek
	Comment 3 • 8 years ago

I don't see anything in the changes between b7 and b8 that look suspicious:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?rev=%27FIREFOX_50_0b7_RELEASE::FIREFOX_50_0b8_RELEASE%27

	(not currently active) Ted Mielczarek
	Comment 4 • 8 years ago

Bleh, that query wasn't quite right. This is close:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_50_0b7_RELEASE&tochange=FIREFOX_50_0b8_RELEASE

	(not currently active) Ted Mielczarek
	Comment 5 • 8 years ago

Looking at the opt builds on the latest pushes they all seem to be displaying this same failure...

	(not currently active) Ted Mielczarek
	Comment 6 • 8 years ago

Seems to be broken back at the b7 builds too:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&fromchange=c216304c306bda09e8b8766f692885fa378ddb3c&selectedJob=1796384

I'll keep looking.

	(not currently active) Ted Mielczarek
	Comment 7 • 8 years ago

I grepped all the build logs, this appears to have been broken all the way back to the 50.0b1 build.

	(not currently active) Ted Mielczarek
	Comment 8 • 8 years ago

Here's the beta 1 tag:
https://hg.mozilla.org/releases/mozilla-beta/rev/FIREFOX_50_0b1_RELEASE

The indexed task for its artifacts:
https://tools.taskcluster.net/task-inspector/#NLkcFlKYSem3H9GbK99a4w/

Its build log:
https://public-artifacts.taskcluster.net/Fc_7lPnlTKWHd2DE7yqYGg/0/public/build/log_raw.log

	(not currently active) Ted Mielczarek
	Comment 9 • 8 years ago

With RyanVM's help I looked at the last Aurora build from the 50 cycle:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&tochange=3fdfaaed6e0fad2d81fc28e49903ff6f0a43b12e&fromchange=2d4867f1d03306471fec3f679896365845985f69&selectedJob=3583159

That build does not display this failure mode, so it's something specific to beta.

	(not currently active) Ted Mielczarek
	Comment 10 • 8 years ago

Okay, I think this is a regression from bug 1244446:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b

One major difference between the aurora and beta configs is that in nightly/aurora we build with --disable-install-strip:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/nightly#3

and on beta and release we don't:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/beta

That means we strip during packaging on beta. Prior to that change, we would do a special stage-package after the build, which explicitly passed PKG_SKIP_STRIP=1:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b#l1.35

With that change, we just do the normal stage-package, and so on beta we stage dist/universal and presumably strip it along the way! We didn't notice this on nightly/aurora because of --disable-install-strip.

	(not currently active) Ted Mielczarek
	Comment 11 • 8 years ago

glandium: we'll need to figure out some workaround here until we can fix symbol dumping to be better. The best thing I can come up with right now is "always make universal builds skip stripping during packaging, make symbolstore.py strip the binaries after dumping symbols".

We could do the stripping in symbolstore.py here:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/toolkit/crashreporter/tools/symbolstore.py#989

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 12 • 8 years ago

This is a blocking issue for Fx50 release.

	(not currently active) Ted Mielczarek
	Comment 13 • 8 years ago

I filed bug 1314137 on making sure this errors in the future.

	Mike Hommey [:glandium] Assignee
	Comment 14 • 8 years ago

AFAICT, before bug 1244446, mac universal builds on beta were, in fact, not much different from a --disable-install-strip build. I'd say: add --disable-install-strip to the release mozconfig, and don't care about stripping after the fact. We weren't caring about doing it before, I don't see a reason to care more just to fix this issue. My interpretation of the situation before bug 1255556 could be wrong, though.

	Mike Hommey [:glandium] Assignee
	Comment 15 • 8 years ago

> My interpretation of the situation before bug 1255556 could be wrong, though.

bug 1244446, not 1255556.

	Mike Hommey [:glandium] Assignee
	Comment 17 • 8 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=abbe85b60688c69188b875b0a924149146599a19

	(not currently active) Ted Mielczarek
	Comment 18 • 8 years ago
mozreview-review

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

https://reviewboard.mozilla.org/r/89774/#review89212

Oh, that is a nice solution! If the try build looks good I'll land this.

	(not currently active) Ted Mielczarek
	Comment 19 • 8 years ago

I checked out the build log and the resulting symbols, they both look good. One fiddly thing in the build log is a bunch of warnings like:
03:54:58     INFO -  warning: cannot open debug object "/builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/js/src/gdb/../libjs_static.a(decContext.o)": No such file or directory

We must be linking some of the binaries under js/src with relative paths to the static libraries, so dsymutil can't find the objects. It's not a big deal, it'll only impact the i386 symbols for those binaries, which probably aren't used anywhere anyway. (Also this will all go away when we disable universal builds.)

	Pulsebot
	Comment 20 • 8 years ago

Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7d59e316811b
Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. r=ted

	(not currently active) Ted Mielczarek
	Comment 21 • 8 years ago

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

Approval Request Comment
[Feature/regressing bug #]: bug 1244446 caused this regression, which didn't show up until beta due to differing build config
[User impact if declined]: No symbols for OS X crash reports
[Describe test coverage new/current, TreeHerder]: None, but this is a build-config only patch.
[Risks and why]: It's possible this could break something in symbol dumping, but that's already broken on OS X on beta. The patch is very targeted.
[String/UUID change made/needed]: N/A

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 22 • 8 years ago

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

This is a blocking issue for 50 release, Aurora51+, Beta50+

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 23 • 8 years ago

https://hg.mozilla.org/releases/mozilla-aurora/rev/45118388b658d19e06fe553cc41654eca677c64c
https://hg.mozilla.org/releases/mozilla-beta/rev/b36718709777b50df48b800c7f05674cfa9e2701
https://hg.mozilla.org/releases/mozilla-release/rev/b36718709777b50df48b800c7f05674cfa9e2701

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 24 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/7d59e316811b