Closed
Bug 578880
Opened 14 years ago
Closed 14 years ago
GLIBCXX_3.4.9 not found during 'make check' on linux builder
Categories
(Release Engineering :: General, defect, P2)
Tracking
(blocking2.0 betaN+)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
blocking2.0 | --- | betaN+ |
People
(Reporter: vlad, Assigned: rail)
References
Details
(Keywords: relnote, Whiteboard: [linux64])
Attachments
(2 files, 4 obsolete files)
(deleted),
patch
|
catlee
:
review+
bhearsum
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
Similar to bug 560894, but somewhat different; nthomas asked for a new bug.
My patch introduces a new symbol dependency from libcxx that has an explicit 3.4.9 version. 'make check' fails for precisely two tests, both of them xpcom:
make[2]: Entering directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'
../../dist/bin/ShowAlignments: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1
Running XPIDL tests
/builds/slave/tryserver-linux64/build/obj-firefox/dist/bin/xpidl -m header /builds/slave/tryserver-linux64/build/xpcom/tests/TestScriptable.idl
rm -f ../../dist/bin/components/compreg.dat; \
\
XPCOM_DEBUG_BREAK=stack-and-abort ../../dist/bin/run-mozilla.sh \
../../dist/bin/TestRegistrationOrder "/builds/slave/tryserver-linux64/build/xpcom/tests/regorder";
../../dist/bin/TestRegistrationOrder: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1
make[2]: Leaving directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'
(From http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1279142586.1279148302.28883.gz)
Note that it's getting "/usr/lib64/libstdc++.so.6".
I don't see LD_LIBRARY_PATH being set anywhere, though nthomas says that he tried doing a manual make check with LD_LIBRARY_PATH set:
18:27 < nthomas> |LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib make -k check| also fails on that slave
Comment 1•14 years ago
|
||
'LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib64 make -k check' does work though, and the scope is limited make check (on try anyway).
Comment 2•14 years ago
|
||
so is this all that's needed? or do we want this for all linux64?
Assignee: nobody → lsblakk
Attachment #457586 -
Flags: review?(nrthomas)
Comment 3•14 years ago
|
||
this is happening for me on the n810s
INFO | automation.py | Application pid: 1602
/builds/fennec/fennec: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/fennec/libxul.so)
TEST-UNEXPECTED-FAIL | automation.py | Exited with code 1 during test run
INFO | automation.py | Application ran for: 0:00:00.195434
INFO | automation.py | Reading PID log: /var/tmp/tmpeC37hipidlog
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!
REFTEST INFO | runreftest.py | Running tests: end.
Comment 4•14 years ago
|
||
Splitting mobile concerns into bug 579186
Comment 5•14 years ago
|
||
Comment on attachment 457586 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 env for try
ANGLE landed on m-c so we'll need that as well, and should probably do this on any branch that's using gcc 4.3.3. And I'd really love if we could do this in the mozconfig, so that changes like bug 559964 can be done in one place.
Attachment #457586 -
Flags: review?(nrthomas) → review-
Reporter | ||
Updated•14 years ago
|
blocking2.0: --- → betaN+
Comment 6•14 years ago
|
||
unassigning myself since I will be on vacation next week and hopefully someone can get on this sooner than I can.
Assignee: lsblakk → nobody
Updated•14 years ago
|
Whiteboard: [linux64]
Reporter | ||
Comment 7•14 years ago
|
||
Any progress on this?
Comment 8•14 years ago
|
||
grabbing this bug to check it's status and even possible land it
Assignee: nobody → bear
Comment 9•14 years ago
|
||
Attachment #464104 -
Flags: review?(lsblakk)
Attachment #464104 -
Flags: feedback?(nrthomas)
Comment 10•14 years ago
|
||
(In reply to comment #5)
> Comment on attachment 457586 [details] [diff] [review]
> set LD_LIBRARY_PATH in linux64 env for try
>
> ANGLE landed on m-c so we'll need that as well, and should probably do this on
> any branch that's using gcc 4.3.3. And I'd really love if we could do this in
> the mozconfig, so that changes like bug 559964 can be done in one place.
Nick,
I mentioned to Ted about making the change attached above and he reminded me that LD_LIBRARY_PATH is not "captured" by configure and persisted thru the to the makefiles.
I also see that in the 32bit version of this change we are setting the value in config.py, both of these are leading me to want to make the change in config.py
Status: NEW → ASSIGNED
Comment 11•14 years ago
|
||
Comment on attachment 464104 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 mozconfig
Bummer, we'll have to use config.py then. We'll need to give Rail a headsup in bug 559964.
Attachment #464104 -
Attachment is obsolete: true
Attachment #464104 -
Flags: review?(lsblakk)
Attachment #464104 -
Flags: feedback?(nrthomas)
Assignee | ||
Comment 12•14 years ago
|
||
(In reply to comment #11)
> Bummer, we'll have to use config.py then. We'll need to give Rail a headsup in
> bug 559964.
Hopefully it won't affect the gcc45, which was compiled with -rpath set to its own directory: http://hg.mozilla.org/build/rpm-sources/file/c8fe426b3bdb/gcc45/centos5-i686/gcc45.spec#l51
Comment 13•14 years ago
|
||
Does not yet include other branches than try - my running of builder_list.py on sm02 is failing and I have to work out why
Attachment #457586 -
Attachment is obsolete: true
Attachment #464401 -
Flags: feedback?(nrthomas)
Comment 14•14 years ago
|
||
Comment on attachment 464401 [details] [diff] [review]
[WIP] set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug
>diff --git a/mozilla/config.py b/mozilla/config.py
>+BRANCHES['tryserver']['platforms']['linux']['env']['LD_LIBRARY_PATH'] = '/tools/gcc-4.3.3/installed/lib'
>+BRANCHES['tryserver']['platforms']['linux-debug']['env']['LD_LIBRARY_PATH'] ='/tools/gcc-4.3.3/installed/lib:%s/dist/bin' % OBJDIR
Is this for parity with 64bit ?
>+BRANCHES['tryserver']['platforms']['linux64']['env']['LD_LIBRARY_PATH'] = '/tools/gcc-4.3.3/installed/lib'
>+BRANCHES['tryserver']['platforms']['linux64-debug']['env']['LD_LIBRARY_PATH'] ='/tools/gcc-4.3.3/installed/lib:%s/dist/bin' % OBJDIR
Should be lib64 I think.
>+BRANCHES['tryserver']['platforms']['linux']['unittest-env'] = {
>+ 'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+}
>+BRANCHES['tryserver']['platforms']['linux64']['unittest-env'] = {
>+ 'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+}
Where does this get used ? If it's old-style unit test builders then those are dead as a dodo for try, and only survive on 1.9.1 and 1.9.2.
The other thing is that we'd want to make these changes just about everywhere except 1.9.1 and 1.9.2. Everything else merges back and forth with mozilla-central.
Attachment #464401 -
Flags: feedback?(nrthomas) → feedback-
Comment 15•14 years ago
|
||
beg'pardon on the other branches, missed your comment on that.
Reporter | ||
Comment 16•14 years ago
|
||
Any progress on this?
Updated•14 years ago
|
Priority: -- → P2
Comment 17•14 years ago
|
||
Looks like this is happening on x86 as well
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284148927.1284150569.14341.gz#err6
Updated•14 years ago
|
Summary: GLIBCXX_3.4.9 not found during 'make check' on linux x86-64 builder → GLIBCXX_3.4.9 not found during 'make check' on linux builder
Comment 18•14 years ago
|
||
My push from 590181 seems to trigger this consistently. Any chance of getting fixed soon?
Comment 19•14 years ago
|
||
Please keep in mind that using LD_LIBRARY_PATH will only work around the root problem, which is that the resulting executable won't run on systems with an old enough libstdc++, which is exactly what happens with these tests: the systems the test run on apparently use an old enough libstdc++.
Now the question is, apart from the build and testing environments, are there users who would be impacted ? More notably, when mozilla switches to gcc 4.5+, there are chances this will get even worse from that perspective, and while libstdc++ from gcc 4.3 is probably available to masses (but maybe not on exotic platforms (android? maemo?)), libstdc++ from gcc 4.5+ is probably not widespread enough.
Reporter | ||
Comment 20•14 years ago
|
||
Any progress on this?
Comment 21•14 years ago
|
||
(In reply to comment #20)
> Any progress on this?
Does 4.3 have -static-libstdc++? if it doesn't then it's a matter of modifying our mozconfig + recompiling gcc with CXXFLAGS=-fPIC just like we do for 4.5 in bug 559964. If that flag isn't available then we should block this on deploying gcc 4.5.
Comment 22•14 years ago
|
||
(In reply to comment #20)
> Any progress on this?
vlad - any response to comment 19? Unless we switch to statically linking libstdc++, users with older stdc++ libraries will be broken.
Comment 23•14 years ago
|
||
(In reply to comment #22)
> (In reply to comment #20)
> > Any progress on this?
>
> vlad - any response to comment 19? Unless we switch to statically linking
> libstdc++, users with older stdc++ libraries will be broken.
I'm not Vlad, but this is only fixed by statically linking stdc++. I don't think there is another way. Unfortunately this requires rebuilding gcc 4.3 with CXXFLAGS=-fPIC(which we do for gcc 4.5 builds already). So I'm hoping we can just switch to gcc 4.5 in bug 559964 real soon and let 4.3 RIP.
Reporter | ||
Comment 24•14 years ago
|
||
What Taras said; I guess we're waiting on bug 559964 which should fix this?
Comment 25•14 years ago
|
||
This is preventing me from landing bug 594489 which is a serious WebGL bug (makes many WebGL demos fail on non-US locales).
Updated•14 years ago
|
Reporter | ||
Comment 27•14 years ago
|
||
This will shortly be preventing the landing of bug 603333, which is a security bug. It sounds like this/bug 559964 is waiting for b7 to ship..
Blocks: 603333
Updated•14 years ago
|
Assignee: bear → nobody
Comment 28•14 years ago
|
||
Going to find an owner for this in the releng meeting today.
Comment 29•14 years ago
|
||
Taras told me that bug 559964 should fix this. Rail, can you confirm that/check?
Assignee: nobody → rail
Comment 30•14 years ago
|
||
The places branch is now perma-orange (linux opt, make check) because of this and the work that caused this to show up is needed. Is there a revised ETA for this to get resolved?
Comment 31•14 years ago
|
||
Also, I'm not seeing this on just linux64, but also the linux32 machines.
Comment 32•14 years ago
|
||
Blocked on switching over to gcc4.5 I believe.
Comment 33•14 years ago
|
||
Please then fill the 'Depends on' field :-)
Reporter | ||
Comment 34•14 years ago
|
||
I'm assuming that's bug 559964, which was until now marked as b2.0-? :(
Depends on: gcc4.5
Comment 35•14 years ago
|
||
So, bug 559964 isn't sticking. Is there anything else we can do about this so I can get the places branch green again (apart from code changes, preferably)
Comment 36•14 years ago
|
||
There's a decent chance some relatively simple code changes could avoid it. That said it's somewhat hard to tell what they would be given that the error on the places branch is in a test that never leaves the build machine (i.e., not packaged).
Comment 37•14 years ago
|
||
(In reply to comment #36)
> There's a decent chance some relatively simple code changes could avoid it.
> That said it's somewhat hard to tell what they would be given that the error on
> the places branch is in a test that never leaves the build machine (i.e., not
> packaged).
Well, I'm quite certain that it's because of the do_check_eq function using string streams:
http://hg.mozilla.org/projects/places/file/ecdd25bb9d90/storage/test/storage_test_harness.h#l82
It's just hard to have useful failure output without using it (AFAIK).
Comment 38•14 years ago
|
||
(In reply to comment #36)
> That said it's somewhat hard to tell what they would be given that the error on
> the places branch is in a test that never leaves the build machine (i.e., not
> packaged).
One could send a hack to try that would add the test to the package manifest.
Comment 39•14 years ago
|
||
(In reply to comment #38)
> One could send a hack to try that would add the test to the package manifest.
But how would packaging it help?
Comment 40•14 years ago
|
||
(In reply to comment #39)
> But how would packaging it help?
Packaging it gives you the binary. With the binary, you can know what symbol exactly needs version GLIBCXX_3.4.9.
Comment 41•14 years ago
|
||
(In reply to comment #40)
> (In reply to comment #39)
> > But how would packaging it help?
>
> Packaging it gives you the binary. With the binary, you can know what symbol
> exactly needs version GLIBCXX_3.4.9.
Mike,
we need static-libstdc++ for other bugs too, so it doesn't matter.
Comment 42•14 years ago
|
||
Rail, what's your feeling on an ETA for this? It's blocking a bunch of performance work on the Places branch.
Comment 43•14 years ago
|
||
(In reply to comment #42)
> Rail, what's your feeling on an ETA for this? It's blocking a bunch of
> performance work on the Places branch.
Do you really want to go ahead with the code knowing that it breaks compatibility with older libstdc++ versions? Until gcc-4.5 with static linking finally sticks, aren't these errors useful indications of code that's going to break on users' machines?
Comment 44•14 years ago
|
||
(In reply to comment #43)
> Do you really want to go ahead with the code knowing that it breaks
> compatibility with older libstdc++ versions? Until gcc-4.5 with static linking
> finally sticks, aren't these errors useful indications of code that's going to
> break on users' machines?
Not in the Places branch case at least. It's test-only code, but there isn't any way to do what I'm doing otherwise without writing lots of code for each type.
Comment 45•14 years ago
|
||
(In reply to comment #44)
> (In reply to comment #43)
> > Do you really want to go ahead with the code knowing that it breaks
> > compatibility with older libstdc++ versions? Until gcc-4.5 with static linking
> > finally sticks, aren't these errors useful indications of code that's going to
> > break on users' machines?
> Not in the Places branch case at least. It's test-only code, but there isn't
> any way to do what I'm doing otherwise without writing lots of code for each
> type.
On the branch you can just copy the appropriate libstdc++.so.whatever into your dist/firefox
Assignee | ||
Comment 46•14 years ago
|
||
> Rail, what's your feeling on an ETA for this? It's blocking a bunch of
> performance work on the Places branch.
We tried to switch to gcc-4.5 last week for mozilla-central, but had to backout.
> On the branch you can just copy the appropriate libstdc++.so.whatever into your
> dist/firefox
...or we can use gcc-4.5 as a default compiler for places branch. What do you think?
Comment 47•14 years ago
|
||
(In reply to comment #46)
> ...or we can use gcc-4.5 as a default compiler for places branch. What do you
> think?
We'll still turn mozilla-central orange once we merge back into it.
Comment 48•14 years ago
|
||
Shawn, by that time we should have 4.5 deployed. I suspect even if the freeze isn't fixed we can figure out which files are causing the hang to avoid it..ie there is no hang without using -O3
Comment 49•14 years ago
|
||
(In reply to comment #48)
> Shawn, by that time we should have 4.5 deployed. I suspect even if the freeze
> isn't fixed we can figure out which files are causing the hang to avoid it..ie
> there is no hang without using -O3
So end of this week is a reasonable timeline?
Assignee | ||
Comment 50•14 years ago
|
||
Attached is refreshed bear's patch applied for tryserver and places branches. The same approach is also used for electrolysis branch (and it uses lib directory for linux64, not lib64).
Attachment #464401 -
Attachment is obsolete: true
Attachment #493200 -
Flags: review?(bhearsum)
Comment 51•14 years ago
|
||
Comment on attachment 493200 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (tryserver and places)
Why does linux64-debug use lib64 and linux64 use lib? Don't we need this for mozilla-central and all of the other branches based on it, too? It's particularly concerning that this patch adjusts tryserver and not mozilla-central.
Do we need to fix the electrolysis entries, too? It's missing LD_LIBRARY_PATH changes for ['linux64']['env'].
Assignee | ||
Comment 52•14 years ago
|
||
(In reply to comment #51)
> Why does linux64-debug use lib64 and linux64 use lib? Don't we need this for
> mozilla-central and all of the other branches based on it, too? It's
> particularly concerning that this patch adjusts tryserver and not
> mozilla-central.
>
> Do we need to fix the electrolysis entries, too? It's missing LD_LIBRARY_PATH
> changes for ['linux64']['env'].
I removed all previous LD_LIBRARY_PATH entries for electrolysis and added a block at the end which sets LD_LIBRARY_PATH for all gcc-4.3 branches (linux, linux64, linux-debug, linux64-debug).
BRANCHES[branch]['platforms']['linux64-debug']['unittest-env'] is not being used anywhere probably, but I set it for consistency.
Attachment #493399 -
Flags: review?(catlee)
Comment 53•14 years ago
|
||
Comment on attachment 493200 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (tryserver and places)
Looks like this one is obsolete now
Attachment #493200 -
Attachment is obsolete: true
Attachment #493200 -
Flags: review?(bhearsum)
Comment 54•14 years ago
|
||
Comment on attachment 493399 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (all gcc-4.3 branches)
So...technically this looks fine.
I'm *REALLY* nervous about switching this on for mozilla-central since it means that code that depends on newer libstdc++ can sneak in.
Attachment #493399 -
Flags: review?(catlee) → review+
Reporter | ||
Comment 55•14 years ago
|
||
it's already sneaking in, because we're using a compiler that requires a newer libstdc++!
Assignee | ||
Comment 56•14 years ago
|
||
According to IRC, we should update system requirements for Firefox 4.0. Setting relnote keyword.
Flags: needs-reconfig?
Keywords: relnote
Comment 57•14 years ago
|
||
Planning to do a reconfig with this tomorrow morning.
Flags: needs-reconfig? → needs-reconfig+
Updated•14 years ago
|
Attachment #493399 -
Flags: checked-in+
Assignee | ||
Comment 59•14 years ago
|
||
All done here. Feel free to reopen if the same problem appears.
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 60•14 years ago
|
||
Doesn't look fixed:
make[2]: Entering directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'
../../dist/bin/ShowAlignments: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1
when re-enabling ANGLE. Only linux-64 opt failed though, linux-32 opt/debug and linux-64 debug both succeeded. Here's the failing log.
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1292286127.1292293295.5858.gz&fulltext=1
on try-linux64-slave04.build.mozilla.org. I see LD_LIBRARY_PATH being set to LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib -- shouldn't that be lib64?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•14 years ago
|
blocking2.0: betaN+ → beta9+
Reporter | ||
Comment 61•14 years ago
|
||
^ ping?
Assignee | ||
Comment 62•14 years ago
|
||
pong. Sorry for the delay with the reply, all hands, beta8, travel, beta8... :)
> on try-linux64-slave04.build.mozilla.org. I see LD_LIBRARY_PATH being set to
> LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib -- shouldn't that be lib64?
Yeah, my fault, just checked it to be sure:
$ file /tools/gcc-4.3.3/installed/lib/libstdc++.so.6.0.10
/tools/gcc-4.3.3/installed/lib/libstdc++.so.6.0.10: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped
$ file /tools/gcc-4.3.3/installed/lib64/libstdc++.so.6.0.10
/tools/gcc-4.3.3/installed/lib64/libstdc++.so.6.0.10: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped
Patch incoming.
Assignee | ||
Comment 63•14 years ago
|
||
Attachment #499446 -
Flags: review?(bhearsum)
Updated•14 years ago
|
Attachment #499446 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 64•14 years ago
|
||
Comment on attachment 499446 [details] [diff] [review]
Use lib64 for linux64
http://hg.mozilla.org/build/buildbot-configs/rev/38e0eed9107c
Attachment #499446 -
Flags: checked-in+
Assignee | ||
Comment 65•14 years ago
|
||
http://hg.mozilla.org/build/buildbot-configs/rev/38e0eed9107c should be landed merged to production.
Flags: needs-reconfig?
Updated•14 years ago
|
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Flags: needs-reconfig?
Resolution: --- → FIXED
Comment 66•14 years ago
|
||
As per today's meeting, beta 9 will be a time-based release. Marking these all betaN+. Please move it back to beta9+ if you believe it MUST be in the next beta (ie: trunk is in an unshippable state without this)
Updated•14 years ago
|
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
You need to log in
before you can comment on or make changes to this bug.
Description
•