https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=ea538d555890a7fa33b97de780bb5d09b230f428&selectedJob=85973554 From the log: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(PkgConfig(Failure { command: "\"pkg-config\" \"--libs\" \"--cflags\" \"x11 >= 1.4.99.1\"", output: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "Package x11 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `x11.pc\'\nto the PKG_CONFIG_PATH environment variable\nNo package \'x11\' found\n" } }), State { next_error: None })', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/result.rs:837 stack backtrace:

Any hints what we need to configure/port? Aryx said something to me on IRC (which I didn't understand), but I didn't log the channel and lost it.

Sounds like missing x11 headers/libraries. I'd expect "configure" to fail, not cargo.

Sorry, am no help here. So this is from https://dxr.mozilla.org/mozilla-central/rev/e03e0c60462c775c7558a1dc9d5cf2076c3cd1f9/third_party/rust/x11/Cargo.toml#39 error: failed to run custom build command for `x11 v2.12.1` process didn't exit successfully: `/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/toolkit/library/release/build/x11-9858dee221fb8695/build-script-build` (exit code: 101) 2.12.0 is 3 months old: https://github.com/rust-lang/crates.io-index

Anyone has any idea what's going on here? Rail moved this to Thunderbird::Build Config, and then said: "Sounds like missing x11 headers/libraries." So where/why/when are they or did they go missing and how do we restore them? After all, this runs on the treeherder servers, or does it also fail locally now?

I don't get this failure locally (on linux). Has it been fixed in m-c since? I have rustc 0.16, not 0.15.1 as on the buid servers.

Aceman and I have been looking at some logs: TB Tinderbox: https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz libX11 i686 1.3-2.el6 centos6 585 k libX11 x86_64 1.3-2.el6 centos6 582 k Required is > 1.4.99 to fit with Rust. On the FF build machines which use Taskcluster we see: https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/logs/live_backing.log [task 2017-03-23T21:57:47.645888Z] 21:57:47 INFO - 0xf14a8000 - 0xf15e1fff libX11.so.6.3.0 which according to the internet equates to a package version of 1.6.3. Sorry, Rail, is this really a Thunderbird::Build config issue? It looks to me that Rust got upgraded and all those Tinderbox Linux machines are now incompatible since they still have X11 1.3. Can they be upgraded? Or is this the end of Thunderbird on Linux unless it moves to Taskcluster?

(In reply to Jorg K (GMT+1) from comment #4) > Anyone has any idea what's going on here? Rail moved this to > Thunderbird::Build Config, and then said: > "Sounds like missing x11 headers/libraries." So where/why/when are they or > did they go missing and how do we restore them? After all, this runs on the > treeherder servers, or does it also fail locally now? I'll take a gander.. not entirely sure what's going on.

(In reply to Jorg K (GMT+1) from comment #6) > Aceman and I have been looking at some logs: > > TB Tinderbox: > https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central- > linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz > libX11 i686 1.3-2.el6 centos6 585 > k > libX11 x86_64 1.3-2.el6 centos6 582 > k > > Required is > 1.4.99 to fit with Rust. > > On the FF build machines which use Taskcluster we see: > https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/ > logs/live_backing.log > [task 2017-03-23T21:57:47.645888Z] 21:57:47 INFO - 0xf14a8000 - > 0xf15e1fff libX11.so.6.3.0 > which according to the internet equates to a package version of 1.6.3. > > Sorry, Rail, is this really a Thunderbird::Build config issue? > > It looks to me that Rust got upgraded and all those Tinderbox Linux machines > are now incompatible since they still have X11 1.3. > > Can they be upgraded? Or is this the end of Thunderbird on Linux unless it > moves to Taskcluster? From what I've discerned from both logs: TB uses mock. Firefox doesn't and ergo uses the x11 libs from the system itself. Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS instances and possibly similar AMIs and ergo similiar operating systems (if not the same). That said, I'm wondering if disabling Mock on Linux32 might work? i.e. https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/thunderbird_config.py#l140 'use_mock': True, change that to 'use_mock': False (Again, just a left-field suggestion since I know absolutely nothing of the stuff that goes in AMIs).

fwiw.. SeaMonkey's also hitting this, but unlike TB, our infra isn't that up to date so I doubt we'll be able to depend on the system's X11 libs.

(In reply to Jorg K (GMT+1) from comment #6) > Sorry, Rail, is this really a Thunderbird::Build config issue? Now it is clear to say that is a build environment issue, not build config. > It looks to me that Rust got upgraded and all those Tinderbox Linux machines > are now incompatible since they still have X11 1.3. > > Can they be upgraded? Or is this the end of Thunderbird on Linux unless it > moves to Taskcluster? Upgrading this would be not trivial. Mock based builds use Centos 6.2, which is very ancient. Probably there is a way to use Centos 6.8 in mock (similar to https://dxr.mozilla.org/build-central/source/puppet/modules/runner/files/mockbuild-config-templates/mozilla-centos6-x86_64.cfg), but it's completely untested. (In reply to Edmund Wong (:ewong) from comment #8) > TB uses mock. > Firefox doesn't and ergo uses the x11 libs from the system itself. > > Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS > instances and possibly similar AMIs and ergo similiar operating systems > (if not the same). Unfortunately, the Tc-based build environment is a bit different from what we use for buildbot. TC uses Centos6 latest (6.8 atm) https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build/Dockerfile, buildbot uses Centos 6.2 (https://s3-external-1.amazonaws.com/mozilla-releng-mock-centos-us-east-1/mirrors/centos/6/latest/centos-version.txt) > > That said, I'm wondering if disabling Mock on Linux32 might work? > > i.e. > https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/ > thunderbird_config.py#l140 > 'use_mock': True, > > change that to 'use_mock': False I don't think this is going to work. With mock disabled, buildbot will use the existing host environment and won't be able to install any packages (it's not running as root). Sounds like we finally hit the wall :/ Fixing the existing build environment is not trivial, neither porting to TC...

Would it be possible to use tooltool or similar to include a version of rustc that works, maybe statically compiled to not depend on the system libX11?

This appears to work (so far, usually stopped after 14 minutes): https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=27f10ccf5e60c61065210b5778e008caec2152c0 Idea of Frank-Rainer Grahl.

Assigned this accidentally.

Landed temporary fix: https://hg.mozilla.org/comm-central/rev/725206ff24cc9e4933df23f1db7ce465114fa22d

OK, let's see whether renaming the file will avoid the "build file copies are not in sync" problem.

I have my doubt as I think whole directories are compared and /build/unix/mozconfig.* are also compared which shows more files not in sync. It seems the files in im/config/mozconfigs/linux mail/config/mozconfigs/linux and suite/config/mozconfigs/linux and the linux64 ones aren't compared. What about trying to move the config option to this files?

Try with my proposal: https://hg.mozilla.org/try-comm-central/rev/2d022f5f1b7c14f2f0930c13af46acc8a136f14e

Comment on attachment 8851035 [details] [diff] [review] 1350011-rename.patch (In reply to Richard Marti (:Paenglab) from comment #17) > How about trying to move the config option to these files? Sure, since my attempt was a complete disaster :-(

Try is green.

(In reply to Richard Marti (:Paenglab) from comment #18) > Try with my proposal: Green as expected. How many files do you want to change? debug, nightly, l10n, perhaps release?, also IB and SM, 32 and 64 bit. 24 files. We can do that. We did already to disable Rust temporarily: https://hg.mozilla.org/comm-central/rev/615bdf95952f15f6fa7020ba8b48e9863f5eabe5

Comment on attachment 8851067 [details] [diff] [review] moveConfig.patch I think you need many more files, see previous comment. Also, let me back out my changeset. I proved the point, but we got lots of oranges as expected.

Added the l10n files. Are the other files used in c-c or only for release etc.?

Backout of first temporary solution: https://hg.mozilla.org/comm-central/rev/01a0dc6906d9ac6f435dd9f089577b559285219e

Comment on attachment 8851068 [details] [diff] [review] moveConfig.patch Review of attachment 8851068 [details] [diff] [review]: ----------------------------------------------------------------- As I said, debug for for debug builds. I'm not sure about release for the time being. And please note my backout.

This should now address all needed files.

Comment on attachment 8851073 [details] [diff] [review] Bug1350011.patch Nice. What are we going to do with the "release" files: 1 in im/, 2 in mail/ and 4 in suite/. I believe they are needed for beta and above. Do we care right now? Anyway, let's land this after the next M-C merge.

Comment on attachment 8851073 [details] [diff] [review] Bug1350011.patch Frank-Rainer, you can r+ this for SM. Do you want the release files treated, too?

Comment on attachment 8851073 [details] [diff] [review] Bug1350011.patch Well we mailed last week about webrender and I prefer to see cursors in edit fields which I lost after enabling it locally so its best that it stays out anyway for now until more mature.

https://hg.mozilla.org/comm-central/rev/6b1f2f9821292d9a46b718d959e177120af0199f Let's close this for now.