Bug 1016529 looks like it may need jdk1.7 available on the builders. I'm glancing at the mock documentation, but I'm not sure from it what you'd need. You can find rpms for jdk1.7 all over, but I assume we need something a bit specialer?

Do we need this on all builders, or just the Linux builders? On Linux, I see the jdk being installed as part of the mock_install process: mock_mozilla -r mozilla-centos6-x86_64 --install ... java-1.6.0-openjdk-devel ... So we'd be looking at regenerating that rpm, unless it's off-the-shelf.

Linux only I think. We can build Android on OSX, but I don't think our builders do.

There are pre-built rpms for CentOS 6.5 for java-1.7.0-openjdk-devel that we might be able to use, assuming they're compatible with 6.2. Otherwise we'll need to grab the srpm and roll our own.

Using b-linux64-hp-0025 to run some tests, I was able to install the pre-built java-1.7.0-openjdk-devel-1.7.0.9-2.3.8.0.el6_4.x86_64.rpm downloaded from http://rpm.pbone.net/. It requires the runtime environment java-1.7.0-openjdk-1.7.0.9-2.3.8.0.el6_4.x86_64.rpm (as 1.6 did, of course). For 1.7 there's an additional dependency, libjpeg-turbo-1.2.1-1.el6.x86_64.rpm (not needed to install the 1.6 package), which will also need to be added to our mirrors.

Since these are pre-built packages with no mozilla customizazion, I will add these to http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64 (see https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages for details)

Just to make sure I am going down the right path, and with the risk of asking something stupid, I need Dustin's help again! :-| I am going to add the rpm's mentioned in comment4 to /data/repos/yum/mirrors/centos/6/latest/os/x86_64 on releng-puppet2.srv.releng.scl3.mozilla.com x86_64 Is the sequence of commands in https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#CentOS:_Landing_Custom_Repository_Changes correct (apart from a different target folder)? Am I missing some relevant steps?

Definitely do *not* modify the mirrors. They're mirrors, after all. The instructions say to add the files to the 'releng' repo, but I think that's actually too generic and starting to cause problems. Instead, it's probably best to add a new custom repository containing these JDK RPMs plus the dependency (libjpeg-turbo), and then modify the mock configs to point to that repository, too. We're moving the repos Mock uses out of PuppetAgain in bug 1022763, by the way, but you'll likely land this before I do. Still, I'll probably ask for your help and advice on testing that change.

Here is where I got the rpm's: ftp.centos.org/6.5/os/x86_64/Packages/java-1.7.0-openjdk-devel-1.7.0.45-2.4.3.3.el6.x86_64.rpm ftp.centos.org/6.5/os/x86_64/Packages/java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64.rpm ftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/libjpeg-turbo-1.2.1-1.el6.x86_64.rpm

whoops: ftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64.rpm ftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/java-1.7.0-openjdk-devel-1.7.0.45-2.4.3.3.el6.x86_64.rpm ftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/libjpeg-turbo-1.2.1-1.el6.x86_64.rpm

As agreed with Dustin, he will put those packages in a new repo (in S3, so that we move in the direction of outlined in Bug 1022763), and I will test by pinning a test box to my puppet master and trying to run some builds to check that the new packages are correctly picked up and installed in the mock environment.

Worth noting that those are all x86_64 packages. Do you want to do the same for i686? If so, we'll need two repositories for the different arches. Currently I've only made one, with no arch in the pathname.

Any chance we can get an eta on this?

hi Wesley, my goal was to close this today, but there are some pending issues while testing Bug 1022763 (which deals with a new setup for packages repositories, and is needed to implement this). I will still try to close this by the end of this week.

Dustin: since Bug 1022763 is currently blocking this, what do you think of deploying the jdk rpm's on the old repos so that devs are not blocked?

Sure -- I put it on the distinguished master, so it should begin propagating soon. You never really answered the question in comment 11..

Sorry for that Dustin! The scope of this request is limited to Android builds, and all mock environments for Android (as far as I can see in buildbot-configs) are for x86_64 arch, so I believe the change for i686 arch is not needed here. Please let me know if there is something shamefully wrong in this line of reasoning!

Nope, that makes perfect sense. So, do you want to land a puppet patch to add this repo to the mock configs, pointing to the old (puppetagain) repo?

Yup, patch is coming!

Comment on attachment 8466246 [details] [diff] [review] Add custom-jdk repo for linux x86_64 slaves to make java 1.7 available Review of attachment 8466246 [details] [diff] [review]: ----------------------------------------------------------------- Can you add a comment indicating that these are currently only used for x86_64?

After the previous patch will be landed, the new java package will be available; this patch is to actually use the new package.

Simone, brad was asking about status here, can you update? (I don't see the puppet patch as landed) We do have http://puppetagain.pub.build.mozilla.org/data/repos/yum/custom/jdk/

The first patch has been landed, so the new package is available. I will land the second in a few hours so that the new package will be actually used in the jobs.

The addition of the jdk repo creates yum conflicts while installing mock packages for other builds (example below is for a b2g-inbound-linux build). Backing the patch out, and trying to find a workaround with dustin's help. Maybe just adding the i686 version of libjpeg-turbo will help, but I will need some tests on a linux box. INFO: installing package(s): autoconf213 python zip mozilla-python27-mercurial git ccache glibc-static.i686 libstdc++-static.i686 perl-Test-Simple perl-Config-General gtk2-devel.i686 libnotify-devel.i686 yasm alsa-lib-devel.i686 libcurl-devel.i686 wireless-tools-devel.i686 libX11-devel.i686 libXt-devel.i686 mesa-libGL-devel.i686 gnome-vfs2-devel.i686 GConf2-devel.i686 wget mpfr xorg-x11-font* imake gcc45_0moz3 gcc454_0moz1 gcc472_0moz1 gcc473_0moz1 yasm ccache valgrind pulseaudio-libs-devel.i686 gstreamer-devel.i686 gstreamer-plugins-base-devel.i686 glibc-devel.i686 libgcc.i686 libstdc++-devel.i686 ORBit2-devel.i686 atk-devel.i686 cairo-devel.i686 check-devel.i686 dbus-devel.i686 dbus-glib-devel.i686 fontconfig-devel.i686 glib2-devel.i686 hal-devel.i686 libICE-devel.i686 libIDL-devel.i686 libSM-devel.i686 libXau-devel.i686 libXcomposite-devel.i686 libXcursor-devel.i686 libXdamage-devel.i686 libXdmcp-devel.i686 libXext-devel.i686 libXfixes-devel.i686 libXft-devel.i686 libXi-devel.i686 libXinerama-devel.i686 libXrandr-devel.i686 libXrender-devel.i686 libXxf86vm-devel.i686 libdrm-devel.i686 libidn-devel.i686 libpng-devel.i686 libxcb-devel.i686 libxml2-devel.i686 pango-devel.i686 perl-devel.i686 pixman-devel.i686 zlib-devel.i686 freetype-2.3.11-6.el6_1.8.i686 freetype-devel-2.3.11-6.el6_1.8.i686 freetype-2.3.11-6.el6_1.8.x86_64 ERROR: Command failed: # ['/usr/bin/yum', '--installroot', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/', 'install', 'autoconf213', 'python', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache', 'glibc-static.i686', 'libstdc++-static.i686', 'perl-Test-Simple', 'perl-Config-General', 'gtk2-devel.i686', 'libnotify-devel.i686', 'yasm', 'alsa-lib-devel.i686', 'libcurl-devel.i686', 'wireless-tools-devel.i686', 'libX11-devel.i686', 'libXt-devel.i686', 'mesa-libGL-devel.i686', 'gnome-vfs2-devel.i686', 'GConf2-devel.i686', 'wget', 'mpfr', 'xorg-x11-font*', 'imake', 'gcc45_0moz3', 'gcc454_0moz1', 'gcc472_0moz1', 'gcc473_0moz1', 'yasm', 'ccache', 'valgrind', 'pulseaudio-libs-devel.i686', 'gstreamer-devel.i686', 'gstreamer-plugins-base-devel.i686', 'glibc-devel.i686', 'libgcc.i686', 'libstdc++-devel.i686', 'ORBit2-devel.i686', 'atk-devel.i686', 'cairo-devel.i686', 'check-devel.i686', 'dbus-devel.i686', 'dbus-glib-devel.i686', 'fontconfig-devel.i686', 'glib2-devel.i686', 'hal-devel.i686', 'libICE-devel.i686', 'libIDL-devel.i686', 'libSM-devel.i686', 'libXau-devel.i686', 'libXcomposite-devel.i686', 'libXcursor-devel.i686', 'libXdamage-devel.i686', 'libXdmcp-devel.i686', 'libXext-devel.i686', 'libXfixes-devel.i686', 'libXft-devel.i686', 'libXi-devel.i686', 'libXinerama-devel.i686', 'libXrandr-devel.i686', 'libXrender-devel.i686', 'libXxf86vm-devel.i686', 'libdrm-devel.i686', 'libidn-devel.i686', 'libpng-devel.i686', 'libxcb-devel.i686', 'libxml2-devel.i686', 'pango-devel.i686', 'perl-devel.i686', 'pixman-devel.i686', 'zlib-devel.i686', 'freetype-2.3.11-6.el6_1.8.i686', 'freetype-devel-2.3.11-6.el6_1.8.i686', 'freetype-2.3.11-6.el6_1.8.x86_64'] Package python-2.6.6-29.el6.x86_64 already installed and latest version Package mpfr-2.4.1-6.el6.x86_64 already installed and latest version Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Package libjpeg is obsoleted by libjpeg-turbo, but obsoleting package does not provide for requirements Error: Package: libtiff-3.9.4-1.el6_0.3.i686 (centos6) Requires: libjpeg.so.62 Available: libjpeg-6b-46.el6.i686 (centos6) libjpeg.so.62 You could try using --skip-broken to work around the problem Error: Package: gtk2-2.18.9-6.el6.centos.i686 (centos6) Requires: libjpeg.so.62 Available: libjpeg-6b-46.el6.i686 (centos6) libjpeg.so.62 Error: Package: jasper-libs-1.900.1-15.el6_1.1.i686 (centos6-updates) Requires: libjpeg.so.62 Available: libjpeg-6b-46.el6.i686 (centos6) libjpeg.so.62 Error: Package: 1:cups-libs-1.4.2-44.el6.i686 (centos6) Requires: libjpeg.so.62 Available: libjpeg-6b-46.el6.i686 (centos6) libjpeg.so.62

I installed the packages from comment 9 manually and the build went fine. Note, I didn't have libjpeg installed, just libjpeg-turbo, and I can't seem to get to jdk1.7 from mock so I just used yum --localinstall for all of those.

:wesj, I was also able to install the packages mentioned in comment 9, but here is the problem: - in order to install java-1.7.0-openjdk, library libjpeg.so.62(LIBJPEG_6.2)(64bit) is required; in particular, the version installed with libjpeg-turbo is needed, while the one in libjpeg.x86_64 does not work. - package gtk2 (used in mock environments for several non android builders) has the opposite requirement: the version of libjpeg.so.62 library provided in libjpeg-turbo does not work and gives the error message reported above; - the problem is the following: as soon as we make libjpeg-turbo available in the mock repos, that is installed in all mock environments because it obsoletes libjpeg.x86_64, therefore all mock environments requiring gtk2 cannot be properly created. This is why we cannot just add the libjpeg turbo to the mock environments: we would break a bunch of non android builds. - we have no way (at the moment, and in my understanding) to choose a different set of repositories for different mock environments, so we cannot keep the libjpeg addition isolated and local to the android builds only - I could not find another package providing the requirements of jdk-1.7.0 without breaking gtk2 installation I spoke to Callek and Dustin, but apparently this is the first time such a conflict happens in our repos :-( I have two ideas at the moment: 1) force the installation of the old package in all builders that require it, so that libjpeg-turbo is not installed in non-android builds (I don't know if this is feasible, this is what I am trying to do right now) 2) (quite an ugly hack): download from tooltool the libjpeg-turbo rpm for Android builds only and install it from the locally-downloaded rpm's instead of taking it from a remote repo (mock_mozilla supports installation of local rpm's). This way, we would be able to make the dependency required for java-1.7 available where it is required without adding it to any of the repos used for mock environment creation and, therefore, without breaking other builds.

I cannot find ways to follow path 1 in previous comment because libjpeg-turbo is always installed because of the "Obsoletes" tag in the spec file used to build it. Another ugly hack which comes to my mind would be to rebuild libjpeg-turbo removing the "Obsoletes" tag: this is very ugly, but it could work to solve this specific problem (did I mention it's an ugly hack?).

An alternative would be not to install java via rpm, but just install a binary version of java and enable it configuring path and java home (which is how I have always installed java in my past jobs). The approach would be similar to the one followed for ant installation, which I proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=971841#c19. The complication here is that there are not official binary releases for openjdk, so we should either build it from source or use the Oracle java (which may not be feasible for other reasons).

We could also set up different mock configurations for different types of builds, including different repos. That may be the easiest way out, if indeed this configuration works fine for b2g but fails for other builders that require gtk2.

:dustin Great! I thought that was not possible (see comment 26, point 3 in the non-numbered list), but if it is, it's the way I would prefer to choose. I will contact in IRC for some help (quite a common pattern recently, hey?)

(recap of our irc conversation) This would involve duplicating and modifying https://github.com/mozilla/build-puppet/blob/master/modules/mockbuild/files/mozilla-centos6-{i386,amd_64}.cfg and arranging to install the modified versions, too. It'd probably be good to rename to something like 'mozilla-firefox-build-env' and 'mozilla-android-build-env' so it's clear what does what. Once the configs are in place, you'll need mozharness changes to update 'mock -r mozilla-centos6-x86_64' to 'mock -r mozilla-{whatever}-build-env', as appropriate to the builder. A few related ideas: 1. If running two build environments will cause a great deal more disk usage, or cause a great deal more downloads from the repositories, then we should be aware of that in advance to avoid causing an outage. 2. It's awkward to need to *write* mock configs with puppet, but then *use* them from mozharness. Is it possible to move the contents of the mock configs into mozharness? I don't know enough about mock to know if this is possible.

Adding a brand new mock configuration just for android, which includes the new custom repo for jdk 1.7.

Patch to use the newly created mock config for android, and update version of java being installed in chroot. New version will be used in all android platforms.

I tested the previous puppet patch by pinning a linux box to my personal puppet environment. Then I configured my staging environment to run android builds using the new buildbot-configs, and I was able to successfully create mock environments - this attachment is the output of the mock creation step.

A few further notes/warnings: 1. I did not investigate yet whether using different mock environments can cause disk or load issues, a risk pointed out by Dustin in comment 31. jford: any comments on this (I ask you as original writer of mock_mozilla)? 2. Since it is not possible to use mock specifying config files which are not in /etc/mock_mozilla, I am now inclined to keep the existing system to configure mock (puppet + buildbot-configs) instead of moving everything to mozharness.

Comment on attachment 8469992 [details] [diff] [review] puppet_02 Review of attachment 8469992 [details] [diff] [review]: ----------------------------------------------------------------- ::: modules/mockbuild/files/mozilla-centos6-x86_64-android.cfg @@ +2,5 @@ > +# License, v. 2.0. If a copy of the MPL was not distributed with this > +# file, You can obtain one at http://mozilla.org/MPL/2.0/. > + > +# Note: Editing this file will trigger a mock scrub. > +# Last refreshed September 12, 2013 This should be removed..

Comment on attachment 8469993 [details] [diff] [review] buildbot-configs_02 Review of attachment 8469993 [details] [diff] [review]: ----------------------------------------------------------------- I have slight concerns around possible fallout in terms of disk-space-used once this lands, since its adding an extra mock environment to the builders. I also am hesitant to deploy this on a friday given possible fallout of the above.

(In reply to Simone Bruno [:simone] from comment #35) > A few further notes/warnings: > > 1. I did not investigate yet whether using different mock environments can > cause disk or load issues, a risk pointed out by Dustin in comment 31. > jford: any comments on this (I ask you as original writer of mock_mozilla)? <coop|triage> simone was going to look into the space himself .. simone, did you look at this yet?

(In reply to Simone Bruno [:simone] from comment #35) > 1. I did not investigate yet whether using different mock environments can > cause disk or load issues, a risk pointed out by Dustin in comment 31. > jford: any comments on this (I ask you as original writer of mock_mozilla)? Each environment used on the machine will take up space in the yum and root fs caches. The root caches are almost certainly not shared between configs, but I think the yum cache might be per repository instead of per machine.

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #40) > Each environment used on the machine will take up space in the yum and root > fs caches. The root caches are almost certainly not shared between configs, > but I think the yum cache might be per repository instead of per machine. Oh and unless you delete the environment after the run, you'll also have the space used by the fully expanded working copy of the environment

Thanks jhford! The addition of a distinct android mock environment implies an extra 1.5G in root cache, and an extra 3.1G for the chroot itself (the expanded working environment). I did not register growth in the yum cache (which seems to be per repository). Android builds are mainly run on AWS r3.xlarge boxes, where AFAIU these extra 4.6G while building should not be a problem.

Dustin, apart from the lines you suggested to remove in the patch you r+'d, I am going to put line: +config_opts['root'] = 'mozilla-centos6-x86_64-android' instead of +config_opts['root'] = 'mozilla-centos6-x86_64', Otherwise we would just have a different config file pointing to the same local environment, which would be useless!

Indeed, probably worse than useless.

Comment on attachment 8469992 [details] [diff] [review] puppet_02 In production with amendments described in comments 36 and 43.

Comment on attachment 8469993 [details] [diff] [review] buildbot-configs_02 Will go to production in next reconfig.

This got backed out: https://hg.mozilla.org/build/buildbot-configs/rev/6446fc04c928 The same patch should apply correctly tomorrow once the new golden AMIs are created.

When this relands, please can the commit message include the bug number, to make it easier to find :-)

:edmorley, sure! I mistakenly omitted it in my previous commit.

Finally we have some working Android builds using java-1.7.0: http://buildbot-master71.srv.releng.use1.mozilla.com:8001/builders/Android%204.2%20x86%20mozilla-central%20build/builds/146 Good news, in particular, in: http://buildbot-master71.srv.releng.use1.mozilla.com:8001/builders/Android%204.2%20x86%20mozilla-central%20build/builds/146/steps/mock-install/logs/stdio

The patches for this went live with: http://hg.mozilla.org/build/puppet/rev/1ccf959136c9 http://hg.mozilla.org/build/buildbot-configs/rev/c2ea3f8a9ae5

Android builds seem to be green and no disk space issues have been experienced so far. Wesj: feel free to mark as RESOLVED FIXED if there are no further issues with this.

Thanks everyone!

The SDK change has been riding the trains, and the java package needs to as well. We missed that for Aurora (bug 1062880) and Beta (bug 1056839), because a bug wasn't filed and hooked up to the appropriate merge day tracking bug. For release I've set up bug 1082325, connected to bug 1082322.