We install `java-1.7.0-openjdk-devel` into the mock environment for Android builds. This is a specific version of java which we may want to ugprade due to bugs or just being too old. As such, it would be better handled via tooltool, where its specification is in-tree and we don't have to run a custom yum/apt repo.

See bug 1176403 -- we *only* install this in mock.

Ted, just looking for your gut instinct here -- better to try to build the jdk from source (following the lines of the RPM .spec), or just unpack the RPM and make a tarball? Neither is a slam-dunk -- the .spec is long and complicated, while the RPM may not be relocatable from /usr/lib/jvm into the build workspace.

If we've been using pre-built packages before I'd say just use an existing package if possible.

https://tools.taskcluster.net/task-inspector/#Q_KK2c5qT4mkrYB00ECjjg/0 { "provisionerId": "aws-provisioner-v1", "workerType": "b2gtest", "created": "2015-06-30T14:53:46.048Z", "deadline": "2015-06-30T15:53:46.048Z", "payload": { "image": "quay.io/mozilla/ubuntu-builder:0.0.2", "command": [ "/bin/bash", "-c", "mkdir build && cd build && curl -o jdk.deb http://mirrors.kernel.org/ubuntu/pool/main/o/openjdk-7/openjdk-7-jdk_7u79-2.5.5-0ubuntu0.14.04.2_amd64.deb && ar -x jdk.deb && tar -xf data.tar.* && cd usr/lib/jvm && tar -zcf /tmp/jdk.tar.gz ." ], "maxRunTime": 600, "artifacts": { "public/jdk.tar.gz": { "path": "/tmp/jdk.tar.gz", "expires": "2016-05-28T16:12:56.693817Z", "type": "file" } } }, "metadata": { "name": "JDK Extract", "description": "Extract the JDK from the Ubuntu package for upload to tooltool", "owner": "dmitchell@mozilla.com", "source": "http://tools.taskcluster.net/task-creator/" } }

[ { "size": 16348210, "visibility": "public", "digest": "9f23451d5b87cbf5fe788749fe585dcd82f63f666d1b6fbcf21507afd9832f4e08142b3dafc3c158951270cb8cb3fe73895b8e480ade6acef987d833960f304a", "algorithm": "sha512", "filename": "jdk.tar.gz" } ]

TODO: code that up in a script like https://bugzilla.mozilla.org/show_bug.cgi?id=1176229#c2

From talking to nalexander, it seems that we need both the JDK and the JRE, which is a different Ubuntu package. However, shipping both in the same tooltool package makes sense. Once that's done, I'll set JAVA_HOME so that the build system can find it.

Creating the new JDK tarball: https://tools.taskcluster.net/task-inspector/#oLT2UkfwQaSND0DeJyv2hA/

[ { "size": 16045724, "visibility": "public", "digest": "934e5eaaecc9f7b61594b402d520b958a6ced82fb93809d6f3266eecae695828fd8a4bcdfd8eda4f8e183aaf7b7e2dd88216ebd02cc81fc51548dc56a373d528", "algorithm": "sha512", "filename": "java.tar.xz" } ]

And with that added to mobile/android/config/tooltool-manifests/android/releng.manifest with unpack: true, I made a try run that sets JAVA_HOME: diff --git a/mobile/android/config/mozconfigs/common b/mobile/android/config/mozconfigs/common --- a/mobile/android/config/mozconfigs/common +++ b/mobile/android/config/mozconfigs/common @@ -5,16 +5,19 @@ # This file is included at the top of all native android mozconfigs if [ "x$IS_NIGHTLY" = "xyes" ]; then MOZ_AUTOMATION_UPLOAD_SYMBOLS=${MOZ_AUTOMATION_UPLOAD_SYMBOLS-1} fi MOZ_AUTOMATION_L10N_CHECK=0 . "$topsrcdir/build/mozconfig.common" +# Java JRE/JDK are installed from tooltool, so we set JAVA_HOME to find them +mk_add_options "export JAVA_HOME=$topsrcdir/java-7-openjdk-amd64" + # Set the most aggressive settings for szip. Not the default because it's # much slower and we didn't want to slow down developers builds. # Has no effect when MOZ_ENABLE_SZIP is not set in mobile/android/confvars.sh. MOZ_SZIP_FLAGS="-D auto -f auto" ac_add_options --enable-elf-hack ANDROID_NDK_VERSION="r8e" https://tools.taskcluster.net/task-inspector/#FHJdhiNdRFWGDCqf-uuMTQ/0 This *almost* works -- unfortunately those two Ubuntu packages don't contain bin/java. Also, I think we should give the toplevel dir a common name without the arch in it, and build both arches at once.

ted suggests I just export JAVA_HOME=.. in the mozconfig

https://tools.taskcluster.net/task-inspector/#SUGE4XlqSFmzoKPq1uyNlg/0 [ { "size": 55391032, "visibility": "public", "digest": "a15fbba949fbb5039bef717f55370d40050a537a2b3a1ffe5e8b655ae90b54419020783c183bba72ce2610f35e3611e259df09b5844a72bd161b512fb54898fb", "algorithm": "sha512", "filename": "java_home-7u79-2.5.5-0ubuntu0.14.04.2-amd64.tar.xz" }, { "size": 57200588, "visibility": "public", "digest": "7cb01855611bb1cf2b329c5692e2fcc7ec84706c09f7969b37c12d8124977a61dc65d90031a655c3ba47f60a2c52b667b199929243ff1568c07242337ff07da0", "algorithm": "sha512", "filename": "java_home-7u79-2.5.5-0ubuntu0.14.04.2-i386.tar.xz" } ]

Nick, do we need the 32-bit Java for android-x86? And ARMv6 for android-armv6? Or does the java arch just have to match the build system's arch? (in which case we just need amd64)

Latest run of a build with java from tooltool: https://tools.taskcluster.net/task-inspector/#H2Gumf4WTlSTwDWRvdsTAw/0

Finding a lot of hard-coded 'java' paths.. some in the Android SDK: 16:05:40 INFO - /home/worker/workspace/build/src/android-sdk-linux/build-tools/22.0.1/dx: line 89: exec: java: not found So I think we're going to have to add $JAVA_HOME/bin to $PATH. Question is, where best to do that.

(In reply to Dustin J. Mitchell [:dustin] from comment #13) > Nick, do we need the 32-bit Java for android-x86? And ARMv6 for > android-armv6? Or does the java arch just have to match the build system's > arch? (in which case we just need amd64) It's my belief that the Java JRE version is only used to execute the various tools (like Proguard, jar, etc) and therefore needs to just run on the build system. I am not aware of any further compatibility requirements, and I'm not sure who would know otherwise. blassey? glandium? snorp?

What nick said.

Thanks! Adding to PATH in the mozconfig gets a green build: https://tools.taskcluster.net/task-inspector/#exx3pihfTzKqk6gaK_Hjjg/0

Bug 1161075: build android with JDK from tooltool; r=nalexander

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander https://reviewboard.mozilla.org/r/12609/#review11077 ::: mobile/android/config/tooltool-manifests/android/releng.manifest:54 (Diff revision 1) > +"size": 55391032, nit: delete trailing whitespace. ::: testing/docker/ubuntu-build/system-setup.sh:20 (Diff revision 1) > nice! I'm no docker expert but I'm happy to see this JDK install at the image level go away. ::: testing/taskcluster/scripts/misc/repackage-jdk.sh:19 (Diff revision 1) > + ar -x the.deb I can't speak to the TC team's script standards but I see no issues. Consider flagging additional eyes if you feel it necessary. ::: mobile/android/base/Makefile.in:166 (Diff revision 1) > - java -cp $(classycle_jar) \ > + $(JAVA) -cp $(classycle_jar) \ Thanks for catching these -- all oversights on my part.

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander https://reviewboard.mozilla.org/r/12609/#review11079 Ship It!

https://reviewboard.mozilla.org/r/12609/#review11083 ::: testing/taskcluster/scripts/misc/repackage-jdk.sh:20 (Diff revision 1) > + tar -C root/ -xf data.tar.* Drive-by review because I saw this in nick's comment, but you can replace ar+tar with dpkg-deb -x

Updated build script based on review comments (although the result should be identical and I won't re-upload): https://tools.taskcluster.net/task-inspector/#U3QNgs2GQy6eqpl2-6t4Rg/

Having some trouble with TC killing the tasks and not keeping the logs.. https://tools.taskcluster.net/task-inspector/#r562_LryQU6VdaxfDM5afw/0

OK, that's not working, but I've punted it to bug 1181174.

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander Bug 1161075: build android with JDK from tooltool; r=nalexander

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander Sorry, carrying forward the r+

Boo, doesn't work in Buildbot: 10:21:09 INFO - /builds/slave/try-and-api-11-000000000000000/build/src/java_home/bin/javac -target 1.7 -source 1.7 -classpath /builds/slave/try-and-api-11-000000000000000/build/src/android-sdk-linux/platforms/android-22/android.jar:/builds/slave/try-and-api-11-000000000000000/build/src/android-sdk-linux/tools/lib/lint.jar:/builds/slave/try-and-api-11-000000000000000/build/src/android-sdk-linux/tools/lib/lint-checks.jar -bootclasspath /builds/slave/try-and-api-11-000000000000000/build/src/android-sdk-linux/platforms/android-22/android.jar -encoding UTF8 -g:source,lines -Werror -d annotationProcessors-classes /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/AnnotationInfo.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/AnnotationProcessor.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/classloader/AnnotatableEntity.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/classloader/ClassWithOptions.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/classloader/IterableJarLoadingURLClassLoader.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/classloader/JarClassIterator.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/CodeGenerator.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/SDKProcessor.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/utils/AlphabeticAnnotatableEntityComparator.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/utils/GeneratableElementIterator.java /builds/slave/try-and-api-11-000000000000000/build/src/build/annotationProcessors/utils/Utils.java 10:21:09 INFO - /builds/slave/try-and-api-11-000000000000000/build/src/java_home/bin/javac: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /builds/slave/try-and-api-11-000000000000000/build/src/java_home/bin/../lib/amd64/jli/libjli.so) Presumably because the libc for CentOS 6.2 is so crazy ancient. Solutions I can see: - include the ubuntu glibc in the tarball (temporarily) - rebuild the tarball to run on CentOS 6.2 (it will likely also work on new ubuntu) - use different tarballs under Buildbot and TaskCluster (which would require different manifests)

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander https://reviewboard.mozilla.org/r/12609/#review11297 Ship It!

- include the ubuntu glibc in the tarball (temporarily) no-go -- the search paths aren't set up for that.

Built from CentOS 6.2 (http://puppetagain.pub.build.mozilla.org/data/repos/yum/custom/jdk/) [ { "size": 30804836, "visibility": "public", "digest": "0c1defcc2df5ffa1a86cf6ff7f41f1b3a2219deff59620fb85f589e755b9efd2ed99e94c016992d412dcf4b62424bd0eeee80f4dc7099a7455a8987ffb98a60c", "algorithm": "sha512", "filename": "java_home-1.7.0-openjdk-1.7.0.45.x86_64.tar.xz" } ]

In hopes I will never do this again, I'm not putting this in a taskcluster task :) dmitchell@rpmpackager1 ~/jdk $ rpm2cpio java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64.rpm | cpio -ivd dmitchell@rpmpackager1 ~/jdk $ rpm2cpio java-1.7.0-openjdk-devel-1.7.0.45-2.4.3.3.el6.x86_64.rpm | cpio -ivd dmitchell@rpmpackager1 ~/jdk $ cd usr/lib/jvm/ dmitchell@rpmpackager1 ~/jdk/usr/lib/jvm $ vi java-1.7.0-openjdk-1.7.0.45.x86_64/VERSION dmitchell@rpmpackager1 ~/jdk/usr/lib/jvm $ mv java-1.7.0-openjdk-1.7.0.45.x86_64 java_home dmitchell@rpmpackager1 ~/jdk/usr/lib/jvm $ tar -Jcvf java_home-1.7.0-openjdk-1.7.0.45.x86_64.tar.xz java_home/

After a few stupid mistakes, one last try to make sure this works in BB and TC: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5fcb33d8bfc&exclusion_profile=false

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander Bug 1161075: build android with JDK from tooltool; r=nalexander

That failed in TC with 21:13:55 INFO - /home/worker/workspace/build/src/android-sdk-linux/build-tools/22.0.1/dx --dex --output=classes.dex jars-proguarded 21:14:14 INFO - trouble writing output: Could not initialize NSS but succeeded on Buildbot: 14:25:03 INFO - /builds/slave/try-and-api-11-000000000000000/build/src/android-sdk-linux/build-tools/22.0.1/dx --dex --output=classes.dex jars-proguarded 14:25:27 INFO - ../../../config/nsinstall -R classes.dex ../../../dist/bin I suspect, again, due to glibc versions. SO CLOSE!

Running that shell script with -x shows the command it's running: + exec java -Xmx1024M -jar /home/worker/workspace/build/src/android-sdk-linux/build-tools/22.0.1/lib/dx.jar --dex --output=classes.dex jars-proguarded trouble writing output: Could not initialize NSS It pauses for several seconds between the two, so it's not a simple executable-loading error.

The last bit from LD_DEBUG=all is 472: calling init: /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so 472: 472: opening file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0]; direct_opencount=1 472: 472: symbol=JNI_OnLoad; lookup in file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] to /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0]: normal symbol `JNI_OnLoad' 472: symbol=Java_sun_security_pkcs11_wrapper_PKCS11_initializeLibrary; lookup in file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] to /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0]: normal symbol `Java_sun_security_pkcs11_wrapper_PKCS11_initializeLibrary' 472: symbol=Java_sun_security_pkcs11_Secmod_nssGetLibraryHandle; lookup in file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] to /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0]: normal symbol `Java_sun_security_pkcs11_Secmod_nssGetLibraryHandle' 472: symbol=dlopen; lookup in file=java [0] 472: symbol=dlopen; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0] 472: symbol=dlopen; lookup in file=/home/worker/workspace/build/src/java_home/bin/../lib/amd64/jli/libjli.so [0] 472: symbol=dlopen; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libj2pkcs11.so [0] to /lib/x86_64-linux-gnu/libdl.so.2 [0]: normal symbol `dlopen' [GLIBC_2.2.5] 472: symbol=Java_java_io_FileOutputStream_writeBytes; lookup in file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/libjava.so [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libjava.so [0] to /home/worker/workspace/build/src/java_home/jre/lib/amd64/libjava.so [0]: normal symbol `Java_java_io_FileOutputStream_writeBytes' 472: symbol=JVM_Write; lookup in file=java [0] 472: symbol=JVM_Write; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0] 472: symbol=JVM_Write; lookup in file=/home/worker/workspace/build/src/java_home/bin/../lib/amd64/jli/libjli.so [0] 472: symbol=JVM_Write; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0] 472: symbol=JVM_Write; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0] 472: symbol=JVM_Write; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 472: symbol=JVM_Write; lookup in file=/lib/x86_64-linux-gnu/libz.so.1 [0] 472: symbol=JVM_Write; lookup in file=/home/worker/workspace/build/src/java_home/jre/lib/amd64/server/libjvm.so [0] 472: binding file /home/worker/workspace/build/src/java_home/jre/lib/amd64/libjava.so [0] to /home/worker/workspace/build/src/java_home/jre/lib/amd64/server/libjvm.so [0]: normal symbol `JVM_Write' [SUNWprivate_1.1] trouble writing output: Could not initialize NSS which looks fine to my (admittedly amateur) eye.

Nick, any ideas here? I'm sort of at a loss.. I suppose another option is to add some conditional logic to the mozconfig which test whether it's running on CentOS 6.2 and uses the system java in that case (in other words, doesn't set JAVA_HOME and add it to PATH), but defaults to the java from tooltool. I'll try that out now.

(In reply to Dustin J. Mitchell [:dustin] from comment #39) > Nick, any ideas here? I'm sort of at a loss.. > > I suppose another option is to add some conditional logic to the mozconfig > which test whether it's running on CentOS 6.2 and uses the system java in > that case (in other words, doesn't set JAVA_HOME and add it to PATH), but > defaults to the java from tooltool. I'll try that out now. Sorry, I don't have much to add here. I searched a little, and see http://www.linuxquestions.org/questions/slackware-14/unable-to-start-apache-tomcat-4175519172/ and https://bugzilla.redhat.com/show_bug.cgi?id=1022017 I wonder if /etc/nss is mis-configured? Or if we need to install the Java Cryptography extension? (I think not, but http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html in case.) This may also be transient, in the sense that it's specific to the version of the JDK/JRE you are using. You could try rolling back a u-version and see if things improve? You should be able to run the dex command without the rest of the build to test; and the input JAR probably doesn't matter at all. (I can give you a test JAR if you want.)

The beauty of doing this in TaskCluster is, I can just run the build locally in a docker container and then do what I like with it, so no need for a test jar. From an strace, I see it opening nss.cfg from within the tooltool download: [pid 476] open("/home/worker/workspace/build/src/java_home/jre/lib/security/nss.cfg", O_RDONLY) = 10 worker@taskcluster-worker:~/workspace/build/src/obj-firefox/mobile/android/base$ cat /home/worker/workspace/build/src/java_home/jre/lib/security/nss.cfg name = NSS nssLibraryDirectory = /usr/lib64 nssDbMode = noDb attributes = compatibility handleStartupErrors = ignoreMultipleInitialisation Switching that to the ubuntu lib64 location: nssLibraryDirectory = /usr/lib/x86_64-linux-gnu doesn't seem to help. It's looking for the version of NSS present on CentOS 6.2, which isn't here: [pid 671] stat("/usr/lib/x86_64-linux-gnu/libnss3.so", 0x2b3a84e0cd60) = -1 ENOENT (No such file or directory) That's provided by the `libnss3` package on Ubuntu, and indeed installing that package makes things work. I'm a little worried about installing that, though -- especially since it's not required for the Ubuntu-derived JRE/JDK. I don't want Firefox accidentally linking against a system version of libnss.. I'm kind of liking my solution from comment 39 -- it's a hack that will be easy to remove once we're not building on Buildbot anymore (and won't hurt if we forget to remove it, and commented nicely for someone else to remove it when they find it). The downside is, it will be one more way that TC and BB builds differ, so one more moving part when we switch the authoritative build service. Also, this solution works -- https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab283602d446&exclusion_profile=false Which would you prefer?

(In reply to Dustin J. Mitchell [:dustin] from comment #41) > The beauty of doing this in TaskCluster is, I can just run the build locally > in a docker container and then do what I like with it, so no need for a test > jar. > > From an strace, I see it opening nss.cfg from within the tooltool download: > [pid 476] > open("/home/worker/workspace/build/src/java_home/jre/lib/security/nss.cfg", > O_RDONLY) = 10 > > worker@taskcluster-worker:~/workspace/build/src/obj-firefox/mobile/android/ > base$ cat /home/worker/workspace/build/src/java_home/jre/lib/security/nss.cfg > name = NSS > nssLibraryDirectory = /usr/lib64 > nssDbMode = noDb > attributes = compatibility > handleStartupErrors = ignoreMultipleInitialisation > > Switching that to the ubuntu lib64 location: > > nssLibraryDirectory = /usr/lib/x86_64-linux-gnu > > doesn't seem to help. It's looking for the version of NSS present on CentOS > 6.2, which isn't here: > > [pid 671] stat("/usr/lib/x86_64-linux-gnu/libnss3.so", 0x2b3a84e0cd60) = > -1 ENOENT (No such file or directory) > > That's provided by the `libnss3` package on Ubuntu, and indeed installing > that package makes things work. I'm a little worried about installing that, > though -- especially since it's not required for the Ubuntu-derived JRE/JDK. > I don't want Firefox accidentally linking against a system version of > libnss.. I would be surprised if this was an issue, since we build NSS in tree (and this configuration must be common). > I'm kind of liking my solution from comment 39 -- it's a hack that will be > easy to remove once we're not building on Buildbot anymore (and won't hurt > if we forget to remove it, and commented nicely for someone else to remove > it when they find it). The downside is, it will be one more way that TC and > BB builds differ, so one more moving part when we switch the authoritative > build service. Also, this solution works -- > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=ab283602d446&exclusion_profile=false > > Which would you prefer? I have no preference. I know little about these automation configuration issues. If the latter works, call it a day!

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander Bug 1161075: build android with JDK from tooltool; r=nalexander

One more round, since there have been a bunch of changes. I'm particularly interested in feedback on the mozconfig hack -- is that too "dynamic" for a mozconfig?

Comment on attachment 8629463 [details] MozReview Request: Bug 1161075: build android with JDK from tooltool; r=nalexander https://reviewboard.mozilla.org/r/12609/#review11439 Ship It! ::: mobile/android/config/mozconfigs/common:13 (Diff revision 4) > +# In TaskCluster, the Java JRE/JDK are installed from tooltool, but that AFAIK, this is all automation specific: lots of hard-coded /build paths, etc. So you should be good.