b2g_mozilla-inbound_linux64_gecko build on 2013-06-27 01:01:24 PDT for push 59f88c16fca7 slave: bld-linux64-ec2-031 https://tbpl.mozilla.org/php/getParsedLog.php?id=24653499&tree=Mozilla-Inbound { 2013-06-27 05:02:40 (4.40 MB/s) - `xulrunner-24.0a1.en-US.linux-x86_64.sdk.tar.bz2' saved [65344299/65344299] tar xjf xulrunner*.tar.bz2 && rm xulrunner*.tar.bz2 test -d profile || mkdir -p profile run-js-command applications-data build/utils.js:219: ReferenceError: GAIA_APPDIRS is not defined build/utils.js:258: TypeError: Gaia is undefined make[7]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/gaia' make[7]: *** [applications-data] Error 3 make[6]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox/b2g/gaia' make[6]: *** [libs] Error 2 make[5]: *** [libs] Error 2 make[5]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox/b2g' make[4]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox' make[4]: *** [libs_tier_app] Error 2 make[3]: *** [tier_app] Error 2 make[3]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox' make[2]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox' make[2]: *** [default] Error 2 make[1]: *** [realbuild] Error 2 make[1]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build' make: *** [build] Error 2 State Changed: unlock buildroot program finished with exit code 2 elapsedTime=3168.958343 ========= Finished compile failed (results: 2, elapsed: 52 mins, 50 secs) (at 2013-06-27 02:02:52.275564) ========= }

NB this only appears to happen on clobber builds.

b2g_mozilla-central_linux32_gecko_localizer nightly on 2013-06-27 03:12:24 PDT for push 3b955f306226 slave: bld-linux64-ix-004 https://tbpl.mozilla.org/php/getParsedLog.php?id=24655145&tree=Mozilla-Central

https://tbpl.mozilla.org/php/getParsedLog.php?id=24657268&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=24654336&tree=Mozilla-Inbound

Talked to :ochameau in #b2g, he suggested bug 848604 might be the cause of this issue. Moving bug to b2g gaia

See bug 848604 comment 70. At the end, I think that bug 848604 as well as bug 853933 are just exposing some issues with build/test scripts.

Hi Ed, I got a loaner machine, Do you know how to build b2g as same as automation build mechanism?

I'm sorry, I don't. You'll need to ask someone from releng.

Hi Chris, do you know about that?

Basically copy/paste from the log. The B2G desktop builds are just like regular builds, except with a different mozconfig. You can copy b2g/configs/mozconfigs/$platform/nightly to .mozconfig to get started.

after investigated I found some thing strange. I use below build to reproduce bug: https://tbpl.mozilla.org/?rev=56423bbe9a13&tree=Mozilla-Inbound target gaia revision of this build is c59267134f80ccb38353a46c51b210009dba0391 which doesn't include changes of bug 848604 but below command copy another revision of gaia code (include commit of bug 848604) and replaced to build/gaia: mock_mozilla -r mozilla-centos6-i386 --cwd /builds/slave/m-in-linux32_g-000000000000000/build --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" CCACHE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" SYMBOL_SERVER_USER="ffxbld" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_dsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" TINDERBOX_OUTPUT="1" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" MOZ_OBJDIR="obj-firefox" MOZ_CRASHREPORTER_NO_REPORT="1" CCACHE_COMPRESS="1" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build '"'"'MOZ_BUILD_DATE=20130717051921'"'"'' but when I executed this command again then everything is fine. hope this information is useful for tracking this bug.

Aki, Do you have any thoughts about that? The try run investigated by Yuren: https://tbpl.mozilla.org/?rev=56423bbe9a13&tree=Mozilla-Inbound Fails here (fedora opt Bg): https://tbpl.mozilla.org/php/getParsedLog.php?id=25376313&tree=Mozilla-Inbound With an exeption message that proves that gaia/build/utils.js file has bug 848604 revision applied: build/utils.js:219: ReferenceError: GAIA_APPDIRS is not defined (GAIA_APPDIRS is a new variable introduce only by bug 848604) *but* this try run is against this mozilla-central changeset 56423bbe9a13: https://hg.mozilla.org/integration/mozilla-inbound/file/56423bbe9a13/b2g/config/gaia.json { "revision": "c59267134f80ccb38353a46c51b210009dba0391", "repo_path": "/integration/gaia-central" } So we should be having following gaia revision: http://hg.mozilla.org/integration/gaia/file/c59267134f80/build/utils.js And you can see that GAIA_APPDIRS doesn't exists in this revision. There is an obvious revision issue in test runner script, where we end up with mixed up revision between gaia Makefile (that is at root gaia folder) and files in gaia's build/ folder. We can suspect gaia.json to badly update the gaia folder, but the logs seems to say that the correct hg commands are being run without any particular error: command: START command: hg pull -r c59267134f80ccb38353a46c51b210009dba0391 http://hg- internal.dmz.scl3.mozilla.com//integration/gaia-central command: cwd: /builds/hg-shared/integration/gaia-central command: output: pulling from http://hg-internal.dmz.scl3.mozilla.com//integration/gaia-central no changes found command: END (0.84s elapsed) Trying to share /builds/hg-shared/integration/gaia-central to /builds/slave /m-in-linux32_g-000000000000000/build/gaia command: START command: hg share -U /builds/hg-shared/integration/gaia-central /builds/slave /m-in-linux32_g-000000000000000/build/gaia command: cwd: /builds/slave/m-in-linux32_g-000000000000000/build command: output: command: END (0.24s elapsed) command: START command: hg update -C -r c59267134f80ccb38353a46c51b210009dba0391 command: cwd: /builds/slave/m-in-linux32_g-000000000000000/build/gaia command: output: 4512 files updated, 0 files merged, 0 files removed, 0 files unresolved command: END (23.33s elapsed) I tend to think there is something going wrong in this code: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#1213 But I have absolutely no idea how to trigger this code, if we want to reproduce this locally. Chris, Aki, Could you either take a look at this, or find someone to do so, and/or help Yuren to debug gaia's folder update/cloning?

A point of clarification: mozilla-inbound != Try, so this is a standard depend build, not a Try build. I kicked off a staging mozilla-inbound build. The slave had no existing build/ directory in /builds/slave/m-in-linux32_g-000000000000000 . The gaia source step updated to c59267134f80ccb38353a46c51b210009dba0391 . I ssh'ed in, and did an |hg ident| in /builds/slave/m-in-linux32_g-000000000000000/build/gaia and got c59267134f80. An |hg diff| gave me nothing. A |sum build/utils.js| gave me "26242 9" and there's no GAIA_APPDIRS in that file. I'm letting it run to see if something in the build updates gaia mid-build.

[cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ sum build/utils.js 48595 8 [cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ !vi vi build/utils.js [cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ hg ident 20b7d51bd03d+ tip During the vi step, I verified that GAIA_APPDIRS exists in that file. After the |hg ident|, I did an |hg diff|; the diff appears to all be l10n-related. My suspicion has been that gaia l10n has been causing these issues; I think I just verified.

Thanks a lot for looking at this! Are you able to reproduce this "GAIA_APPDIRS is not defined" exception?

(In reply to Alexandre Poirot (:ochameau) from comment #29) > Thanks a lot for looking at this! > > Are you able to reproduce this "GAIA_APPDIRS is not defined" exception? Yes. Stas: any way we can prevent the gaia-l10n makefile step from updating gaia?

Oh would it be because of this command? https://github.com/mozilla-b2g/gaia/blob/master/Makefile#L299 (I'm not a hg expert, does hg update, just reset everything to the tip?) That seems to be executed only when we are using hg instead of git *and* when we pass a LOCALE_BASEDIR variable.

That looks right.

So I think we do want another hg command here. But we should really find resources/time to come up with gaia objdir in order to prevent this kind of hack...

Comment on attachment 778060 [details] pullrequest 11056 Looks good to me, r=me, I think revert is what we want. Using `hg update --clean .` with '.' as revision would have worked, too. The differences between the two are subtle, and according to hg help only affect ongoing merges. http://stackoverflow.com/questions/4681939/revert-vs-update-for-uncommited-changes backs that up.

sry, gotta hide from tbpl. needinfo me if you need further feedback.

master: 4183c938541b4a3aeb8ba67eb81c86f6e35e6625

Thanks again Aki to help us figuring out what was going wrong!!

It looks like the patch that was merged was broken. It had 2 spaces rather than a tab. I just pushed: https://github.com/mozilla-b2g/gaia/pull/11136 https://github.com/mozilla-b2g/gaia/commit/79d65c8ee9a49e6cbeb66cb2586badd7d7665bc4 which replaces the spaces with a tab.