Following from bug 1182116. Even with the right env variables to make an engineering build, you can't have the spark apps and the engineering apps within the same build. I wanted to investigate more, but I got stuck with bug 1187294. I'll investigate the makefile.

Each configuration is defined in a list, like this one[1]. The path of this file is evaluated at [2] in the variable called GAIA_APP_CONFIG. But, in the case of spark, we override this variable at [3], by that app list[4] Potential solutions: 1. Either we create a new app list, like [4]. This will introduce more duplication 2. Or we change the makefile to load 2 app lists if we need to 3. Or we change [5], to allow to import some other .list files. [1] https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/build/config/phone/apps-engineering.list [2] https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/Makefile#L356 [3] https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/Makefile#L392 [4] https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/distros/spark/apps.list [5] https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/build/scan-appdir.js

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master I started with the first solution. This is not the most beautiful one, but this will unlock 17 tests on Aries. Ricky, do you think we can go with this first workaround?

Could you tell me what's the real use case about that? If you just want to override app.list with your specific app.list base on different env flags at https://github.com/mozilla-b2g/gaia/blob/29f363d6236bf7db8141d7a1f1185a1dcd809bf7/Makefile#L392. Try to modify GAIA_APP_CONFIG := $(DISTRIBUTION_APP_CONFIG) to GAIA_APP_CONFIG ?= $(DISTRIBUTION_APP_CONFIG) Would it meet you requirement?

The use case for automation is: To have, on the same build, the engineering and dogfood apps on Aries. At the moment we can't run 17 tests on Aries because the engineering apps are not present on any aries build. If I understand your proposal correctly, we'll only have the engineering apps, because we won't override "GAIA_APP_CONFIG" with the the spark distro. Is that right?

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #5) > The use case for automation is: To have, on the same build, the engineering > and dogfood apps on Aries. At the moment we can't run 17 tests on Aries > because the engineering apps are not present on any aries build. > > If I understand your proposal correctly, we'll only have the engineering > apps, because we won't override "GAIA_APP_CONFIG" with the the spark distro. > Is that right? Yes, commont 4 would be a solution for you so that you can try to specify GAIA_APP_CONFIG=xxx in Aries build script

I'm not exactly sure what's behind the solution you proposed, but it works! I updated the PR. Manual testing done: * make reset-gaia => OK. Regular and engineering apps are present on the homescreen. * GAIA_DISTRIBUTION_DIR=distros/spark make reset-gaia => OK. Regular + Spark + Engineering apps are on the homescreen. I started some of them, they run. * PRODUCTION=1 make reset-gaia => OK. Only the regular apps are displayed. * PRODUCTION=1 GAIA_DISTRIBUTION_DIR=distros/spark make reset-gaia => Regular + Spark apps are shown. Is there an automated test I can add, Ricky?

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #7) > Is there an automated test I can add, Ricky? We will appriciate if you could do that! I wonder why your manual testing works lol. My proposal recommends you to set GAIA_APP_CONFIG=your-app.list. So you could try `make clean` every time when you invoke any type of make command.

(In reply to Ricky Chien [:rickychien] from comment #8) > We will appriciate if you could do that! I wish I could, but I'd like to know what we currently have to test the makefile itself. Do we already have some tests running, so I can add the same type of test? > So you could try `make clean` every time when you invoke any type of make command. I tested again with `make clean && make really-clean &&` in front of every command I mentioned in comment 7. They all work like I expect. Could you try the PR on your side, so we're sure that there's nothing wrong with my set up?

I was afraid my patch was not related to the results I saw, so I performed some more testing. Without the patch, if I run: * make clean && make really-clean && GAIA_DISTRIBUTION_DIR=distros/spark make reset-gaia => I only see the regular + spark apps. I don't see any engineering app. Hence, the patch actually does something, but I'm not familiar with the Makefile enough, to know what rules we trigger.

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master Just add an integration test would be fine. See an example at https://github.com/mozilla-b2g/gaia/blob/master/build/test/integration/distribution_test.js Please set r?=rchien again once your integraion test is ready.

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master I have a first version of the test. There are 3 main issues in it: * The tests take a long time before the feedback is given (at least 1 minute per test). This is due to the fact that we do the whole compilation and also that we download the spark apps each new time. * We don't see the engineering apps in config.grid. * 4 of the spark apps (Customizer, Hackerplace, Studio, Sharing) return null in config.grid. I'm not exactly sure if I'm testing it the right way, then. What do you think, Ricky?

That test looks very great! LGTM for that patch. However, I forgot there are some network issues in our CI infra which is probably unstable if we're trying to access network resources during testing. Here are two questions about network in our CI: 1. Is that ok to download third-party NPM package now (ex: lodash) without updating gaia-node-modules? 2. For the case of spark apps (ex: twitter, facebook..etc), is that stable enough to access these online apps in our integraion test? Gareth, do you know about that? thanks!

Note to myself: Check that FM radio is NOT present in when PRODUCTION=1 and spark is there (Naoki's idea).

I just tested this scenario, the FM radio is present between SWOOOPS and Firesea IRC.

(In reply to Ricky Chien [:rickychien] from comment #13) > That test looks very great! > > LGTM for that patch. However, I forgot there are some network issues in our > CI infra which is probably unstable if we're trying to access network > resources during testing. > > Here are two questions about network in our CI: > > 1. Is that ok to download third-party NPM package now (ex: lodash) without > updating gaia-node-modules? Yep! gaia-node-modules is dead forever. > 2. For the case of spark apps (ex: twitter, facebook..etc), is that stable > enough to access these online apps in our integraion test? > > Gareth, do you know about that? thanks! You shouldn't assume online access for integration tests. It'll be available in some places (like gaia-try) but it won't be around on mozilla-inbound I believe.

Unfortunately, due to unavailability of online access for integration tests, the better solution for tests would keep it in codebase but disable by default on CI server. Let's put [Network Require] keyword in test suite description, namely: suite('Specidfic apps should be present in given build configuration [Network Require]', ... to notify CI server to skip this test but it is able to run locally.

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master I investigated a bit more the reasons of the issues found in comment 12: * Concerning the network, I added [Network Require] * About the engineering apps not present on the homescreen config file. I did a recursive grep and they are present nowhere. Then, we can't perform an integreation test this way. * Concerning the 4 apps showing null, it was an actual bug of my changes. In fact, these apps were not present anymore on the homescreen. And like said in comment 15, FM radio was back in Spark. Hence, I prefer to not use the ?= operator. I'm not exactly sure why these apps disappeared and FM radio is back. Then I'd prefer to switch back to the simpler solution I proposed. I did some more testing with this patch, we have the apps we want. I also updated the tests to no verify the engineering apps, as we have no way to test them, at the moment. What do you think, Ricky?

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master If there is still some problems in this mechanism, I'm fine to fall back to your original patch. I don't see [Network Require] and other changes from your patch (it's likely you forgot to update the patch?) Please make sure update your patch and r?=me again =)

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master Oops, sorry about that. Here's the updated PR.

Comment on attachment 8657818 [details] [gaia] JohanLorenzo:bug-1187330 > mozilla-b2g:master LGTM.

Landed in master at https://github.com/mozilla-b2g/gaia/commit/6a5ee1f361ebc48355aa7f233ee22bbdd0fbfd63. jshint was passing locally but not on treeherder. I reverted it immediately: https://github.com/mozilla-b2g/gaia/commit/1f226126dcba8af16c8db08556130b0bb04bb8d3

Green build with no JSHint errors: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=f94285f9fbce75f9a2a8c86d0b22b63aca734789 Landed in master at https://github.com/mozilla-b2g/gaia/commit/0f59a94987f15f87f0b0115d53bdad4cb6ace57e

sorry had to revert this, seems this cause bustages like https://treeherder.mozilla.org/logviewer.html#?job_id=2955804&repo=b2g-inbound

Based on [1], it only fails the build of b2g-desktop on Linux. I'll look into it. [1] https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=9c2190137b04

https://treeherder.mozilla.org/logviewer.html#?job_id=2958201&repo=b2g-inbound too (OS X) too it seems

The logs show a file is not found. I tried to reproduce the build error locally, but when I try to build gecko, I get another compilation error, early at the beginning. Ricky, have you ever seen this kind of issue before? > 22:50.17 2015-10-06 09:51:12 URL:http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2015/03/2015-03-05-16-02-02-mozilla-central/b2g-39.0a1.multi.linux-x86_64.tar.bz2 [99820869/99820869] -> "b2g-39.0a1.multi.linux-x86_64.tar.bz2" [1] > 23:00.46 Exception: Error: -*- build/utils.js: Invalid file path () > 23:00.46 [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsILocalFile.initWithPath]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js :: getFile :: line 145" data: no] > 23:00.46 > 23:00.46 getFile@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js:160:11 > 23:00.46 exports.execute@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/app.js:116:18 > 23:00.46 CommonjsRunner.prototype.run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:168:5 > 23:00.46 run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:183:3

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #28) > > 22:50.17 2015-10-06 09:51:12 URL:http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2015/03/2015-03-05-16-02-02-mozilla-central/b2g-39.0a1.multi.linux-x86_64.tar.bz2 [99820869/99820869] -> "b2g-39.0a1.multi.linux-x86_64.tar.bz2" [1] > > 23:00.46 Exception: Error: -*- build/utils.js: Invalid file path () It shows the path is empty > > 23:00.46 [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsILocalFile.initWithPath]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js :: getFile :: line 145" data: no] > > 23:00.46 > > 23:00.46 getFile@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js:160:11 > > 23:00.46 exports.execute@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/app.js:116:18 It seems that options.STAGE_DIR is empty at gaia/build/app.js line 116, but I don't have idea currently why 3 lines of Makefile changes caused build bustage... > > 23:00.46 CommonjsRunner.prototype.run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:168:5 > > 23:00.46 run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:183:3 How to reproduce it? It passed on my Mac after trying your patch and `make` to build gaia like as treeherder log

With the help of :mrrrgn and folks, I set up a b2g-desktop build on TC, that contains the patch I had before the backout[1]. It has been built thanks to Taskcluster-Github[2]. The configuration used is [3]. On that set up, the build error doesn't show up[4]. I hope this is close enough from the production b2g-desktop builds. I'm wondering if the file missing error, was not just that the workspace wasn't in a clean state that day. Tomcat, do you think it's possible to try to land the Gaia patch and to watch if the build error is still there? [1] https://github.com/mozilla/taskcluster-test-gaia/pull/2 [2] http://docs.taskcluster.net/services/taskcluster-github/ [3] https://github.com/mozilla/taskcluster-test-gaia/blob/tc-gh-base/.taskcluster.yml [4] https://tools.taskcluster.net/task-graph-inspector/#WnZBuMKGRRaFv1-tU3aoeQ/VNGuvUBdRKyjJSbjgb5WlQ/0

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #30) > With the help of :mrrrgn and folks, I set up a b2g-desktop build on TC, that > contains the patch I had before the backout[1]. It has been built thanks to > Taskcluster-Github[2]. The configuration used is [3]. > > On that set up, the build error doesn't show up[4]. I hope this is close > enough from the production b2g-desktop builds. I'm wondering if the file > missing error, was not just that the workspace wasn't in a clean state that > day. > > Tomcat, do you think it's possible to try to land the Gaia patch and to > watch if the build error is still there? > > [1] https://github.com/mozilla/taskcluster-test-gaia/pull/2 > [2] http://docs.taskcluster.net/services/taskcluster-github/ > [3] > https://github.com/mozilla/taskcluster-test-gaia/blob/tc-gh-base/. > taskcluster.yml > [4] > https://tools.taskcluster.net/task-graph-inspector/#WnZBuMKGRRaFv1-tU3aoeQ/ > VNGuvUBdRKyjJSbjgb5WlQ/0 sure we can try :) which PR do you want get landed both ?

PR landed in master at: https://github.com/mozilla-b2g/gaia/commit/7a67098e9150509468df612261ede16341035294 I warned the current sheriff, KWierso, on IRC.

I'm reverting this for some weird build failures: https://queue.taskcluster.net/v1/task/LQ4u2GHeR3WbixZ_3V5YyA/runs/0/artifacts/public/logs/live_backing.log https://github.com/mozilla-b2g/gaia/commit/f9b6b87d9f3af8db9e5d4c5351cc0809582396df 78[1G[J[34m 0:46.45(B[m /home/worker/workspace/gecko/b2g/branding/unofficial/locales/en-US/brand.properties: WARNING: no useful preprocessor directives found 78[1G[J[34m 0:46.75(B[m /home/worker/workspace/gecko/b2g/locales/en-US/chrome/overrides/aboutCertError.dtd: WARNING: no preprocessor directives found 78[1G[J[34m 0:46.75(B[m /home/worker/workspace/gecko/b2g/locales/en-US/chrome/overrides/appstrings.properties: WARNING: no useful preprocessor directives found 78[1G[J[34m 0:47.01(B[m gmake[6]: warning: -jN forced in submake: disabling jobserver mode. 78[1G[J[34m 0:47.40(B[m gmake[6]: warning: -jN forced in submake: disabling jobserver mode. 78[1G[J[34m 0:47.59(B[m Test SDK directory: /home/worker/workspace/gaia/b2g_sdk/39.0a1-2015-03-05-16-02-02 78[1G[J[34m 0:47.83(B[m Exception: Error: -*- build/utils.js: Invalid file path () 78[1G[J[34m 0:47.83(B[m [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsILocalFile.initWithPath]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js :: getFile :: line 145" data: no] 78[1G[J[34m 0:47.83(B[m 78[1G[J[34m 0:47.83(B[m getFile@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/utils-xpc.js:160:11 78[1G[J[34m 0:47.83(B[m exports.execute@resource://gre/modules/commonjs/toolkit/loader.js -> file:///home/worker/workspace/gaia/build/app.js:116:18 78[1G[J[34m 0:47.83(B[m CommonjsRunner.prototype.run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:168:5 78[1G[J[34m 0:47.83(B[m run@/home/worker/workspace/gaia/build/xpcshell-commonjs.js:183:3 78[1G[J[34m 0:47.83(B[m @-e:1:1 78[1G[J[34m 0:47.83(B[m 78[1G[J[34m 0:47.85(B[m gmake[6]: *** [app] Error 1 78[1G[J[34m 0:47.85(B[m gmake[5]: *** [libs] Error 2 78[1G[J[34m 0:47.85(B[m gmake[4]: *** [b2g/gaia/libs] Error 2 78[1G[J[34m 0:47.85(B[m gmake[3]: *** [libs] Error 2 78[1G[J[34m 0:47.85(B[m gmake[2]: *** [default] Error 2 78[1G[J[34m 0:47.85(B[m gmake[1]: *** [realbuild] Error 2 78[1G[J[34m 0:47.85(B[m gmake: *** [build] Error 2

Let's see if try can show off the failure: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae8823bfcf93

The try build in comment 35 passed. Naoki pointed out that it might only happen on b2g-inbound[1], it passed too[2]. It doesn't seem related to the tree I'm building on. Hence, I'm guessing there are differences between the way my custom build are done, and the one that reports to Treeherder. Morgan, would you know who I can ping to know the differences between a build on try and a build on b2g-inbound? [1] https://github.com/mozilla/taskcluster-test-gaia/blob/71afcc3c170d0728b2a976125f854de5f6a20d90/.taskcluster.yml#L34 [2] https://tools.taskcluster.net/task-graph-inspector/#b_uPL7zxSaS_IKIVyEf7zA/UHN3Eq1fRN2km3SPo7Kjog/0

From what I'm seeing now, this must be an issue with the environment variables. Dustin would know best about how this works, but I'll see what differences I can find and report back.

Ahh, I see the problem: >> [taskcluster-vcs] run end : hg update -C default (0) in 4083 ms The TaskCluster GitHub job that's been setup here needs environment variables so that the build script knows what revision to check out. It's just using default now.

In short, http://hg.mozilla.org/mozilla-central/file/9605da94e75d/testing/docker/builder/bin/checkout-gecko is trying to checkout revisions using environment variables which aren't defined in the job. Also, it's trying to use HG. Because the job runs against GitHub pushes and pull requests, it doesn't have any way of knowing about hg revisions. you'll want to remove checkout-gecko and check out the project yourself in the job's run command.

This patch turned out to be harder to debug, than I expected. Moreover, the spark distro is supposed to disappear, at some point. Bug 1196992 is showing good progress in order to make the spark apps available in the regular tree. Bugzilla Lite has already landed in the regular build, for instance. This is currently the only spark app we perform end-to-end test against. I discussed with Naoki ans we agreed that there was no point in continuing to debug this temporary use case. Let's close this bug as WONTFIX. To get these apps, we have to change the build on TaskCluster. In this regard, I filed bug 1218452.