The gaia build process, currently has a set of rules inside the Makefile & build/utils.js which define how all the apps are built / delivered on Firefox OS. The problem with aforementioned system is that it is inflexible, and requires new directories to be added and new logic to be added each time you want to build a slightly different configuraiton.

The basic system I am proposing is the following basic file format which stores the information about the apps being installed. config/dogfood.json or engineering.json { "apps" : [ {"internal" : "apps/bluetooth"}, {"internal" : "apps/browser"},...etc.etc {"external" : "external-dogfood-apps/hoststubtest"}, } The make process reads in this file, and copies everything into a _mkapps/internal or _mkapps/external directory. make pass the GAIA_APP_SRCDIRS two directories always _mkapps/internal & _mkapps/external no more need to propogate further information to utils.js re: production / dogfood. fabrice generally want to know if this design seems generally acceptable, as I am nearing having something workable.

Generally will make things backward compatible by choosing the config file based upon what environment variables are set ..eg production / dogfood, or whichever config file is set as an enviornment variable.

Fabrice, Would like you to consider this patch, it offloads the apps / directory choices to a json file, now having said that there is a lot more we can do with this. Packages of apps, do we want build configuration to be stored in the json file as well. Ability to pull from github for off source apps.

Yuren - I know you've done quite a lot work on partner customizations. Will this patch impact some of the customization work you've built?

Yurenju, If you could annotate bug with your design docs that would be great. Please note this bug is just a general proposal, still ironing out how we want to accomplish this task.

looks good for me. CC Rudy since he is working on bug 838098 for preload apps from GAIA_DISTRIBUTION_DIR. It would be great if we can read config file from GAIA_DISTRIBUTION_DIR/apps.json while it exists. and please set feedback? to me when you submit patch/pull request for landing this work, I will check compatibility with GAIA_DISTRIBUTION_DIR environment variable. thanks!

Looking at the patch that is on bug 838098, I am not really happy with that approach. ---- GAIA_DISTRIBUTION_DIR/apps.json <--- This isn't listed on https://wiki.mozilla.org/B2G/MarketCustomizations#Customization_Overview so my thought is that this is a new requirement, I will add this to my patch to assume apps.json is there, could someone update the wiki so it is a known expectation, because the build will fail if apps.json is not there, and that environment variable is set. (In reply to Yuren Ju [:yurenju] (PTO during 3/15-3/18) from comment #6) > looks good for me. > > CC Rudy since he is working on bug 838098 for preload apps from > GAIA_DISTRIBUTION_DIR. > > It would be great if we can read config file from > GAIA_DISTRIBUTION_DIR/apps.json while it exists. and please set feedback? to > me when you submit patch/pull request for landing this work, I will check > compatibility with GAIA_DISTRIBUTION_DIR environment variable. > > thanks!

Details of this patch: 1) GAIA_DISTRIBUTION_DIR if this is set, it will attempt to use GAIA_DISTRIBUTION_DIR/apps.json 2) If PRODUCTION / DOGFOOD / DEMO is set we will use the associated json file 3) If GAIA_APP_CONFIG is set we will use that. 4) If none of these are set an engineering build with all apps will be created. This should be backward compatible with current usage.

(In reply to dclarke@mozilla.com [:onecyrenus] from comment #8) > Created attachment 725163 [details] [diff] [review] > updated patch uses GAIA_DISTRIBUTION_DIR > > Details of this patch: > 1) GAIA_DISTRIBUTION_DIR if this is set, it will attempt to use > GAIA_DISTRIBUTION_DIR/apps.json > 2) If PRODUCTION / DOGFOOD / DEMO is set we will use the associated json file > 3) If GAIA_APP_CONFIG is set we will use that. > 4) If none of these are set an engineering build with all apps will be > created. > > This should be backward compatible with current usage. Anyhow, is this really a 1.0.1 blocker?

Comment on attachment 725163 [details] [diff] [review] updated patch uses GAIA_DISTRIBUTION_DIR Review of attachment 725163 [details] [diff] [review]: ----------------------------------------------------------------- ::: Makefile @@ -86,3 @@ > endif > > -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),) What happened to GAIA_OUTOFTREE_APP_SRCDIRS? We uses this here to support apps that are not sitting within the gaia/ subdirectory. Is this still supported via other means?

Comment on attachment 725163 [details] [diff] [review] updated patch uses GAIA_DISTRIBUTION_DIR Review of attachment 725163 [details] [diff] [review]: ----------------------------------------------------------------- ::: Makefile @@ -86,3 @@ > endif > > -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),) What happened to GAIA_OUTOFTREE_APP_SRCDIRS? We uses this here to support apps that are not sitting within the gaia/ subdirectory. Is this still supported via other means?

Comment on attachment 725163 [details] [diff] [review] updated patch uses GAIA_DISTRIBUTION_DIR Review of attachment 725163 [details] [diff] [review]: ----------------------------------------------------------------- ::: Makefile @@ -1,1 @@ > -############################################################################### nit: no need to remove that line @@ -86,3 @@ > endif > > -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),) I agree, we need a way to let people add apps from somewhere else. @@ +233,5 @@ > + type=$${A[0]}; \ > + app_name=$${A[1]}; \ > + cp -rf $$app_name $(MAKE_APPS_DIR)/$$type/ ; \ > + done; \ > + ln -s $(CURDIR)/shared $(MAKE_APPS_DIR)/ Why do we need that? ::: build/read-configs.py @@ +18,5 @@ > + data = json.load(json_data) > + json_data.close() > + for app in data['apps']: > + for key, value in app.iteritems(): > + print key + ':' + value nit: trailing ws ::: configs/dogfood.json @@ +27,5 @@ > + {"internal" : "showcase_apps/crystalskull"}, > + {"internal" : "showcase_apps/cubevid"}, > + {"internal" : "showcase_apps/twittershare"} > + ] > +} I'm not a big fan of this json structure. What about something like: { "internal": ["apps/bluetooth", "apps/browser", ...], "external": ["external-apps/marketplace", "external-apps/m.here.com", ...] }

(In reply to Fabrice Desré [:fabrice] from comment #12) > Comment on attachment 725163 [details] [diff] [review] > updated patch uses GAIA_DISTRIBUTION_DIR > > Review of attachment 725163 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: Makefile > @@ -1,1 @@ > > -############################################################################### > > nit: no need to remove that line > > @@ -86,3 @@ > > endif > > > > -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),) > > I agree, we need a way to let people add apps from somewhere else Wouldn't the way to add an app to the build, be to edit the json file, and add it there. You can maintain your own json file for the apps you want to push to the device. ? > > @@ +233,5 @@ > > + type=$${A[0]}; \ > > + app_name=$${A[1]}; \ > > + cp -rf $$app_name $(MAKE_APPS_DIR)/$$type/ ; \ > > + done; \ > > + ln -s $(CURDIR)/shared $(MAKE_APPS_DIR)/ > > Why do we need that? > This is part of the build process that is already there, ex: look at the gallery app Makefile, it includes resources from ../../shared/js.... so possibly would need to pass in a shared directory location to the makefile in order for the process to work as expected. > ::: build/read-configs.py > @@ +18,5 @@ > > + data = json.load(json_data) > > + json_data.close() > > + for app in data['apps']: > > + for key, value in app.iteritems(): > > + print key + ':' + value > > nit: trailing ws > > ::: configs/dogfood.json > @@ +27,5 @@ > > + {"internal" : "showcase_apps/crystalskull"}, > > + {"internal" : "showcase_apps/cubevid"}, > > + {"internal" : "showcase_apps/twittershare"} > > + ] > > +} > > I'm not a big fan of this json structure. What about something like: > > { > "internal": ["apps/bluetooth", "apps/browser", ...], > "external": ["external-apps/marketplace", "external-apps/m.here.com", ...] > } Can change the structure, that is fine. Including out of src apps, i generally feel that this should be done through the json file mechanism...

(tef- as I won't see why this is mandatory for v1.0.1. Please feel free to re-nom with justification if desired)

Michael, Purpose of this patch is to allow partners to be able to customize which apps go on the phone. imho should be considered part of the device customization framework. Unsure what our partner requirements are, but if they include deciding what apps to deploy on the device, then i think this is tef+. Currently the only mechanism available is to delete the apps from the gaia source tree, and build / deploy.

Thanks. "Unsure what our partner requirements are" is a little scary because it implies we don't know that what we are building will ultimately be useful for v1.0.1. Where are the requirements for this bug coming from then? Also GAIA_OUTOFTREE_APP_SRCDIRS is one mechanism that is currently available to add new apps outside of the gaia/ directory, but looks like this is being removed in this patch? This will cause churn on v1.0.1 that I'd like to avoid if at all possible.

Michael, So to answer your question for GAIA_OUTOFTREE_APP SRCDIRS, you can include whichever apps you want inside the json file. The other part to GAIA_OUTOFTREE apps that doesn't work, and was the reason for me removing it is there is no ability to have an external out of gaia tree app. So you end up with a name that hints you can include any kind of out of tree app, but in reality only handles internal apps. My perspective is that GAIA_OUTOFTREE_APP_SRCDIRS is confusing and generally masks the problem, which is as an app developer i need a way to push my internal / external app to the device. I believe this process should be separate from gaia make.

I don't think Michael uses GAIA_OUTOFTREE_APP_SRCDIRS to push apps to the device, but to build images that include apps that are not part of gaia. That is a very legitimate use case, and we really need to support that with as few changes on their side as possible.

Fabrice, do you mean no changes from their side or as few changes from their side. This functionality is still supported, but you'd maintain the list of apps you wanted on device inside the json file, and they can most definitely be out of the src directory. For expediency sake i'll just add this back in.

(In reply to dclarke@mozilla.com [:onecyrenus] from comment #19) > Fabrice, do you mean no changes from their side or as few changes from their > side. As few as possible. > This functionality is still supported, but you'd maintain the list of apps > you wanted on device inside the json file, and they can most definitely be > out of the src directory. Can you provide an example? That would help a lot I think.

(tef-, again :( Sorry guys, I'm really not feeling this at all right now. We are still talking about perturbing the v1.0.1 build for a new feature with unclear requirements. That makes three reasons to take a pass at this point.)

Comment on attachment 725163 [details] [diff] [review] updated patch uses GAIA_DISTRIBUTION_DIR clean feedback flag because this patch got r-. set again when you have another patch.

Fixed what i believe to be the general concerns #1) It is possible for you to include out of source tree apps using environment variables. Ex: export GAIA_OUTOFTREE_APP_SRCDIRS=/media/b2g/twittershare2/ Symlinks the out of source app try into your make directory _mkapps/internal #2) It is possible for you to include out of source tree apps by editing the configuration json file ex: .... "apps/video", "/home/myuser/myapp/" ] This will copy the app into the build directory into _mkapps/internal each time you rebuild. Slight semantic differnce, but wanted to preserve all previous notions of the environment variable. (My belief is that apps are small enough they should be copied into _mkapps/internal. As you will then have a record of exactly what was pushed to device even if you are mucking around in the source tree. #3) It is possible to include apps by linking to their url. "external" : [ "external-apps/marketplace", "external-apps/m.here.com", "http://marketplace.firefox.com/manifest.webapp"] Creating a dummy metadata.json file with the origin set. #4) Also this patch moves the entire build process to a build directory, so that the make process isn't touching source.

Will also run this through try tonight, and post the results.

(In reply to dclarke@mozilla.com [:onecyrenus] from comment #24) > Will also run this through try tonight, and post the results. Gaia changes don't have a mechanism to be pushed to the try server ....

Same patch as before but in the form of a gaia pull request.

removed the need to specify external or internal in the configuration file.

This last patch basically subsumes the work done by yurenju in the below commit https://github.com/yurenju/gaia/commit/a62d990266c6f3661a5c712bc5d52eb49d6cdf52

Any eta on this ?

Hey David, sorry for being late there. I'll try to do it this week. I like a lot the idea of moving everything out to a build/ directory btw.

Hi dclarke, Yuren and I had a lengthy conversation to sync up what we had done v.s. your approach here. I am looking into your work right now to see if it fits all the requirements, especially around distributions. Our approach in Taipei was progressive enhancement -- land changes in different bugs one-by-one to ensure everyone could follow up -- but if you have already done all the changes and we could still make sure it's safe enough for v1.0.1, then it doesn't make sense not to land it.

Tim, That is great, would appreciate any extra eyes on this bug.

Comment on attachment 730956 [details] pointer to gaia pull request It looks like Tim started to review on github

I wrote what we plan for this bug on mozilla wiki: https://wiki.mozilla.org/B2G/MarketCustomizations#Plan feel free to update wiki page when this feature available :)

Hi all, I can help look into this to see what's left to do if no one is actively working on this. Please take this back if any concerns. Thanks.

Rudy, sounds good this patch got stuck in review, so probably a fresh pair of eyes and a different perspective is what is needed.

steal from Rudy ;-)

Don't forget to update https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Gaia/Hacking after this lands.

Pointer to Github pull-request

Comment on attachment 763428 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427 There are two majar changes by this pull request: 1. use build/apps-[production|engineering|dogfood|demo|.list for picking apps. 2. use GAIA_DISTRIBUTION_DIR/apps.list if the environment variable defined

remind myself: we also need to update https://wiki.mozilla.org/B2G/MarketCustomizations

you also can use specific app list to pick apps, download the attachment and type: GAIA_APP_CONFIG=<path-to-apps.list> make reset-gaia you will have a gaia only contains system, homescreen & settings app :)

Comment on attachment 763428 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427 The patch looks generally alright as we have addresses all the use cases we could think of/find. Please add the part we discussed so we could backward-compatible GAIA_APP_SRCDIRS with a warning message.

PR updated accroding Tim's review

Comment on attachment 763428 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427 I'm fine with what Tim and Alex think are OK with. The current system does not scale in many situations (mostly my bad) so do whatever you think is smarter :)

Comment on attachment 763428 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427 Thanks for this great work and sorry for the delay in reviewing the patch. I spent some time testing various environment and it appears to break Windows support (yes, some people build gaia on Windows ;)). That would be really great if we can keep it running on this platform.

I'll find a windows VM to test it and thank you for investigating on windows!

I have updated PR, but we still need James's feedback on Pull Request page. also set ni? here. https://github.com/mozilla-b2g/gaia/pull/10427#discussion_r4862925

Looks good to me!

Comment on attachment 763428 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427 Thanks for the fixes, you should go ahead and land this. Do not forget to watch TBPL results, this is the kind of changes that can easily break test environments!

I'll watch TBPL for this and Askeing will help us to verify on Linux & Windows again. Thank you Alex, Tim & Vivien! Merged! https://github.com/mozilla-b2g/gaia/commit/f1a33dc54c644dd07e5503a0de66c7ba0b7679d9

I think this change caused bug 887761

(In reply to Kim Moir [:kmoir] from comment #53) > I think this change caused bug 887761 Have reverted this for now: https://github.com/mozilla-b2g/gaia/commit/b716b8391a8c9ad1ed97a9abc326c8edeb992556 The manifest auto-updater will soon check into birch, we'll need to wait for a desktop B2G clobber build after that to confirm it has fixed it.

Thank you Kim & Ed, I'm investigating. do we have the way to test on TPBL before merge to gaia master?

I talked to Hal, he said you could specify a different gaia rev in the gaia.json file (in gecko/firefox) and then run a build on try.

I can't reproduce on my local machine by: - make -C <GAIA_DIR> profile - <B2G_DIR>/build.sh - pull gecko from mozilla-central & build with a mozconfig from https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Using_the_B2G_desktop_client Kim, Where can I find build script for TPBL? I would like to reproduce it on local machine.

I got it. http://hg.mozilla.org/build/mozharness/file/84187194b050/scripts/b2g_build.py

actually I don't have gaia rev before push to gaia master. talked to kanru and he said we may use repo_path.

this is the same Pull request as attachment 763428 [details] but rebase to master & resolved some conflict. I can't reproduce TBPL error on bug 887761, and then I triggered those TBPL build again I got another errors about webapp-optimize. - https://tbpl.mozilla.org/?rev=3b955f306226 - https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound - https://tbpl.mozilla.org/?rev=e5f664c8b66d&tree=Mozilla-Inbound Alexandre & Tim, Do you have any suggestion?

The following error: build/webapp-optimize.js:487: ReferenceError: GAIA_CONCAT_LOCALES is not defined is most likely due to merge issue against bug 853933. But looking at your pull request, this GAIA_CONCAT_LOCALES seems to be correctly set.

I am not familiar with TPBL :'( Jonathan, would you be able to help out and take a look of the log?

Note the error only occurred on clobber builds.

I'm not sure Jonathan can really help us figuring out what is being broken in gaia build system only on tbpl. I think what would help is giving access to Yurun access to a build slave and help him build gaia the same way our automated scripts do. It should be quite easy to figure out what is going wrong once we reproduce it and can instrument gaia.

sounds helpful if I can access to build slave and reproduce this bug :D and I'm also clone tbpl and try to setup environment, maybe I can reproduce on my local machine.

(In reply to Yuren Ju [:yurenju] from comment #65) > and I'm also clone tbpl and try to setup environment, maybe I can reproduce > on my local machine. TBPL is just a dashboard for displaying results, so isn't what you want to set up locally. Your best bet is to request access to a releng machine. See: https://wiki.mozilla.org/Platform/2013-07-02#Automation_Corner_.28ctalbert.2Fjoduinn.29

Having access to a build slave won't help you produce the builds in the same way that buildbot does. It would be quite difficult to try and reproduce that locally, even using a build slave, I believe, unless the error is just a matter of the build slaves being "slow" compared to the typical developer machine. I'm cc'ing some rel-eng folks who are involved in the builds, maybe they can help. catlee, do you have any idea about what's going on with the B2G build error in e.g., https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound

bug 892318 filed for requesting a releng machine.

Oh, there is really something weird with tbpl tests. I'm seing the GAIA_CONCAT_LOCALES exception on a push-to-try (that has nothing to do with gaia!): https://tbpl.mozilla.org/?tree=Try&rev=736c69ef34dd Can it be that bug 853933 regressed tbpl, but that we don't see it unless pushing manually to try???

Ok, I don't know what is going on with the test runner, but something bad happens. This following run: https://tbpl.mozilla.org/?tree=Try&rev=736c69ef34dd is based on an old gecko, around June 25th, its gaia.json targets a gaia revision around the same date. But we are seeing an exception about GAIA_CONCAT_LOCALES, that is something brand new, that only exists in gaia since bug 853933 landed on July 4th. There is no way to see this tring appear on a gaia from June 25th. I have no idea what exactly happens, but it looks like gaia/Makefile file is from June 25th, but all files from gaia/build/ folder is from master (or from a revision after July 4th) I've just pushed the same gecko patch but against a recent gecko/gaia just to see what will happen: https://tbpl.mozilla.org/?tree=Try&rev=b71104170ebf Given these strange behaviors, I don't think it has anything to do with your patch Yuren.

sounds not my fault :D Ed, can I land this commit again?

(In reply to Yuren Ju [:yurenju] from comment #71) > sounds not my fault :D > > Ed, can I land this commit again? If the issue is fixed, sure :-) (Though there aren't b2g desktop builds in the comment 70 try run...)

> catlee, do you have any idea about what's going on with the B2G build error > in e.g., > > https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound I don't sorry. That looks like it's deep in the gaia build system, which I don't know much about.

I have same conclusion as Alexandre said on comment 70, bug 887761 looks like a test runner bug because Makefile and scripts in build/ came from different date, that's why I can't reproduce bug 887761 on my local machine even I use the same commit hashtag from gaia-integration with mercurial. remove dependency with bug 887761 and I'll land this soon.

(In reply to Yuren Ju [:yurenju] from comment #74) > I have same conclusion as Alexandre said on comment 70, bug 887761 looks > like a test runner bug because Makefile and scripts in build/ came from > different date, that's why I can't reproduce bug 887761 on my local machine > even I use the same commit hashtag from gaia-integration with mercurial. > > remove dependency with bug 887761 and I'll land this soon. Note even if this is a test runner bug, if this patch is enough to trigger it, then we'll have to backout again, removed dependency or not.

Merged again. https://github.com/mozilla-b2g/gaia/commit/b54c694

got new result. now we have webapp-optimize error but not applications-data. https://tbpl.mozilla.org/?rev=3b955f306226

(In reply to Yuren Ju [:yurenju] from comment #77) > got new result. > > now we have webapp-optimize error but not applications-data. > > https://tbpl.mozilla.org/?rev=3b955f306226 That TBPL link is from June?

Oh sorry I gave a wrong link. I thought retrigger build process will get recent gaia from '/integration/gaia-central', but looks not.

With this change applied, I can't have apps as symbolic links anymore (it was working before). That is, I used to be able to do: cd apps; ln -s /whereverMyNewAppIs/myappDir .; cd ..; make profile and I would get a myappDir.gaiamobile.org app included on the profile. Now it doesn't work.

Hi Antonio, could you file a bug for it and CC me? thanks!

(In reply to Yuren Ju [:yurenju] from comment #81) > Hi Antonio, could you file a bug for it and CC me? thanks! Created bug 902402.