Per Candice Serran's email: Closing the loop on this from discussion in-person with Wilfred, Doug, Tim, and Ken. Plan for 2.5: Doug to implement bumper bot solution for Bugzilla Lite and Buddy Up Tim to review Estimate: 2-3 days of engineering development

Patryk, where should these apps go on the Homescreen?

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master For reference only. Holding off on review until comment 1 is answered.

Technically app hierarchy is a product question, so since this is for an official (2.5) release. Wilfred would be the best to comment on comment 1.

as sent via email: Dialer SMS Contacts Email Browser Camera Gallery Music Video Marketplace Calendar Clock Settings FM Vaani (TBC) Webmaker Buddy Up Bugzilla Lite Facebook Twitter Notes Calculator Costcontrol

Rob, Wilfred has asked via email for you to make the final decision on the Homescreen layout. Would you like to change anything in comment 6?

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. The only comment I have is that we'll want to add all the apps that wilfred mentioned in comment 6 that isn't in gaia currently. (or whatever the final list is )

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. Needs to be updated with Jacqueline's answer.

Here's a conversation I had with sfoster about using late customization instead of bumper bot for pulling hosted and/or Marketplace apps: http://logs.glob.uno/?c=mozilla%23gaia&s=27+Aug+2015&e=27+Aug+2015#c152607

Via email from Rob Macdonald [:robmac], > The order of the list that Wilfred has proposed generally makes sense to me. > We may want to tweak the order slightly depending on (a) whether app grouping > is available for the default home screen and (b) whether we ship with the > three column or four column grid. > > Also, I assume this is the default on master but that partners can customize > based on the needs of their users. > > Jacqueline came up with the current default order in 2.0 and may have more to > add. > > Thanks! > Rob Via email from Jacqueline Savory [:jsavory], > This arrangement looks good to me as well, the priority of the apps and > grouping of them seems appropriate so I wouldn't make any major changes. > > The only tweak I might make would be after we have confirmation between the > three or four column grid, as it might change which apps are on a row together. > Currently, I think this looks good on the three column grid, but could use a > little moving around for the four. > > Thank you!

Thanks Doug for updating the bug with the email feedback. Removing ni?, but feel free to flag me again if the default icon set is four columns instead of three.

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. I am not in charge of bumper bots.

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. Kevin has historically reviewed Spark-related patches like this for me.

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master For hosted apps, I prefer to check in as folder rather than binrary files, and do package during build time. Bumper bot will regularly pull new binrary files into gaia that would increase the repo size very fast.

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. I feel like a build peer needs to review this as it feels like it's part of the build to me. I also don't know bash too well, and personally would prefer a node solution (as most gaia developers are JS hackers). Going to flag Ricky on this one instead if that's ok.

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master I tried unrolling the zip files that 'preload.py' downloads, but there are several problems with that: * 'metadata.json' no longer does anything, so we don't get updates for the preloaded apps via Marketplace. * The manifests include `package_path` fields, so we'd have to parse them and strip this out. While we could do that, it's very likely that this will break in the future. * The app origin is broken, so if it has to send requests back to a server, it doesn't work. As much as I'd like to be able to do this, I don't think that unrolling the zips will work very well. I looked for information about how Git handles binary diffing to see if, every time one of these zip files gets updated, it'll blow up the size of the repo. It seems that over time, the repo gets packed and Git does binary diffs between versions, [1] [2] so most of the cost is up-front. If this is true, then zipping vs unzipping will add roughly the same amount of overhead to the repo. [1] http://stackoverflow.com/questions/17976067/will-git-store-diffs-of-binary-files-that-change-in-content-but-never-change-si [2] http://stackoverflow.com/questions/4697216/is-git-good-with-binary-files

(In reply to Doug Sherk (:drs) from comment #19) > Comment on attachment 8651956 [details] > [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master > > I tried unrolling the zip files that 'preload.py' downloads, but there are > several problems with that: I was confused here, Fred told me preload.py merely generate a new folder instead of a zip, so why you need to unroll them? In my imagination, bumperbot will pull hosted apps into gaia everyday and these apps should be folder format. So that gaia build system is able to pack hosted apps as gaia apps. > * 'metadata.json' no longer does anything, so we don't get updates for the > preloaded apps via Marketplace. > * The manifests include `package_path` fields, so we'd have to parse them > and strip this out. While we could do that, it's very likely that this will > break in the future. > * The app origin is broken, so if it has to send requests back to a server, > it doesn't work. I don't understand that why above reasons will make us need to check in zips into gaia. > As much as I'd like to be able to do this, I don't think that unrolling the > zips will work very well. > > I looked for information about how Git handles binary diffing to see if, > every time one of these zip files gets updated, it'll blow up the size of > the repo. It seems that over time, the repo gets packed and Git does binary > diffs between versions, [1] [2] so most of the cost is up-front. If this is > true, then zipping vs unzipping will add roughly the same amount of overhead > to the repo. > > [1] > http://stackoverflow.com/questions/17976067/will-git-store-diffs-of-binary- > files-that-change-in-content-but-never-change-si > [2] http://stackoverflow.com/questions/4697216/is-git-good-with-binary-files

(In reply to Ricky Chien [:rickychien] from comment #20) > I was confused here, Fred told me preload.py merely generate a new folder > instead of a zip, so why you need to unroll them? > In my imagination, bumperbot will pull hosted apps into gaia everyday and > these apps should be folder format. So that gaia build system is able to > pack hosted apps as gaia apps. I'll be very specific, because I think we're talking about two different things here. Marketplace is capable of serving both packaged and hosted apps. In both cases, 'preload.py' preloads some metadata via the 'metadata.json' file. Note that this is the only file that is custom written by 'preload.py'. The rest are all served statically by Marketplace, or whatever other source we're pulling them from. There are also special cases, which are as follows - Packaged apps: 'preload.py' preloads a stub 'update.webapp' manifest and the app package zip file. This file points to the package zip file, which has a 'manifest.webapp' manifest inside it. - Hosted apps: 'preload.py' preloads a full 'manifest.webapp' manifest. Note that the apps that we're preloading in this bug are a mix of hosted and packaged apps. Here's a link to BuddyUp, an example of packaged app being served on Marketplace: https://marketplace.firefox.com/app/buddyup?src=search Here's the manifest for that app: https://marketplace.firefox.com/app/8d979279-a142-4fee-993b-8e7797b221a5/manifest.webapp Note also that this manifest has a `package_path` field. If you go here from the Firefox OS browser, you'll see that it when you download/install it, it gives you a download size. This is because it's just displaying the size of the zip file that it's serving you. Hosted apps just prompt you to install them without providing a file size, like Bugzilla Lite for example. Based on our seeking a way to have packaged apps unrolled in the Gaia build, I tried unrolling the zipped 'application.zip' files that Marketplace serves us, but then we lose the following benefits: - Updates from Marketplace. - We have to change the manifest and strip out the `package_path` field before checking it into Gaia, which is hacky and prone to breaking. - Metadata associating the app with its actual origin. - Random things break, like image paths and such. I'm open to alternatives here, but I don't see any reasonable ones. Do you have any thoughts on this?

Thanks for your specific explanation here! It's really a problem when pulling hosted apps and packaged apps, we have to figure out how to deal with it for future build system. So I think it's reasonable solution to check in zip files into gaia repo as we check in marketplace app.

Taking a review of this, I think we need to account for the apps that we are placing on the system partition. This will cause an issue with flame OTA. I think we need options for install for flame so that we keep the system partition size managable. We already ran out of space once and it cause issues trying to get people upgraded. bug 1181372, bug 1184980

Adding Alexandre so that he's aware and if he would care to comment. (if not no worries)

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master Could we get this tested as an OTA update, so we can confirm whether or not comment 23 will be a problem? This would involve starting from any relatively recent image, then serving this Gaia branch as part of an OTA update.

Anyone from QAnalysts, feel free to take testing comment 25 from Naoki, but he should still make the final call.

https://bugzilla.mozilla.org/show_bug.cgi?id=1199863 We already run into this issue on Flame now.

I have no background on this problem, but here's my understanding. The Aries device doesn't have this problem, because the system partition is big enough by default. The Flame, though, is running out of space on the system partition (bug 1199863). We've run into this problem in the past, and to fix it then, we expanded the system partition size. Alex, is there anything we can do here? Can we expand the system partition again? Are there any other options? Is there anyone else whom we can ask for help on this problem? Even if we wanted to land these changes for Aries only, we have no way of making device-specific app lists, so that's not an option.

Looks like there's nothing QA can do for here as this is definitely a problem, based on bug 1199863.

I assume the problem here applies to engineering builds since user builds are a fair bit smaller. On a Flame using the v4 base image by default we're left with ~100M after having flashed an engineering build; for the record that's ~30MiB less than a user build. This is not enough to perform an OTA update because we need to unpack the ZIP file entirely in the free space before replacing the existing system. There's two solutions for this: - Increase the /system partition size. While this is technically possible (we could eat into /data which is 2G currently) I don't think we can get another base build from T2M- - Use FOTA instead of OTA. We've been saying this with Alexandre for ages, FOTA works a lot better than OTA as it needs less space and can do more. From a user-experience POV it's less nice (while we perform the installation the user cannot use the phone) but it's a ton faster and doesn't require extra space available. Things are finally moving in this direction and we've even scheduled regular meetings for discussing rolling out FOTA (minutes are here for those interested https://etherpad.mozilla.org/fxos-update ) so the issue of the space on Flame might become moot soon enough.

(In reply to Doug Sherk (:drs) from comment #28) > I have no background on this problem, but here's my understanding. > > The Aries device doesn't have this problem, because the system partition is > big enough by default. > > The Flame, though, is running out of space on the system partition (bug > 1199863). We've run into this problem in the past, and to fix it then, we > expanded the system partition size. > > Alex, is there anything we can do here? Can we expand the system partition > again? Are there any other options? Is there anyone else whom we can ask for > help on this problem? > > Even if we wanted to land these changes for Aries only, we have no way of > making device-specific app lists, so that's not an option. Two things. First, no, the trick we used safely cannot be used anymore. Repartitionning now will need to actually move partitions, which is much more dangerous. Second, it's not clear what is the issue. Did you actually run into the issue ? We added an extra 52MB, I think it allows us to have 26MB more of apps/content when doing an OTA. How much are we actually getting now ? Are you just referring to bug 1199863 ? Because without any logcat and proper checking we can only /infer/ this is a partition size issue, and not confirm it.

(In reply to Alexandre LISSY :gerard-majax from comment #31) > Two things. First, no, the trick we used safely cannot be used anymore. > Repartitionning now will need to actually move partitions, which is much > more dangerous. > > Second, it's not clear what is the issue. Did you actually run into the > issue ? We added an extra 52MB, I think it allows us to have 26MB more of > apps/content when doing an OTA. How much are we actually getting now ? Are > you just referring to bug 1199863 ? Because without any logcat and proper > checking we can only /infer/ this is a partition size issue, and not confirm > it. Naoki and I just tried this on a fresh build. We started on the latest Flame build, then OTA updated to one with the bumper bot apps included. The update failed with the same error code as bug 1199863. My suggestion for now is to just include the hosted apps only, and not the packaged apps, until we figure out a better solution. Perhaps that solution will be the one in comment 30.

I've updated the PRs to update only the hosted apps. Naoki, can you give QA approval on this?

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. Just as a quick look, of looking at the code : Landing just the hosted apps of the bugzilla lite, twitter and facebook seems ok with me. I would need to test the build process at some point as long as this doesn't break the building of gaia, it should be good to go. + for landing.

Comment on attachment 8651954 [details] [spark] Add support for bumping hosted and Marketplace apps into Gaia. https://github.com/fxos-eng/spark/commit/de4f1f65b8cf5d731759361229143c89a143136b

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master https://github.com/mozilla-b2g/gaia/commit/03387127ae7eb865919cf3a0909036c874431778

Comment on attachment 8651956 [details] [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master Partially reverted in https://github.com/mozilla-b2g/gaia/commit/60e212543329ee48f2921e0cd92aaca4b4ce2f6a due to build failures. Ricky and I are investigating. Ricky suggested that it may be because the `origin` attribute in 'metadata.json' isn't prefixed with "http" or "app".

Ricky, please leave any info you find here. I'll investigate more tomorrow.

Errors arised at https://github.com/mozilla-b2g/gaia/blob/master/build/test/integration/helper.js#L60-L63 The webapps `origin` attribute should begin with `app://` protocal unless mochitest app. However, those origin of hosted apps bebin with `https://` which generates from preload.py. An appropriate fix is patch preload.py to generate app:// origin with a specific flag.

Ricky, something isn't adding up here. The "app://" scheme is for packaged apps, but the apps that we're including for now are all hosted. I don't think that the fix is to switch to the "app://" scheme.

Comment on attachment 8658916 [details] [gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master Ricky, I made the checks for schemes a little bit less restrictive. Content-level apps can now use either the "http(s)?://" or "app://" schemes. This makes more sense to me as it is functionally correct.

Comment on attachment 8658916 [details] [gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master LGTM

Comment on attachment 8658916 [details] [gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master https://github.com/mozilla-b2g/gaia/commit/1f7b19915f8146bcf1e9f938e1b9c6e4bce0bc95

I'm no longer working on this directly as I don't know how to deal with the issue in comment 28. If it turns out that I can help and finish this with whatever solution we come up with, I'll take this again.

Note that I landed bug 1214623 yesterday. This may free some room on Flame. For example, the package for the SMS app is now half the previous size (I haven't checked other apps).

Naoki tells me that this is now safe to take to completion using the original plan. We're going to FOTA the next Flame update, so we sidestep bug 1199863.

FYI, This is the bug that will add Buddyup to the main gaia apps.

Comment on attachment 8674513 [details] [spark] Add BuddyUp, Notes, Calculator to the bumped apps. It seems not impact on gaia build. r+.

Comment on attachment 8674514 [details] [gaia] DouglasSherk:1196992-hosted-packaged > mozilla-b2g:master r+, but you should probably also make the relevant changes in apps/homescreen/build/default-homescreens.json

Had spun a build on friday, but realized that I needed a fingerprint; creating a build today. Was out sick yeterday.

Agreed with Cwiis. We want to move to the new homescreen by default for pin the web + other stuff. Could this change be made please?

I tried patching the second patch [https://github.com/DouglasSherk/gaia/commit/72699d8b5086d4984ecf155fff34b440b235bba3 ] the first patch was already committed [ https://github.com/DouglasSherk/gaia/commit/c64b41cdf188a1e51f45e8d65f088f151e320784 ] and made my own gaia but I don't see the buddyup app listed in the files.

Naoki and I discussed this on IRC. The problem was that the apps are missing from the Gaia repo, since we haven't run spark-bot's check-in process for these apps yet.

Comment on attachment 8674513 [details] [spark] Add BuddyUp, Notes, Calculator to the bumped apps. For updating the hosted files, this seems to work. Not sure if I'm a big fan of all the separate gaia and upload; too many parts that could potentially break there. Multiple FOTAs seem to work ok on the Flame device currently; having said that I haven't add tons to the profile and there's too many varying variables.

Comment on attachment 8674513 [details] [spark] Add BuddyUp, Notes, Calculator to the bumped apps. https://github.com/fxos-eng/spark/commit/38c667be5dcec18eee180b3fbfc232fca020a530

Bumped remotely hosted apps for the first time using attachment 8674513 [details] here: https://github.com/mozilla-b2g/gaia/commit/ec2b1bbba3d23f1c224874b3bae928b1b2d675fa

Comment on attachment 8674514 [details] [gaia] DouglasSherk:1196992-hosted-packaged > mozilla-b2g:master https://github.com/mozilla-b2g/gaia/commit/aac4d416b6665e908589ae28f47f3388e17fca3f

This broke a few tests, so backing out for now. Please use a pull request to re-land and ensure tests are green. Backout: https://github.com/mozilla-b2g/gaia/commit/90d30c756b5c4cb9c4968e676d42c5da38b86ef0 For breakage: https://treeherder.mozilla.org/logviewer.html#?job_id=580026&repo=gaia-master Backout: https://github.com/mozilla-b2g/gaia/commit/db3b2e26c7f360a024088ccf073d454af9b842bc For breakage: https://treeherder.mozilla.org/logviewer.html#?job_id=580030&repo=gaia-master Some of these tests are a bit strange for having hard-coded values. I think we can probably just update them, but I'm not entirely sure about the strategy to bump/update these going forward. Doug - I'll leave it to you, thanks.

Hi all, Drs is sick with the flu today and I imagine he will need a couple of days to recover. Do we have any back up plans to get this fixed. Unfortunately, we are really cutting it close for getting Bugzilla Lite into the 2.5 release with this bug and need this resolved by Friday. Any ideas? I'm not to sure who to need info so please need info accordingly.

I'll take a look.

Thank you, Kevin!

Re-landed: https://github.com/mozilla-b2g/gaia/commit/2d8465a779fbddfa5a3d748fc40f0382686f5be5

Which apps should start appearing in the future FOTA updates?