It appears that the Template application is not present on the Tarako builds we're running performance tests on. 2014-03-06 12:36:32,679 B2GPerfRunner INFO | Running B2GPerfLaunchTest Traceback (most recent call last): File "/var/jenkins/workspace/b2g.tarako.mozilla-aurora.v1.3.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 353, in run self.test() File "/var/jenkins/workspace/b2g.tarako.mozilla-aurora.v1.3.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 394, in test raise AppLaunchError() AppLaunchError: Error launching app If it's intentionally not present, then we should remove this app from the cold launch tests we're running.

The template apps + others were causing the the device to over fill, so I think they weren't placed on? I can't seem to find the portion in the build command line that would cause that. Somewhere in the log should show how to get the production version of gaia on there even though it's marked VARIANT=eng for tarako : https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/mozilla-b2g28_v1_3t-tarako-eng/20140513080502/logs/log_raw.log search for ./build.sh and you should find the line that would control that. Maybe Aki might know off the top of his head?

I don't think I understand the question. Maybe that means I don't know off the top of my head?

I think the Tarako devs removed things, which may include apps. I don't think Releng does anything to specify that; I think it's keyed off of eng or user or w/e.

Could this be related to bug 994985? We're using the tinderbox builds for Tarako, which after a full flash does not include Template or any other testing apps. Is there a way we can include just the Template app without having all the other testing apps?

I'm not sure, it could very well be; what I do know is that we had run into the same issue that Stephend had run into ( bug 973046 ). I can't seem to find the bug in regards to reducing space for fonts and a few other things in order to make the system.img smaller. I am not sure what happened to the tinderbox engineering build for the build commandline.

From bug 1009456: Perhaps we need a new apps-*.list file for test automation? The 'engineering' list contains more than we need, and the 'production' list misses template and others we may need (such as UI Test?). Alternatively, maybe we could slim down 'engineering' and create an 'all' or 'dev' list. Do you think it's worth raising a bug for this?

Just to clarify, this is what Dave is talking about from the other bug: $ cp build/apps-production.list build/apps-perf.list $ echo test_apps/template >> build/apps-perf.list $ GAIA_APP_TARGET=perf GAIA_OPTIMIZE=1 NOFTU=1 make clean reset-gaia It was his solution to get things to work. I agree. I think we just need to push the new list into the gaia repo? http://mxr.mozilla.org/gaia/source/build/config/phone/ Maybe Kevin Grandon could help with this? Kevin, is this the modification we would need to create a new apps list for the phone?

Is it possible for you guys to create your own config for this device? It should be possible to pass a GAIA_APP_CONFIG=/path/to/file.list with a list of desired apps for the gaia build. I don't think it's a good idea to check this into the gaia repo. I would recommend having: apps/* dev_apps/template and anything else you need from dev_apps.

:kgrandon we can create our own config but that's too late to do anything with. We're using the releng tinderbox builds and need to influence the apps present on those.

Aki, would it be possible to have a different apps list for the tarako builds for the engineering build? See Comment 7 and Comment 8

I don't recommend going the route of a special app config file that isn't checked into the tree for a releng-supported CI build like tarako-eng; this is fragile and hard to debug and ends up wasting a bunch of time for everyone. I don't understand why we can't specify this in the repo?

Kevin, could you respond to comment 11 please?

We could, but we would still need to specify GAIA_APP_CONFIG=<some config file> during the gaia build step. I also didn't want to have to add some device specific .list file to the codebase as that's also has potential to get out of sync. Aki - what about copying our .list file and appending the template app during the build step? E.g., cp build/config/apps-production.list ~/apps-tarako-perf.list echo dev_apps/template >> ~/apps-tarako-perf.list GAIA_APP_CONFIG=~/apps-tarako-perf.list make I will also investigate adding a config file to gaia.

Alternative approach if we don't want to script anything, though we'd still need to specify the right file during the gaia build.

(In reply to Kevin Grandon :kgrandon from comment #13) > We could, but we would still need to specify GAIA_APP_CONFIG=<some config > file> during the gaia build step. I thought we had a per-device config setup, during which we could check if VARIANT==eng and switch our GAIA_APP_CONFIG to an otherwise-unused tarako-eng-specific config file. Is that a false assumption? > I also didn't want to have to add some > device specific .list file to the codebase as that's also has potential to > get out of sync. I think that'll get more out of sync if it's outside the codebase. Is there a way to enable an "include" type statement in the .list files? > Aki - what about copying our .list file and appending the template app > during the build step? E.g., > > cp build/config/apps-production.list ~/apps-tarako-perf.list > echo dev_apps/template >> ~/apps-tarako-perf.list > GAIA_APP_CONFIG=~/apps-tarako-perf.list make Our automation has to handle all configurations, and is already far more complex than is prudent. > I will also investigate adding a config file to gaia. I'll take a look.

I think I'm thinking of https://github.com/mozilla-b2g/B2G/blob/master/config.sh#L136 .

Comment on attachment 8424114 [details] Github pull request I think this works, with the config.sh change.

Comment on attachment 8424114 [details] Github pull request LGTM

I have a couple of concerns. Firstly, I don't think this patch would apply to v1.3t, which is the branch we're currently using against Tarako devices. The other concern I have is that we'd need this to be used when the engineering builds are produced, and these builds are not only used for performance testing. I think it would be better to either adjust the apps installed on an engineering build, or produce a new build that only has apps suitable for test automation.

I think what Aki is stating is that we need another patch to specify for the tarako build, we could potentially check for VARIANT=USERDEBUG, ENG and use the build/config/phone/apps-tarako-perftest.list patch that Kevin made. I think this would make it specific to Tarako Engineering builds for all builds. As for the versioning, I believe we would want this on master 2.0 ( in the event that we need to support it or something similar in the future) and need to push this to the 1.3t branch. I think this is what David's concern is... is that correct?

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #20) > I think what Aki is stating is that we need another patch to specify for the > tarako build, we could potentially check for VARIANT=USERDEBUG, ENG and use > the build/config/phone/apps-tarako-perftest.list > patch that Kevin made. I think this would make it specific to Tarako > Engineering builds for all builds. I think so. https://github.com/mozilla-b2g/B2G/blob/master/config.sh#L136 Could be something like "tarako") if [ $VARIANT = "eng" ] ; then echo GAIA_APP_CONFIG=path/to/apps-tarako-perf.list >> .tmp-config; fi echo DEVICE=sp6821a_gonk >> .tmp-config && echo PRODUCT_NAME=sp6821a_gonk >> .tmp-config && repo_sync $1 ;; (I haven't checked that if statement for validity, just an example) > > As for the versioning, I believe we would want this on master 2.0 ( in the > event that we need to support it or something similar in the future) and > need to push this to the 1.3t branch. I think this is what David's concern > is... is that correct? afaict config.sh isn't branched, so this would take effect for tarako-eng across all versions.

I believe that David is referring to the gaia pull request.

Well, I'm just concerned that the engineering builds are not a 1-1 mapping with performance testing. There's already a apps list for engineering builds, however I guess we're not currently using this for Tarako because of the space limitations? Perhaps just having a Tarako specific 'eng' app list (rather than using 'perf', which is very narrow). Do we know who the consumers of the engineering builds are, besides test automation? It's possible we could just slim down the apps in this variant as not many of the extra ones are needed for automation purposes. Template and UI Tests are the only two that come to mind. (In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #22) > I believe that David is referring to the gaia pull request. Yes, the config would only be present in Gaia's master branch unless it's uplifted.

We're no longer running performance tests against Tarako.