I'd like a way to manage a configurable list of apps to be ignored. We would have both a hard coded list (for the things that are not app) and a configurable list (for apps that are flaky to be disabled). That list would be configured at runtime using an env variable. This way we could always run everything on eg Travis, or locally, but still disable some apps on jenkins for datazilla so that it's reliable there.

I should morph this as "configure per branch the apps to test". Since we run 1.3T test using master. And this should become P1 because of bug 846909 comment 22

Also Julien said in a review: "maybe we could try to look up the role in manifest.webapp instead ? This way we could automatically ignore even future not launchable apps, like the homescreen does."

Bug 1005917 will make this less of a problem as we'll ignore "system" apps.

This ability was provided when test-perf introduced a config.json file which contains configurable whitelists and blacklists.