At the moment to disable a jetpack test (in the case of high intermittent failure rate), the disabling has to occur upstream and then another import of jetpack's repo performed. We really need a disabled tests manifest in mozilla-central (outside of the imported source directory), so sheriffs can disable misbehaving tests (eg bug 882867) without having to resort to upstream changes (or making changes to mozilla-central that will just get overwritten on next import). ie jetpack only partially adheres to: https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#9.29_Supports_the_disabling_of_individual_tests

Irakli what do you propose here? I was suggesting using a `jetpack-test-options.json` file in bug 915376 https://github.com/mozilla/addon-sdk/pull/1241, which would have covered this, but you vetoed that and implemented preferences instead because you said you didn't want to see extra files in the add-on, so I'm not sure what you want to see here.

Note that the manifests added in bug 1035512 fix this when we turn those tests on.

Looks like we have solution as suggested by mossop above.

(In reply to Ed Morley [:edmorley] from comment #0) > We really need a disabled tests manifest in mozilla-central (outside of the > imported source directory), is this "outside of the imported source directory" part really necessary? we uplift changes from m-c to github.

The reasoning being, that people can disable problematic tests immediately (and in a repo they gave access to/checked out locally), rather than needing to wait for the next uplift. That said, a possible workflow might be: disable test in mozilla-central's copy of the addonsdk directory manifest, but also make sure someone makes the equivalent change in the external repo, so the change doesn't get clobbered on next uplift. Either way, I'm happy to defer this call to RyanVM now, since I'm now on the treeherder team, rather than the sheriffs :-)

(In reply to Ed Morley [:edmorley] from comment #5) > That said, a possible workflow might be: disable test in mozilla-central's > copy of the addonsdk directory manifest, but also make sure someone makes > the equivalent change in the external repo, so the change doesn't get > clobbered on next uplift. This is what we already do as Erik said.

(In reply to Dave Townsend [:mossop] from comment #6) > (In reply to Ed Morley [:edmorley] from comment #5) > > That said, a possible workflow might be: disable test in mozilla-central's > > copy of the addonsdk directory manifest, but also make sure someone makes > > the equivalent change in the external repo, so the change doesn't get > > clobbered on next uplift. > > This is what we already do as Erik said. That's not how comment 4 read to me, the key part is the addonsdk team being happy for the simultaneous landing both in both repos, not that things are just uplifted. If that's already happening, great :-) Either way I don't really have a horse in this race any more, I'm no longer a sheriff

we should still complete this bug, it seems like we don't have any requirement on where the file is now though.

Now mochitest-jetpack is running the tests can be disabled via the manifest so I'd be inclined to close this as fixed and open other bugs if other manifests for jpm/travis are needed. Make sense Erik?

(In reply to Dave Townsend [:mossop] from comment #9) > Now mochitest-jetpack is running the tests can be disabled via the manifest > so I'd be inclined to close this as fixed and open other bugs if other > manifests for jpm/travis are needed. Make sense Erik? Sure that sounds fine to me.