Right now we run jetpack tests on mozilla-central and mozilla-aurora. However, we do not run them on mozilla-beta or mozilla-release, which means we're not testing what we actually ship, and we don't run them on any of the project branches that merge into mozilla-central, which means if they break on a project branch we'll have a huge amount of code to bisect (which really means anybody making changes that could affect jetpack should not be using project branches). We should run these tests on beta, release, and project branches.

This makes sense, and will happen as soon as we have a bit more stability on the jetpack test suite which is currently going orange (or red) too frequently to be of much use on the more 'stable' beta/release branches. bug 629263 is tracking the work on this.

fwiw, the suite also runs on try.

putting this to p4 since it's waiting for another bug to be fixed first.

Can we at least run the jetpack tests on Inbound (hidden by default)? A big inbound -> central merge is causing compartment mismatch assertions in debug builds (bug 738956), and I can't tell what exactly caused it without seeing the jp test results on the individual pushes to the inbound branch.

Sigh. Doesn't entirely depend on bug 629263, since as comment 4 demonstrates there's value in running a test even if that test doesn't result in immediate backouts, but since Armen is looking for builders to thrown under the bus to be able to add a few, I suspect it depends on bug 712244.

kwierso, the code to add jetpack to inbound is here: http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla-tests/config.py#l1170 but as philor points out I have to refactor some code to reduce the # of builders. We are lucky enough to have few branches that have been abandoned and I will reach the project branch owners to remove them and give room to run jobs were we need them.

I keep watching as things make more room, and then things bounce because there's no room, and not having the slightest idea when I can get away with this. It's just one puny little builder per OS, maybe we can afford that this week?

Comment on attachment 619349 [details] [diff] [review] inbound http://hg.mozilla.org/build/buildbot-configs/rev/25ea9b2c950a which was just fast enough to catch a reconfig, and we're now running it on inbound, https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=jetpack

We'll be wanting to see what Ionmonkey does to Jetpack sooner than when it comes crashing into mozilla-central.

Comment on attachment 656339 [details] [diff] [review] ionmonkey Thanks philor but I think a further discussion needs to happen or the owners of the Jetpack project to say on which branches to run it on. Adding Jetpack to Ionmonkey to be hidden does buy much but not as much as taking machine cycles. I just don't know how to feel about none closing tests when we are having such bad wait times. I wonder if we had a tool that went to a development branch and triggered jobs on that branch to find a regression range. I will add it to my list of items to be discussed on our next meeting.

If mozilla-central is running a set of tests to determine whether a changeset is valid, then I think IonMonkey should run the same tests so there are no surprises when we merge.

My memory of 2010 is getting really fuzzy, but I think that the reason we run the Jetpack tests at all is because we found out far too late that Jaegermonkey, our last big JS engine rewrite before Ionmonkey, broke Jetpack. If the problem is that Jetpack is hidden, then a) people need to get over that, Jetpack does exactly what it needs to do while hidden, the system works just fine, and hidden does not mean ignored, and b) because Ionmonkey is a project branch with a strong tolerance for visible test failures, I had no intention of hiding it there. It's a nearly-done rewrite of the JS engine, which needs to know what it will break, which is a very different situation than when an intentional change which breaks Jetpack's assumptions lands on mozilla-central/mozilla-inbound.

Comment on attachment 656339 [details] [diff] [review] ionmonkey Thanks for clarifying and helping me understand it better. Want to land? or want me to?

If you can, that'd be great, I'm hours away from having a tree to push from, and with the number of reconfigs happening this week, I might luck into production at any time ;)

Comment on attachment 656339 [details] [diff] [review] ionmonkey http://hg.mozilla.org/build/buildbot-configs/rev/abe15f89c3d8

Made it to production today.

Fine if we close it and open when we want to add more branches?

Sure, if you don't mind closing it as WONTFIX, saying that even in the glorious future of infinite slaves in the cloud, we will never be willing to run jetpack tests on every tree where we run tests.

I assume we would want to run them even if they are not tier1 to make Jetpack's life easier. It will be awhile until we have enough capacity. Another alternative would be to find a way of making Jetpack jobs to run on idle times except for current active branches. This is not a feature we have and would require lots of work but might be the best way of using resources.

Jetpack tests are still too expensive to run by default everywhere, but we can continue to enable them on branches where the branch owners will do something with the results. WONTFIXing for now. We can revisit in a few months when we (hopefully) have iX machines running tests.