We are enabling the FX UI updates for the betas. We currently test all locales for all releases since the first Gecko 38 version. We're currently splitting this up in 10 chunks but even then, they will be too many. The runtime is going to be pretty long and it will become even longer as more betas are added. We will need to trim this up. I decided to spin this out from bug 1148546 to reduce the noise. The reduced matrix is discussed in here: https://github.com/mozilla/mozmill-ci/issues/535 The code to alter is in determine_testing_configuration. [1] Anyone familiar with Python can easily pick this up. STR: hg clone http://hg.mozilla.org/build/mozharness cd mozharness python scripts/firefox_ui_updates.py --cfg update_tests/mozilla-beta.py --cfg developer_config.py --determine-testing-configuration --run-tests --dry-run True [1] http://hg.mozilla.org/build/mozharness/file/ecd1d24223d3/scripts/firefox_ui_updates.py#l164

Current beta matrix: 16:54:21 INFO - ##### Running run-tests step. 16:54:21 INFO - ##### 16:54:21 INFO - Running pre-action listener: _pre_run_tests 16:54:21 INFO - Running main action method: run_tests 16:54:21 INFO - About to run 20150330154247 /firefox/releases/38.0b1/mac/%locale%/Firefox 38.0b1.dmg - 89 locales 16:54:21 INFO - About to run 20150406174117 /firefox/releases/38.0b2/mac/%locale%/Firefox 38.0b2.dmg - 89 locales 16:54:21 INFO - About to run 20150409144858 /firefox/releases/38.0b3/mac/%locale%/Firefox 38.0b3.dmg - 89 locales 16:54:21 INFO - About to run 20150413143743 /firefox/releases/38.0b4/mac/%locale%/Firefox 38.0b4.dmg - 89 locales 16:54:21 INFO - About to run 20150416143048 /firefox/releases/38.0b5/mac/%locale%/Firefox 38.0b5.dmg - 89 locales 16:54:21 INFO - About to run 20150420134330 /firefox/releases/38.0b6/mac/%locale%/Firefox 38.0b6.dmg - 89 locales 16:54:21 INFO - About to run 20150427090451 /firefox/releases/38.0b8/mac/%locale%/Firefox 38.0b8.dmg - 89 locales 16:54:21 INFO - About to run 20150429135941 /firefox/releases/38.0b9/mac/%locale%/Firefox 38.0b9.dmg - 89 locales 16:54:21 INFO - About to run 20150508094354 /firefox/releases/38.0/mac/%locale%/Firefox 38.0.dmg - 89 locales 16:54:21 INFO - About to run 20150511143336 /firefox/releases/38.0.5b1/mac/%locale%/Firefox 38.0.5b1.dmg - 89 locales 16:54:21 INFO - About to run 20150514163436 /firefox/releases/38.0.5b2/mac/%locale%/Firefox 38.0.5b2.dmg - 89 locales 16:54:21 INFO - About to run 20150518141916 /firefox/releases/38.0.5b3/mac/%locale%/Firefox 38.0.5b3.dmg - 89 locales 16:54:21 INFO - About to run 20150523155636 /firefox/releases/39.0b1/mac/%locale%/Firefox 39.0b1.dmg - 89 locales 16:54:21 INFO - About to run 20150601171003 /firefox/releases/39.0b2/mac/%locale%/Firefox 39.0b2.dmg - 89 locales 16:54:21 INFO - About to run 20150604162752 /firefox/releases/39.0b3/mac/%locale%/Firefox 39.0b3.dmg - 89 locales

FYI https://github.com/mozilla/mozmill-ci/issues/535 has nothing to do with updates. This is only for our CI system and running functional and remote tests. Updates should be tested for each locale!

I thought that not reducing the matrix for locales was clear as discussed by bug 1148546 comment 92.

Henrik, I don't want to reduce the matrix of the *locales* but determine which of the previous releases do need all locales. If https://github.com/mozilla/mozmill-ci/issues/535 is not to be used; can we discuss in here what is the right matrix? I'm happy to WONTFIX this, however, that is not what I read from bug 1148546 comment 92. I'm going to use simplified numbers to make the math easier. Once we're in, let's say Gecko 58, and we do 10 betas per release and we have 90 locales. That would be 20 releases (starting from Gecko 38) x 10 x 90 = 18,000 runs and at 1 minute each of them that is 300 hours. At the moment, we're now aiming to split this in 10 chunks which would take 30 hours to run 20 releases from now. Or in other words, we add approximately 9 minutes to each chunk after each beta. We want to test all locales but we should not for every beta. Or is this what is needed?

I think this is more of choosing which releases and which locales to run for every build. Once the tests are running, someone needs to make a decision and adjust the parameters of what is run. Maybe for now that isn't an issue, but as Armen points out it will get messy in the near future.

Ok, I see the misunderstanding now. So onn the referenced CI issue you can indeed find the following line: > Oldest version of Firefox to use will default to the oldest supported ESR (train) So as defined this should be the hard limit, but I think we should bring Kairo into the boat for his feedback on.

I'm fine if we always only go back 4-5 versions or so. And we do not need to test every locale for every single beta build. As defined in the github issue, we should do random sampling out of them so that in total we cover them all.

So for each version splitting up all the locales across the supported platforms should do it? Then we would test all of them with a way lower overhead. Or we might not have to run the same locale for e.g. each Windows version but only a single one.

Splitting across the platform is not so much what I'd like, but for each platform, splitting them across the Firefox and platform versions, that is fine.

We already do something very similar for the existing update verify tests. We run the full tests on the complete set of locales for the previous release, but only run the full tests on a few locales for all prior versions. This helper function on the UpdateVerifyConfig class would return only the "full" test version/locale combinations: https://github.com/mozilla/build-tools/blob/master/lib/python/release/updates/verify.py#L140

Any chance we can implement sooner than later ? Philor would probably say it blocks moving to the test slaves.

We won't run the tests on beta until we fix this. I'm not saying that I will work on it. I'm really hoping someone in releng will pick this up since it is really a releng deliverable to relman. If in a couple of weeks this project is still a priority and no one picks it up I will move the needle a bit more, however, I want to handover asap.

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #13) > We won't run the tests on beta until we fix this. > I'm not saying that I will work on it. That's not true, but it could be. We disabled _notifications_ for these in https://github.com/mozilla/build-buildbotcustom/commit/ecf2598e9da3ea0ba87b9a0eb6444e3b9b5e056a. If there's no (or little) value in running the tests we should disable them completely for now.