According to the plan at

https://docs.google.com/document/d/1oRPkwP3l7QzdQYj0Wn7d_3EfTZaakocA_i_pGKlG0dI/edit#heading=h.wao3tap2swke

we can stop running Fennec tests on mozilla-central at the start of the Gecko 71 development cycle - next week.

Stop running Fennec tests:

• stop running all Android 4.3 tests
• switch Android 7.0 cppunit from the fennec apk to the geckoview apk (no difference
  in behavior expected)
• stop running Android 7.0 marionette tests, since they also run against fennec
• remove android-em-4.* references from taskcluster configs
• remove android instance: extra-large references from taskcluster configs,
  since they only affect aws, which is no longer used for android

Stop running all Raptor tests that run against Fennec. Raptor tests running
against geckoview and geckoview products continues.

I announced these changes on #geckoview yesterday and also checked in with David Bolter and Stefan Arentz today. Stefan asked that we be sure not to disrupt tests on esr68 (shouldn't be affected by my patches).

https://hg.mozilla.org/mozilla-central/rev/97cb5cb1f175

Hi, the Raptor Fennec64 tests are try-only, and in the past have been used to compare Fenix and GVE with Fennec64. Are we sure we want to remove this ability to run Fennec64 Raptor tests on try? Just double-checking, thanks! :)

(In reply to Robert Wood [:rwood] from comment #6)
Glad you brought this up, as I was not aware of that use case.

But I think it is okay: As I understand it, we will only release Fennec from esr68 going forward, so shouldn't we be using the esr68 branch of Fennec64 for comparisons with Fenix/GVE? We will still be able to run Fennec raptor tests on try after this change, as long as the try push comes from an esr68 context.

As long as we keep the ability to run Fennec raptor tests on try I'm good with the change. Is there any work that needs to happen to ensure the try push comes from esr68?

Existing tests that work with Fennec64 right now will work with esr68? And updates to tests should not impact this release?

(In reply to Vicky Chin [:vchin] from comment #8)

As long as we keep the ability to run Fennec raptor tests on try I'm good with the change...

:gbrown, your patch will completely remove the raptor-fennec*.yml configs, so we won't be able to run them on 'try' either. Can you just change the configs for all the raptor fennec tests to run on 'try' only instead of deleting them? Thanks!

Existing tests that work with Fennec64 right now will work with esr68? And updates to tests should not impact this release?

Sorry I don't know the answer to this, will need to defer to the others, thanks.

(In reply to Robert Wood [:rwood] (PTO back Sep-24) from comment #9)

(In reply to Vicky Chin [:vchin] from comment #8)

As long as we keep the ability to run Fennec raptor tests on try I'm good with the change...

:gbrown, your patch will completely remove the raptor-fennec*.yml configs, so we won't be able to run them on 'try' either. Can you just change the configs for all the raptor fennec tests to run on 'try' only instead of deleting them? Thanks!

My patch will completely remove tasks from mozilla-central, and you won't be able to run them on try from mozilla-central, but that will not affect the ability to run raptor tests on try from esr68: clone esr68, then 'mach try ...' from that esr68 clone and you'll pick up raptor tasks on esr68.

Fennec is now shipped only from esr68. When we test fennec on try from mozilla-central, we are testing code that will not be shipped. Also, according to the ESR68 Transition Plan, there is no guarantee of working Fennec builds on mozilla-central beginning with Gecko 71 (now!).

However, when I look closer at esr68, I see:

• there are no raptor tests run in automation
• not all fennec raptor tests currently available for mozilla-central/try are available on esr68/try - we probably need to uplift some patches.
  I'll need to sort through those issues before moving forward here.

Should this be announced on dev-platform?

I think so, but have been holding off because of the discussion about raptor tests. It is a muddled message right now: "Most fennec tests are turned off, but some raptor fennec are still running, but I'm trying to turn them off soon..." See how today goes?

https://groups.google.com/forum/#!msg/mozilla.dev.platform/k-irJtmCcqg/8HTMITxZAgAJ

(In reply to Geoff Brown [:gbrown] from comment #10)

However, when I look closer at esr68, I see:

• there are no raptor tests run in automation
• not all fennec raptor tests currently available for mozilla-central/try are available on esr68/try - we probably need to uplift some patches.
  I'll need to sort through those issues before moving forward here.

:bebe can you work with :gbrown to uplift patches to esr68 so that we're able to run raptor tests against esr68 using try?

We are also running tests against a released version of fennec (v64), which was primarily used as a way to measure Fenix against release criteria. Now that we've shipped Fenix 1.0 we no longer need this comparison. We also now have 3 months of weekly results against fennec v64 where the apk did not change. The only reason I can see to continue running these tests is if we plan to use it as a comparison point again for a future release, which we are not. Let's remove these jobs and the cron that triggers them.

took a look over the diff between esr68 and central regarding fennec tests.

Currently there is a small bat of tests that run on fennec in esr68

There are a lot of changes that need to be uplifted from central to esr68 to bring the esr68 repo to the current state.
These changes will include the mobile re-recordings, mitmproxy 4.0.4 updates and the split of config files in treeherder.

Do we want all these changes to be uplifted?

From a first view the bugs that need to be uplifted are:

Test files
Bug 1563164
Bug 1563162
Bug 1554679
Bug 1564040
Bug 1556760

Raptor
Bug 1539983

Mitmproxy
Bug 1539983

Note:
Not sure if these are all the bugs or if they are "upliftable"

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #15)

Do we want all these changes to be uplifted?

Maybe? I know very little about raptor. I was hoping you would know!

I had a look at the raptor fennec tests available to 'mach try fuzzy --full' on esr vs mozilla-central. On m-c there's a long list and on esr there's a very short list. But if I eliminate 'fennec64' tests (comment 14) from the m-c list, compare only tests (not the build platforms), and eliminate those already on esr, the list of tests to add to esr is just:

raptor-scn-cpu-memory-idle-bg-fennec-1proc
raptor-scn-cpu-memory-idle-fennec-1proc
raptor-scn-cpu-memory-power-idle-bg-fennec-1proc
raptor-scn-cpu-memory-power-idle-fennec-1proc
raptor-scn-power-idle-bg-fennec-1proc
raptor-tp6m-[1..27]-fennec-cold-1proc

Does that seem correct? If that's the case, do we need all those bugs uplifted?

Those bugs contain recording and proxy updates to the tests.

raptor-scn-cpu-memory-*-fennec-* tests where developed in Bug 1555799 So we need to add it to the list.

Is there a way I can patch my local branch with all these bugs so we can test everything works fine?

In theory we can apply patches to our local esr branches and push to try. The tricky part seems to be applying the right patches in the right order, since there is so much discrepancy now between esr68 and trunk 71. I haven't gotten very far at all today.

I would suggest to setup up queries for bugs as fixed in 69, and 70 first. Mostly sorting by last time changed might help to nearly get the right order for grafting those onto the 68esr branch. For 71 it shouldn't be that many.

:bebe can you please follow up here? how did your testing of esr68 from mozilla-central via try go? I don't see a link to your try push here or in phabricator.

:gbrown do you have any concerns using this approach? it would leave a set of fennec tests in mozilla-central for the sole purpose of running via an apk available via a specific taskcluster index pointing at the latest esr68 build.

No concerns. I think this is brilliant.

managed to setup and run a small try to make sure the setup is working as expected in:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=603a7a5da3d1531ebdc15053086820917fcc2177&selectedJob=266832157

Started a complete try build in:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=73531a9ac284ced26cad6dc6e4f19d6c37aea8b9

try job are downloading the builds from:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-esr68.latest.mobile.android-api-16-opt/artifacts/public/build/target.apk

If all looks good in the review we are good to go with this without the need of all the uplifts

:bebe please add documentation on how to run tests against fennec-esr68 to https://wiki.mozilla.org/TestEngineering/Performance/Raptor

:davehunt
Done in:
https://wiki.mozilla.org/TestEngineering/Performance/Raptor#Running_Fennec_ESR_68_tests

:davehunt :gbrown

Should we schedule a cron job to run fennec esr68 on a daily/weekly build??

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #28)

:davehunt :gbrown

Should we schedule a cron job to run fennec esr68 on a daily/weekly build??

I think we should replace the current weekly Fennec v64 jobs with Fennec ESR68 jobs.

Whatever is needed. I just wouldn't run them very often given the scarcity of android-hw resources and the stability of esr68.

https://hg.mozilla.org/mozilla-central/rev/5c94e7e6e153

:bebe - I'm eager to finish this up. Can you give the last patch one more review?

:gbrown there is a small number of Fennec tests that need to be removed but the rest looks ok

I have updated the ini files in the latest revision.

https://hg.mozilla.org/mozilla-central/rev/7a69e92c05ee