Now that there are no more Fennec nightlies built from the 69 branch, we can no longer use mozregression -n fennec to find regression ranges for Android-specific platform regressions.

The current alternative seems to be mozregression -n gve which runs the GeckoView Example app, but this app is very primitive and lacks many browser features like history.

It would be good to add Fenix as a supported app so we can find regression ranges using Fenix nightlies.

I did look at this a little while ago since it seemed likely to be useful soon, but struggled to see the needed abilities in the GitHub API to get pushes instead of just commits. Unless Fenix is brought into hg.mozilla.org I don't see this happening any time soon.

It might be easier to enhance GVE to have more features.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #2)

It might be easier to enhance GVE to have more features.

Won't Fenix need to have a regression window finding story anyways, for front-end regressions? Or regressions where it's not clear if it's front-end or platform?

Yeah, that's true.

In the case where it turns out to be a platform regression, the fenix regression window will likely just point to a "bump m-c from version X to version Y" commit, and then we'll need to further bisect m-c using GVE to isolate between X and Y. And if GVE doesn't have enough features to do that bisection it would still be bad. So we will need both - features in GVE to allow bisecting usefully, and the ability to bisect the final fenix product.

the fenix regression window will likely just point to a "bump m-c from version X to version Y" commit, and then we'll need to further bisect m-c using GVE to isolate between X and Y.

This is correct. Fenix currently only updates its GV snapshot every few days to a week, so mozregression will need to support GVE to bisect Gecko platform regressions on Android.

The priority flag is not set for this bug.
:wlach, could you have a look please?

For more information, please visit auto_nag documentation.

Note: it appears that fenix is being built with taskcluster these days, so at least the process of getting builds might not be incredibly difficult. You can see builds appear on treeherder here:

https://treeherder.mozilla.org/#/jobs?repo=fenix

As :Kwan mentioned in comment 1, we would indeed still have to sort out how to get mozregression to refer to github commits instead of mercurial pushes.

:catlee tells me that :jlund is the person to ask about Fenix-related automation questions. CC'ed him on this bug.

One thought I had before was that as an interim measure we could maybe do at least a date-based bisection, since that's doable with just taskcluster routes like so: https://tools.taskcluster.net/index/project.mobile.fenix.v2.nightly.2019.9.16/latest

Looking at the cadence of commits that might even be more than enough.

Of course I'm not sure how woven-in mozregression's assumptions about the existence of a pushlog are, and thus how doable that actually is.

(In reply to Ian Moody [:Kwan] (UTC+1) from comment #8)

One thought I had before was that as an interim measure we could maybe do at least a date-based bisection, since that's doable with just taskcluster routes like so: https://tools.taskcluster.net/index/project.mobile.fenix.v2.nightly.2019.9.16/latest

Looking at the cadence of commits that might even be more than enough.

That would be an excellent start! ++

Of course I'm not sure how woven-in mozregression's assumptions about the existence of a pushlog are, and thus how doable that actually is.

I suspect this part shouldn't be too bad, at worst I would think we could just create a simplified codepath for dealing with git repositories.

Did some more investigation on this, I think this is doable without too much trouble. We would have to teach mozregression about pulling commit data from github instead of mercurial: seems doable with the commits API -- we can get the commit closest to a specific date by using the since parameter, and paginate through longer results by using the page parameter.

Looking at taskcluster, it appears that pushes to Fenix are relatively rare-- mostly I see single pushes (this is pretty standard for github projects). In any case, mozregression is pretty good at falling back if a particular revision happens to be missing during bisection, so we can probably just assume that commit=push and see how it goes...

We probably need to report more than just the Github pushes. Botond was asking for the Gecko revisions in comment 0.

Providing the Application Services, Android Components, Glean version on the good/bad builds would be helpful too.

(In reply to Kevin Brosnan [:kbrosnan] from comment #11)

We probably need to report more than just the Github pushes. Botond was asking for the Gecko revisions in comment 0.

Providing the Application Services, Android Components, Glean version on the good/bad builds would be helpful too.

Agreed this would be very useful -- though I'm not sure offhand how to look this data up. I didn't see it in my cursory examination of the taskcluster jobs in treeherder or the task inspector. If anyone has any pointers, please add them to this bug!

Maybe Callek knows or can redirect?

(In reply to William Lachance (:wlach) (use needinfo!) from comment #12)

(In reply to Kevin Brosnan [:kbrosnan] from comment #11)

We probably need to report more than just the Github pushes. Botond was asking for the Gecko revisions in comment 0.

Providing the Application Services, Android Components, Glean version on the good/bad builds would be helpful too.

Agreed this would be very useful -- though I'm not sure offhand how to look this data up. I didn't see it in my cursory examination of the taskcluster jobs in treeherder or the task inspector. If anyone has any pointers, please add them to this bug!

Needinfo'ing a few people who have contributed to .taskcluster.yml and might know the answer to this. Is there any current way of determining what revision of GeckoView, Application Services, and Glean were used to build a specific version of Fenix in CI? I see the info in the about screen but can't seem to find it in the logs or anywhere else.

If not, is this something we could add? Perhaps using buildhub.json files, like we use for desktop and fennec (we could file a new bug for that):

https://firefoxci.taskcluster-artifacts.net/Lf_yN9B7RBSEfcb8gsm9-Q/0/public/build/buildhub.json

Hey there!

Is there any current way of determining what revision of GeckoView, Application Services, and Glean were used to build a specific version of Fenix in CI?

For release builds, yes. For regular CI builds, not yet. That said, we're close to have reproducible CI builds thanks to bug 1620585. Let me give your more context.

Android builds are built like old Java apps: they rely on maven packages. For instance, we make Geckoview release available on [1] and same thing for android-components or application-services. The last 2 though, leverage another feature of maven packages: snapshots. Glean has multiple 37.0.0-SNAPSHOT versions[2]. This helps developers to remain up to date without having to bump dependencies each time[3]. Although, over the past year, this paradigm has caused more trouble than it solved issues. That's why bug 1620585 started. Thanks to it, we will have nightly builds with deterministic version number.

Then, we can add buildhub.json there to be able to track revisions.

Regarding release builds, each release of a-c or a-s is tied to a Github release. Thus, we can get the revision from it too. That said, buildhub.json seems a more streamlined way of exposing the data, in my opinion.

Please let me know if I can provide more information 🙂

[1] https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview/
[2] https://snapshots.maven.mozilla.org/?prefix=maven2/org/mozilla/components/service-glean/37.0.0-SNAPSHOT/
[3] https://github.com/mozilla-mobile/fenix/blob/63517a52e4446880735407751cf674f80c263368/buildSrc/src/main/java/Dependencies.kt#L36

Thanks Johan, super helpful. I filed bug 1622948 specifically about adding these buildhub.json files, hope that's ok.