Is this something we still care about and are willing to prioritize?

Reset the priority and assignee as now this ping regains the interest.

Hey chutten, could you take a look at this new collection?

You can find the document here.

Thanks!

Hi; I'm a data scientist working on automated experiment analysis infrastructure, and this provides a critical improvement in our ability to analyze these experiments quickly and rigorously, so thank you for working on it.

I left some comments in phabricator with a couple of small questions about the documentation, but I had a few broader questions. I'm not sure whether a distinction is being made between cohorts and branches; can you explain? What does it mean to qualify for a cohort? What's the difference between qualification and experiment enrollment?

For this to be useful, this event has to be sent at the moment when a client in the control branch would have seen a message, if they had been enrolled in any of the variant branches.

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #9)

I left some comments in phabricator with a couple of small questions about the documentation, but I had a few broader questions. I'm not sure whether a distinction is being made between cohorts and branches; can you explain?

Hi Tim, thanks for the comments/questions! There is no distinction between them, we use them interchangably. But I agree, we should be more consistent on the wording. Will review the event schema with our DS (:shong), and unify the wording in the code/docs.

What does it mean to qualify for a cohort? What's the difference between qualification and experiment enrollment?

There are two levels of targeting for Messaging Experiments (including Firstrun (about:welcome), CFR, Moments Page, What's New Panel, and Snippets). The first level is for the experiment enrollment, like those filters defined in the Normandy experiment recipe. Here is an example for a CFR experiment, this is the enrollment targeting. The second level is for message targeting. For instance, we only show a recommendation to users who have met certain conditions. Note that each branch has its own targeting, whether or not to show the message is up to which branch that user was enrolled in.

For this to be useful, this event has to be sent at the moment when a client in the control branch would have seen a message, if they had been enrolled in any of the variant branches.

Agreed. The current implementation will report all the qualified branches (including the enrolled one) from all the users enrolled in the experiment. Will confirm that with Su to ensure we're collecting the right thing.

Thanks for the examples; that's really helpful. This addresses message targeting, right? And enrollment targeting and telemetry is handled by ExperimentManager?

The two branches here seem to have identical targeting and trigger expressions; is it the case that all clients would send a reach event containing both branch names when the CFR is displayed? That would be great.

I'm concerned that there's an 80 character limit on event values; firing a separate event for each branch instead of concatenating branch names would avoid the risk of failing to send telemetry for complex experiments with many branches, and also has better query ergonomics for the use cases I have in mind.

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #11)

Thanks for the examples; that's really helpful. This addresses message targeting, right? And enrollment targeting and telemetry is handled by ExperimentManager?

Exactly, this ping only cares about the message targeting. The enrollment targeting and telemetry are indeed handled by ExperimentManager, which, in turn, relies on Normandy. I've added this bit in the patch.

The two branches here seem to have identical targeting and trigger expressions; is it the case that all clients would send a reach event containing both branch names when the CFR is displayed? That would be great.

Indeed, their targeting are exactly the same. Since we wanted to run this as an A/A test for ExperimentManager.

I'm concerned that there's an 80 character limit on event values; firing a separate event for each branch instead of concatenating branch names would avoid the risk of failing to send telemetry for complex experiments with many branches, and also has better query ergonomics for the use cases I have in mind.

Yeah, we're aware of those length limitations of Events telemetry, and understanding the risk of string truncation. Will finalize the strategy with Su for a more robust solution. Thanks for the reminder!

Have gone through the proposed schema and other details with Su, and we believe that the current implementation is sufficient for us to get started.

We will also be using the comma-separated string to record the qualified branch IDs for now, and we will mitigate the truncation risk by experiment review and monitoring.

https://hg.mozilla.org/mozilla-central/rev/8d47bff523b1

I have verified that the "messaging_experiments#reach#cfr" scalar is successfully displayed in the "about:telemetry" page using the following steps:

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Open a new tab am click the "Facebook" top site.
3. Repeat the previous step 4 times.
4. Navigate to the "about:telemetry" page and go to the "Keyed Scalars" section.
5. Observe the scalars displayed in the "telemetry.event_counts" section.

Verified using the Firefox Nightly 77.0a1 and latest Firefox Nightly (78.0a1 Build ID - 20200504205324) installed on Windows 10 x64, Mac 10.15.4, and Ubuntu 18.04 x64.