The reach ping currently only evaluates the branch targeting to see whether the enrolled user qualifies for the branch. It might be useful to check the trigger as well.

Extra care needs to be taken since some triggers could potentially leak the user's browsing history.

Good point about the security and privacy concern. But if the CFR being tested in a branch of the experiment passes the security and privacy criteria (since we're collecting impressions from the users that do see the CFR), then wouldn't the same logic apply to the reach ping for users that would have seen the CFR?

Commenting on the adding trigger to this ping:

Currently, this reach ping is reporting the users that could have seen a given CFR (match the targeting criteria), but that is different from the users that would have seen a CFR (since that depends on user behavior, i.e. hitting a certain website).

What that gives us is something like this (each category is superset of the one below):

• experiment population
  • (includes everyone)
• filter eligible [what we have]
  • users who send reach ping
  • (includes those who hit trigger those who don't)
    • (for control, that would be both users who would have seen the CFR in question, as well as those who wouldn't)
    • (for the treatment of the CFR in question, that would be users who send impression / see the CFR as well as those who don't)
• filter and trigger eligible [we don't have]
  • users who send reach v2 ping described in section below
  • (only includes users who have or would have seen CFR in question)

(IMO) I think it's fine to go forward for now with the reach ping as it's currently implemented and evaluate it's usefulness in reducing noise in experiments.

If this allows us to reduce the analysis populations so that it's mostly reflective of the population we want (for example, once we reduce it, then 90% of the reduced population would have seen the CFR, estimated from the impressions in treatment) and we're not concerned about noise washing out experiment effects, then we should be good as is.

If not, however, then we would need a more specific reach ping taking triggers into account. I've laid out some specs or what that would look like below:

The requirements for a reach ping v2 that includes the trigger are thus:

• It should tell us who would have seen a given CFR for an experiment (filter and trigger match)
• It should tell us when they would have seen a given CFR (since we'll need to be able to differentiate pre-exposure and post exposure timelines / data)
• It should have coverage for every CFR defined in the experiment, for every user enrolled in the experiment
  • note: careful consideration should be made to distinguish between branches with the same CFR that has different parameters, as well as when a treatment branch is modifying / removing an existing CFR (in which case we'll need a reach ping in the treatment for the unmodified / unremoved CFR).

That would require some significant re-design (changing the checking timing, etc.) so I think it would make sense to implement as a new ping (as opposed to modifying the existing reach ping).

It turns out I'm mostly restating what Su said above, but: yes, an event that responds to the trigger is important. There are a couple of different ways that we can measure outcomes in an experiment. We can measure the effect of deploying a product experience, or we can measure the effect of encountering a product experience. We can convert between these outcomes by understanding the rate at which the product experience is encountered (activation) by users to whom the experience is available (enrollment). The activation-based approach is most sensitive because it's not diluted out by a mass of users that didn't receive the experience under test and it specifies when the experience was delivered, which allows us to understand how the intervention changes user behavior over time with higher precision.

The landed version of the reach ping allows us to refine the eligible population, which is good, but this still isn't quite an activation signal. [1] The availability of these messages for both the control and treatment branches controls the precision of the analysis both in time and in effect size. A message that a user has actually seen a CFR is not useful, except for diagnostics, unless a symmetric message is available in the control indicating that the user actually would have seen the CFR at that moment.

I agree with Su that the privacy concern is upstream of this enhancement and already present in the product.

[1] with the caveat that if virtually all users that meet the enrollment criteria also, more or less immediately, meet the trigger criteria, then the distinction is unimportant (though it suggests we could stop using targeting).

https://hg.mozilla.org/mozilla-central/rev/1f9c7ec6706b

[Tracking Requested - why for this release]:

This telemetry enhancement would enable us to do far more informative analysis on the CFR experiments that we're planning on conducting in 77.

STR for QA:

This is a follow up of bug 1471318, which only evaluates the CFR targeting for the reach ping. Aside from targeting, this also checks if the experiment users fire any of the CFR triggers, and only records the "reach" ping if both targeting and trigger conditions are met.

Prerequisites

• Install RemoteSettings devtools in nightly
• The test CFR for the Picture-in-Picture feature is located in Stage-Preview, please use RS devtools to switch the RS environment from Prod to Stage-Preview
• Still on RS devtools, click on clear local data and then force sync buttons for the messaging-experiments collection
• Restart the browser so that Experiment Manager can enroll you into the experiment

Steps to test

• This CFR has a list of video sites as the trigger (such as youtube.com, twitch.tv), the targeting is set to true for branches
• Since this experiment has two branches, control and treatment, respectively. The reach ping should only be recorded from the control branch. So we should verify both cases as follows:
  • when enrolled into the control group:
    • visit youtube.com
    • make sure you do not see the CFR chicklet in the address bar
    • go to "about:telemetry", make sure there is an entry called "messaging_experiments#reach#cfr" in the "Keyed Scalars" section, and the value should be 1
    • note that currently reach event doesn't have impression cap enabled, so if you visit youtube.com again or any other site on the trigger list, it will record another reach event
  • when enrolled into the treatment group:
    • visit youtube.com
    • make sure you see the CFR chicklet in the address bar
    • make sure there is no entry "messaging_experiments#reach#cfr" recorded in the "Keyed Scalars" section

I have verified the following by using the STR from comment 8 on the latest Firefox Nightly 78.0a1 (Build ID: 20200528032513) on Windows 10 x64, macOS 10.15, and Ubuntu Linux 18.04:

• If enrolled in the “control” branch, there is a “messaging_experiments#reach#cfr” ping in the “Keyed Scalars” section with the value set to 1 and there is no “CFR chicklet” displayed on “youtube.com”.
• If enrolled in the “treatment” branch, there is no “messaging_experiments#reach#cfr” ping in the “Keyed Scalars” section and there is a “CFR chicklet” displayed on “youtube.com”