From an email:

I met DS and survey folks on Friday to discuss the survey for Interventions, and they have a few asks for the survey.

Timing

They've asked that we trigger the survey directly after the user types one of the identified searches. Since we're already recording event telemetry on each of the control and variant branches, I hope we can do this relatively easily.

Survey groups

They've asked that we show the survey to users in three distinct cases:
Treatment: When they engage with an intervention
Treatment: When they are shown an intervention but ignore it
Control: When they would have seen an intervention but we didn't show it

Survey group size

We need to aim for 250+ responses per survey group to get useful results. Typically we'll get about 2% of users to respond to a survey, so that means we'll need to get at least 12500 users to click on an intervention. We'll see how the experiment goes, but we can keep it running if we don't get enough results.

As the first case will be the least common, we'll probably want to prompt 100% of the time. For the other cases, we'll want to randomly sample so we get about the same number of surveys in each group. As an example we think we'll see a CTR of 2%, we should survey about 2% of the other groups to keep it balanced. We don't need to be perfect here, we just need to make sure we hit 250+ without over-surveying the second and third groups.

To implement, the add-on needs to know the URLs to open. Related:

Usually we pass in a value via URL param that tells the platform which branch the user was in.

To implement the "random sample" part for the other two cases, the add-on needs to know the number of users in the experiment.

I have a few UX questions I emailed to the list that I'm blocked on for now.

Unblocked by email discussion this morning.

Merged