Closed Bug 1603564 Opened 5 years ago Closed 5 years ago

[Experiment] Add-On Experiment: Search Tips aka Nudges Fx 72.0 to 74.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox72+ fixed, firefox73+ fixed, firefox74+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox72 + fixed
firefox73 + fixed
firefox74 + fixed

People

(Reporter: experimenter, Assigned: mconnor)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Add-on experiment

    What are the branches of the experiment:

- Treatment control 50%:

Standard address bar experience
        
- Control treatment 50%:

When one of the landing pages of either Google, Bing or DuckDuckGo are visited, a nudge may be shown, depending on the amount of times you saw it before and whether there's already a different notification present.
        

    What version and channel do you intend to ship to?

2% of Release Firefox 72.0 to 74.0

    Are there specific criteria for participants?


No additional filters

Countries: all

Locales: English (Australian) (en-AU) English (British) (en-GB) English (Canadian) (en-CA) English (New Zealand) (en-NZ) English (South African) (en-ZA) English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Jan 14, 2020 - Mar 23, 2020 (69 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

Primary effect is an increase in tagged search share.  Tagged Search Share is the percentage of total search (tagged+organic) going through our access points:

Formula is:

tagged / (tagged + organic)

We also have SABER to conduct automated analysis of this experiment.

    Who is the owner of the data analysis for this experiment?

bmiroglio@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

QA finished the TestRail cases on 12/5/2019. Testing is ongoing.

This experiment is signed off as GREEN - SHIP IT. For more information please see the following link: https://experimenter.services.mozilla.com/experiments/search-tips-aka-nudges/#comment580

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/search-tips-aka-nudges/

Search Tips aka Nudges

Search Tips are designed to increase engagement with the QuantumBar instead of in-content/organic search. The initial design is lightweight, but the heuristics can evolve to introduce new features and capabilities, or even win back users who've switched to organic...

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/search-tips-aka-nudges/

Search Experiment: Nudges
Firefox Release 72

We have finished testing the Search Experiment: Nudges experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We have logged multiple issues of which most of them are fixed and verified.
  • There are also 3 bugs with "Won't fix" status:
    • Bug 1600324 is an edge case,
    • Bug 1600329 is related to the megabar design (not implemented in Release 72),
    • Bug 1605738 would probably generate regressions on the telemetry side of the experiment.
  • None of the "Won't fix" issues has a high severity that would affect the end user.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Ubuntu 16.04 x64
  • macOS 10.14.5

Tested Firefox versions:

  • Firefox Unbranded Beta 72.0bx (en-US)
  • Firefox Developer Edition 72.0bx (en-US, en-CA, en-GB)

[Tracking Requested - why for this release]: Experiment is ready to go, and part of our plan to ship in the 74 cycle.

tracking-firefox72: --- → ?
User Story: (updated)
Summary: [Experiment]: Add-On Experiment: Search Tips aka Nudges → [Experiment] Add-On Experiment: Search Tips aka Nudges Fx 72.0 to 74.0 Release
Start Date: 2020-01-14 End Date: 2020-03-23

Search Experiment: Nudges
Firefox Release 73

We have finished testing the Search Experiment: Nudges experiment for Firefox 73.
The study is already live in Firefox 72, but since it's running for 69 days, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.

Quality status: GREEN

Why is this feature green?

  • During testing we haven’t found any issues.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac 10.15.1
  • Ubuntu 16.04 x64

Tested Firefox versions:

  • Firefox Beta 73.0b11 en-US
  • Firefox Beta 73.0b11 en-GB
  • Firefox Beta 73.0b11 en-CA
User Story: (updated)
Whiteboard: [no-nag]

Search Experiment: Nudges
Firefox Release 74.x

We have finished testing the Search Experiment: Nudges experiment for Firefox 74.
The study is already live, but since it's running for 69 days, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.

Quality status: GREEN

Why is this feature green?

  • During testing we haven’t found any issues.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac 10.15.1
  • Ubuntu 16.04 x64

Tested Firefox versions:

  • Firefox Beta 74.0b8 en-US
  • Firefox Beta 74.0b8 en-GB
  • Firefox Beta 74.0b8 en-CA
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.