[meta] CFR Machine Learning Experiment
Categories
(Firefox :: Messaging System, enhancement)
Tracking
()
People
(Reporter: k88hudson, Unassigned)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Keywords: meta)
Attachments
(8 files, 12 obsolete files)
(deleted),
text/x-google-doc
|
Details | |
(deleted),
application/x-xpinstall
|
Details | |
(deleted),
application/x-xpinstall
|
Details | |
(deleted),
text/plain
|
tdsmith
:
data-review+
|
Details |
(deleted),
application/x-xpinstall
|
Details | |
(deleted),
application/x-xpinstall
|
Details | |
(deleted),
application/x-xpinstall
|
Details | |
(deleted),
application/x-xpinstall
|
Details |
Reporter | ||
Comment 1•5 years ago
|
||
Updated•5 years ago
|
Uploading control add-on v0.1.0 for test-signing
Uploading control add-on v0.1.1 for test-signing
Uploading treatment add-on v0.1.1 for test-signing
@jvehent These are the control and treatment condition addons for the CFR study. Requesting signing whenever/however possible so that we can have preliminary QA performed on just the client-side functionality in parallel with continued refinement of the end-to-end testing plan and further signoffs as per the RRA.
Comment 7•5 years ago
|
||
Comment 8•5 years ago
|
||
Updated•5 years ago
|
Updated•5 years ago
|
Comment 10•5 years ago
|
||
Comment 11•5 years ago
|
||
Comment 12•5 years ago
|
||
Updated•5 years ago
|
Comment 13•5 years ago
|
||
Comment 14•5 years ago
|
||
Comment 15•5 years ago
|
||
By the version number bump I assume that this is intended for a final signature now? Have QA and a Firefox Peer signed off on this version?
Comment 16•5 years ago
|
||
By the version number bump I assume that this is intended for a final signature now? Have QA and a Firefox Peer signed off on this version?
It is intended for final QA, but not production signing yet. For us, v1.0.0 means fully implemented. Subsequent rounds of QA will lead to 1.0.1, 1.0.2 etc until a version has been green-lighted by QA, after which no final version bump will be made, only a re-signing with the production key. This way Normandy can ship the same build that was green-lighted by QA, removing any risks associated with rebuilding the add-on.
Comment 17•5 years ago
|
||
Sounds great. Thanks for the info.
Comment 18•5 years ago
|
||
Comment 19•5 years ago
|
||
Comment 21•5 years ago
|
||
Is https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/master/docs/TELEMETRY.md the correct public documentation? Please update either the documentation or the request to match what you plan to collect; thanks!
Comment 22•5 years ago
|
||
(In reply to Tim Smith 👨🔬 [:tdsmith] from comment #21)
Is https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/master/docs/TELEMETRY.md the correct public documentation? Please update either the documentation or the request to match what you plan to collect; thanks!
Yes. Everything not listed in the data steward review is a telemetry feature that is already collected in release or a computed value based on that data - does that need to be added to the doc?
Comment 23•5 years ago
|
||
Comment 24•5 years ago
|
||
No additional legal review needed. Alicia Gray will sign off for both Trust/Security and legal.
Comment 25•5 years ago
|
||
Comment 26•5 years ago
|
||
Comment 27•5 years ago
|
||
Uploading control and treatment add-on v1.0.1. This changes the name of the cfr-ml-model
collection to cfr-ml-models
Comment 28•5 years ago
|
||
Comment 29•5 years ago
|
||
Updated•5 years ago
|
Comment 30•5 years ago
|
||
[PI-438] CFR personalization experiment
Firefox Release 72.x
We have finished testing the CFR personalization experiment.
We have found the following issue which we consider a blocker for the experiment:
Bug 1608803 The user is wrongly unenrolled from the study due to unexpected “CAUGHT ERROR” exception after browser restart - https://bugzilla.mozilla.org/show_bug.cgi?id=1608803
QA’s recommendation: RED - DON’T SHIP
Reasoning:
- The found issue is considered blocker for the release from QA perspective due to the fact that users enrolled in the study will be unenrolled after a browser restart.
- Testing was blocked for the treatment branch (main branch) of the add-on therefore we were not able to test any of the functionalities specific to this branch.
- The "treatment" branch scores are not updated after each “cfr-ml-model” Remote Settings bucket update, due to the same "Invalid telemetry payload" error displayed in the console. Might have the same root cause as the issue logged, but we cannot say for sure.
- We haven’t received steps to naturally trigger the CFR recommendation and all the testing performed until now was based on manually overriding the prefs values.
What needs to be done?
- In order to receive a GREEN sign-off we will need the blocker issue to be fixed and we will require testing steps to update the value of the “browser.messaging-system.personalized-cfr.scores” preference in order to naturally trigger a CFR recommendation.
Testing Summary:
- Verified that the control add-on is installed and it correctly modifies the targeted preferences.
- Tried to install the treatment add-on via Normandy and encountered Bug 1608803 (https://bugzilla.mozilla.org/show_bug.cgi?id=1608803).
- We could not test the add-on using Firefox Dev Edition due to Bug 1608832 (https://bugzilla.mozilla.org/show_bug.cgi?id=1608832) that blocks Normandy stage recipes from executing on the channel.
- Tried to change the value of the “browser.messaging-system.personalized-cfr.scores” preference by force updating the “cfr-ml-model” collection, adding an override to shorten the polling period, and waiting an hour for the collection to naturally update without success.
Tested Platforms:
- Windows 10 x64
- macOS 10.15.1
- Ubuntu 18.04
Tested Firefox versions:
- Firefox Nightly 74.0a1
- Firefox Unbranded Release 72.0.2
- Firefox Dev Edition 73.0b4
Regards,
Patriciu
Updated•5 years ago
|
Updated•4 years ago
|
Description
•