Split out from https://bugzilla.mozilla.org/show_bug.cgi?id=1520262

I need to figure what's involved with porting the v2 personalization into the new discovery stream code base. At a glance, this has more complex timing considerations than v1.

To test:

1. Create a brand new profile
2. Set browser.newtabpage.activity-stream.asrouter.devtoolsEnabled to true
3. Set browser.newtabpage.activity-stream.discoverystream.endpoints to "http,https"
4. Set browser.newtabpage.activity-stream.discoverystream.spocs-endpoint to "https://e5e8374d-fffa-4a77-a604-31603852b6fd.mock.pstmn.io/spocs/v2"
5. Open about:newtab#devtools-ds
6. Ensure "show_spocs" is checked
7. Scroll down a bit and click "Enable V2 Personalization"
8. Scroll up and click "Trigger Idle Daily"
9. You should now have all the attachments from remote settings downloaded.
10. Wait a second or two, then click "Expire Cache"
11. Wait a second or two, then click "Trigger System Tick"
12. Wait a second or two, then refresh
13. Look for a spoc under "spoc data" and click "collapse/open" on any of those spocs.

Expected:

You're looking for values in the spoc data score to be different than item_score. item_score is what comes from adzerk, and is the starting score.

score is the updated score after personalization.

It is because we used an empty profile that it didn't produce an interesting personalization result. You can try this on an older profile with some history to get more interesting results, but it's harder to re trigger the setup from scratch more than once.

Ideally we test this using regular browser usage to trigger all the cache clears and data downloads on natural intervals. But that can take 24 plus hours because of remote settings and Discovery Stream relying on daily idles, worst case it's 48 hours before we start waiting for a system tick and cache expire, which can take up to another 12 hours, this isn't really practical. So we forced it to happen now using devtools.

Pocket JIRA shadow ticket https://getpocket.atlassian.net/browse/DL-34

Some updates.

1. I have a newest pull request where I've moved most of the switcher and provider logic into a new feed. This is probably my best pr to date.
2. I've tested the new code using personalization v1 using these steps: https://github.com/mozilla/activity-stream/pull/4864#issuecomment-479962039 and it produced the expected results. original score: 0.2036, personalized score: 0.992036
3. I then used the above steps with v2 and also produced new, but different results. original score: 0.2036, personalized score: 0.4090260872193703

Step 3 is interesting, producing a different and lower score than v1. Not particularly conclusive, as a lower score doesn't mean less or more accurate scoring, but def means something has changed. This is in contrast to if the score was the same between v1 and v2. Dunno what this all means yet, but stuff IS happening that we need to figure out the best way to get data from.

https://hg.mozilla.org/mozilla-central/rev/4cb264a0a617

I have verified that the "item_tags", "nmf_tags", and "nb_tags" values are present in the spoc data for the v2 personalization, using the steps from comment 2. Verified using the latest Firefox Nightly (74.0a1 Build ID - 20200127213955) installed on Windows 10 x64, Mac 10.15.2, and Ubuntu 18.04 x64.