Once Disconnect has annotated their list with DNT information, we will need to split their list into the trackers that honor DNT and those that don't.

Disconnect has submitted a PR with a sample annotated entry: https://github.com/mozilla-services/shavar-prod-lists/pull/14 It follows the format I suggested in https://public.etherpad-mozilla.org/p/dnt-loophole

Here are the new lists to derive from the Disconnect upstream list: - base-track-digest256 (DNT="", all categories except content) - basew3c-track-digest256 (DNT==W3C, all categories except content) - baseeff-track-digest256 (DNT==EFF, all categories except content) - content-track-digest256 (DNT=="", content category) - contentw3c-track-digest256 (DNT==W3C, content category) - contenteff-track-digest256 (DNT==EFF, content category) Given the latest PR from Disconnect (https://github.com/mozilla-services/shavar-prod-lists/pull/14), all of the lists should be empty except for: - base-track-digest256 (roughly equivalent to mozstd-track-digest256) - baseeff-track-digest256 (should have exactly 1 entry) - content-track-digest256 (should be the difference between mozfull-track-digest256 and mozstd-track-digest256)

Having read thru the lists2safebrowsing.py script now ... We only need the new separate lists for the tracking-protection section? I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists?

(In reply to Luke Crouch [:groovecoder] from comment #3) > We only need the new separate lists for the tracking-protection section? > I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists? Correct, we're not touching the plugin blocklist or the whitelist. Looking at the prod config file (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/prod.ini), we'll be adding these new sections: - tracking-protection-base - tracking-protection-baseeff - tracking-protection-basew3c - tracking-protection-content - tracking-protection-contenteff - tracking-protection-contentw3c and not removing any of the existing ones because of backwards compatibility with old clients. We'll add the same new lists to stage (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/stage.ini). Note that we're not changing the end-to-end stage lists (tracking-protection-testing and entity-whitelist-testing) because they exist to exercise the entry addition/removal part of the list creation script only.

:ckolos - Can you verify again here (IRC is ephemeral and I've lost track of it since last Wednesday) that Travis used/is-using the readme-and-comment-updates branch of shavar-list-creation to make the lists for the shavar.stage.mozaws.net environment?

Travis is not involved in this in the least. Jenkins, however, is using the following branch: <snip> Repo: https://github.com/mozilla-services/shavar-list-creation/ Branch Specifier (blank for 'any'): */readme-and-comment-updates The rules for the branch spec are as follows: <Wildcards> The syntax is of the form: REPOSITORYNAME/BRANCH. In addition, BRANCH is recognized as a shorthand of */BRANCH, '*' is recognized as a wildcard, and '**' is recognized as wildcard that includes the separator '/'. Therefore, origin/branches* would match origin/branches-foo but not origin/branches/foo, while origin/branches** would match both origin/branches-foo and origin/branches/foo.

I think I found this myself ... In https://deploy.mozaws.net/view/Shavar%20Stage/job/Shavar%20List%20Creation%20Stage/17522/git/ I see refs/remotes/origin/readme-and-comment-updates listed under "Revision" ... I also see refs/remotes/origin/master listed under "Built Branches" ... Are there docs about what each of these mean? In any case, the console output shows "git checkout -f f361b44e28755739bf13dac1d1d857ca5192b7ed" as the last command for the repo.

Oops, meant Jenkins. Thanks for the confirmation.

The effective change is here, along with the script output as a comment: https://github.com/mozilla-services/shavar-list-creation/commit/1f1d26cc553914e119e931a8947451fd4b65bc48 Will go thru some more verification & QA before sending the PR.

(In reply to Luke Crouch [:groovecoder] from comment #9) > The effective change is here, along with the script output as a comment: > > https://github.com/mozilla-services/shavar-list-creation/commit/ > 1f1d26cc553914e119e931a8947451fd4b65bc48 > > Will go thru some more verification & QA before sending the PR. The first part of that change (honoring the new dnt attribute) looks good. The other part is to change the content lists so that they only include entries from the content category (i.e. not anything that's in the base list). I assume you'll be tackling that in a later commit?

Ah, I had misunderstood then. Is that only for the new DNT-separated lists? Or do the old lists need to be updated as well? It looks like the old tracking-protection-full list included Content AND the other categories together.

(In reply to Luke Crouch [:groovecoder] from comment #11) > Is that only for the new DNT-separated lists? Or do the old lists need to be > updated as well? Only for the new lists. Lists are essentially frozen in time as soon as they get picked up by a Firefox release. So when we want to change them, we have to create new ones and leave the existing ones untouched. Then once there are no more users pulling them down, we can delete the old lists (e.g. bug 1277446). > It looks like the old tracking-protection-full list > included Content AND the other categories together. Correct. That was unnecessary and we'll be fixing that in the new lists.

Okay, updated to include *only* the content category in the new tracking-protection-content list.

Here: https://github.com/mozilla-services/shavar-list-creation/commit/4d57e3a9a4dac501f9b04ac092bc37c44b444f3d

:francois, :rtilder - so, I think the combination of https://github.com/mozilla-services/shavar-list-creation-config/pull/11 and then https://github.com/mozilla-services/shavar-list-creation/pull/37 should deploy the new DNT lists to stage shavar?

Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation https://github.com/mozilla-services/shavar-list-creation/commit/10c424f868b58f1d636cbe4fcd77ff1096f73829 bug 1258038: start dnt list config; define SECTION constants https://github.com/mozilla-services/shavar-list-creation/commit/1f1d26cc553914e119e931a8947451fd4b65bc48 bug 1258038: check org DNT value matches list value https://github.com/mozilla-services/shavar-list-creation/commit/3e55a64add628879d08e901ad9f3e5ebdd5d83e2 bug 1258038: SKIP|INCLUDE|ONLY logic for content https://github.com/mozilla-services/shavar-list-creation/commit/786519fb638dd041071a12b0eeebef08f1c6333d bug 1258038: set sample disconnect_urls to mozilla-services https://github.com/mozilla-services/shavar-list-creation/commit/2fc9f84f7e2750e0e851cab277815a7636045f0c bug 1258038: final updated README https://github.com/mozilla-services/shavar-list-creation/commit/98f5594d533aa87cca0c9bd5a41fc3ad83f0d5a2 Merge pull request #37 from mozilla-services/start-list-split-1258038 bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic

Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation https://github.com/mozilla-services/shavar-list-creation/commit/97704040300725180d1f2285b2644e556a90f0d9 Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic" https://github.com/mozilla-services/shavar-list-creation/commit/1cecfb3d9049d474bb7ea50589494701b4f57267 Merge pull request #38 from mozilla-services/revert-37-start-list-split-1258038 Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"

Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation https://github.com/mozilla-services/shavar-list-creation/commit/db50ce50660de5bbec7b4bf927a8989e7f31fa15 Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"" This reverts commit 97704040300725180d1f2285b2644e556a90f0d9. https://github.com/mozilla-services/shavar-list-creation/commit/db237a99a20aa72da684400b1a716551eb99a1d1 Merge pull request #39 from mozilla-services/start-list-split-1258038 Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INC…