There is a strong concern that having safe browsing pref-ed on by default will incur a very high cost of downloads from the shavar service (more details in bug 1196016 comment 1). A workaround would be to only refresh the lists from shavar when the user enters private browsing (or opt-in to safe browsing in normal mode). Francois suggested that, rather than changing the sensitive safe browsing code, it should be possible to set the safe browsing pref to false, and flick it to true when the user opens a private browsing window, which would then automatically download a fresh copy of the whitelist and blacklist. This approach would greatly help the shavar infrastructure sustain the load of downloads.

(In reply to Julien Vehent [:ulfr] from comment #0) > Francois suggested that, rather than changing the sensitive safe browsing > code, it should be possible to set the safe browsing pref to false, and > flick it to true when the user opens a private browsing window, which would > then automatically download a fresh copy of the whitelist and blacklist. Just to be clear, my suggestion was: 1. set privacy.trackingprotection.pbmode.enabled to false by default 2. the first time a user opens a private browsing window, we flip it to true 3. we never touch that pref again (it's up to the user to enable/disable) #2 is likely to require a new pref like "privatebrowsing.neveropened = true" so that the logic would look like: On NewPrivateWindow: if privatebrowsing.neveropened: privacy.trackingprotection.pbmode.enable = true privatebrowsing.neveropened = false else: // nothing to do Testing note: we should remember to test the case where Firefox is always in Private Browsing mode.

While the scalability concerns are understandable, it's highly unlikely we can store any state locally that depends on whether the user has ever entered Private Browsing on a computer (even if there is a way to flip that state manually elsewhere). Unfortunately, from whichever angle I look at this, I cannot see a way to fulfill this privacy requirement without serving updates even to user who don't use Private Browsing mode. Marco, should I nominate this bug for triage at our daily or is this already tracked for the Platform meeting?

(In reply to :Paolo Amadini from comment #2) > While the scalability concerns are understandable, it's highly unlikely we > can store any state locally that depends on whether the user has ever > entered Private Browsing on a computer (even if there is a way to flip that > state manually elsewhere). Unfortunately, from whichever angle I look at > this, I cannot see a way to fulfill this privacy requirement without serving > updates even to user who don't use Private Browsing mode. > > Marco, should I nominate this bug for triage at our daily or is this already > tracked for the Platform meeting? Thanks Paolo. Adding bug to triage so we can discuss today.

We spoke about this in our desktop standup today. Unless we misunderstand this request, we can't do it because it compromises the local safety of PBM. We've had a few other cases where we've had to back away from desired functionality because doing so would save state on a user's machine that they'd visited PBM. In this case, we have a new pref pbm.neveropened showing a true/false state. If the concern is shavar load, is there another way to do to this? Could we throttle Shavar updates so we're not constantly updating people who aren't using the feature? Maybe, everyone gets the list (staggered) every 2 weeks *BUT* we also check for the latest list everytime you open PBM? I'm guessing this touches some of the sensitive code though.

Sorry, I just saw that *I'm* the linked commenter in the other thread. I appreciate the effort to help us cut costs. If it's financial (and not service stability) at issue, I'm a lot less concerned. I'll work w/ leadership to make sure we have proper budget. We did get a better entity list from Disconnect today and that should dramatically cut the size of the list we serve, I'll share that as soon as they share a link so we can forecast a bit.

It would really help me firm up finance and operational numbers if someone could give me a better idea of the audience size. I'm using our current AMU as a worst case scenario, but if someone can point me to "a" correct-ish ADU number, that would help immensely.

> If the concern is shavar load, is there another way to do to this? Could we throttle Shavar updates so we're not constantly updating people who aren't using the feature? One approach that wouldn't require changing any of the list update code is to throttle the feature similar to what we did with Hello. Something like: Default values: - privacy.trackingprotection.pbmode.enabled = false - privacy.trackingprotection.pbmode.throttled = true On NewPrivateWindow: if not privatebrowsing.pbmode.throttled: privacy.trackingprotection.pbmode.enable = true else: // nothing to do We can start gradually pushing out "privacy.trackingprotection.pbmode.throttled = false" to clients and then in a future release we can take this hack out entirely.

francois: I just realized the problem with my suggestion to throttle. This is a part of a big 42 launch, so everyone needs to get it when they download. ckolos MAUs is probably correct, or weeklies. Thanks for doing this. I think we just need to run the worst-case numbers, have the conversation regardless whether it feels scary or manageable. I don't currently see a short-term alternative to just paying and making sure the VPs are ok with it. We do have have mitigation strategies for cost beyond near-term. The new pared down entity whitelistlist is about 1/6 the size of the old by Francois' numbers at ~ 322KB. We also know that incremental shavar updates is a med term possibility

How about the following scheme: Default values: - privacy.trackingprotection.pbmode.enabled = false - privacy.trackingprotection.pbmode.throttled = true On FirefoxStartup: if (random 10% chance): privacy.trackingprotection.pbmode.enable = true privacy.trackingprotection.pbmode.throttled = false else: // nothing On NewPrivateWindow: if privacy.trackingprotection.pbmode.throttled: privacy.trackingprotection.pbmode.enable = true privacy.trackingprotection.pbmode.throttled = false else: // nothing In other words, every time you start Firefox, you have a 10% chance of being "un-throttled" and downloading the list immediately. Once you are un-throttled, you are like that forever. Also, if you open private browsing, then we un-throttle you immediately. But that doesn't reveal anything to anyone because you could just have been un-throttled automatically. Of course, this is just to minimize the load at launch. We get rid of this hack entirely in 43 or 44.

Given that we have addressed the cost issues in other ways and that the server folks are not worried about load on day 1, we can close this issue.