User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36

Steps to reproduce:

I set the following AutoConfig files.

C:\Program Files\Mozilla Firefox\defaults\pref\autoconfig.js
// IMPORTANT: Start your code on the 2nd line
pref("general.config.filename", "firefox.cfg");
pref("general.config.obscure_value", 0);

C:\Program Files\Mozilla Firefox\firefox.cfg
// IMPORTANT: Start your code on the 2nd line
defaultPref("browser.newtabpage.activity-stream.feeds.section.topstories", false)

I closed Firefox, deleted "C:\Users\admin\AppData\Roaming\Mozilla" and restarted Firefox.

Actual results:

After launching Firefox, I clicked on the new tab and saw the Top Stories section is displayed. If I closed Firefox and relaunched it then it would hide the Top Stories. So the value doesn't take effect until the second time you start Firefox.

Also, if you use LockPref instead of defaultPref then it'll take effect on the first run.

Expected results:

It should prevent Top Stories from being displayed the first time Firefox is started.

The component has been changed since the priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

This is breaking the new Firefox Home policy.

Ed, can you point me somewhere to debug this?

I did notice that on first run, browser.newtabpage.activity-stream.feeds.section.topstories is explicitly set to true, so it's being set after policy sets it to false.

Do you know where this pref gets set?

Note that we had fix all of this a while ago, so it regressed. I can't find that bug.

Found it - https://bugzilla.mozilla.org/show_bug.cgi?id=1496241

Clearing priority so we talk about this regression in triage today. Thank you for flagging :mkaply!

FYI, the reason I didn't build a test for this is I couldn't easily do it because pocket comes in late on the new tab so I couldn't check for the presence of the pocket section.

Hmm.. Did this AutoConfig ever work for browser.newtabpage.activity-stream.feeds.section.topstories ? I could see dynamic (geo/region-based prefs) always replacing the default pref value:
https://searchfox.org/mozilla-central/rev/250f5cc9fb8bdcbb6b23d2a06acfd48addb2f99b/browser/components/newtab/lib/ActivityStream.jsm#728-736

The behavior here is on startup, if geo pref is not ready yet, set some generic default, and later on GEO_PREF change, it'll replace that default with the geo-specific value with a new default. (both changes are to the default pref branch)

The fix from bug 1496241 seems to only handle not replacing the default pref on init:
https://searchfox.org/mozilla-central/rev/250f5cc9fb8bdcbb6b23d2a06acfd48addb2f99b/browser/components/newtab/lib/ActivityStreamPrefs.jsm#66-70

I believe AutoConfig behaves by having a default pref on the default pref branch before activity stream init happens, and the fix above worked by just checking if a default pref value existed already.

I thought it worked before, but I guess for the Pocket stuff it must not have.

Autoconfig (and policy) definitely work by setting the default value on the branch.

Isn't the generic default set to true?

Can't we just not do the geo if the default value is false?

(In reply to Mike Kaply [:mkaply] from comment #7)

Isn't the generic default set to true?

Nah, topstories is default false as most locales and regions don't show it, and we don't want to briefly show stories then remove them.

Can't we just not do the geo if the default value is false?

Yeah, seems reasonable to add the similar default.get() !== undefined also to the dynamic prefs section.

Is this something we want to uplift to 70 beta? Is there a good STR for QA? (I've been locally changing firefox.js for development builds.)

Is this something we want to uplift to 70 beta? Is there a good STR for QA? (I've been locally changing firefox.js for development builds.)

Yes.

And for QA, that can just add this policy and create a new profile

https://github.com/mozilla/policy-templates/blob/master/README.md#firefoxhome

STR:

1. download a mac en-US build and extract/copy the application from the dmg
2. in finder, right-click the extracted application -> Show Package Contents
3. go into Contents then Resources
4. create a new folder named distribution and open it
5. create/copy in a new file policies.json (attached), so there should now be a Nightly.app/Contents/Resources/distribution/policies.json with:

{
  "policies": {
    "FirefoxHome": {
      "Pocket": false
    }
  }
}

6. open the application with a new profile
7. open a new tab (probably already showing about:home)

expected: no pocket section shown

actual: pocket section shown

Probably good to just double check that about:config shows browser.search.region as US. Note: this bug only happens on the first run as once the region pref is set, subsequent starts of Firefox will correctly hide the pocket section.

For windows, I believe distribution directory goes next to firefox.exe and similarly on linux, distribution goes next to firefox-bin

[Tracking Requested - why for this release]: Autoconfig/Policy turning off Pocket in new tab still shows stories on first run

I guess this affects esr68 as FirefoxHome policy was added in bug 1548080

https://hg.mozilla.org/mozilla-central/rev/eba9b9f96abc

I tried to verify this issue with the steps from the description and also the ones from comment 12, but I encountered the following problems:

• If I add a user.js before starting the profile with the browser.search.region preference set to US I am not able to reproduce the initial issue on affected builds.
• I also tried using a VPN in order to have the browser.search.region preference set by the browser, but the browser.search.region preference was not added even after a restart. I tried this scenario with Windscribe and Tunnelbear.

@bryanh Could you please verify if you can reproduce the issue on Latest Firefox Nightly? You can find the build here or a direct link to the zip.

@Ed, do you think I am missing something or I am blocked by the problems I am hitting?

The bug is indeed triggered by the dynamic changing of browser.search.region, so setting it via user.js ahead of time will prevent it from changing. I believe cmuresan has been able to get the region set dynamically via network requests as that's how Pocket section decide to show or not. ?

Sadly I've only done the dynamic change to verify spoc endpoint changes, but in that case I didn't have to have the pref set beforehand so I've just changed it manually.
I'm really at a loss as to what we could do if using a VPN fails to set the pref for us.

Should we just create a separate preference for this?

The bug happens when a new profile has browser.search.region pref unset then during first run, it dynamically gets set to US.

cmuresan/acupsa, do you know if you're using the same VPN? Maybe it needs to wait longer to set the region pref? I suppose a sanity check of does some IP lookup website show that your region is indeed US? E.g., https://whatismyipaddress.com/

@Ed we did test using the same VPNs, we are actually office colleagues (desk colleagues even).

I verified the IP displayed on the https://whatismyipaddress.com/ website while having either VPN (Tunnelbear, Windscribe and Express VPN - Paid) connected and it displayed the correct location. I tried with 3 different US locations just to be sure.
I also tried waiting for ~10 minutes to see if the browser.search.region preference is set, and also restarted the browser, but sadly the preferences was still not set.

Actually, if it's consistently not setting the pref, then you should be able to just manually change browser.search.region from (unset) to US which should normally cause the Pocket section to appear with no AutoConfig/Policy. Then when using a policy to turn off the pocket section, without the fix from this bug, the Pocket section appears (and subsequent runs hides it); and with the fix here, the Pocket section should not appear during the first run when you manually set region pref as well as subsequent runs.

I tried again to reproduce the issue with the additional information provided but it just seems that I am out of luck. I tried with 2 Firefox builds on which the issue was supposed to reproduce, Firefox 70.0b1 Build ID:20190805220030 and Firefox 71.0b1 Build ID: 20190910095613, on Windows 10 x64 and Mac 10.14.5.

I tried with the following two scenarios, and in both cases, I blocked updates using the channel-prefs.js file in order to avoid unwanted browser updates:

Scenario 1 - adding the policies file after creating the profile:
1. Added a user.js with the browser.search.region preference set to US
2. Opened the profile and opened a new tab -> The pocket section is displayed
3. Added the distribution folder in the firefox folder.
4. Restarted the browser.
Result: The Pocket section is not displayed.

Scenario 2 - adding the policies file before creating the profile:
1. Added the distribution folder in the firefox folder.
2. Added a user.js with the browser.search.region preference set to US.
3. Opened the profile and opened a new tab.
Result: The Pocket section is not displayed.

Ah! I think I see the confusion. When I say set browser.search.region in comment 22, I meant in the running Firefox instance from about:config so that it simulates the pref changing dynamically instead of the pref already being set.

1. create a new profile
2. open new tab and verify no pocket section (because no region set)
3. open about:config and create browser.search.region set to US
4. existing and new new tab should show pocket section

And for testing this bug with policies file:
0) add distribution folder with policies.json under firefox folder

1. create a new profile
2. open new tab and verify no pocket section (because no region set)
3. open about:config and create browser.search.region set to US

expect with fix: existing and new new tab should still not show pocket section

The reason why comment 23 both scenarios did not trigger the bug is that by setting user.js to have US, the region never changes from (unset) to US in a running instance of firefox which behaves differently from "first launch of a new profile"

Oh! I just found some code added this year that does some timezone checks before storing US region in the pref:
https://searchfox.org/mozilla-central/rev/7531325c8660cfa61bf71725f83501028178cbb9/toolkit/components/search/SearchService.jsm#128-131

I'm guessing if you created a new profile and set browser.search.log to true before starting the profile, you'll probably see something like:

*** Search: _fetchRegion got success response in 700ms: US
*** Search: storeRegion mismatch - US Region, non-US timezone

acupsa, could you try changing your clock to Pacific time before launching a new profile with US VPN?

@Ed thanks for all the help and patience! It finally worked.
I managed to reproduce the issue on Firefox 70.0b1 Build ID:20190805220030 on Windows 10 x64, Mac 10.14.5 and Debian 9 with the following scenarios:
Scenario 1 with policies:
Prerequisites:

• Updates are blocked from the channel-prefs.js file.
• Have VPN turned on and set on the US.
• The device's timezone is set to the US.
• The distribution folder with policies.json is added under the firefox folder (Resources folder for Mac OS).
• Before opening the profile have a user.js with the browser.search.log preference set to true added to it.

Steps:

1. Open the profile from the prerequisites.
2. Open a new tab.

Result:

• The Recommended by Pocket section is displayed on the first launch of the profile. The Pocket section is not displayed after a browser restart.

Scenario 2 with the cfg and auconfig. files:
Prerequisites:

• Updates are blocked from the channel-prefs.js file.
• Have VPN turned on and set on the US.
• The device's timezone is set to the US.
• The autoconfig.js file is added to the Mozilla Firefox\default\pref folder.
• The firefox.cfg file is added under the firefox folder (Resources folder for Mac OS).
• Before opening the profile have a user.js with the browser.search.log preference set to true added to it.

Steps:

1. Open the profile from the prerequisites.
2. Open a new tab.

Result:

• The Recommended by Pocket section is displayed on the first launch of the profile. The Pocket section is not displayed after a browser restart.

Following the scenarios from above, I verified the issue on Firefox Nightly 71.0b1 (Build ID: 20190919184237) on Windows 10 x64, Mac 10.14.5 and Debian 9.
The Recommended by Pocket section is not displayed on the first launch of the profile and neither after a couple of restarts.

Should we go ahead and uplift this?

https://github.com/mozilla/activity-stream/commit/58f5a67facf8e44ddc53802adef18615c5585a0f

Comment on attachment 9094697 [details]
Bug 1559185 - Use existing default prefs for dynamic prefs too

Beta/Release Uplift Approval Request

• User impact if declined: AutoConfig and Policy deployments incorrectly show Pocket on first run.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Comment 26
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small JS change to the handling of dynamic prefs (which depend on geo/region) to match the AutoConfig/Policy behavior of non-dynamic prefs. This was a little bit tricky to verify on nightly given the geo/VPN/time-zone + firstrun + custom policy requirements, but we finally figured it out for comment 26.
• String changes made/needed:

Also ESR68 please :).

Comment on attachment 9094697 [details]
Bug 1559185 - Use existing default prefs for dynamic prefs too

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Prevents enterprise deployments from turning off Pocket for firstrun
• Fix Landed on Version: 71
• See comment 29

Comment on attachment 9094697 [details]
Bug 1559185 - Use existing default prefs for dynamic prefs too

Fix for new tab issue, verified in nightly, adds new tests.
OK for uplift for beta 10.

https://hg.mozilla.org/releases/mozilla-beta/rev/dc09238320ee

Comment on attachment 9094697 [details]
Bug 1559185 - Use existing default prefs for dynamic prefs too

Approved for 68.2esr.

https://hg.mozilla.org/releases/mozilla-esr68/rev/59b26012d530

I have verified this issue with the same steps as in Comment 26 on Firefox Beta 70.0b11 (Build ID: 20190930132843) and on Firefox 68.2.0esr (Build ID: 20191002005440) downloaded from treeherder.
The Recommended by Pocket section is not displayed on the first launch of a new profile and neither after a couple of restarts.