[Environment:] Windows 8.1 x64 62.0a1 20180516220130 61.0b5 20180514150347 [Prerequisites:] 1. Set the app.normandy.dev_mode preference to true to run recipes immediately on startup. 2. Set the app.normandy.logging.level preference to 0 to enable more logging. 3. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90. 4. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1 [Steps:] I. 1. Open Control Center (https://normandy-admin.stage.mozaws.net/) 2. Create a rollout recipe with the blob: { "preferences": [ { "preferenceName": "test.int.1", "value": 1 } ] "slug": "rollout-test" } 3. Save, Approve, Publish the recipe. 4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+) 5. Open Browser Console and notice the logs. OR II. 1. Open Control Center (https://normandy-admin.stage.mozaws.net/) 2. Disable a published recipe. 3. Publish again the disabled recipe. 4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+) 5. Open Browser Console and notice the logs. [Actual Result:] 1526549697907 app.normandy.recipe-runner ERROR Could not fetch recipes from https://normandy.stage.mozaws.net/api/v1: "Error: recipe signature is not valid" [Expected Result:] Recipes should be signed correctly. [Note:] 1. If all the recipes that have been updated/created today are disabled, Normandy works. 2. The recipe signature error is returned for any type of new/updated recipe (rollout / prefs exp.)

I've just noticed that there has been a Normandy server upgrade, but I'm not sure if this is related to it or a signature issue. Either way, my guess would be that this affects both production and staging environment + all FF clients that run Shield/Normandy.

This may also be related to the Autograph update, which happened recently. CC Ulfr. I'll look into this. It isn't known if this is affecting prod yet, or why the signatures aren't valid.

This sounds like a blocker.

We've determined that this is not related to Autograph, and is likely caused by the recent update to Normandy server. We have a fix incoming.

Fix is here: https://github.com/mozilla/normandy/pull/1378

The fix from comment 5 has been merged and deployed to stage. I've verified that the STR from comment 0 no longer cause the problem. The problem still exists on prod, and we'll be deploying the fix there soon.

This has been deployed to prod.

I still can reproduce the issue on staging for both new and existing recipes. For example if you disable/publish recipe https://normandy-admin.stage.mozaws.net/recipe/422/ and run pre-requisites the error is hit again.

Commits pushed to master at https://github.com/mozilla/normandy https://github.com/mozilla/normandy/commit/79475df13d6796b453e135521c8a4c4860b8c0b3 Bug 1462290 p1 - Adding failing test for signatures during enable https://github.com/mozilla/normandy/commit/87f2ac282f2fc593e75a1b4b67f5d9aa7d522d45 Bug 1462290 p2 - Refresh recipe during enabling, so it is enabled when signed https://github.com/mozilla/normandy/commit/a2b146cdb638dd38df9259582a44fcee3d6b42b5 Merge #1382 1382: Bug 1462290 - Correctly update signatures when enabling recipes r=rehandalal a=mythmon Splitting this into two commits, one with a failing test, and one with the fix. Co-authored-by: Mike Cooper <mythmon@gmail.com> Co-authored-by: Rehan Dalal <rehandalal@gmail.com>

Rehan and I found another problem that affects this in a different situation that the one I tested yesterday. We have fixed it, and deployed that fix as v92 to stage.

v92 has now been deployed to prod. Adrian, can you verify the fix on stage?

(In reply to Michael Cooper [:mythmon] from comment #12) > v92 has now been deployed to prod. > > Adrian, can you verify the fix on stage? Sure. Since this issue was server side issue I've verified that the new/update issues are now handled correctly and recipes signed accordingly: Firefox clients (60/61/62) runs new/updated recipes successfully. Note that this issue cannot be verified per se on production since we are lacking rights and also it would be bad practice to create test recipes on prod. However, indirectly, there are new (>05.18) functioning recipes on Normandy prod., therefore we can safely assume that the production Normandy deploy fixed the issue there as well. environment: 60.0.1 20180516032417 (ESR) 61.0b6 20180517141400 62.0a1 20180520220103 Windows 8.1 Ubuntu 16.04