[Environment:] Windows 8.1 x64, OsX 10.12 62.0a1 2018-05-28 [Prerequisites:] 1. Set the app.normandy.dev_mode preference to true to run recipes immediately on startup. 2. Set the app.normandy.logging.level preference to 0 to enable more logging. 3. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90. 4. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1 [Steps:] 1. Open Control Center (https://normandy-admin.stage.mozaws.net/) 2. Create a rollout recipe targeting at least 2 preferences. 3. Approve & publish the above rollout recipe. 4. Run FF client with the prerequisites set. 5. Create, approve and publish a rollback recipe on the rollout slug for the step 2 recipe. 6. Restar/reopen the FF client with the same profile that has been used on step 4. 7. Update the Step 2 rollout recipe by updating preferences values for example. 8. Restart/reopen the FF client. [Actual Result:] After the rollout is updated (step7), on step 8 the following happens: app.normandy.recipe-runner INFO Executing recipe "Adi/ 3 types -rollout" (action=preference-rollout) app.normandy.action.PreferenceRolloutAction DEBUG updating adiz: aa value changed from changed13 to changed13x [Show/hide message details.] Updated pref rollout in unexpected state: rolled-back PreferenceRolloutAction.jsm:65 app.normandy.recipe-runner INFO Executing recipe "rollback adiz" (action=preference-rollback) app.normandy.action.PreferenceRollbackAction INFO Rolling back adiz The problem above is that only one pref. value is listed as being updated, when it should've listed all the pref. values which the rollout recipe got updated with. Running the above scenario a few times, the number of updated preferences vary, so I don't think it's a Normandy logging issue, but a race between the rollout update and the rollback. Additionally rollout update and rollback telemetry are logged for the two events. [Expected Result:] In terms of architecture, based on discussing this with :mythmon , currently "the code assumes rollouts won't change once a rollback is issued", so I'd guess the expected result should be a TBD.