We should have an add-on Hotfix that can be rolled out to all users on unsupported Versions of Firefox that gives these users a warning that they are out of date, a quick link to a full installer to update them to the latest version of Firefox (since their updater is likely broken), and that allows them to submit their update logs to us for analysis. rstrong should give input here on what logs would be beneficial for us to solving potential bugs in the updater.

Note: it should be possible to use the existing update ui that notifies users to manually download the installer. This way it is already localized, etc. The files are the same as previously. In the update directory, active-update.xml, updates.xml, updates/backup-update.log, updates/last-update.log, and updates/0/update.log if they exist for the updater. If updates aren't found then messages from the error console starting with AUS: for the update component.

just a change, instead of a link to the full installer it would be nice to use the internal updater mechanism to just force a download and install of a full build. Linking to the installer and expecting the user to go through the install is alot of steps and will hinder the success of this hotfix.

That assumes the updater works, though, which is quite unlikely in the relevant problem scenarios.

Is there a way we can force the updater to clear all previously downloaded partials (basically reset it) and then force it to download and install a full-installer through a more fail-safe way? Rob?

There wouldn't be a way to have update run an installer (they are apples and oranges) and there is no reason to think that clearing out a downloaded update (it only downloads one at a time) would help even if we could run an installer from it or that whatever is mucking things up which has yet to be identified wouldn't muck that up.

The hotfix addon could be written to download and run the installer itself.

Definitely

That works then. I just want to avoid the situation of making users (who are probably less technical) having to click a link, save a file, find that file, run it, and go through a full install.

At least in some if not most cases they will still need to interact with UI in that they will need to at the very least confirm elevation on Windows.

That's fine, I believe we can live with a 2-3 click process rather than a 5-6 click process.

To further streamline things, the hotfix addon could ask for permission to install the upgrade and then run the installer in a zero-click mode. (not silent; still show a UI, just without a second confirmation of install or options) This could get it down to 1-2 clicks.

One thing that would also be a good idea would be to have the hotfix addon re-prompt the user later if they agree to install the upgrade but it does not get completed fully. (e.g. didn't get permission to install) We'd then need a dedicated help page to give to these people on the second go-around.

Adding wireframe from bug 994882. As Smedberg notes in bug 994882#c26, we should redirect to a SUMO page if the installer failes.

Adding Saptarshi. We should be in a position to measure the effect of this rollout when it happens. Quick question -- this hotfix will reach *all* users from Fx 12 onwards, correct?

(In reply to John Jensen from comment #14) > Quick question -- this hotfix will reach *all* users from Fx 12 onwards, > correct? No, I believe this was just targeting Windows initially: "This quarter we are focusing on Windows users. If that is successful, we will likely attempt to deploy a similar hotfix for Mac users." Also, the hotfix doesn't get deployed to users with application updates disabled or to users on older versions without the new hotfix cert fingerprint who didn't get the hotfixes to rotate the fingerprint before the cert expired. See bug 803596 and bug 874513.

(In reply to Matthew N. [:MattN] from comment #15) > Also, the hotfix doesn't get deployed to users with application updates > disabled It doesn't? Why not? That seems like a serious problem for this effort... > or to users on older versions without the new hotfix cert > fingerprint who didn't get the hotfixes to rotate the fingerprint before the > cert expired. See bug 803596 and bug 874513. That should be a very small set of users in theory, right?

(In reply to :Gavin Sharp (email gavin@gavinsharp.com) from comment #16) > (In reply to Matthew N. [:MattN] from comment #15) > > Also, the hotfix doesn't get deployed to users with application updates > > disabled > > It doesn't? Why not? That seems like a serious problem for this effort... Hotfixes are essentially providing updates so hotfix checks were tied to application update preferences[1]. Adding a new preference just for hotfixes that managed environments would have to change for their existing user base wouldn't be less ideal IMO. On firefox-dev it sounded like the focus was on helping people whose updates were not working and not forcing people with updates disabled to upgrade so I don't think this is that serious. > > or to users on older versions without the new hotfix cert > > fingerprint who didn't get the hotfixes to rotate the fingerprint before the > > cert expired. See bug 803596 and bug 874513. > > That should be a very small set of users in theory, right? Well, we cut it pretty close at least one of the two times IIRC so computers that are only used occasionally (e.g. on weekends or a few times per month) very possibly missed one of those cert fingerprint rotations. If the user already had update issues at that time, they wouldn't have the new fingerprint. [1] https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/AddonManager.jsm?rev=0552c02f54b0&mark=1149-1154#1139

(In reply to Matthew N. [:MattN] from comment #17) > > > or to users on older versions without the new hotfix cert > > > fingerprint who didn't get the hotfixes to rotate the fingerprint before the > > > cert expired. See bug 803596 and bug 874513. > > > > That should be a very small set of users in theory, right? > > Well, we cut it pretty close at least one of the two times IIRC so computers > that are only used occasionally (e.g. on weekends or a few times per month) > very possibly missed one of those cert fingerprint rotations. If the user > already had update issues at that time, they wouldn't have the new > fingerprint. > > [1] > https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/ > AddonManager.jsm?rev=0552c02f54b0&mark=1149-1154#1139 Are you certain that we actually look at the certificate's expiry date? For some reason I thought we only looked at the cert fingerprints... I agree though, the set of users that don't have the latest hotfix fingerprints should be small. They were shipped with Firefox 24 (https://bugzilla.mozilla.org/show_bug.cgi?id=803531), and the hotfix was made available for older installs.

Hello, I've attached a figure that has for the last 15 days, the % of ADI on different versions. Each panel is a version. FYI recent release dates are 2014-04-29 - FF 29 2014-03-18 - FF 28 ... 2013-10-29 - FF 25 Things to note 1. % on version 27 and 26 is on the down as people move of 27,26 to higher versions 2. The shape of the curves are a) similar for panels 12 ... 25 b) The percentage of ADI from these versions hovers in the 0.5% ... 1.2% c) the trend line (red line) fairly horizontal (no downward trend) From 2(a,b,c) i would say profiles on versions 12-25 for some reason continue to stay on that version and without intervention (e.g. this bug) they will continue to stay on the version. For "success", we can keep this chart around. Everyone has to concur what absolute/relative percent difference is a success? - is it 0.1% change (difference between before and after) - is it 10% relative change (after-before)/after ? Yes, we can see more FHR profiles come online but it will be tricky to know if they came for this hotfix (the numbers are so small). It would be nice if the FHR profile has a one bit of information : _did they come from this hotfix prompt?_ (boolean)

Paul, can you please take ownership of ensuring this gets tested and signed-off?

Sure

Can we get a status update on the testing of this? Thanks.

Actually I'm currently waiting for bug 1014194 and bug 1014187 to get fixed first.

This is a two page pdf that shows the percent of ADI on different versions. Each panel is the % contribution to ADI vs date by the version in the panel. Steady constant decline over time.

https://hg.mozilla.org/releases/firefox-hotfixes/rev/2e4cc857dd67 https://hg.mozilla.org/releases/firefox-hotfixes/rev/d29a3d24405c

The update hotfix is now in production. I don't see any further use of this bug, so closing.

Paul, can you do some quick smoketesting to make sure this is working in Production?

I got one crash while the hotfix was installing on FF 27.0.1 on Win XP. https://crash-stats.mozilla.com/report/index/d15249a6-4367-4d60-a276-9cde62140717 Don't know if it's related or not to the hotfix, I also couldn't reproduce the crash again. Otherwise everything works fine in production. Tested on FF 10, 19, 28 Win 7, FF 15, 20.0.1, 27.0.1 Win XP.

Can't tell much from the crash report: it's missing module information. I'm inclined to ignore it for now.

Updated data as of 2014-07-20. Note, version 24 is an ESR hence the stable (non decreasing) %.

Going forward, view this dashboard: https://metrics.mozilla.com/protected/dashboards/adi/ (LDAP protected) It will be updated about 2-3 times a week (automated) There is a steady decline of % on older versions, i dont see evidence of that being hastened. FHR will not capture this information - if it is the profiles first time on FHR, we dont capture the version they updated from.

This hotfix is causing major problems for one of the large Australian Banks. The Bank uses our software which is based on Firefox running on a USB stick. https://www.commbank.com.au/business/online-business-services/commbiz/netlock.html The hotfix is causing a Firefox update to these devices which is making them inoperable, and requiring a USB hardware replacement. A lot of clients are being affected so this is a major problem. Our software uses a custom update server (set using app.update.url.override), but it seems the hotfix is being applied anyway. So questions 1/ Is it possible to update the hotfix so that it does not apply when app.update.url.override is set 2/ How do we best configure Firefox in future so that hotfixes like this will never be installed.

This is going to be tricky since there are everyday users that set that pref and forget to unset it that we want to target with this. I'll try to come up with a method for your use case. Are you creating your own update mar files or using Mozilla's?

If I can get a copy of your custom Firefox distro I might be able to find something else to key off on. Thanks!

and what Firefox version are you providing and updating users to at this time?

You can download from: http://keyvault.com.au/Download/cba/Token_3.0.1500.0.7z Just extract to a USB key. To launch, lick on NetLock.exe. This performs a quick integrity check on the USB file contents before launching Firefox. The issue is that following a Firefox update, the integrity check fails. On launch we go to an activation page. Clients authenticate using bank credentials and then client keys and certificates are generated and installed, for mutual SSL there after. Activation also updates the mozilla.cfg and prefs.js settings, so for example updates are enabled. This image uses Firefox 14.0.1, which granted is old. The bank is currently switching USB hardware to a more sophisticated device that support programmatic read<>read/write switching. The new version uses the latest Firefox. And thanks for the quick response!

Benjamin, do you know of a better way to handle the use case in comment #36?

Some day's ago a reported a nearly simalar problem in an other bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1042099 There my portable profile got to an Update from Firefox.

Comment 36 is an interesting edge case, but I'm not sure we should fix it. You are after all shipping a branded version of Firefox but then not updating it. We *could* check the .url.override pref and avoid doing hotfix update if it's pointing at a non-Mozilla server. However, one of the use-cases that we explicitly wanted to fix with the hotfix was users broken because that pref got altered somehow (by addons, 3rd-party software, malware, whatever). I don't think this case is really distinguishable from that case!

Can you at least implement a workaround so that the hotfix is not installed if app.update.url.override starts with https://update.commbiz.commbank.com.au? This is affecting a lot of clients for us and requires a hardware replacement in each each, so it's a big deal. Also, what's the recommended approach in future to disable all updates from anything other than our update server?

Guys, sorry to be pushy but are you able to help us here? Thanks.

For the existing hotfix that was already deployed that ship has already sailed. I'll provide more info in the next day or two after I have time to consider the approaches available.