The Swiss Federal Institute of Technology (ETH) and Google Switzerland used Google's server logs to monitor 75% of the world's internet users for over a year to see if users were installing security updates for their Web browser. They found that on any given day in 2007, only 80% of Firefox users were running the latest version. A copy of their study is attached. I believe we can get our minor version update number from 80% to very close to 100% by doing two things: 1) Make sure Firefox is literally capable of updating itself (either have per user installations so the limited account will have the privileges to run the update, or run a service with administrator privileges to check for updates). Issues related to this are starting to be discussed in bug 477424, bug 407875, bug 318855 and bug 404541 2) Make the minor update process transparent to the user I've filed bug 489138 which discusses the user experience of encountering update notifications, and proposes that we remove them entirely

"Based on HTTP user- agent header information stored in anonymized logs from Google’s web servers, we measured the patch dynamics of about 75% of the world’s Internet users for over a year... ...the maximum share of the latest, most secure version never exceeded 80% for Firefox users"

mconnor points out on irc that these numbers sound similar to what we are seeing with blocklist pings, but I'm not sure on the specifics (also some fraction of users will have blocklist checks turned off)

btw: the add-on blocklist pings don't have ui to turn them off and the ping occurs even when there are no add-ons installed so the fraction should be extremely low. I don't think bug 404541 has anything to do with this bug since that is for the installer and not update.

This analysis is great! I believe that it would be a useful analysis for us to have constantly available internally, and I also believe we should try to find a way to possibly make the data available to the community as well. CC'ing some metrics folks.

Just adding to comment #2... If the question is, "What percentage of Fx3 users are on the latest version of 3.0?", the answer is about 87% (according to our recent blocklist numbers). If the question is, "Of the total universe of Fx users, what percentage are on the latest version of 3.0?", the answer is about 77% (according to our blocklist numbers). It looks like the study cited in comment #1 considers the latter question, while security studies (I'm sort of guessing here) generally focus on the former analysis (i.e., uptake of our releases).

Also please note that there are some surprisingly drastic differences between: A - blocklist update vs AUS numbers B - firefox users with a "valid" update channel vs firefox users without a "valid" update channel For A, this implies that there is a group of users who have either turned off their AUS checking or that checking is otherwise blocked. I think it's reasonable to assume that this group is mostly the "expressed interest in not having updates" for one reason or another. For B, this implies custom distributions of Firefox, altered distributions (through some add-on), etc. We cannot guarantee an update to these users would work, and IMO, shouldn't be considering them as a missed target. (In reply to comment #0) > I believe we can get our minor version update number from 80% to very close to > 100% by doing two things: > > 1) Make sure Firefox is literally capable of updating itself (either have per > user installations so the limited account will have the privileges to run the > update, or run a service with administrator privileges to check for updates). > Issues related to this are starting to be discussed in bug 477424, bug 407875, > bug 318855 and bug 404541 > > 2) Make the minor update process transparent to the user > I've filed bug 489138 which discusses the user experience of encountering > update notifications, and proposes that we remove them entirely These are ways of ensuring that updates happen for users who want them to happen automatically. We ship with automatic update as the default, so the only way that updates aren't happening is: - bugs (you mention some candidates in your first point), or - user choice to not receive updates I agree that there is much to do in order to make updates less intrusive and costly to users (not interrupt users, not hijack user tasks, not require restart, maintain compatibility with add-ons) but I wanted to make sure that we weren't suggesting removing the option of user choice. Do we have any formal analysis of why users choose to disable automatic updates, and what percentage that contributes to our non-updated users? My deep suspicion is that's it's caused by: - frequency / pestering nature of update UI (which we can address) - worry that updates will change features (which we can address, somewhat) - inability to roll back if user doesn't like an update (which we can address) - potential add-on incompatibility (which we cannot fully address) - user choice (which we cannot fully address)

(morphing title slightly to make this easier to find/dupe against, and indicating that this is a [meta] bug; clearing blocking flag, please nominate specific issues, we try not to block on metas - to be clear, I agree fully with this bug!)

we gathered some anecdotal survey data freom brazil (one of the countries where major and security updates are lagging) and most of the concern seem to be around beltzner's second point. - worry that updates will change features [and introduce bugs] The worry seem to be not so much around Firefox specifically, but the users seem to have been been by OS and other application updates that caused them problem in the past. There was also some concern about the updates actually being malware attacks and the users were worried that by accepting the update they would put the security of the system at risk. I think these perceptions mean that we also have some user education and outreach work to do. Especially if we are going to continue to allow users to choose to opt out of updates, which I think is very important.

as far as the study goes it seems like there are a couple of good things to measure. 1) "update acceleration" which would be to measure the speed at which users that have continuously opted in to updates are actually getting updated. and 2) "update opt-in rates" if we are going to allow user choice we should measure how many people are making the "right choice" that help them to keep there browser (and its plugins) secure. The first becomes a lot of work around our infrastructure and continued good work to make the process streamlined and error free for users. The second might become more of an educational problem and less of a technical problem. On update acceleration I think we are doing pretty good. In just 6 days we get over 85% and in 8 days we get over 90% of 3.0.7 users moved to 3.0.8 date 3.0.8 users 3.0.7 users tl 307+308 users pct of3.0.8 03/28/09 3892860 55514360 59407220 0.07 03/29/09 30260627 30999525 61260152 0.49 03/30/09 44886582 28825003 73711585 0.61 03/31/09 57355239 15999301 73354540 0.78 04/01/09 60678955 11474087 72153042 0.84 04/02/09 62559234 9017299 71576533 0.87 04/03/09 61284414 7339499 68623913 0.89 04/04/09 52747220 5508855 58256075 0.91 04/05/09 55865320 5029271 60894591 0.92 04/06/09 66932172 5597929 72530101 0.92 we top out at around 97% after about 20 days 04/17/09 68663644 2409859 71073503 0.97 04/18/09 58708715 1843714 60552429 0.97 04/19/09 61147516 1792059 62939575 0.97 04/20/09 72953299 2328876 75282175 0.97 04/21/09 71629765 2160907 73790672 0.97 04/22/09 71091816 2134465 73226281 0.97

@beltzner >- user choice to not receive updates > - inability to roll back if user doesn't like an update (which we can address) If we provide these two things, that seems like pretty full coverage of enabling user control. Requiring preemptive consent for each update is sort of illogical given that the user has no idea ahead of time if the update is going to cause an issue until they have actually tried the updated version. We would probably also want to: 1) Disable automatic updates as soon as the user rolls back to an older minor version (still updating them to the next minor version would get annoying pretty fast). 2) Ask the user to tell us why they are rolling back and locking in to the older version so we can actually attempt to address the specific issue. @chofmann >1) "update acceleration" which would be to measure the speed at which users >that have continuously opted in to updates are actually getting updated. I agree that acceleration is a pretty good metric, and it also can result in neat graphs for people who want to report security metrics :) http://www.techzoom.net/publications/silent-updates/figure-3.png

(In reply to comment #10) > 1) Disable automatic updates as soon as the user rolls back to an older minor > version (still updating them to the next minor version would get annoying > pretty fast). I'm not sure this is a good idea. If a user upgrades, has problems, and figures out how to roll back, we've likely lost them on whatever version they rolled back to forever. From a security standpoint, this is horrible and puts users at risk for however many versions they don't upgrade to, mostly because we never tell them new versions are available. > 2) Ask the user to tell us why they are rolling back and locking in to the > older version so we can actually attempt to address the specific issue. Definitely should do this if we decide to help users rollback to an older version (which I disagree with from a security standpoint).

re:comment 11 If a user upgrades and has problems and can't roll back we create other kinds of problems. Users like this seem just as likely to turn off updates, or even just give up on firefox altogether if the frustration and problem and problem level are high enough. From the fact gathering in Brazil we saw both actual problems and also a lack of confidence in the update system to do the right thing as reason for people not upgrading. it seems like we not only need the "ask users why they are rolling back", but then we need an additional "tell users we think the update coast is clear - please try again" communication channel. that last communication channel also implies that we get a big more rigorous in analyzing update failures and update reliability.

Most "minor-update problems" seem to be unrelated to the content of the update. Instead, users have firewall issues, issues with the update mechanism (which rollback might not address), and *lots* of pure coincidences. Rollback wouldn't fix most of the problems users complain about, so it's a bit of a footgun.

Every once in a while, we actually do ship a buggy update. Perhaps we should intentionally stagger updates, sacrificing initial update speed for scaring fewer users away from updates when we introduce a bug. There are several ways we could do this: * Get a lot more users on the minor-update beta channel, maybe with a "Include beta updates" checkbox. * Give update-paranoid users an option to only update after the update has been out for 3 days. * Exponentially ramp up every update offer: before we add the 40000th user, 20000 users have tried it for an hour. Any severe problem we detect in an hour based on N users will only reach 2N users.

the rollback and user reporting feature would be really valuable to use minor-update beta channel and early stages of a roll out like jesse talks about in comment 14. we have a handle on items in comment 13, but we would probably learn more and get a more precise definition of known problems if we had the rollback and comment system. maybe once we have the first 1 to 3 million updates released under the rollback and comment system, have time to analyze the feedback and are ok with it, then we switch over to no-rollback. it would be interesting to see what pct of users might use rollback, and what the motivations might be, and track this release to release. we would also want to test if the availability of roll back actually increases user confidence of taking new updates. that would be the one of the goals in trying to do this. in comment 11 ss thinks rollback might have no effect or a bad effect on user confidence in accepting updates. I could imagine it might go either way, but the only real way to find out would be to test. user education of how rollback works, when you should us it, and why you need to continue to keep taking updates also would play a big roll in this feature if its going to be successful.

btw: we are considering implementing MSIs on Windows for Firefox.next which may require the use of its own update implementation. It also solves many of these problems for Windows including rollback.

Would there be a way to find out if longer release cycles had higher acceptance rates than the shorter cycles, from the data we have? I often get complaints from friends who think we are releasing updates "all the time" and how annoying that is to them.

What if we try to look at this at a different angle? Instead of trying to analyze why the user doesn't upgrade, why not try to look into this as "lets try to make more users upgrade". I'd say it's relatively safe to assume that they don't upgrade because they choose not to - as opposed as problem in the upgrade. There's one simple test we can do; in one of the following releases, put more focus on the security part, like a message on the lines of "Every new firefox release introduces new *security fixes*" if the uses decides not to upgrade, with an option to continue like "No, at my own risk". I'd say that we'd be getting users that don't upgrade just because they don't bother. Focusing on the security risks of being left behind would make them think twice

(In reply to comment #17) > Would there be a way to find out if longer release cycles had higher acceptance > rates than the shorter cycles, from the data we have? I often get complaints > from friends who think we are releasing updates "all the time" and how annoying > that is to them. This is indeed a very interesting question! I have heard exactly the same complaint. The first thing we can do here is to make them be less annoying for the user. I think having less notifications here will only help. Not sure there is much we can do for performance of the update? The other thing we should think about is if we're doing the regularly scheduled updates too often. I.e. how many users are we permanently loosing, vs. how many days of less exposure do we have for users.

anecdotally "update fatigue" is something that we all experience, and see other experience; but, its not clear from the data that we have that its having a *big* impact on uptake of minior releases; at least not yet. It's also clear that we need better data and more analysis in that area and to watch the trends in this area more closely. Here is about the best we can do now for watching minor version uptake. http://people.mozilla.com/~chofmann/security/malware/3.x-3.5.x-minor-update-uptake.png The vertical slope of the lines shows pretty fast and complete adoption for just about every 3.0.x maintenance release over the last year as users move from older maintenance release to the newer. The height of the lines shows the steady trend of additions to the user base and we were able to keep the "update acceleration" moving pretty fast despite having nearly 80 million users on 3.0.10. just before 3.0.11 came out. There is pool of users that we just can't get to update, and as we do each release we might be adding to that pool. Those are the more horizontal lines stacking up along the bottom of the chart. This pool doesn't seem to be growing as fast as the growth in the userbase from an eyeball of the chart. To get a more clear picture for this area we need some better metrics around Number of users offered a major update + # accepted, # rejected Number of users offered a minor update + # accepted, # rejected Estimated number of users opted out of the update system Number of continuing users that use downloads instead of updates. Number of new users entering the system. There is some discussion about going on in future metrics planning work, but some of those metrics maybe really hard to get at.

We treat minor and major updates the same so this bug is likely no longer applicable. We are working on other bugs that should increase the update rate for both minor and major updates. Bug 318855 is about both major and minor updates. The process for updating is the same now for both minor and major updates so Bug 489138 isn't applicable to this bug.