User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36 Steps to reproduce: This one is somewhat hard to reproduce because the most common problem seems to be automatic updates of XPI download via AMO that cause the issue. However, it also seems to be lost at other times, such as when the browser crashes. I understand simple-prefs is also an option, but RES uses simple-storage instead of simple-prefs for two key reasons: 1) it's a cross-browser extension, we only have the manpower to build one console/interface to our settings, not 4, so we needed a common way to store our data on all browsers. simple-prefs doesn't really fit the bill here. 2) even if we put in the work on browser specific consoles, there's business logic to RES's settings that simple-prefs forms doesn't really support (e.g. options that depend on other options). For this reason, we'd end up setting all the prefs to hidden anyway. It seems that the data loss typically occurs when Firefox crashes, and/or when the user receives an updated version of the addon via AMO. Actual results: store.json / data stored in simple-storage is blown away Expected results: store.json / data stored in simple-storage should not be lost when user updates his/her addon.

Which version of Firefox are you testing with?

this has been a longstanding issue since RES was released 3 years ago (Firefox 18? maybe lower?) and only now am I finally filing a bug for it because our userbase has grown enough that it's a source of great ire, rather than an occasional "oh, darnit"... To answer your question, though: users are on FF33, maybe some on 32. However, it's definitely not a new problem at all. We even have a FAQ about it, imploring users to back up their data.

I can confirm described behavior. I recently lost all of my RES settings (it may have been after a crash). I restored store.json from backup and got them back.

Also happening with Ghostery, we've never been able to pinpoint the cause of settings being reset but it must be this.

(In reply to Brad McDermott from comment #4) > Also happening with Ghostery, we've never been able to pinpoint the cause of > settings being reset but it must be this. Yes, I can confirm this. It will annoy me after every update. Sometimes just closing the Fox is enough to lose the files.

We recognize this is an issue with simple-storage based on these reports, but it's still unclear what the cause is without a way to reproduce the issue. simple-storage as-is has not proved to be a that great a way to store data. If you can, I'd like to hear more about your use cases for ghostery, RES and other add-ons. It seem to us that add-on developers should consider the following two options: 1. for really simple stuff, use prefs, ideally simple-prefs 2. for anything above this, use IndexedDB. Now that we have jpm and the possibility of publishing and using 3rd party modules via npm, I wonder if the right thing to do is to build an alternative implementation of simple-storage that uses IndexedDB. I'm mainly suggesting this as an alternative to simple-storage because I intuitively think using a database with transaction support is simply going to be better than writing to a file on disk. ¯\_(ツ)_/¯

So, the way RES uses simple storage (and the reason) is relatively straightforward, I think: 1. RES is a cross browser extension, meaning that localStorage was an extremely attractive option for storing options because it's commonly available across modern browsers. Anything that has an API that looks/works just like localStorage makes for a really easy abstraction. 2. RES has a custom settings console because of both the fact that it's a cross browser extension and the fact that it stores a lot more data than just simple key/value pairs at times. RES stores json objects to aggregate different types of data. An example is "user tags" -- tagging usernames so that when you run across them, you recall something about them (e.g. "this guy is a troll" or "this guy posts really useful information") For these reasons, simple-prefs didn't seem to fit the bill, and IndexedDB wasn't really a thought - simple-storage looks just like localStorage and works pretty much just like it, so it made sense to use. The key times people seem to complain of data loss: - a Firefox crash - Manual task killing seems to be a way to possibly reproduce this issue, I haven't tried yet and I'm currently traveling so I shouldn't even really be replying here :-) - occasionally, but not always, browser and/or extension auto-update -- this one's dicey because sometimes people *think* they got an RES update when in fact they lost their data and therefore the flag that says "you've loaded this version before" - so it's unclear how accurate those reports are unfortunately. We can certainly look at using indexedDB - but I would argue very strongly (especially as a kinda-sorta contributor to Mozilla in that I'm an AMO reviewer) that simple-storage should be deprecated and/or very blatantly labelled as "cannot be trusted" if there's not going to be effort put into stabilizing it / preventing data loss. Remember that in Mozilla's docs / the way it's described it's not "writing to a file on disk" - it's "use this simple storage API that has get and set commands for key value pairs", so jetpack add-on developers really aren't thinking of it as "writing a file to disk" so much as "here's how you store data". I hope that provides a little of the info / background that you requested - happy to answer any other specific questions you may have!

Pretty much the same exact reasons for Ghostery as well. Also, indexeddb wasn't supported when we started using simple storage. In addition to the usual times Steve mentioned, a few of our users complain of their settings being reset on rebooting their machine (usually using some flavor of linux). That may or may not be related to this bug.

(In reply to Jeff Griffiths (:canuckistani) from comment #6) > simple-storage as-is has not proved to be a that great a way to store data. > Now that we have jpm and the possibility of publishing and using 3rd party > modules via npm, I wonder if the right thing to do is to build an > alternative implementation of simple-storage that uses IndexedDB. I think this could be a really smart way of dealing with this problem. It is unclear what directly causes the problem, right? Maybe (I cannot estimate how effortful it is) it would indeed just better to implement it your way. My question: Is the perfomance of IndexedDB pervasive better than simple-storage? [Well, maybe this isn't even so important if we can prevent data loss.]

(In reply to quadronom from comment #9) ... > My question: Is the perfomance of IndexedDB pervasive better than > simple-storage? > [Well, maybe this isn't even so important if we can prevent data loss.] I think perf for simple-storage-sized use cases should be fine. Just don't store movies or music files in there, or do high-frequency updates. Both file and database operations are known to be expensive, generally, even if they're local.

(In reply to Steve Sobel from comment #7) Thanks for the detailed reply - really helps. ... > We can certainly look at using indexedDB - but I would argue very strongly > (especially as a kinda-sorta contributor to Mozilla in that I'm an AMO > reviewer) that simple-storage should be deprecated and/or very blatantly > labelled as "cannot be trusted" if there's not going to be effort put into > stabilizing it / preventing data loss. Remember that in Mozilla's docs / > the way it's described it's not "writing to a file on disk" - it's "use this > simple storage API that has get and set commands for key value pairs", so > jetpack add-on developers really aren't thinking of it as "writing a file to > disk" so much as "here's how you store data". As I mentioned previously, I wonder how a simple-storage-compatible implementation based on IndexedDB would do. I'm going to start hacking on something like that over here and will publish it to npm as soon as it starts passing tests: https://github.com/canuckistani/simple-storage-2

I'd like to chime in on this as well. Since we started using simple-storage well over a year ago we've battled issues with users data getting lost, or maybe more accurately the jetpack folder gets into a state where it cannot be read from. Not sure whether or not our issue is related to Steve Sobel's, but as he pointed out it's hard to reproduce. What I can attest to is that as our user base is growing we're seeing more and more complaints about it. We've even started seeing negative reviews on AMO. Here is the problem I posted in the Jetpack group over a year go: https://groups.google.com/forum/#!searchin/mozilla-labs-jetpack/potential$20problem/mozilla-labs-jetpack/x2wJZr5Y0hY/_CkQI1v5TG0J We found one case where a user was using Dropbox to sync their profiles folder. This caused their Jetpack folder to go into a "write only" mode. So our extension was no longer able to read data from it and was triggering repeated behavior as they just installed our extension. We're working diligently with some of our users to see if we can find the steps to reproduce this problem. I'm not 100% certain our issue is directly related, but it seems to have some correlation. I'll post back when we learn more.

Not meaning to add additional noise, but hoping to add some value... Since my last comment (about a month ago) I've experienced this same dataloss (with RES) three additional times. The most recent two within about 36 hours of each other after what seemed otherwise uneventful Firefox upgrades (no crashing or killing). I don't know if my profile is more susceptible or what, but it does seem a little extreme and I can understand how some would abandon ship over such behavior.

Hi, I've been confronted to this bug as a user of the Ghostery addon : every now and then, my settings were erased for no apparent reason. Thanks to these comments, I've finally managed to understand what caused the problem. - If I close Firefox before shutting down my computer, everything is fine. - But if I shut down or reboot my computer while Firefox is still opened, upon restart my Ghostery settings are lost. Firefox must have interpreted this as a crash, thus triggering the data loss. Usually, I like leaving Firefox opened when I shut down my computer, because upon restart, Firefox will propose me to restore my previous session, which I find handy. I'm using Firefox 33 on Ubuntu.

there hasn't been a status update on this in some time, so I just wanted to ask if any further progress might have been made? I certainly appreciate that Jeff G got a start on it here: https://github.com/canuckistani/simple-storage-2 Sadly, that repo is still empty, but I know he's got other work to do so the complaint I'm about to lodge isn't at him personally - I appreciate your help, Jeff... It's frustrating as a developer to be stuck in limbo for this long, and to visit the docs page and still see simple-storage labeled "stable" - https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/simple-storage I'd love to know if / when a decision might be made to either: - complete an IndexedDB-based simple-storage API and allow us some way to migrate data from store.json to that - or, as much as I'd prefer the former, deprecate simple-storage and at least stop having new developers make use of it. The suggestion that we should be using IndexedDB, by the way, is especially bothersome considering it's still labeled "experimental" - https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/indexed-db Ultimately I've got about 220k daily users (not sure on total / non-daily #) on Firefox, many of whom are screaming at me every time they either get a Firefox update or an update of my addon - both of those seem to be the biggest causes of the issue - with Firefox crashes being a close #3. A light at the end of the tunnel would be great to see... I'm hesitant to even work on an indexedDB wrapper when Mozilla's own documentation still calls it "experimental"

(In reply to Steve Sobel from comment #15) > I certainly appreciate that Jeff G got a start on it here: > > https://github.com/canuckistani/simple-storage-2 > > Sadly, that repo is still empty, but I know he's got other work to do so the > complaint I'm about to lodge isn't at him personally - I appreciate your > help, Jeff... Well it wasn't much help to be honest! ... > I'd love to know if / when a decision might be made to either: > > - complete an IndexedDB-based simple-storage API and allow us some way to > migrate data from store.json to that This is going to be difficult to do while keeping the faux-synchronous api simple-storage has. One could do a simple-storage2 module that was promise / callback based though, is would just be very different. At that point you might as well use: https://www.npmjs.org/package/async-storage > - or, as much as I'd prefer the former, deprecate simple-storage and at > least stop having new developers make use of it. > > > The suggestion that we should be using IndexedDB, by the way, is especially > bothersome considering it's still labeled "experimental" - > https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/indexed-db Yeah, noted. We just got the same complaint about the new ui / button apis, and really at this point indexedDB should be marked as stable, IMO. Logged bug 1132168

I think that is worth adding to Blocks Bug 840246, to make it easier to not lose. Especially because it's not the first bug in simple-storage

Will this instability be fixed in the new WebExtensions storage API?

With only looking into this issue for a , I'm pretty sure this is due to truncating the file before writing instead of writing to a new file and then renaming. https://github.com/mozilla/addon-sdk/blob/3527a27ac240c1a5b7fde68af94e00a8e69c545d/lib/sdk/simple-storage.js#L163 https://github.com/mozilla/addon-sdk/blob/3527a27ac240c1a5b7fde68af94e00a8e69c545d/lib/sdk/io/file.js#L140 Practically *every* use of O_TRUNCATE is a bug. The sole exception is if you are writing to a cache or tempfile that you *know* you are perfectly happy destroying casually. All files open for writing should either use O_APPEND (e.g. logfiles) or O_EXCL on a new name in the same directory, followed by an (atomic) rename. Apparently there is a even a fixed version: mozilla.org/network/safe-file-output-stream but that raises the question of why the unsafe version even *exists*.

@ b.r.longbons@gmail.com "Apparently there is a even a fixed version: mozilla.org/network/safe-file-output-stream but that raises the question of why the unsafe version even *exists*." Does that mean this bug will be finally fixed ? Since Firefox 45 (Windows 32 bit) it is worst for me, 2 time on 3 time I close Firefox store.json from RES addon is nuked (0K). Before version 45 it was longtime since I had trouble with the jetpack folder. If you need somebody to test it I will be happy to help you.

Firefox 45 on Ubuntu 14.04. I faced this issue with the OneTab add-on in Firefox. Every time my Firefox is not running while I perform an update through Ubuntu Software Center, it deletes all the data OneTab has saved. I mean, when Firefox browser is running when the update with Ubuntu Software Center is performed, the data is not lost. But when I run the update without running Firefox process, it wipes out the data. Hope this helps. And, I mean, WTF - an issue from 2014, affecting a whole bunch of people and add-ons and it's still not fixed? Looking at some comments, the cause of the issue is even known, so even more WTF?!

(In reply to radoslaw.paluszak from comment #21) > Firefox 45 on Ubuntu 14.04. > > I faced this issue with the OneTab add-on in Firefox. Every time my Firefox > is not running while I perform an update through Ubuntu Software Center, it > deletes all the data OneTab has saved. > I mean, when Firefox browser is running when the update with Ubuntu Software > Center is performed, the data is not lost. But when I run the update without > running Firefox process, it wipes out the data. > > Hope this helps. > > And, I mean, WTF - an issue from 2014, affecting a whole bunch of people and > add-ons and it's still not fixed? Looking at some comments, the cause of the > issue is even known, so even more WTF?! Firefox 45.0.2 on Ubuntu 14.04 Hi, me again. Just wanted to confirm that I've just updated Firefox from 45.0.1 to 45.0.2 through the Ubuntu Software Updater with running Firefox and OneTab data remained untouched (it's still there). Looks like it's the way to go.

On my machine it almost always happens when shutting down Linux (Xubuntu 14.10). I normally leave the browser open and use the XFCE shutdown function or sometimes sudo poweroff. I have multiple tabs and almost everytime when starting firefox (after shutting down as described) I see the restore window (which is not a problem of the plugin but maybe the source of the problem). Few days ago, I recovered all my OneTab tabs from a backup and restarted firefox multiple times successfully, but when I started ff again the next day, everything was lost again. Package: firefox Architecture: amd64 Version: 45.0.1+build1-0ubuntu0.14.04.2 lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.4 LTS Release: 14.04 Codename: trusty

It happen to me few times a month under Windows with RES (Reddit Addon), it's a pain in the A**SS ! I no longer use One Tab cause I lost to many useful tabs :( I use Tabs Groups instead https://addons.mozilla.org/fr/firefox/addon/tab-groups-panorama/ it's not exactly the same but it's the same purpose and I no longer lose my tabs.

So as far as I understand, the issue (mostly) is in writing a JSON file when it cannot be written, e.g. no space left. The built-in session storage implemented (mostly by :yoric AFAIR) a save-a-JSON-file routine with rotating backups that could IMHO be adapted/generalized for simple storage and could soften the loss-of-data issue. Could also be an interesting base implementation for any WebExtension API.

@Florian In THIS bug report, you are the first person writing about the no-space issue. This bug is not about no-space. We all have plenty of free space and still lose our data. No-space is a known problem (seen in the OneTab plugin user feedbacks) but another issue or not even reported yet.

@DanielAlder As the only one who has looked at the technical details of this bug, I am convinced that running out of space (like crashing) simply increases the probability of this bug to 100% instead of, say, 10% if only the write-vs-exit race condition occurs.

I just want to avoid that debugging goes into the wrong direction. This bug could also appear because the wrong structure is saved (for example that the save event occurs before the first load event in a browser session). In this case it's a completely different thing than out-of-space where the problem should be solved by writing a temporary file and then renaming

The whole point of the actual bug here is that you should *always* write to a temporary file and then rename.

I totally agree that the way how files are saved should be changed. But I believe changing this will not help solving this issue. And now let's the developers do their work

(In reply to b.r.longbons from comment #27) > @DanielAlder As the only one who has looked at the technical details of this > bug, I am convinced that running out of space (like crashing) simply > increases the probability of this bug to 100% instead of, say, 10% if only > the write-vs-exit race condition occurs. Fyi. This bug has occurred and still occurs to me regularly (in my case with Mailvelope) and it is definitely NOT an issue of running out of disk space on the volume where my home partition is. Also, as far as I can tell, Firefox doesn't crash or shows other irregular behaviour when the issue occurs. The data is just vanished at some (but not all) browser restarts, updating FF or updating extensions is just one of the circumstances when this appears to occur.

Here is my scenario: in Kubuntu 15.04/16.10, when leaving the session (logout, reboot, shutdown) Firefox is not closed properly (I don't know if KDE does not wait properly that Firefox stopped, or if Firefox wrongly notifies KDE that it has stopped), and so at the next Firefox start I get the screen for restoring the session after a crash, and also the data of my addon "Mykanji" stored using 'simple-prefs' is lost (most of the time). Note that I have a lot of opened tabs and many add-ons. If I close the tabs before the logout, then the problem disappears. I can reproduce the problem in a VM. For now, I tried to investigate why Firefox does not stop properly, and KDE logs that it has stopped it properly (which may not be the case). I find it strange to lose data even when I did not modified anything. From a preference facility, I would expect them to be save after a modification, or after a delay following a modification.

(In reply to Fabimaru from comment #32) > Here is my scenario: in Kubuntu 15.04/16.10, when leaving the session > (logout, reboot, shutdown) Firefox is not closed properly Firefox abruptly terminates itself when Linux asks it to exit (Bug 336193) so you really need to quit manually and wait for the process to exit first.

Sounds about right but why "abruptly terminates itself" will cause a total loss instead of losing of unsaved data?

But isn't the real problem here, that even if you quit it manually (atleast at Windows) the data loss occurs? Because it does here. Randomly and not always, but it does...

(In reply to Kestrel from comment #33) > Firefox abruptly terminates itself when Linux asks it to exit (Bug 336193) > so you really need to quit manually and wait for the process to exit first. I searched a bit and Firefox seems to support XSMP for session management. In the file "./toolkit/xre/nsNativeAppSupportUnix.cpp", I can see in the function "nsNativeAppSupportUnix::Start" that Firefox registers callbacks for the session management messages from the X-Window Session Manager (for saving its state and closing). When the later ("die") is received, it calls "eForceQuit". I don't know which piece is missing so Firefox exits properly, if Firefox does not do something properly so it get terminated by Linux, or if Linux Session Manager kills Firefox by mistake (which may be the Bug 336193).

hey guys i have the same problem , two things can be reproduced ... change your addon options .. kill firefox ... new data is lost after restart close firefox with tons of tabs .. wait for it to crash .. restart .. simple-storage store-json is now []/empty ! a solution i have found is just writing a backup of your simplestorage array as a json file with a distinctive name to the profile dir. those are not many lines of code actually and it uses the same underlying stuff aka file/io. here i´ve made a pastebin you can use easily use as a template for all the lost souls which come by here , http://pastebin.com/1p2mcEtW .. it simply backups your actual simple storage(not the file but the actual data) to a file in the profile dir also someone above mentioned writing files is heavy -> its NOT heavy to write a few KB´s to a textfile , lol not even close to heavy .. its not even 1% of my cpu ( so small i can´t measure it)

I feel it's worth updating this ticket to note: The advice I was given from Mozilla was essentially "this isn't super maintained anymore, it's not a priority, move to IndexedDB"... to which I responded "but it's still marked experimental - which is why I didn't..." I was then informed that it's really not experimental anymore, and should be OK / more stable to use. There's even a bug report to mark it as stable, which hasn't happened yet: https://bugzilla.mozilla.org/show_bug.cgi?id=1132168 So, we switched to IndexedDB and are having problems with that, too -- see here: https://bugzilla.mozilla.org/show_bug.cgi?id=944918 It seems important to me at this point to update this ticket and note to anyone reading: IndexedDB is not an ample replacement for simple-storage, and neither of them are stable. I do wonder if it's coincidental, or if there might be some underlying data I/O issue that causes both to break?

Neither this ticket nor the other ticket I linked in my last comment have seen updates in 6 months. Extension developers who need more than a minimal amount of storage need a stable platform to store that data, and it seems that both simple-storage and IndexedDB are unstable. Can we get any sort of an update on prioritization here? Even if it's "it's not a priority right now at all", I think any information would be better than none.

(In reply to Steve Sobel from comment #39) > Can we get any sort of an update on prioritization here? Even if it's "it's > not a priority right now at all", I think any information would be better > than none. Sorry to bear bad news, but at this point the add-on sdk is pretty much unmaintained and sdk add-ons won't even load beginning with Firefox 57 so its hard to justify spending any time on it. I didn't read the entire indexedDB bug but it is quite old, there are many people using indexedDB from webextensions successfully, that's the ideal route if it is feasible for you.

We already switched over to IndexedDB and we have users with problems all the time where the db gets corrupted or can't be created. We're working on a port to WebExtensions, and I suppose we'll ultimately be using the .storage API there -- is the backend of that IndexedDB? I sure hope not.

The webextensions storage APIs use a different backend than indexedDB. Sorry to hear you've been having so much trouble with indexedDB, as always, specific bugs with concrete steps to reproduce are extremely helpful in getting the problems resolved...

unfortunately it's usually unrelated to the extension when either store.json or indexedDB get corrupted/messed up. something seemingly related to browser crashes, unexpected system restarts, etc. Repro steps become extremely difficult to dig up in those cases. That said, if things are moving toward a totally separate backend, I suppose we'll just migrate there and see what happens.. hopefully 3rd storage mechanism is a charm :)