User Agent: Mozilla/5.0 (X11; Linux i686; rv:6.0.2) Gecko/20100101 Firefox/6.0.2 Build ID: 2011090700 Steps to reproduce: On Linux (openSUSE 11.4) and Firefox 6.0.2 places.sqlite-wal and places.sqlite-shm not removed on exit. They hang around most times, however sometimes they are removed on exit. Actual results: see above. Expected results: Not 100% sure. On Windows, these files are always removed, so shouldn't the same happen on Linux?

It happens on Windows as well. Apparently, sometimes the sqlite library chooses to keep the updates in sqlite-wal and sqlite-shm and does not (yet) apply them to sqlite upon close. On the next time the program exits, the files usually are removed.

I've never seen it happen on Windows. The background is this: I am using Windows and Linux in a dual-boot setup. Since it's a hassle to synchronise the bookmarks, the Linux installations uses a (symbolic) link to the Windows drive, where places.sqlite for the Windows installation is stored. That works most of the time, however, at times, the database gets corrupted. I would like to get to the bottom of this. Obviously, if the database is stored in a "main" file and some auxiliary files (wal, write-ahead-log, and shm (?)) I'm not surprised that corruption can occur if at exit time these auxiliary files are left behind.

I so see it on Windows with Seamonkey 2.3.3 You may want to use the workaround I wrote in bug 662858 After vacuum, the two extra files are gone and all data is in the main file. (which is also much smaller)

Thanks. I used the sqlite3 places.sqlite vacuum Immediate benefit is that the file is much smaller now. I'll try it on Linux and see whether the wal and shm files go away.

Another important point is that the file needs to be on a local disk! Networked filesystems do not work for sure. sqlite3 will create -wal and -shm files and will never remove them (not with vacuum and also not with 'pragma wal_checkpoint' which is a command to apply the -wal and -shm files without shrinking the file). I don't know if a symlink is a problem.

may be related to bug 687726, btw removing or not the file may be an explicit choice of SQLite.

I don't think there's nothing really specific we should do here, SQLite can figure whether those files should be kept or removed by itself. It's possible some add-on open a connection and doesn't close it properly though, but that's again not our bug.

Hello. Others and I are currently seeing this problem in both Firefox v34 and SeaMonkey v2.31 in both Linux (Debian stable's Iceweasel for me) and Mac OS X (10.8.5 for me) OSes. I did not have this problem with older versions like SeaMonkey v2.26.1. We have to delete places.sqlite-shm and places.sqlite-wal to be able to use manually copied places.sqlite files. :( Please kindly read mozilla.support.seamonkey's places.sqlite newsgroup thread on news.mozilla.org news server. Thank you in advance. :)

I'm seeing the same issue on Linux with Firefox 34.0 too. > I don't think there's nothing really specific we should do here, SQLite can figure whether those files should be kept or removed by itself. The modification date of places.sqlite doesn't change even if there were changes. Maybe Firefox should ensure committing all changes on shutdown (with pragma wal_checkpoint as I read above).

if the wal is not merged on shutdown, it gets merged at the next start. deleting the -wal file is also wrong, if you copy the database you should also copy the wal.

> if the wal is not merged on shutdown, it gets merged at the next start. On my system places.sqlite-wal does still exist after a restart of Firefox. Also its modification time is still newer than of places.sqlite. Interestingly I'm seeing that webappsstore.sqlite-wal gets merged with webappsstore.sqlite on every shutdown.

(In reply to sworddragon2 from comment #11) > > if the wal is not merged on shutdown, it gets merged at the next start. > > On my system places.sqlite-wal does still exist after a restart of Firefox. the wal file is not deleted in the merge on start case, cause it would have to be created again. its contents are merged. the file is removed only if the merge on shutdown succeeds. > Interestingly I'm seeing that webappsstore.sqlite-wal gets merged with > webappsstore.sqlite on every shutdown. that journal is very small and the shutdown path is much simpler, not comparable.

> the wal file is not deleted in the merge on start case, cause it would have to be created again. its > contents are merged. > Also its modification time is still newer than of places.sqlite. The merge hasn't happened on startup. Or is it maybe not guaranteed too?

(In reply to sworddragon2 from comment #13) > The merge hasn't happened on startup. Or is it maybe not guaranteed too? Did you read comment 12? on merge only the file contents are merged, the file itself doesn't change.

That sounds implausible. If places.sqlite-wal gets merged into places.sqlite the file should indeed change.

My twos cents worth of wisdom: On Windows 7 (64 bit) places.sqlite-wal and places.sqlite-shm didn't get merged, neither on shutdown nor on startup of Firefox. In the end I ran "sqlite3 places.sqlite vacuum" and now it gets merged again at shutdown. So something was fishy.

Actually, I take that back partially. After starting FF a few more times, the WAL and SHM files hang around again and don't get merged at all.

(In reply to Marco Bonardo [::mak] -- catching up, delayed answers -- from comment #7) > I don't think there's nothing really specific we should do here, SQLite can > figure whether those files should be kept or removed by itself. > It's possible some add-on open a connection and doesn't close it properly > though, but that's again not our bug. I am confused by the status of this bug report. We are using FF 35 under Linux (RHEL 6) and having the exact same problem. The -shm and -wal files are *NOT* being cleaned up, *EVER*. This was not a problem when we were using FF 24. How could this be related to our installation of SQlite when we haven't changed Sqlite at all (all we did was try to run FF 35 instead of 24). And we tested this with a completely clean profile and no addons or extensions. How is this not a Firefox problem?

it's clearly possible Firefox is unable to cleanly close the database for a lot of reasons, we can only do our best to minimize shutdown work and ensure proper ordering of the operations. But apart from the files sticking around, that should have no consequences on the product. If you have a case where a clean profile does never remove those, then it might be worth investigation, and reproducing the problem with a debug build could help figuring issues (there should be warnings if the connection can't be closed). Is RHEL 6 using system sqlite for Firefox?

(In reply to Marco Bonardo [::mak] -- catching up, delayed answers -- from comment #19) > But apart from the files sticking around, that should have no consequences on the product. It is true that it appears to be processing them but just not removing them (we have no corruption issues... yet). And if that is the case, then it isn't doing much harm, other than leaving hundreds of unnecessary files on our system and using some additional disk and backup resources (which is not a big deal). In our case, all the files are always stamped with the date/time of the start of Firefox and remain that way. Example (with user not logged in, although looks the same even when running Firefox): # ll pla* -rw-r--r-- 1 crxssi infosys 10485760 Mar 9 13:30 places.sqlite -rw-r--r-- 1 crxssi infosys 32768 Mar 9 13:30 places.sqlite-shm -rw-r--r-- 1 crxssi infosys 721456 Mar 9 13:30 places.sqlite-wal > If you have a case where a clean profile does never remove those, then it > might be worth investigation, and reproducing the problem with a debug build > could help figuring issues (there should be warnings if the connection can't > be closed). Is RHEL 6 using system sqlite for Firefox? This is with RHEL 6's sqlite: sqlite-3.6.20-1.el6.x86_64

(In reply to crxssi from comment #20) > In our case, all the files are always stamped with the > date/time of the start of Firefox and remain that way. That is normal, the file is merged at startup and recreated. > This is with RHEL 6's sqlite: sqlite-3.6.20-1.el6.x86_64 So, we don't use system sqlite there since wal was introduced with Sqlite 3.7.

(In reply to Marco Bonardo [::mak] -- catching up, delayed answers -- from comment #14) > (In reply to sworddragon2 from comment #13) > > The merge hasn't happened on startup. Or is it maybe not guaranteed too? > > Did you read comment 12? on merge only the file contents are merged, the > file itself doesn't change. (In reply to sworddragon2 from comment #15) > That sounds implausible. If places.sqlite-wal gets merged into places.sqlite > the file should indeed change. After rereading some posts I have noticed that we got into a communication error but basically at this point I reported this state: - The contents of places.sqlite-wal hasn't got merged into places.sqlite on shutdown. - The contents of places.sqlite-wal hasn't got merged into places.sqlite on startup (which I was noticing with the newer modification date of places.sqlite-wal).