Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Description • 15 years ago

During our todays website testing testday we got a notice from a tester that a Turkish banking site gets blocked by our phishing protection. See the given URL.

Given the fact that Google classifies it as a forgery site too this was ok for us for the first time:

http://safebrowsing.clients.google.com/safebrowsing/diagnostic?client=Firefox&hl=en-US&site=http://www.turkishbankgroup.com/

But further testing brought up some unexpected behavior. We tried to get the warning with Firefox 3.0.11 and the same profile too but the real website was shown and hasn't been blocked. So I went back to Firefox 3.5 RC2 and tried again. Now it even doesn't get blocked!? So something has probably changed on the profile. But even I used time machine to revert the state of the profile to a timestamp before we started testing the warning doesn't come up anymore for me.

Google safebrowsing still qualifies this website as malicious so shouldn't we block it? I wonder how many websites could slip through?

	juan becerra [:juanb]
	Comment 1 • 15 years ago

When I toggle between 3.0x and 3.5(rc2), the urlclassifier file gets regenerated. After a few minutes of use on either version other phishing sites get detected. I suspect that once the urlclassifier data is regenerated in its entirety, the site in question will be detected again, it means that there a few hours where you could be vulnerable.

	Johnathan Nightingale [:johnath]
	Comment 2 • 15 years ago

Is this on a fresh profile? It can take several hours - sometimes even days - to get a complete database. The most recent additions are served first, but it's quite plausible, likely even, that jumping between profiles (especially ones that are new, or ones that are not often used) would show different behaviours, here.

The alternative, which would guarantee that we always got up to date answers, would be to query google for each page load - but that's really unacceptable from a privacy point of view.  For typical users, whose profiles are long-lived, this update behaviour isn't a problem, obviously, since they have long since gotten a DB laid down, and are now just picking up deltas.

If you're seeing this on long-lived profiles, that's a different story, assuming it's still a live listing.  Otherwise, though, I think this bug is INVALID, since it describes a known and intended behaviour of the system.

	Johnathan Nightingale [:johnath]
	Comment 3 • 15 years ago

Given Juan's comment 1, I am more inclined to think, as mentioned in comment 2, that this behaviour is being caused by an incomplete malware DB, which is an artifact of the testing process.

	Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Comment 4 • 15 years ago

Johnathan, I have always used the same profile. Here a short overview what I did:

1. Run with Fx3.5: blocked
2. Restart with Fx3.5: blocked
3. Run with Fx3.0: not blocked
4. Run with Fx3.5: not blocked
5. Restored profile with Time Machine from before step 1.
6. Run with Fx3.5: not blocked

So this is more than a simple update problem.

	Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Comment 5 • 15 years ago

Now after Fx 3.5 RC2 has been running for about 1h with this profile the file is still 32768 byte in size and hasn't been altered since yesterday (MESZ timezone):

-rw-r--r--@ 1 henrik  staff   32768 18 Jun 01:20 urlclassifier3.sqlite

	Johnathan Nightingale [:johnath]
	Comment 6 • 15 years ago

(In reply to comment #5)
> Now after Fx 3.5 RC2 has been running for about 1h with this profile the file
> is still 32768 byte in size and hasn't been altered since yesterday (MESZ
> timezone):
> 
> -rw-r--r--@ 1 henrik  staff   32768 18 Jun 01:20 urlclassifier3.sqlite

Which directory are you looking at, there?  On Mac, we store the urlclassifier3.sqlite file in the Caches area, not the Application Support area.

~/Library/Caches/Firefox/Profiles/<profilename>

Though we used to store them in the main profile dir.

	Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Comment 7 • 15 years ago

That's a great tip. Thanks. Seems like it has been changed a while back. So checking this folder I see that the urlclassifier3.sqlite file is getting deleted when you run a profile with Firefox 3.0.11. Users have to download everything from scatch. Is this an expected behavior?

	Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Comment 8 • 15 years ago

Ok. So lets update the summary and make it a security core bug.

	:Gavin Sharp [email: gavin@gavinsharp.com]
	Comment 9 • 15 years ago

I don't know that it really needs to be hidden, attackers don't really gain anything from knowing about this.

	Daniel Veditz [:dveditz]
	Comment 10 • 15 years ago

I agree: it's not an exploit and if anything publicity may help some users protect themselves (unlikely as that is). Maybe a warning about this should even be in the relnotes and on SUMO.

	Mike Beltzner [:beltzner, not reading bugmail]
	Comment 11 • 15 years ago

I think the real key is to understand why we're deleting the file; it might be a required artifact of it moving.

Either way, I don't think this blocks, as it's not a common user behaviour to switch back and forth between versions.

	Henrik Skupin [:whimboo][⌚️UTC+2] Reporter
	Comment 12 • 15 years ago

Mike, this will affect any users who run a major update. After they have updated from 3.0.x to 3.5.x the database is empty.

	Samuel Sidler (old account; do not CC)
	Comment 13 • 15 years ago

I think you wanted to renominate it based on that information.

	Mike Beltzner [:beltzner, not reading bugmail]
	Comment 14 • 15 years ago

Sure, but that would be why we should change it on 1.9.1, not 1.9.2, unless we changed the location again.

	kimsland
	Comment 15 • 15 years ago

(In reply to comment #13)
> Either way, I don't think this blocks, as it's not a common user behaviour to
> switch back and forth between versions.

Sorry to bring up an old possible (fixed?) bug (I'm also unsure if it was resolved or not)
But the "urlclassifier3.sqlite" file IS being deleted on thousands of User's computers everyday!

The free cleaning utility TFC by OldTimer (quoted on all malware removal forums, by malware removal specialists) Recommend, to run this cleaning utility, that subsequently removes: %userprofile%\Local Settings\Application Data\Mozilla\Firefox\Profiles\<profilename> Folder completely!
Including files: urlclassifier3.sqlite; XUL.mfl and XPC.mfl

So my question is this good or bad? 
Did the "bug" get resolved? 
Do you feel this file should be moved BACK to: %APPDATA%\Mozilla\Firefox\Profiles\<profilename> ?

I'm thinking Yes.
But I would like your thoughts on the matter...

	Samuel Sidler (:ss)
	Comment 16 • 15 years ago

kimsland: What you're talking about is a different but and you should file it separately. (It's likely an outreach-type bug, since it's not anything Firefox itself is doing.)

	David Dahl :ddahl
	Comment 17 • 15 years ago

(In reply to comment #17)

> The free cleaning utility TFC by OldTimer (quoted on all malware removal
> forums, by malware removal specialists) Recommend, to run this cleaning
> utility, that subsequently removes: %userprofile%\Local Settings\Application
> Data\Mozilla\Firefox\Profiles\<profilename> Folder completely!
> Including files: urlclassifier3.sqlite; XUL.mfl and XPC.mfl
> 
> So my question is this good or bad? 

If this "utility" is removing the entire profile directory, then it is a very serious issue. This would remove all extensions, themes and bookmarks as well. 

I am not sure who at Moz contacts third parties about this, but they should. Do we have a keyword or status whiteboard to flag this issue?

	Johnathan Nightingale [:johnath]
	Comment 18 • 15 years ago

(In reply to comment #19)
> If this "utility" is removing the entire profile directory, then it is a very
> serious issue. This would remove all extensions, themes and bookmarks as well. 

I suspect it is not - this is on modern versions of windows that separate out the "Roaming" app data and "Local" app data - with Roaming being the stuff you might want to keep around (the nomenclature is clearly pointing to corporate Windows deployments where you might be using many different workstations within a domain, and want your user data pushed to whatever machine you might be on), and Local being stuff that needs hard drive space, but not network push, like cache files.

Nevertheless, I disagree with OldTimer's comments in that support forum - deleting the urlclassifier.sqlite is not only something that can take a significant while to recreate, it is also a security risk, since users are left unprotected in the meantime. It is certainly something a person could try to delete when attempting to diagnose a problem, but it's not "good housekeeping" to get rid of it.

Moreover, the times he reports for re-population of the file are underestimates. I suspect he deleted it, started Firefox, and watched it to see how long until it stopped growing. But we populate this file in several pulls, to avoid overburdening potentially constrained network connections. So a user running this clean up every few days could basically end up never having phishing/malware protection in full effect.

The forum doesn't seem to want to let me post this, but yeah, I would strongly suggest people not use his clean up tool until it stops deleting protection software. It's very much akin to "cleaning up" someone's downloaded virus definitions for an antivirus program, and is dangerous regardless of whether or not the definitions eventually come back.

	kimsland
	Comment 19 • 15 years ago

(In reply to comment #20)
> (In reply to comment #19)
> ...
> The forum doesn't seem to want to let me post this, but yeah, I would strongly
> suggest people not use his clean up tool until it stops deleting protection
> software. It's very much akin to "cleaning up" someone's downloaded virus
> definitions for an antivirus program, and is dangerous regardless of whether or
> not the definitions eventually come back.

I don't understand what you mean about the "forum doesn't seem to want to let me post this"
If you are posting on the "forum topic" (I created) then your post WILL go through
If you are posting reply in the downloads area (specifically TFC.exe) then Yes this will be removed, as this is not a forum support topic, and comments are only related to the download of the file.

I would ask that you try again. Obviously free sign up required.
Personally I feel its worth it, as "Malware Removal Specialists" and even Members, will have a chance to make up their own mind.
And NOT just on the geeks forum, but on every forum that deals with Malware removal. (through back link reference)

"urlclassifier3.sqlite" is removed on thousands of user's computers everyday (on multiple forums quoting to use TFC.exe).
This itself may give Mozilla cause enough to be concerned, as they are the ones consistently re-supplying the updated database (through their server or provided dedicated server)

My concern is for the poor old member, who feels they are being better protected (through the process of Malware removal)by running TFC.exe
I would also like to note that TFC (Temp File Cleaner) is quoted on EVERY forum and in EVERY malware guide, EVERYwhere worldwide. It is the present standard used utility, by all support.

	kimsland
	Comment 20 • 15 years ago

Edit:

Oh, this is the old bug report topic, the new one I created is here: https://bugzilla.mozilla.org/show_bug.cgi?id=550722

Including external links
Maybe Mods here could move these relevant posts here to my created on topic bug report

	Justin Dolske [:Dolske]
	Comment 21 • 12 years ago

The sqlite DB is gone now.