Mozilla intentionally (I think) hides the referer when going from one HTTPS page to another in a different domain. This is a *very* bad design decision, as it prevents using at least one major online credit card processing service which looks for referer information (along with other info) for security checks. Currently we have been telling our customers that they have to use IE because mozilla/netscape simply does not work. We do not want to do this! I understand that this is done to improve security, since some sites place private information in plain text in URLs. Consider, though, that this is a problem with the site and cannot be fixed by the browser. For example, if someone walks by a person using the insecure site there private information will be right there on the screen ready to read (in the URL). The decision to make fundamentally insecure sites more secure is causing the browser to be *much* less usefull overall since it can often not be used to place orders online, forcing users to switch back to IE. By default, referer information should be included when jumping from one HTTPS page to another in a different domain. If you want to go beyond that and add the option (not selected by default!) of not providing the referer in this case, fine. I hope this descision will be reconsidered! Thanks

Reporter (Alan): what site or sites does this bug affect? According to bug 128213, there's a hidden pref an advanced user can set to allow referer information to be sent from one https site to another. That's just wrong in so many ways: 1. Yet another (hidden) pref. Hurts startup time, complicates code, doesn't do anything useful. 2. Why does it matter whether the target is http or https? 3. Giving sites a way to prevent the referer from being sent is, IMO, a security hole. A minor one, but enough of a hole that many porn sites refuse image requests when the referrer is not sent instead of assuming that the request is from a bookmark. I think the best solution is to send the referer but strip off the pathname, query string, and hash. For example, a link from https://answers.google.com/answers/main?cmd=threadview&id=8149 would have its referrer set to https://answers.google.com/.

Yeah, that last solution you propose was what badami, darin, mstoltz all seemed to agree to last time I mailed them about it... (send referrer from https to https, but strip everything but the hostname out of the url).

Alan, thanks for your report and that you care about Mozilla-based products. Which site are we talking about? Which information does it exactly what to see in the referrer? E.g. is *any* referrer (even a wrong one) sufficient? Is the hostname of the referring site sufficient? DOes it need the full, valid referrer? Any site which relies on the Referrer header to be present is broken and violates the HTTP specification. The spec is ultra-clear that the referrer is something optional that can be disabled on the user's discretion. Please notify the site about that. Reference: RFC 2616, sections 14.36, 15.1.2 and 15.1.3. The latter also says: Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol. This is not exact the case discussed here, but might be worth consideration. Sending the hostname of the referring host as Referrer between 2 https hosts might strictly be a violation of the HTTP spec. The hostname is not "the address (URI) of the resource from which the Request-URI was obtained". Apart from that, I currently don't know a reason not to send the hostname in this case.

The credit card processor in this case is authorize.net. Perhaps there are others as well. Thousands of sites (including my site, FreeFind.com) use this service to handle credit card transactions, but it does require a referer to be set. To guarantee complete compatibility with what's out there now, you'd need to send the complete URL. I would guess that stripping just the query string would probably work for most cases. Only sites which have them in their order processing sequence would need to update their credit card processing security handshake info. If you set the referer to just the domain name it would force essentially all sites using authorize.net to reconfigure their security handshake info before mozilla would work. Ideally they would (and I certainly would), but I think that practically speaking most would not. In a way, this does sound safest (if your willing to have your users take the compatibility/usability hit), but in some cases it might be less safe. For instance, the referer for two different sites: https://lotsasites.com/~userA/ and https://lotsasites.com/~userB/ would be the same... (and again, yes I know that the referer shouldn't be part of a security mechanism, but what is out there being used is what the browser needs to work with..) Again, where I've noticed this is going from a HTTPS page on one site (my site) to an HTTPS page on a different site (the credit card processor's site). And this may be more widespread than the thousands of sites using authorize.net. It's easy to envision lots of services having a referer check as part of their security mechanism. I have notified authorize.net of the problem. We've been experiencing this for months now, but they seem to be asleep at the wheel on this. For me, this underlines the importance of having mozilla work with the way things are currently set up on the net and not force people (both developers and users) to reconfig/recode/conform in order to support mozilla. I think that's very important to success. Given the above, I would recommend (and I hate to say this..gag) referer behavior that is 100% compatible with IE, at least by default. If including query strings is truly out of the question then at least keep everything else (the path). Think "always compatible" and thanks again, Alan

will address this for moz 1.0.1

Before sending the full path, I can't we try to advocate the broken site not to require a referrer? Move to evang?

[diatribe] Even if you could get authorize.net to update their system to support mozilla (and I think there are real issues with getting them to abandon referers, which are already intergrated into their system), what about the thousands of other sites that you don't know about that rely on the referer for some check/feature/etc? How do you track those sites down and tell them to re-code too? Even if you could track them down, would they? or would you just be systematically offending the internet community? (While your at it, you could tell them to get all the syntax errors out of their HTML pages, too ;) This is a basic compatibility issue. How compatible you want your browser to be? Do you think that most people will keep on using it if they have a small but steady stream of problems with it? [/diatribe] :)

mass futuring of untargeted bugs

Alan, there are not thousands of sites. This is a corner case any this site is the only problem we know of. I just called customer support of Authorzie.net, which declined to let me speak with a developer, claimed that this is working fine with Netscape 6.2 and refused to change their site. Well, their problem, I'd say.

And it seemed to work fine with their test account for WebLink with Beonex Communicator with the fake 3rd party referrers option enabled (bug 55477).

*** Bug 151640 has been marked as a duplicate of this bug. ***

Ben, I just tried again with the latest Mozilla 1.1 alpha (build 2002061104) and the very latest netscape (6.2.3) with the same (failing) results. It is my understanding that authorizenet has thousands of customers, each of whom is a webmaster with their own site(s), so this problem is probably greater than a single site. It is very difficult to get complete/accurate feedback from users (of all types), so I would hesitate to conclude anything about the scope of the problem based on lack of feedback. Typically only a minute percentage of people provide feedback even when it would be trivial to do so. When there are additional obstacles to providing feedback (registering for bugzilla and figuring it out, for example) you can expect even less than a minute percentage. Furthermore in the case of browser software, first a small percentage of the end users will be complaining to the webmaster with the problem ("your website doesn't work"), and then a small percentage of those webmasters will be then be downloading mozilla, testing, and then bugzillaing the feedback on to you. To make the challenge of writing a compatible browser even more difficult, it is difficult if not impossible to comprehensively "test the web" to make sure your browser is compatibile, especially when the test may involve buying things online! Because it is impossible to truly test the web and because getting a statistically significant quantity of feedback from users is problematic, it seems that a good strategy to ensure compatibility for web users would be to compare mozilla's precise behavior with ..I truly hate to say this.. IE, since that is the current standard. Insisting that companies re-write their code to be techinically correct is clearly a non-starter. One phone call to authorizenet (or more, as I have made!) demostrates this conclusively. Technically unconforming sites might be "their problem" but in fact it is "your problem" because if mozilla doesn't work everywhere they go the users will not keep using it regardless of "whose problem" it actually is. So my recommendation for the success of Mozilla is to be maximally compatible - work like (...) IE, even if IE is wrong. ..hmmm...there's a option for you! (x) IE compatibility mode ( ) mozilla RFC validator mode As for my specific case, all *I* personally need is for mozilla to by default send something (anything!) in the HTTP referer field in this specific case. Even "this referer was not set due to security concerns" would work. Then I could adjust my authorizenet account to accept it. :) Thanks again

My company's icon technology makes specific use of the browser's referrer field. We are yet another example of a company that uses the browser's referrer field for added security and to provide a service. Our model has proven to work in all browsers until this more recent change to the Netscape source to prevent referrers from being passed from one HTTPS domain to another. The omission of the referrer results in our icon not being displayed to the end user(the browser) making the browser much less useful. I too would not like to suggest to all of our customer's customers to not use any Netscape browser products. Even if the browser only sent over the referring domain I could work out solution. GeoTrust's icons get millions of hits per week, and provide valuable information to the end user. I do not see anywhere in RFC2616 where it states the referrer should be omitted in HTTPS if another server is also HTTPS. If the user feels he has privacy concerns and wishes to disable referrer sending, and therefore disable services like GeoTrust's True Site, that is ok. But by default the browser should be sending this vital information along. This is not the only case where referrer sending has been left to the programmer's interpretation. Another example of where the referrer is not sent would be in SCRIPT SRC tags. Again Netscape browser's do not send a referrer, while other browsers do. Making the Netscape browsers a little less useful again.

> Another example of where the referrer is not sent would be in SCRIPT SRC tags Irrelevant to this bug, and fixed in bug 64248 a while back... 7.0beta1 should be sending "Referer:" in that situation, modulo this bug.

*** Bug 154062 has been marked as a duplicate of this bug. ***

*** Bug 157054 has been marked as a duplicate of this bug. ***

we're also suffering from this problem. our website uses third party for a drug info. they provide us with a drug description and info http://cp.gsm.com/ we link to them from regular pages and from secured pages. the latter links don't work :( they use HTTP_REFERER as some sort of authorization. yesterday i filed a bug 157054 that is a dup of this one.

It seems from the comments that it is a authorize.net problem. The referer field is optional in the HTTP specs; So I agree 100% I agree with Comment #3 From Ben Bucksch. It seems to me also, from the doc of authorize.net (see: https://secure.authorize.net/docs/developersguide.pml ) that this strange behaviour should occur only in WEBLINK mode (and perhaps ADC RELAY RESPONSE). It should not occur in ADC DIRECT RESPONSE, because the commnunication process relies only on the server (not on the browser). Am I right?

The issue isn’t about whether or not the Referrer is optional or not. The issue is that many applications use this field as an added security check. If someone wishes to disable Referrer passing, and therefore not have access to the application that checks it, so be it. But until recently all browsers passed Referrer regardless if they were under HTTP or HTTPS. I agree with the suggestion to not pass GET data in the Referrer, but a Referrer should be passed by default in my opinion. As a site administrator I want to share my own content between domains. But I don’t want unauthorized sites to use my content, and my bandwidth. So I put a simple check in my configuration to state which domains (Referrers) are allowed to get at the content. With the recent change I can’t make this check, and therefore I can’t get at my own content. This is merely one of many examples. It is not necessarily a fault of the application. The application acts as expected: The security check fails therefore the action is not performed. But as an end user, or the client of the application, this is not the desired behavior, and there is no way to fix it other than using a “working” browser (IE, Netscape <4, Opera, etc).

John, this is *clearly* a fault of the application. The HTTP standard says that the Referer is optional, meaning that the site *must not* rely on it. If it does, it does not comply to HTTP anymore. Even worse are browsers passing Referers from HTTPS to HTTP. I don't know, if any one does that, but it would be a clear violation of HTTP, and a potential security problem. All we are talking about is *maybe* adding a workaround for a certain egde case. But the fact remains that the site is at fault.

> a fault of the application. s/application/website/

> Even worse are browsers passing Referers from HTTPS to HTTP. what about going from HTTPS to HTTPS on another host? i think it should pass the referer in that case. what does the HTTP spec sez about it?

I am not fighting to change the standard at the moment. I understand that the Referrer is optional. But many WWW sites provide functionality based upon the Referrer. The example case I listed above is used by many web administrators, mainly to restrict content down to a single domain, but sometimes to a list of domains. Search around and you will find many examples on how to enable this "security" feature for your content. If I were to follow your point to completion, why send a referrer ever? It is optional, so why not just delete it completely? The answer is because it provides the WWW server being called valuable information it can use to make simple security checks, fraud checks, etc. I am largely responsible for the icon technology we use here at GeoTrust. I don't consider my application broken. It does just what it should, if I am not sent a referrer I have no idea which icon I should render, therefore I render nothing. I followed the specification, I don’t crash because I don’t get a referrer, but I also can’t provide the client with the functionality they want. I have not seen an example where an HTTPS page sends a referrer to an HTTP page. This would be a clear violation of the specification. But HTTP to HTTP, or HTTPS to HTTPS, is not a violation, and is what browsers have been doing since the beginning of the WWW. I obviously do not believe this is a fault of the application. The application looks for information it gets from “working” browsers, and acts upon that information. If the information is missing, it does nothing. That is following the specification to the letter. By not sending referrer the browser is merely proving to be less useful to clients and servers. Therefore consumers will simply use a browser that provides them the content/features/actions they desire. We now have about a million hits a day on our icon. And currently 90% of the browsers hits are from IE. Personally I’d like to see an alternative, but this issue is an example of why there isn’t right now.

For what it's worth, Bank of America's eStores Settle Up Credit Card processing system also uses HTTP_REFERRER to authorize credit card transactions: "When a customer places an order, the address of the site generating the order is compared to this list of sites authorized by the merchant before the transaction is submitted for payment processing. If the addresses do not match, the order is rejected and no credit card authorization is obtained. When the addresses match, the order is submitted for credit card authorization and finalized accordingly." BofA's system returns a "site not authorized to post" error. Sure enough, in using their debugging system, HTTP_REFERRER is not passed.

i think we should settle on a solution for this bug by mozilla 1.2 ... our current solution (of a hidden pref) is just not satisfactory.

i'm treating this as an enhancement bug right now. our current security policy prevents sending a referrer in this case. i'm not sure if we can reasonably do anything... sending a only the prepath as a referrer is breaking the spec (lying to the website) and sending the complete referrer is really putting the user at risk. i'm also not sure how many sites break because of our restrictive security policy. all this makes it very difficult to determine the best solution for this bug.

I for one would prefer to see half of the truth as apposed to nothing. Does the specification state that the entire GET string must be sent as part of the Referrer? I didn't infer that, the spec states: Referer = "Referer" ":" ( absoluteURI | relativeURI ) Example: Referer: http://www.w3.org/hypertext/DataSources/Overview.html The definition of absoluteURI shows Query String as optional. So isn't sending the complete referrer without the query string acceptable? I don't see this as being a security risk. If you want to provide a paranoid option for turning off referrer that's fine. But by default the solutions already suggested seem appropriate. Every day I come across another cgi/web application that makes use of referrer. Luckily for you most applications are running under HTTP so this bug doesn't show its ugly head often. But more and more people are realizing they do need SSL for these applications. We had hoped this bug would have been resolved many months ago. Certainly before the Mozilla 1.1, and Netscape 7 were released. Webmasters and end users are getting frustrated, and our only work around is to suggest not using Mozilla based browsers. You are losing many loyal fans because of issues like this. Our icon is now getting a couple of million hits a day. We are still receiving complaints about all of the new Mozilla based browsers. More alarming to you should be the statistics we are noticing, before this bug was introduced we had about 86% IE hits, now we are seeing closer to 93% IE hits.

I've always thought that https->https should send referrer. The spec says: (rfc2616, 15.1.3): Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol. so HTTPs is not a "(non-secure HTTP request)", so the refererer should be sent. I think that the main purpose of this is so that a GET login to an https page doesn't then get sent out in the clear subsequently. This does leave open the case of a link on the original server to an https page which grabs the referer in some malicious way. GET login requests are evil anyway. ISTR getting outvoted last time, htough...

Any bad guy can run an https site...

Right, but you'd ahve to get that 'bad guy' to get a link to their page onto the other site's https page. Not impossible, but its not something 'anyone' can do

What about XSS (Cross Site Scripting)? It would be hard, but not impossible to create a link that would appear legit.

isn't cross site scripting a same origin violation? so wouldn't the user have to explicitly allow it (assuming signed js were involved)? given that the attacker would have to add their link to the legitimate site's pages, i'm inclined to agree with bradley. what is the real risk of enabling cross site HTTPS referrers? websites can anyways link to other HTTPS sites indirectly via a HTTP redirect if they really don't like this behavior, which i'm sure they're well acquainted with since IE sends cross site HTTPS referrers. are we really helping by preventing cross site HTTPS referrers?

fixed summary to state the problem instead of just one of the proposed solutions.

I assume we send no referrer when the user types something in the url bar, right? If so, I'm also leaning toward allowing this....

right, no referrer should be sent in that case.

> isn't cross site scripting a same origin violation? It's not scripting, but just linking. What about e.g. webmail? Not all webmail sites are sane... I still think that any site depending on referrers is broken per the HTTP spec, especially in this case. IMO, this is an Evang issue, not a browser issue. Why can't we first try to solve this via Evang, like we do with many other cases?

Don't forget that this can't happen automatically, because if js was enabled, then the malicious page just has to grab the current rul via js. If js is disabled, then you have to entice someone to click on an https page to an https page on another domain, where the current URL has sensitive information encoded as part of a GET url.

> If js is disabled, then you have to entice someone to click on an https page to > an https page on another domain No clicks needed. Think of images in HTML mail sent to webmail accounts.

Darin states that he's not sure how many sites are affected. Every single online store that uses authorize.net's Automated Direct Connect Relay Response mechanism will not work. I don't know how many customers authorize.net has, but my guess would be that it is in the thousands. I'm tired of having MSIE as a "backup browser". I get so frustrated when I have to revert to it to conduct business on the Web. We can sit back and blame the site designers or we can accept the reality of today's imperfect Web and accomodate some of the major mistakes. If I understand correctly, the reason you don't send the referrer is to prevent sensitive info in the URL from being transmitted to an untrusted site. Encoding sensitive information in a URL is pretty brain-dead. Isn't this also a failure of the site designer? So we're working around one class of mistake but refusing to accomodate another, and the end result is a browser that is less useful to the end user on today's Web. I know that mass popularity is not the end-all goal of the Mozilla team. But if you don't achieve some sort of critical mass, MS will continue to drive the de facto standards of how sites are constructed, and Mozilla will always play catch-up. I love Mozilla passionately and evangelize at every opportunity. But the reality is that even I have to keep MSIE on standby. Most of the time, I need it because some Frontpage idiot can't design with an eye toward standards. In those cases, I either skip over the site entirely or begrudgingly open IE. But in this case, I find it more difficult to blame the site owners when the referrer behavior of major browsers has been consistent for the past 5-6 years. Suggestion: I know Mozilla needs to be secure out of the box. But if you implement a "fix" for this "bug", I feel that a checkbox is not enough; Mozilla needs to send the referrer on HTTPS _out of the box_. Otherwise, it will never gain mainstream acceptance. Believe me, if you saw the authorize.net error message, you'd never think that there was a browser setting you could adjust to correct the problem. Joe User would just say "Mozilla sucks" and go back to IE. How about a baseline security policy that is "mostly secure"; at install time, the power user could check something that would adjust the overall policy toward "manic paranoid", locking down anything and everything (including referrer on HTTPS, Javascript popups, etc). Just a thought.

I have to agree with Jason Priebe above. There are countless sites that are using Referrer checks as part of a normal security/sanity/fraud check. GeoTrust's customer list is in the many thousands right now. The discussion about Webmail is an interesting one, but sure sounds like you are trying to invent a problem for the current "solution". Ask yourself two things; 1) How many webmail sites use HTTPS? AND 2) How many of those webmail sites encode the password in a GET query string. Are there any? Certainly none of the ones I am familiar with do(EXs: Yahoo, Hotmail, Outlook Web Access). Since almost all webmail systems use HTTP, if they simply encoded your username/password in the GET query string this bug doesn't add any security anyway.

benb: I forgot about <img>. hmm.... Any site using the referer as a security check is broken, since its trivially forged. Using it as a sanity check is only slightly less broken, because people may choose not to send it for privacy/etc reasons. (Using it as a sanity check to reject incorrect, as opposed to non-existant, referers may be more useful, although that can again be set by the user) Using it in combination with other things (ie username/password) doesn't make it less broken, since you then have an extra layer of authentication making the referer stuff almost useless. _However_, there are sites which care about this, and since the spec doens't forbit us to do this, and a large % of users have browsers (including ns4) which do so, and there are sites which rely on it, no matter how broken they may be.... Can we just strip off the query paramaters? That would get rid of most potnetially sensitive info. People who code sensitive stuff into the path (making it work via server side support of some sort) are likely to be rare. I dunno - thoughts?

A site checking the referrer as a security check is not "broken". How else are webmasters supposed to prevent the mass stealing of their content and bandwidth by other sites? The answer for the last 10 years has been they simply check the referrer to make sure it is one of thier own servers making the request. It isn't trivial to hack every end user's client browser to pass a specific referrer when calling a particular url. In fact I think that would be very difficult. If you could hack the client's browser, you could do a lot more harm than sending a bogus referrer. My situation is an extension of the above. We host thousands of images using one image tag. We use the referrer to figure out which image to send. If we don't get a referrer, we send an empty image.

Well, its 'broken' in that people can and do disable referer strings. Personally, I don't, but there are those that do, so.... Note that what you're doing is not 'security' in the sense that I was using the word. I meant that relying on the referer to make sure that someone only visits your page from the login page, or has completed part 1 of the form before part 2, or something like that, is silly, and broken. You're trying to protect yourself against a malicious thrid-party page, but an uncaring user, rather than a malicious user, which is a different situation. Anyway, I'm still leaning towards opening this up, both because the spec doesn't forbid it, and for 4xp/iecompat reasons. Darin pointed out to me that this is actually a pref ATM, so we'd just be swapping the default value. (BTW, Bug 89995 was the other bug I was refering to earlier, where this check was *way* too strict.)

ok, here's the patch to enable cross-site HTTPS referrers. i also thru in a minor perf tweak suggested by bbaetz.

Comment on attachment 99018 [details] [diff] [review] v1 patch >+// This pref controls whether or not we send HTTPS referrers >+// cross domain. By default, this is enabled for compatibility >+// with other browsers and many websites (see bug 141641). I'd prefer the comment be more explicit that both hosts are HTTPS so that "cross domain" isn't misinterpreted as https->http // Controls whether we send HTTPS referres to other HTTPS sites. // By default this is enabled for compatibility (see bug 141641) sr=dveditz

dveditz: yeah, that's much clearer.. thx!

Comment on attachment 99018 [details] [diff] [review] v1 patch This treats all HTTPS servers as equally trusted WRT referrees, which seems absurd, but I recognize the compatibility issues here and I think the actual risk is small. Ditto on adding a more explicit comment. Other than that, r=mstoltz.

fixed-on-trunk

Comment on attachment 99018 [details] [diff] [review] v1 patch a=rjesup@wgate.com for 1.0 branch. Change mozilla1.0.2+ to fixed1.0.2 when checked in

hello, wrt comments #41 and #42: stripping of the query string can easily be worked around using path info. when I had my https-/referrer problem, I solved it this way. leaves as much information as before to be shipped in the referrer... wrt mass stealing of content: in theory, not accounting for server load and application complexity, using time stamped urls for your images could prevent most bogus image requests since the urls time out after a while (just like time stamped email addresses which bounce after their time out period).

This is needed by a major bank customer in Japan. CC'ing jaimejr and buckland. Please make sure that this will be in the next NS release.

tever: pls verify this as fixed on the trunk. thanks!

This is what I am seeing: Referers passed from HTTPS to HTTPS (original problem now working) " " " HTTP to HTTP " " " HTTP to HTTPS NOT passing Referers from HTTPS to HTTP which from my understanding is how it is supposed to work. internal tests at http://bubblegum/tever/refererSSLb.html and http://bubblegum/tever/refererSSLa.html. verified trunk - 09/18/02 builds - winNT4, linux rh6, mac osX

Adding a dependency to bug 150046, as it is needed by a major embedding customer in the JA market.

In and email I asked Kat if fixing this bug was both necessary AND suffient to get Mitsui-Sumitomo to support us. I interpret the answer below to mean that if we fix this bug all known blocks to their support are cleared. Kat wrote: We have word from Japan that this is the blocking issue and they have no other issue currently. We want this for getting Mitsui-Sumitomo to support us. This is a vital step in our overall plan to get top 24 banks to support us. If the committee needs to talk to me about this, I will be happy to bring any info you need.

adt1.0.2+, already has drivers approval -- please land on the 1.0 branch.

fixed1.0.2

tever: can you pls verify this as fixed in tomorrow's 1.0 branch builds? thanks!

verified1.0.2 - 09/24/02 builds - winNT4, Linux rh6, mac osX

ver 0.8.13 the extension conflicts with a plugin of this page. so error message open continuously. even if it is off, error causes. http://sejin90.tistory.com/227