NTLM fails under certain HTTP2/TLS variants. I have tested the following variants in the current nightly (58.0a1 (2017-10-23) (64-Bit)): HTTP1.1/TLS 1.1 OK HTTP1.1/TLS 1.2 OK HTTP2/TLS 1.1 OK HTTP2/TLS 1.2 FAIL FAIL means that there is no NTLM Auth header sent from firefox. To enforce Firefox to use only TLS 1.2 i set the property "security.tls.version.max" to 2. To enforce HTTP/1.1 i disabled http2 on the server side. FYI - We are using tomcat (version 8.5.23) on the server side. for further details i provide networklogs for each case as an attachment.

(not sure if this could somehow relate to bug 1409858)

log_http2_tls12.clean has a channel (https://host.example.com:10009/intergator-nightly-head/web/_auth/6951a6de/?ret=%2Fintergator-nightly-head%2Fweb%2Fitl%2Fdebug%2Frequest.itl) that asked for credentials because of 401 and, because the prompt has been canceled in 1.5s, the channel showed the 401 response (expected). the HTTP2/TLS 1.2 combo: - connects apparently to a completely different server that asks for basic auth ('intergator' realm) instead of NTLM - 2017-10-24 08:13:53.843 │ Socket Thread │ nsHttpConnection::EnsureNPNComplete 000000001B4BFC00 [.S.....[tlsflags0x00000000]host.example.com:10009] negotiated to 'h2' for the HTTP2/TLS 1.1 case I don't see h2 being used: - 2017-10-24 08:11:45.926 │ Socket Thread │ nsHttpConnection::EnsureNPNComplete 00000000186C9800 [.S.....[tlsflags0x00000000]host.example.com:10009] negotiated to 'http/1.1' This could be a server config problem...

What doesn't work ist automatic NTLM negotiation over HTTP/2. So as you said it prompted me and i canceled the prompt, as the prompt isn't the expectation. The automatic negotiation does work over HTTP1.1 as you can see in the logs. We see this behaviour since firefox 54. You are right HTTP2/TLS1.1 seems to really speak HTTP/1.1 . I dont know why, but in these combo automatic ntlm negotiation works. Could this be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1346392? A server problem is possible - but unlikely. But to verify what can i check?

(In reply to Frank Taffelt from comment #3) > What doesn't work ist automatic NTLM negotiation over HTTP/2. That's not the correct interpretation of my findings. What happens is that in the HTTP2/TLS 1.2 case the server is not asking for NTLM at all, it ask only and only for Basic auth. There is no Server response header in any of the responses, so hard to say to which server we connect in what case. Also according how the WWW-Authenticate header name is capitalized differently, I believe we are talking to a completely different server in the HTTP2/TLS 1.2 case. Hence, a server or some SSL hardware/balancer middleware issue. I don't think Firefox is in fault here. Patrick, Nick or Dragana (all cc'ed) may know about some nuances that influences npn with regard to tls version. > So as you said > it prompted me and i canceled the prompt, as the prompt isn't the > expectation. The automatic negotiation does work over HTTP1.1 as you can see > in the logs. We see this behaviour since firefox 54. > > You are right HTTP2/TLS1.1 seems to really speak HTTP/1.1 . I dont know why, Would be a good start to find out. > but in these combo automatic ntlm negotiation works. > > Could this be related to > https://bugzilla.mozilla.org/show_bug.cgi?id=1346392? Fixed on 55 and according my findings - no.

I also just noticed that the URL asking auth is different in every log: the faulty one... log_http2_tls12.clean: 2017-10-24 08:13:53.824 ⁃ nsHttpChannel ⁃ 2a93d000 ⁃ finished ⁃ 401 ⁃ https://host.example.com:10009/intergator-nightly-head/web/_auth/6951a6de/?ret=%2Fintergator-nightly-head%2Fweb%2Fitl%2Fdebug%2Frequest.itl ...is apparently a different URL from all others... log_http11_tls12.clean: 2017-10-24 08:02:37.667 ⁃ nsHttpChannel ⁃ 1b894000 ⁃ finished ⁃ 401,401,200 ⁃ https://host.example.com:10009/intergator-nightly-head/web/itl/debug/request.itl;jsessionid=788B7681235ECE16A645A2C6FE8F9BBD log_http11_tls11.clean: 2017-10-24 08:33:37.627 ⁃ nsHttpChannel ⁃ 2ae17000 ⁃ finished ⁃ 401,401,200 ⁃ https://host.example.com:10009/intergator-nightly-head/web/itl/debug/request.itl;jsessionid=087E10B7924FD09D594B7F6E8177AFCC log_http2_tls11.clean: 2017-10-24 08:11:46.016 ⁃ nsHttpChannel ⁃ 29939000 ⁃ finished ⁃ 401,401,200 ⁃ https://host.example.com:10009/intergator-nightly-head/web/itl/debug/request.itl;jsessionid=F213D6A7A67DC7131CCCC08C31E05CD6 Closing as invalid based on that.

And one more note: apparently the server in the log_http2_tls12 case (which is the only one that negotiated h2) decides it can't use NTLM auth for the request in question, because I can see this in the log log_http2_tls12.clean: 2017-10-24 08:13:53.690 ⁃ nsHttpChannel ⁃ 1bd16000 ⁃ finished ⁃ 200 ⁃ https://host.example.com:10009/intergator-nightly-head/web/itl/debug/request.itl;jsessionid=5E37AE0241A9C67A8E75F8A7A9237BEA i.e. no 401 response. this is not the first channel on the h2 session and no 401 NTLM auth response happened on it before this response.

new testcase with failing NTLM.

(In reply to Frank Taffelt from comment #7) > Created attachment 8921895 [details] > new network logs with HTTP/2 and TLS1.2 > > new testcase with failing NTLM. 2017-10-25 14:00:12.681 ⁃ nsHttpChannel ⁃ 1c655000 ⁃ cancelled ⁃ 307 ⁃ https://host.example.com:10009/intergator-nightly-head/web/api/1.0/http-auth/rest/session/getSessionInfo ⁃ 804b0003,804b0003 2017-10-25 14:00:12.736 ⁃ nsHttpChannel ⁃ 23f45000 ⁃ cancelled ⁃ 307 ⁃ https://host.example.com:10009/intergator-nightly-head/web/api/1.0/http-auth/rest/session/getSessionInfo;jsessionid=2C577008630712EBD47427561F9F9B96 ⁃ 804b0003,804b0003 2017-10-25 14:00:12.827 ⁃ nsHttpChannel ⁃ 204fd000 ⁃ finished ⁃ 500 ⁃ https://host.example.com:10009/intergator-nightly-head/web/api/1.0/http-auth/rest/session/getSessionInfo ⁃ 0 A redirect chain, I don't see any NTLM/401 here...

i'm not sure what you mean? searching for NTLM in my latest log brings: 2017-10-25 14:00:12.758000 UTC - [Socket Thread]: V/nsHttp index 61: www-authenticate NTLM 2017-10-25 14:00:12.758000 UTC - [Socket Thread]: V/nsHttp index 62: :status 401 yes the 401 is part of the redirect chain. the redirect chain is something like: URL -> 307 -> URL;jsessionid=XXX -> 401(NTLM) -> URL;jsessionid=XXX -> 307 -> URL

Now we are getting somewhere! Thanks for this log. Please next time just add an explanation of what you are submitting and what in your opinion we should look for - would save some time. What I meant in my last comment was that I didn't see an nsHttpChannel instance (above and transparent to h2) that would be handling that 401. The more interesting line in the last log is: 2017-10-25 14:00:12.757000 UTC - [Socket Thread]: I/nsHttp Http2Decompressor 0000000026777B08 connection-based auth found in www-authenticate Triggered at [1]. This has been introduced in bug 1360574. ni? on Nick then, looks like the h1 fallback still doesn't work. The issue with some slightly different symptoms might be present pre-55 too. [1] https://dxr.mozilla.org/mozilla-central/rev/a124f4901430f6db74cfc7fe3b07957a1c691b40/netwerk/protocol/http/Http2Compression.cpp#649

Hmm... looking more to the log, we *do* restart the transaction on a new h1 connection. But the response it gets is not 401, it's the following instead: 2017-10-25 14:00:12.772 │ Socket Thread │ http response [ 2017-10-25 14:00:12.772 │ Socket Thread │ HTTP/1.1 307 2017-10-25 14:00:12.772 │ Socket Thread │ location: /intergator-nightly-head/web/api/1.0/http-auth/rest/session/getSessionInfo 2017-10-25 14:00:12.772 │ Socket Thread │ Pragma: no-cache 2017-10-25 14:00:12.772 │ Socket Thread │ Cache-control: no-store,no-cache 2017-10-25 14:00:12.772 │ Socket Thread │ Expires: -1 2017-10-25 14:00:12.772 │ Socket Thread │ Transfer-Encoding: chunked 2017-10-25 14:00:12.772 │ Socket Thread │ Date: Wed, 25 Oct 2017 14:00:11 GMT 2017-10-25 14:00:12.772 │ Socket Thread │ OriginalHeaders 2017-10-25 14:00:12.772 │ Socket Thread │ location: /intergator-nightly-head/web/api/1.0/http-auth/rest/session/getSessionInfo 2017-10-25 14:00:12.772 │ Socket Thread │ Pragma: no-cache 2017-10-25 14:00:12.772 │ Socket Thread │ Cache-control: no-store,no-cache 2017-10-25 14:00:12.772 │ Socket Thread │ Expires: -1 2017-10-25 14:00:12.772 │ Socket Thread │ Transfer-Encoding: chunked 2017-10-25 14:00:12.772 │ Socket Thread │ Date: Wed, 25 Oct 2017 14:00:11 GMT 2017-10-25 14:00:12.772 │ Socket Thread │ ] So, to me this still looks like a server config issue. Sorry. log notes: nsHttpChannel @23f45000 created 2017-10-25 14:00:12.736 nsHttpTransaction @157ebc00 created 2017-10-25 14:00:12.744 Http2Stream @c6c48a0 created 2017-10-25 14:00:12.745 restarted on: nsHttpConnection @16168400 created 2017-10-25 14:00:12.764

Yes maybe you restart a new transaction on a h1 connection, but IMHO there is no negotiation of the NTLM auth on firefox side. The request doesn't contain any : Authorization: NTLM TlRMT... request header. Looking at the my last log (HTTP/2): 2017-10-25 14:00:12.757000 UTC - [Socket Thread]: I/nsHttp Http2Decompressor 0000000026777B08 connection-based auth found in www-authenticate looks good, but i didn't find any entries of the form: 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::ProcessAuthentication [this=000000001BD53040 channel=000000002AE17730 code=401 SSLConnectFailed=0] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::PrepareForAuthentication [this=000000001BD53040 channel=000000002AE17730] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp proxy continuation state has been reset 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=000000001BD53040 channel=000000002AE17730] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=000000001BD53040 channel=000000002AE17730 proxyAuth=0 challenges=NTLM] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetIdentityFromURI [this=000000001BD53040 channel=000000002AE17730] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpAuthCache::GetAuthEntryForDomain [key=https://host.example.com:10009 realm=] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: V/nsHttp nsHttpNTLMAuth::ChallengeReceived [ss=0000000000000000 cs=0000000000000000] 2017-10-24 08:33:37.669000 UTC - [Main Thread]: V/nsHttp Force use of generic ntlm auth module: 0 2017-10-24 08:33:37.669000 UTC - [Main Thread]: V/nsHttp Default credentials allowed for host: 1 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp identity invalid = 0 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/nsHttp nsHttpChannel::ConnectionRestartable this=000000002AE17000, restartable=1 2017-10-24 08:33:37.669000 UTC - [Main Thread]: V/nsHttp nsHttpNTLMAuth::GenerateCredentials 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/negotiateauth nsAuthSSPI::Init 2017-10-24 08:33:37.669000 UTC - [Main Thread]: D/negotiateauth AcquireCredentialsHandle() succeeded. which are present in my working http/1.1 logs? Of course i did set the "network.automatic-ntlm-auth.trusted-uris" config property ;-) On the serverside i turned on request logging and it also shows that there is no request header like "Authorization: NTLM TlRMT..." on http/2 Sidenote: This was working in Firefox until version 53 and is also working in current Chrome and Edge.

I don't know what exactly you expect Firefox should do, but my interpretation of how we and the server should behave is the following: - initially establish an h2 session - potentially, make whatever number of requests on that sessions, while server is not asking for authentication - do a request R for which the server returns 401 with a connection based schema (like NTLM) - the browser then creates a new connection to the same origin (supposedly the same server) disabling h2 on it - the http transaction (representing the request R) is then carried our again on this new connection, the same way as it was made before, i.e. we send the SAME request R including the headers again on the new h1 connection - (1) the server is expected to answer with EXACTLY the same response (401 with NTLM) now on the new h1 connection, as it has responded just before on the h2 session - the browser then does as usual, obtains credential (using default in your case), sends Type 1 message, expects 401/Type 2 response, sends Type 3 and receives 200 with the content ; since then the connection is authenticated But in the last log I can see that at (1) the h1 server answers with 307 (a redirect) instead of with 401/NTLM. That proves you expect something else from the browser to do. I believe you expect we process the 'NTLM' challenge and on the new h1 connection we send a request with Type 1 message. Then let you know - we don't. I don't know how exactly other browser behave (Chrome/Edge/IE). This is how we have chosen to do this since it seems like most logical way for most common scenarios. Simply repeat the same request on an h1 connection with expectation the server responses with (the same) 401 again. It could also be explained as: the 401/NTLM response received on an h2 session is completely ignored by the browser. I assume there is no specification on how browsers should exactly handle the h2->h1 connection based schema fallback. Nick, do you know about any spec/other browsers behavior as you were working on the h1 fallback? (The bug remains as INVALID, tho)

(In reply to Frank Taffelt from comment #12) > This was working in Firefox until version 53 Frank, could you please use mozregression [1] to find the exact changeset that broke this? I would be interested, since I really don't know from the top of my head (and bugzilla search) which patch could break this. (This has a lower priority than the next query, tho) > and is also working > in current Chrome and Edge. I would be interested in a PCAP from Chrome and Edge to see what exactly those two browsers do in that case. Would you be wiling to provide it? You can also send the PCAPs directly to my bugzilla email privately to prevent exposing them here publicly. Thanks. [1] https://github.com/mozilla/mozregression/releases

(In reply to Honza Bambas (:mayhemer) from comment #13) > I assume there is no specification on how browsers should exactly handle the > h2->h1 connection based schema fallback. > > Nick, do you know about any spec/other browsers behavior as you were working > on the h1 fallback? > > (The bug remains as INVALID, tho) There's no strict specification. Basically, it's a combination of RST_STREAM with the error code HTTP_1_1_REQUIRED (https://tools.ietf.org/html/rfc7540#section-7), and discussions about NTLM and similar connection-based auth on the http-wg mailing list saying "yeah, those won't work with http/2, so here's the error code we've come up with for that situation". I, too, would be interested in the pcaps & mozregression results. We had positive interop on HTTP_1_1_REQUIRED with IIS back when we first added that to the spec, so either this is some other server that behaves differently, or something changed that unexpectedly broke our HTTP_1_1_REQUIRED implementtion.

As soon as our security gives a go , i'll provide the pcaps. Also i had some talks with the guy that did the authentication web endpoint for our product and has to work with all the browsers and that’s what he has to say (he is not on bugzilla): Hello. Reading the comment above about the way firefox handles downgrades I now understand why it is working with other browsers but not Firefox. The other browsers do send a NTLM authorization header after downgrade, Firefox intentionally (as described) does not. If ones goes by the definition that NTLM is a connection based „session“ and transparent to anything running on the web server, and Tomcat is the one doing the NTLM server part, then this might be correct. Alas it’s not always as simple as that and the web server might simply be a proxy to an authentication server somewhere else. Connected via some other protocol. In this case the browser believes it is working connection based, but the web server actually uses something else, i.e. a session id in cookie or url part to tie the requests together and doesn’t care about the connection. In this case the real endpoint doesn’t expect to receive the intial request (R it was called), i.e. w/o NTLM authorization header, a second time after responding with the www-authenticate header to the first request. It will see this as „client does not support NTLM“ and fail. It’s a „we go by the definition“ vs „the world is vast, be flexible“ . I need to and can be flexible. I will work around that by setting a flag for ntlm + http2 protocol to disregard that duplicate request. Not fun, but doable and an insurance for me if other user clients do the same. It’s simply another piece in the long list of browser peculiarities for me. That is also the reason for that 307 redirect confusing people. Sorry about that. It’s simply there to persist the session cookie (the ;jsessionid path part is a byproduct for certain cookie less environments) . It has nothing (the 401 response follows after that) but then anything to do with authentication: one browser ignore(d) a set-cookie header on a 401 response because the NTLM authentication is/was more important. Another forgot it when changing protocols and displaying a self signed certificate warning. Or send that NTLM header like a basic auth header on any subsequent request for that path (preemptive authentication), corrupting any http post doing so. So if this was not a test case the web server would actually redirect to an artificial path to do NTLM later on while persisting that cookie. Kind regards, Gunnar Brand

(In reply to Frank Taffelt from comment #16) > As soon as our security gives a go , i'll provide the pcaps. > Thanks you! I expect to see in the logs what's described just below - other browsers send the request on the new h1 connection with the auth request headers Type 1 message. But it's still worth confirm this 100%. > Also i had some talks with the guy that did the authentication web endpoint > for our product and has to work with all the browsers and that’s what he has > to say (he is not on bugzilla): > > Hello. > > Reading the comment above about the way firefox handles downgrades I now > understand why it is working with other browsers but not Firefox. > The other browsers do send a NTLM authorization header after downgrade, > Firefox intentionally (as described) does not. > > If ones goes by the definition that NTLM is a connection based „session“ and > transparent to anything running on the web server, and Tomcat is the one > doing the NTLM server part, then this might be correct. Alas it’s not always > as simple as that and the web server might simply be a proxy to an > authentication server somewhere else. Connected via some other protocol. In > this case the browser believes it is working connection based, but the web > server actually uses something else, i.e. a session id in cookie or url part > to tie the requests together and doesn’t care about the connection. In this > case the real endpoint doesn’t expect to receive the intial request (R it > was called), i.e. w/o NTLM authorization header, a second time after > responding with the www-authenticate header to the first request. It will > see this as „client does not support NTLM“ and fail. Yes, that's exactly that. > > It’s a „we go by the definition“ vs „the world is vast, be flexible“ . > I need to and can be flexible. I will work around that by setting a flag for > ntlm + http2 protocol to disregard that duplicate request. Not fun, but > doable and an insurance for me if other user clients do the same. It’s > simply another piece in the long list of browser peculiarities for me. It is a particularity, but today trend is to make browsers behave in a more unified and predictable way, on any level. Hence, if all major browsers but Firefox behave in some way, we should probably change our code to be in parity. It will be a bit more work and I'm afraid the priority will be low. Let's track this at least as a known issue and difference from other browsers. > That is also the reason for that 307 redirect confusing people. Sorry about > that. It’s simply there to persist the session cookie (the ;jsessionid path > part is a byproduct for certain cookie less environments) . It has nothing > (the 401 response follows after that) but then anything to do with > authentication: one browser ignore(d) a set-cookie header on a 401 response > because the NTLM authentication is/was more important. Another forgot it > when changing protocols and displaying a self signed certificate warning. Or > send that NTLM header like a basic auth header on any subsequent request for > that path (preemptive authentication), corrupting any http post doing so. So > if this was not a test case the web server would actually redirect to an > artificial path to do NTLM later on while persisting that cookie. This is perfectly OK and I understand what's going on. Not the first time I can see a mechanism like this, it's not uncommon to redirect to itself to add a cookie or add query strings. > > Kind regards, > Gunnar Brand Thank you. This confirms my last findings. I will update the summary, reopen and reprioritize. This broke on 53, but the actually fallback code we are going to fix is on 54.

Mass bug change to replace various 'parity' whiteboard flags with the new canonical keywords. (See bug 1443764 comment 13.)

I don't believe this classifies as a regressions. Before implementing the fallback feature we didn't have h1 ntlm fallback at all.

I assumed it was a regression from comment 12 ("This was working in Firefox until version 53 ...").

(In reply to Marco Castelluccio [:marco] from comment #21) > I assumed it was a regression from comment 12 ("This was working in Firefox > until version 53 ..."). Aha! Hmm.. the fallback was implemented in 55, so I'm a bit puzzled, this should simply never work before 55, IMO. I never got any feedback to comment 14 asking for regressions range finding, so hard to say.