Closed Bug 1440479 Opened 7 years ago Closed 4 years ago

Duplicate GETs and POSTs when interoperating with Windows 2016 IIS 10.0 + SSL + HTTP/2 + "Accept Client Certificates" set

Categories

(Core :: Networking, defect, P3)

57 Branch
Unspecified
All
defect

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox58 --- affected
firefox59 --- affected
firefox60 --- affected
firefox80 --- affected
firefox81 --- affected
firefox82 --- affected
firefox83 --- affected

People

(Reporter: lbukys, Unassigned)

References

Details

(Whiteboard: [necko-triaged])

Attachments

(1 file)

Attached file demo1.aspx (deleted) —
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 Build ID: 20171226003912 Steps to reproduce: To reproduce: Serve up a page from Windows 2016 IIS 10.0 with SSL, with HTTP/2 enabled (the default), and this setting: SSL Settings > Client Certificates > Accept Example already running at https://52.15.117.97:666/FirefoxBug/demo1.aspx Actual results: Symptom observable on server: In the server logs: All GETs and POSTs are doubled. The first one has an sc-win32-status of 64 indicating that the server's response failed with a network error. The second one succeeds in sending the HTTP response back to the client. Symptom observable on client: Client network log shows only one request (duplication is happening either down in the HTTP/2 state machinery, or in the IIS HTTP/2 machinery). Meanwhile, here is a piece of server-side code that make the problem observable on the client: it increments a counter on every GET or POST, which allows the client to see whether the count goes up by one or by two on every click. There's a copy of this running at https://52.15.117.97:666/FirefoxBug/demo1.aspx -- clicks on the GET link or the POST button should increment the counter by 1. Increments by 2 indicate double GETs and POSTs. Source code is included below. Firefox (57 and higher) is the only browser demonstrating this problem. My recent testing has been with 58.0.2 on Windows, and 57.0.4 on Mac. If you get the "which client cert" dialog, cancel out without selecting one. The correlation with server's "SSL Settings > Client Certificates > Accept" is (I think) new information. Previous discussions circa August 2017 elsewhere identified the problem and settled on the only known workaround being turning off HTTP/2 in IIS (registry changes in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters), or turning off HTTP/2 in Firefox (network.http.spdy.enabled.http2 false) -- both solutions draconian and unpalatable. See https://forums.iis.net/t/1236747.aspx?Firefox+double+loading+SSL+pages+I+turn+off+SSL+and+it+works+fine and https://groups.google.com/forum/#!topic/mozilla.support.firefox/MRo1snUdoWo Expected results: Each GET or POST from the client only results in one GET or POST on the server.
More information: I decided to try a curl client against the IIS server, for variety: curl -v -k -b cookies.txt -c cookies.txt --http2 https://52.15.117.97:666/FirefoxBug/demo1.aspx interestingly, what I observed was: curl: (92) HTTP/2 stream 1 was not closed cleanly: HTTP_1_1_REQUIRED (err 13). By switching between --http2 and --http1.1, I found that the http/2 GET didn't succeed in returning a response, but the server-side script did increment the session variable. So the backoff from HTTP/2 plus another request sent with HTTP/1.1 could explain the multiple GET/POST. This bears some resemblance to Bug 1411193 "NTLM/Negotiate HTTP/2 -> HTTP/1 fallback re-sends the initial request, other browsers send Type 1 auth request header" Also resembles Chromium case https://bugs.chromium.org/p/chromium/issues/detail?id=431306 "Implement HTTP/2 error code HTTP_1_1_REQUIRED." about which there seems to have been some disagreement about correct implementation.
I can confirm this is infact sending duplicate, atleast 2 get/post requests to the webserver.
on windows 7 professional x86 on firefox 58.0
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Unspecified → Windows
57.0.4 on Mac also exhibits the problem.
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 Firefox: 60.0a1, Build ID: 20180301024724 I have managed to reproduce this issue on Windows 10 x64, Ubuntu 14.04 x64 and Mac 10.12 with the latest Nightly (60.0a1) installed. I've opened the browser, navigated to the example page from the description and values are duplicated each time the GET link or the POST button are pressed. Also using the "Mozregression" tool I have managed to find a regression window: Last good revision: ced1402861b8 (2014-11-26) First bad revision: cef590a6f946 (2014-11-27) Pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=ced1402861b8&tochange=cef590a6f946 Considering the above I will assign a component for this issue. Feel free to change it if is not the correct one.
Component: Untriaged → Security
OS: Windows → All
Product: Firefox → Core
Component: Security → Networking
Looks like this started with h2 basically, which seems reasonable since client certs aren't functional over h2. Any further insights, Nick?
Flags: needinfo?(hurley)
Priority: -- → P3
Whiteboard: [necko-triaged]

On this page: https://support.mozilla.org/en-US/questions/1290085 the issue is also reported and "resolved" by changing settings in IIS such that HTTP/2 is no longer used.

I am experiencing the same issue on a shared hosting service where disabling HTTP/2 is a "no go", so I would like to have the issue actually fixed. I see the same symptoms as reported on the mentioned page and in this bug and was able to check the following:

  • Using other browsers than Firefox (i.e. Edge, Chrome, Safari), the issue cannot be reproduced
  • When making the request via HTTP (non-SSL) with Firefox, the issue cannot be reproduced
  • Using different certificate suppliers (used Let's Encrypt and Sectigo) on the server does not change the behaviour: issue can be reproduced in both cases

Thus, it means that it is really a Firefox issue, when using HTTP/2 via HTTPS.

I am using the latest stable version of Firefox (version 80) on Windows 10

Currently, with the latest official release, version 83, on Windows 10 I have tested and narrowed it down to some certain cases.

I have set up a small test web site on IIS 10.0, using Perl scripts and using the Perl CGI.pm module to create the reponse. The script adds a record to a simple database every time it is executed. It will then report in the response (in simple text) how many times it was executed from the same IP address within the last 5 seconds.

Now I have found the following, using this setup:

  • When the script is named nph-<something>.pl and CGI.pm is instructed to generated headers according to the Non-parsed headers CGI definition, AND the script is executed via HTTPS, the script is executed twice
  • When the script is named <something>.pl (without the "nph-") and CGI.pm is instructed to generate headers according to the Non-parsed headers CGI definition, AND the script is executed via HTTPS, the script is executed once.
  • When the script is executed via HTTP (insecure), the script is executed once.

Having said this, the double execution only happens with Firefox (only tested on Windows), but not with Edge, Chrome or Safari (on iOS). Other browsers I did not test.

The strange thing is that in the Network-tab of the Developer Tools, only one request is listed (the original one) and (in my case) only one entry is logged in the IIS server logs. This makes me wonder what is happening here. I do not have access to the system IIS logs (I am on a hosting provider with limited access/configuration possibilities), so I do not know what is happening under the hood why the script is executed twice. On the other hand, since it only happens with Firefox, there seems to be something in the Firefox request that makes this happen.

The test scripts can be accessed via the following URLs (until further notice):

  • http(s)://ff.famversteeg.nl/nph (for the script that starts with "nph-" and is executed twice when accessed via HTTPS)
  • http(s)://ff.famversteeg.nl/non-nph (for the script that does not start with "nph-" and is executed once)

How can I support in being able to debug/solve this issue? There seems to be a work-around (renaming all my scripts), but I prefer not to do that for one single browser as that requires a lot of additional testing.

Flags: needinfo?(jstutte)

Hi Marcel, I'd suggest to try to record the network traffic with wireshark or similar. This allows you to see:

  • if there are really two requests
  • if there are any differences in the requests sent dependending on the naming and/or the browser used
  • if there are any differences in the responses sent from IIS depending on the naming and/or the browser used
    There is a never clarified suspect in bug 1411193 that this might be more a server side configuration problem and an in-depth analysis of the traffic might help to understand this.
    Thank you for your support
Flags: needinfo?(jstutte) → needinfo?(mozilla)

Hi Jens,

Using Wireshark I saw the following happen when requesting the script that starts with "nph-" in Firefox:

  1. There is a TLS handshake between the browser and the server
  2. Then Firefox does a HTTP2 request to the server
  3. Then IIS sends back a HTTP2 response saying: "Error: HTTP_1_1_REQUIRED (13)"
  4. Then Firefox does a HTTP/1.1 request to the server (which is not shown in the Network tab of the developer tools)
  5. The correct response from the script is sent back.
    This happens for every HTTPS request done to this script.
    I did test again with Microsoft Edge and found out that when I completely clear its cache and restart the browser and request the NPH script, Edge does the same thing. The difference is that Edge remembers it should use HTTP/1.1 for the following requests and does not do a HTTP2 request anymore.

When I make a trace in Wireshark of the request to the script that does not start with "nph-" the the following happens:

  1. There is a TLS handshake between the browser and the server
  2. Then Firefox does a HTTP2 request to the server
  3. Then IIS sends the correct response

The strange thing is, that in both situations CGI.pm is instructed to do NPH, but for some reason this is ignored by IIS because the script does not start with "nph-".

So my conclusion is as follows:

  1. It would be nice if Firefox remembers that it should not retry a HTTP2 request once a HTTP_1_1_REQUIRED response is received for a specific server/URL
  2. I need to find a way to handle HTTP2 correctly in my scripts.

Thank you for the hint of using Wireshark for tracing.

Flags: needinfo?(mozilla)

The underlying problem/unexpected behavior seems to have its root cause on the server side. Nevertheless I filed bug 1679494 to check whether we should cache this particular error response.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(u408661)
Resolution: --- → WORKSFORME

(In reply to Marcel V from comment #8)

The test scripts can be accessed via the following URLs (until further notice):

  • http(s)://ff.famversteeg.nl/nph (for the script that starts with "nph-" and is executed twice when accessed via HTTPS)
  • http(s)://ff.famversteeg.nl/non-nph (for the script that does not start with "nph-" and is executed once)

This is just to let people know that I have taken down these test scripts

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: