Open Bug 1848013 Opened 1 year ago Updated 1 year ago

HTTP/2 WebSocket CONNECT missing headers when proxied (RFC8441)

Categories

(Core :: Networking: WebSockets, defect, P2)

Firefox 116
defect

Tracking

()

People

(Reporter: christopher.staite, Unassigned)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [necko-triaged][necko-priority-next])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36

Steps to reproduce:

Set up a transparent proxy (e.g. Squid).
Set the proxy in Firefox.
Attempt to connect to a HTTP/2 WebSocket.

Actual results:

Note a 426 response because Sec-WebSocket-Version is not presented.

Notably, if you force the Sec-WebSocket-Version on the server then it causes Firefox to crash reliably.

The headers are shown in the devtools network tab, however the headers on the wire are shown in WireShark as:

    [Header Length: 365]
    [Header Count: 6]
    Header: :method: CONNECT
    Header: :path: /websocket
    Header: :authority: host
    Header: :scheme: https
    Header: :protocol: websocket
    Header: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/116.0

Expected results:

A 101 response and the WebSocket functions (as occurs without the proxy).

Captured headers when not proxied:
[Header Length: 1077]
[Header Count: 18]
Header: :method: CONNECT
Header: :path: /websocket
Header: :authority: host
Header: :scheme: https
Header: :protocol: websocket
Header: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/116.0
Header: accept: /
Header: accept-language: en-GB,en-US;q=0.7,en;q=0.3
Header: accept-encoding: gzip, deflate, br
Header: sec-websocket-version: 13
Header: origin: ORIGIN
Header: sec-websocket-extensions: permessage-deflate
Header: cookie: COOKIES
Header: sec-fetch-dest: empty
Header: sec-fetch-mode: websocket
Header: sec-fetch-site: cross-site
Header: pragma: no-cache
Header: cache-control: no-cache

The Bugbug bot thinks this bug should belong to the 'Core::Networking: HTTP' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Networking: HTTP
Product: Firefox → Core

Hi Christopher, could you provide more info regarding your setup?
We'd appreciate a public proxy to test with or a list of steps necessary for the setup.

Notably, if you force the Sec-WebSocket-Version on the server then it causes Firefox to crash reliably.

Could you also explain what you mean by "forcing the header" ?
A link to the crash would be quite useful too - you can find the crash reports in about:crashes.

Thanks!

Component: Networking: HTTP → Networking: WebSockets
Flags: needinfo?(christopher.staite)

Hi, thanks for the info on how to find the crash report, it's available here: https://crash-stats.mozilla.org/report/index/8949f2f9-a6c0-4c62-aa4f-d98120230809

The setup that I was testing on was:
Squid -> HAProxy -> Python Tornado

I have created a public test server that's able to reproduce the issue that is locked down to access the single domain only.

Steps to reproduce:
Set HTTP/HTTPS proxy to h2.menlotest.com port 3128
Navigate to https://libwebsockets.org/testserver/
Note that the connection to the websocket fails.

This does not force the headers at the server, so there is no crash, it simply fails to start the websocket.

NOTE: Setting network.http.http2.websockets = false shows the function working.

Flags: needinfo?(christopher.staite)

I realised I failed to explain what I meant by force the headers. Since the Sec-WebSocket-Version was missing, the server was dropping the WebSocket CONNECT with a 426 due to the missing header.

I added the header (Sec-WebSocket-Version) manually in a re-write rule in HAProxy such that the WebSocket was started (i.e. the CONNECT succeeded without any headers).

That is what caused the reliable crash.

I followed your instructions in comment 3 and I was able to reproduce the failed connection. Can you configure the server to cause the crash?

I have created a server so you can reproduce the crash, however I believe this is a separate issue and should have a separate bug report.

Steps to reproduce crash:
Set HTTP/HTTPS proxy to h2.menlotest.com port 3128
Navigate to https://h2.menlotest.com/websocket-test/
Note that Firefox crashes immediately

Interestingly, if the page is loaded on startup it doesn't crash, only if you navigate to it.

Thanks. Let's use bug 1667047 for the crash, and we'll use this one for the WS connection failure.

Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P2
Whiteboard: [necko-triaged][necko-priority-new]
Summary: RFC8441 broken when WebSocket is proxied → HTTP/2 WebSocket CONNECT missing headers when proxied (RFC8441)

I wonder, is it worth disabling network.http.http2.websockets when a proxy is in use until this is fixed? Perhaps that's an easy win for now?

I was just looking through 1728734 and I've discovered that this might be a dupe of 1800533?

Flags: needinfo?(edgul)

That does look likely, but we don't have a lot of information on Bug 1800533 as far as I know.

Flags: needinfo?(edgul)
Whiteboard: [necko-triaged][necko-priority-new] → [necko-triaged][necko-priority-next]
You need to log in before you can comment on or make changes to this bug.