Closed Bug 1770742 Opened 2 years ago Closed 2 years ago

Hard refresh breaks the network connection

Categories

(Core :: Networking, defect, P2)

Firefox 102
defect

Tracking

()

VERIFIED FIXED
103 Branch
Tracking Status
firefox103 --- fixed

People

(Reporter: github, Assigned: dragana, NeedInfo)

Details

(Whiteboard: [necko-triaged])

Attachments

(3 files)

Attached file devtools.zip (deleted) —

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

  • Navigate to google.com (and confirm that the page is working)
  • Hit Ctrl+F5 (or Ctrl+Shift+R)
  • Try using the search box or clicking the links on the page (e.g. "About" in the top left)

Actual results:

The page appears broken. The search bar doesn't work. Most of the links don't work.

Some observations:

  • Looking at the network tab in the developer tools, the request doesn't seem to complete at all

  • Wireshark shows a TCP reset (see the attached screenshots)

  • Navigating to a different page works. Navigating back to google.com after does not work

  • Closing and reopening the browser fixes the issue until the next hard refresh

  • Opening a new tab in a different container works as well

  • The same issue is present when loading libraries on a different page that are included from Googles CDN (ajax.googleapis.com)

  • setting "security.tls.version.max" to "3" seems to resolve the issue

I'm using Firefox Nightly (2022-05-22) on Windows.

Expected results:

The request should complete and clicking on the links should navigate to the linked pages.

The Bugbug bot thinks this bug should belong to the 'DevTools::Netmonitor' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Netmonitor
Product: Firefox → DevTools
Component: Netmonitor → Networking
Product: DevTools → Core

I don't think this has anything to do with DevTools, but is more of a networking issue. I'll just put it here, but move it as you see fit.

Could you try to capture a http log for this?
Thanks.

Flags: needinfo?(github)

HTTP logging being turned on seems to prevent the issue from occurring.
I was able to reproduce it immediately after hitting "Stop Logging".

I kept logging turned off, recreated the issue, turned logging on and then tried to use the page.

Flags: needinfo?(github)

Some additional observations:

This would explain, why the issue isn't present if TLS1.2 is forced (since HTTP/3 requires TLS1.3).
Enabling HTTP logging apparently also causes Firefox to use HTTP/2 on a hard refresh(?), explaining why I'm unable to reproduce it with logging turned on.

  • Opening Google and cloudflare-quic.com in two different tabs and then hard refreshing one of them also breaks the other tab.

  • I have been able to reproduce the issue on other company-provided machines, but not on my own, personal one.

All of the companies machines use Sophos. Other people seem to be reporting similar issues with Sophos and Google-owned websites on Firefox [1]. This might very well be the same issue. I've checked with our IT department if Sophos can be turned off for further testing and will report back, if I get a response.

[1] https://community.sophos.com/intercept-x-endpoint/early-access-program/f/feedback-and-issues/134147/mozilla-firefox-trouble-with-google-gmail-web-based-access

Sophos is the root cause of this, but I will try to make Firefox more resilient to the issue.
Can you please help me narrow down the problem:

  • please go in about:config and change network.http.http3.enable_0rtt to false. Does this solve the problem?
Flags: needinfo?(github)

Changing network.http.http3.enable_0rtt to false did not help.
Changing security.tls.enable_0rtt_data to false did help.

Flags: needinfo?(github)

Thanks.

The log you have attach also gave me more clues. What is happening:

  • Firefox use 0RTT and it gets SSL_ERROR_BAD_MAC_ALERT (this is an issue at Sophos)
  • We already have an retry mechanism for this error due to another firewall thatwas doing a wrong thing with 0RTT. For that firewall the retry was the fix, but here it is not.
  • The request will be retries without 0RTT but H2 connection will use 0RTT. The second reuse of 0RTT returnd NET_RESET error probably from Sophos. This error will not be retry.

The solution is to disable 0RTT completely not only for a transaction.

I will also add a mechaism to disabe 0RTT competely if multiple such errors happen in a row.

Assignee: nobody → dd.mozilla
Severity: -- → S3
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Priority: -- → P2
Whiteboard: [necko-triaged]

Also, the fact that we use 0RTT for Http2 session on retry is a bug in the retry code.

Pushed by ddamjanovic@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3b17ed20e45f Disable 0RTT properly for SSL_ERROR_BAD_MAC_ALERT error r=necko-reviewers,kershaw
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 103 Branch

I also discussed the problem on Sophos forum: https://community.sophos.com/intercept-x-endpoint/f/discussions/134116/firefox-especially-gmail-cannot-complete-some-requests-no-responses-are-returned-zero-bytes-assume-that-because-of-endpoint-agent

Disabling of their "Intercept X Endpoint" product resolves the issue.

We opened support request and they promised to fix it within few months.

As one recommended "security.tls.enable_0rtt_data" fixes Firefox. I suffered for months without any solution.

Flags: qe-verify+

I tried to reproduce this issue on a 2022-05-22 Nightly build on Windows 10 but I was unsuccessful. Would you be so kind as to verify the fix on the latest beta/nightly channel?
Thank you.

Flags: needinfo?(dd.mozilla)

(In reply to Ardelean Oana from comment #14)

I tried to reproduce this issue on a 2022-05-22 Nightly build on Windows 10 but I was unsuccessful. Would you be so kind as to verify the fix on the latest beta/nightly channel?
Thank you.

I was not able to reproduce the issue again.

Thanks for confirming! Marking the bug as VERIFIED and removing the qa-verify+ flag based on Comment 15.

Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: