Closed Bug 1767808 Opened 3 years ago Closed 3 years ago

Crash in [@ memcpy_repmovs | NS_CopySegmentToBuffer]

Categories

(Core :: Networking: HTTP, defect)

Unspecified
All
defect

Tracking

()

VERIFIED FIXED
102 Branch
Tracking Status
firefox-esr91 --- unaffected
firefox100 --- unaffected
firefox101 --- unaffected
firefox102 + fixed

People

(Reporter: gsvelto, Unassigned)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: crash, regression)

Crash Data

Crash report: https://crash-stats.mozilla.org/report/index/d4a6c32b-2e4c-4035-839b-c19350220504

Reason: EXCEPTION_ACCESS_VIOLATION_WRITE

Top 10 frames of crashing thread:

0 vcruntime140.dll memcpy_repmovs d:\a01\_work\4\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm:40
1 xul.dll NS_CopySegmentToBuffer xpcom/io/nsStreamUtils.cpp:766
2 xul.dll static mozilla::FunctionRef<nsresult  mfbt/FunctionRef.h:180
3 xul.dll mozilla::ipc::data_pipe_detail::DataPipeBase::ProcessSegmentsInternal ipc/glue/DataPipe.cpp:365
4 xul.dll mozilla::ipc::DataPipeReceiver::ReadSegments ipc/glue/DataPipe.cpp:622
5 xul.dll NS_ReadInputStreamToBuffer netwerk/base/nsNetUtil.cpp:1650
6 xul.dll NS_ReadInputStreamToString netwerk/base/nsNetUtil.cpp:1685
7 xul.dll mozilla::net::HttpChannelParent::OnDataAvailable netwerk/protocol/http/HttpChannelParent.cpp:1369
8 xul.dll nsInputStreamPump::OnInputStreamReady netwerk/base/nsInputStreamPump.cpp:378
9 xul.dll mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/ipc/glue/DataPipe.cpp:644:25'>::Run xpcom/threads/nsThreadUtils.h:531

Ignore the volume on Thunderbird under this signature, this is a nightly/beta crash in Firefox that seems to be a regression. CC'ing a mozillian who can repro.

Adding signatures for other Windows versions an Linux/Ubuntu

Crash Signature: [@ memcpy_repmovs | NS_CopySegmentToBuffer] → [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer]

Can someone please provide the STR?
Thanks.

(In reply to Kershaw Chang [:kershaw] from comment #2)

Can someone please provide the STR?
Thanks.

STR were:

  • reboot my laptop for another unrelated reason
  • restart my firefox

It seemed to be related to me trying to access support.lenovo.com, but now I can access it without reproducing the issue (I dont repro at all now)

I hit this crash in Nightly 102 after clicking a mailto: link in Google Calendar's meeting editor, but I haven't been able to reproduce the crash again.

bp-dac4a16c-a7fa-464e-8797-fb5450220505

I also just saw this crash while I wasn't even focused in the Firefox window (so Firefox was idle from a user interaction perspective). bp-1f7f42d2-4e6c-44ca-b3e8-57f4e0220505

I'm not sure if it's related, but earlier today, the Firefox parent process hung a few seconds after startup and i had to kill the process. This happened several times, but after a few restarts, it seemed to stop. The mozilla::net::HttpChannelParent::OnDataAvailable is what made me think it might be related. Stack from WinDBG:

0:119> ~0 kp 30
 # Child-SP          RetAddr               Call Site
00 000000ce`247fe3c8 00007fff`8da8a797     ntdll!NtWaitForAlertByThreadId+0x14
01 000000ce`247fe3d0 00007fff`8b45d3f9     ntdll!RtlSleepConditionVariableSRW+0x137
*** WARNING: Unable to verify checksum for mozglue.dll
02 000000ce`247fe450 00007fff`74faefa3     KERNELBASE!SleepConditionVariableSRW+0x29
*** WARNING: Unable to verify checksum for xul.dll
03 000000ce`247fe490 00007fff`29e39ead     mozglue!mozilla::detail::ConditionVariableImpl::wait(class mozilla::detail::MutexImpl * lock = <Value unavailable error>)+0x13 [/builds/worker/checkouts/gecko/mozglue/misc/ConditionVariable_windows.cpp @ 50] 
04 (Inline Function) --------`--------     xul!mozilla::OffTheBooksCondVar::Wait(void)+0x14 [/builds/worker/workspace/obj-build/dist/include/mozilla/CondVar.h @ 58] 
05 (Inline Function) --------`--------     xul!mozilla::Monitor::Wait(void)+0x14 [/builds/worker/workspace/obj-build/dist/include/mozilla/Monitor.h @ 35] 
06 (Inline Function) --------`--------     xul!mozilla::detail::BaseMonitorAutoLock<mozilla::Monitor>::Wait(void)+0x14 [/builds/worker/workspace/obj-build/dist/include/mozilla/Monitor.h @ 134] 
07 (Inline Function) --------`--------     xul!`anonymous namespace'::BufferWriter::WriteAsync(void)+0x2dd [/builds/worker/checkouts/gecko/netwerk/base/nsNetUtil.cpp @ 1533] 
08 (Inline Function) --------`--------     xul!`anonymous namespace'::BufferWriter::Write(void)+0x3be [/builds/worker/checkouts/gecko/netwerk/base/nsNetUtil.cpp @ 1412] 
09 000000ce`247fe4c0 00007fff`29e3a0de     xul!NS_ReadInputStreamToBuffer(class nsIInputStream * aInputStream = <Value unavailable error>, void ** aDest = 0x000000ce`247fe5a8, int64 aCount = <Value unavailable error>, unsigned int64 * aWritten = 0x000000ce`247fe5b0)+0x49d [/builds/worker/checkouts/gecko/netwerk/base/nsNetUtil.cpp @ 1643] 
0a 000000ce`247fe580 00007fff`29edd619     xul!NS_ReadInputStreamToString(class nsIInputStream * aInputStream = 0x00000218`d8a54680, class nsTSubstring<char> * aDest = 0x000000ce`247fe660, int64 aCount = 0n24451, unsigned int64 * aWritten = 0x000000ce`247fe5b0)+0x8e [/builds/worker/checkouts/gecko/netwerk/base/nsNetUtil.cpp @ 1687] 
0b 000000ce`247fe5f0 00007fff`29e353d9     xul!mozilla::net::HttpChannelParent::OnDataAvailable(class nsIRequest * aRequest = <Value unavailable error>, class nsIInputStream * aInputStream = 0x00000218`d8a54680, unsigned int64 aOffset = <Value unavailable error>, unsigned int aCount = <Value unavailable error>)+0x129 [/builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelParent.cpp @ 1370] 
0c (Inline Function) --------`--------     xul!nsInputStreamPump::OnStateTransfer(void)+0x207 [/builds/worker/checkouts/gecko/netwerk/base/nsInputStreamPump.cpp @ 549] 
0d 000000ce`247fe6f0 00007fff`2bb24036     xul!nsInputStreamPump::OnInputStreamReady(class nsIAsyncInputStream * stream = <Value unavailable error>)+0x4a9 [/builds/worker/checkouts/gecko/netwerk/base/nsInputStreamPump.cpp @ 378] 
0e (Inline Function) --------`--------     xul!mozilla::ipc::DataPipeReceiver::AsyncWait::<lambda_8>::operator()(void)+0x2e [/builds/worker/checkouts/gecko/ipc/glue/DataPipe.cpp @ 648] 
0f 000000ce`247fe7a0 00007fff`2afb923f     xul!mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/ipc/glue/DataPipe.cpp:644:25'>::Run(void)+0x36 [/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h @ 532] 
10 (Inline Function) --------`--------     xul!mozilla::RunnableTask::Run(void)+0x35b [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 467] 
11 000000ce`247fe7e0 00007fff`2acb44ee     xul!mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(class mozilla::detail::BaseAutoLock<mozilla::Mutex &> * aProofOfLock = <Value unavailable error>)+0x111f [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 780] 
12 (Inline Function) --------`--------     xul!mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(class mozilla::detail::BaseAutoLock<mozilla::Mutex &> * aProofOfLock = <Value unavailable error>)+0xb [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 612] 
13 (Inline Function) --------`--------     xul!mozilla::TaskController::ProcessPendingMTTask(bool aMayWait = <Value unavailable error>)+0x17 [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 390] 
14 (Inline Function) --------`--------     xul!mozilla::TaskController::InitializeInternal::<lambda_1>::operator()(void)+0x26 [/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp @ 124] 
15 (Inline Function) --------`--------     xul!mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:124:7'>::Run(void)+0x26 [/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h @ 531] 
16 000000ce`247fef60 00007fff`2afe8fd4     xul!nsThread::ProcessNextEvent(bool aMayWait = <Value unavailable error>, bool * aResult = 0x000000ce`247ff330)+0xeae [/builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp @ 1180] 
17 (Inline Function) --------`--------     xul!NS_ProcessNextEvent(class nsIThread * aThread = <Value unavailable error>, bool aMayWait = <Value unavailable error>)+0x29 [/builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp @ 465] 
18 000000ce`247ff300 00007fff`29f592ef     xul!mozilla::ipc::MessagePump::Run(class base::MessagePump::Delegate * aDelegate = 0x00000218`cab3e2e0)+0xc4 [/builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp @ 85] 
19 (Inline Function) --------`--------     xul!MessageLoop::RunInternal(void)+0x16 [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 380] 
1a 000000ce`247ff3a0 00007fff`295a124e     xul!MessageLoop::RunHandler(void)+0x2f [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 374] 
1b 000000ce`247ff3f0 00007fff`297097c8     xul!MessageLoop::Run(void)+0x4e [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 356] 
1c 000000ce`247ff450 00007fff`2970877f     xul!nsBaseAppShell::Run(void)+0x28 [/builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp @ 139] 
1d 000000ce`247ff490 00007fff`2dc591b1     xul!nsAppShell::Run(void)+0x2f [/builds/worker/checkouts/gecko/widget/windows/nsAppShell.cpp @ 609] 
1e 000000ce`247ff600 00007fff`2dcc7046     xul!nsAppStartup::Run(void)+0x41 [/builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp @ 296] 
1f 000000ce`247ff650 00007fff`2dcc7ead     xul!XREMain::XRE_mainRun(void)+0xab6 [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5740] 
20 000000ce`247ff8f0 00007fff`2b5b1ea3     xul!XREMain::XRE_main(int argc = <Value unavailable error>, char ** argv = <Value unavailable error>, struct mozilla::BootstrapConfig * aConfig = <Value unavailable error>)+0x2ed [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5925] 
*** WARNING: Unable to verify checksum for firefox.exe
21 000000ce`247ff9b0 00007ff7`1f0eaa91     xul!XRE_main(int argc = 0n1, char ** argv = 0x00000218`cab030a0, struct mozilla::BootstrapConfig * aConfig = 0x000000ce`247ffbe0)+0x43 [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5992] 
22 (Inline Function) --------`--------     firefox!do_main(int argc = 0n1, char ** argv = <Value unavailable error>, char ** envp = <Value unavailable error>)+0xef [/builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp @ 225] 
23 (Inline Function) --------`--------     firefox!NS_internal_main(int argc = <Value unavailable error>, char ** argv = 0x00000218`cab030a0, char ** envp = <Value unavailable error>)+0x3cd [/builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp @ 395] 
24 000000ce`247ffb00 00007ff7`1f0fdb98     firefox!wmain(int argc = <Value unavailable error>, wchar_t ** argv = 0x00000218`ca879bf0)+0x661 [/builds/worker/checkouts/gecko/toolkit/xre/nsWindowsWMain.cpp @ 167] 
25 (Inline Function) --------`--------     firefox!invoke_main(void)+0x22 [d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
26 000000ce`247ffe60 00007fff`8cdc54e0     firefox!__scrt_common_main_seh(void)+0x10c [d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
27 000000ce`247ffea0 00007fff`8da2485b     kernel32!BaseThreadInitThunk+0x10
28 000000ce`247ffed0 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

I also saw a similar parent process hang on my first startup today. Didn't happen again after I force-quit and restarted Nightly.

Fallout from bug 1754004 maybe?

Flags: needinfo?(nika)

I get this every time I open Firefox on the latest Nightly while it's loading my pinned tabs. I'm not sure which tab it is yet but I suspect one of multiple Gmail or Slack tabs.

https://crash-stats.mozilla.org/report/index/bb143af9-86b4-4093-8c85-38ccd0220505
https://crash-stats.mozilla.org/report/index/f3c0514d-d18e-4b61-b3e9-abe990220505

Update: As soon as I opened a Gmail tab it crashed again: https://crash-stats.mozilla.org/report/index/f32ae1cf-840c-4c07-a4fd-e7ddd0220505

I hit this same crash with a slightly different crash signature on macOS: bp-4d54b95a-4242-4cd2-ab5c-49de40220505

Crash Signature: [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] → [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] [@ NS_CopySegmentToBuffer]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)

Fallout from bug 1754004 maybe?

Based on looking at relevant information, that seems likely. As far as I can tell there is some issue where the HTTP stream believes there is more data available in the input stream as reported by the pump compared to the amount of data actually provided by the pump, which leads to NS_ReadInputStreamToString blocking internally on a monitor waiting for some more data to arrive which will never be sent...

I was surprised that we were seeing a DataPipe here anyway, so I think it might be something to do with service workers manipulating POST data streams on network requests, and somehow bypassing upload stream normalization, which is quite unfortunate... I haven't updated to the latest nightly yet and it's very late when I noticed this, but I'll look into fixing it early tomorrow.

Given the severity, it's probably worth backing out bug 1754004 and related bugs until I can figure out what happened here, and roll out a fix, as I don't immediately know what's going on and will not be able to fix it until tomorrow afternoon at the earliest.

Blocks: clouseau

Fixed by backout of bug 1754004.

Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(nika)
Resolution: --- → FIXED
Target Milestone: --- → 102 Branch
Crash Signature: [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] [@ NS_CopySegmentToBuffer] → [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] [@ NS_CopySegmentToBuffer] [@ __memmove_avx_unaligned_erms | NS_CopySegmentToBuffer ]
Crash Signature: [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] [@ NS_CopySegmentToBuffer] [@ __memmove_avx_unaligned_erms | NS_CopySegmentToBuffer ] → [@ memcpy_repmovs | NS_CopySegmentToBuffer] [@ memcpy | NS_CopySegmentToBuffer] [@ __memcpy_avx_unaligned_erms | NS_CopySegmentToBuffer] [@ NS_CopySegmentToBuffer] [@ __memmove_avx_unaligned_erms | NS_CopySegmentToBuffer ] [@ __memcpy_ssse3 | NS_CopySe…
Flags: qe-verify+
Regressed by: 1754004
Has Regression Range: --- → yes

I could not reproduce the crash on Win10x64 using Fx build 102.0a1 (20220504234551) and steps from comments.
Can you please confirm issue is not reproducing on latest Nightly and Beta (https://archive.mozilla.org/pub/firefox/candidates/102.0b5-candidates/). Thank you.

Flags: needinfo?(gsvelto)

Re-directing, Alexandre can you still report this one?

Flags: needinfo?(gsvelto) → needinfo?(lissyx+mozillians)

It was during the time nika landed that patch that badly broke some network code, I have had no repro at all since it was backed out as mentionned in comment 11.

Flags: needinfo?(lissyx+mozillians)

Marking as verified based on comment#15.

Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.