Closed Bug 1545196 Opened 5 years ago Closed 5 years ago

Crash in [@ mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer] due to OOM copying indexedDB::ObjectStoreAddPutParams

Categories

(Core :: Storage: IndexedDB, defect, P2)

Product:
Core
Component:
Storage: IndexedDB
Platform:
x86
Windows 10
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla74
Tracking Flags:
Tracking Status
firefox-esr68 --- wontfix
firefox67 --- unaffected
firefox68 --- wontfix
firefox72 --- wontfix
firefox73 --- wontfix
firefox74 --- fixed

People

(Reporter: marcia, Assigned: sg)

References

(Regression)

Details

(Keywords: crash, regression)

Crash Data

Attachments

(2 files)

Bug 1545196 - Avoid copying large RequestParams sub-object in ObjectStoreAddOrPutRequestOp. r=#dom-workers-and-storage
(deleted), text/x-phabricator-request
Details
Bug 1545196 - Fix lifetime dependency issue in RemoteWorkerController. r=perry,#dom-workers-and-storage
(deleted), text/x-phabricator-request
Details

This bug is for crash report bp-af102580-fc42-4088-856a-769030190331.

Seen while looking at nightly crash data, crashes started in 20190329220047: https://bit.ly/2VPLBID

All crashes have MOZ_RELEASE_ASSERT(data.Append(aOther.data)) (out of memory). similar to Bug 1519123. At least 2 of the Nightly 68 reports had Ghostery, but there also appear to be reports that don't have that addon installed.

Possible regression range based on Build ID: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4e2ea1a75e878ae392e4775f2eddd9f83d1b008e&tochange=bd1e28b0143bdcff0798b0e6a4f54791c41192e8

If this is an IPC bug, Bug 1539542 was the only one in that regression range but it involved removing stuff from the exclusions list.

Top 10 frames of crashing thread:

0 xul.dll mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer ipc/glue/IPCMessageUtils.h:81
1 xul.dll mozilla::dom::IDBObjectStore::AddOrPut dom/indexedDB/IDBObjectStore.cpp:1632
2 xul.dll struct already_AddRefed<mozilla::dom::IDBRequest> mozilla::dom::IDBObjectStore::Put dom/indexedDB/IDBObjectStore.h:179
3 xul.dll static bool mozilla::dom::IDBObjectStore_Binding::put dom/bindings/IDBObjectStoreBinding.cpp:478
4 xul.dll mozilla::dom::binding_detail::GenericMethod<mozilla::dom::binding_detail::NormalThisPolicy, mozilla::dom::binding_detail::ThrowExceptions> dom/bindings/BindingUtils.cpp:3150
5 xul.dll js::InternalCallOrConstruct js/src/vm/Interpreter.cpp:534
6 xul.dll static bool InternalCall js/src/vm/Interpreter.cpp:589
7 xul.dll js::SpreadCallOperation js/src/vm/Interpreter.cpp:5088
8 xul.dll static bool Interpret js/src/vm/Interpreter.cpp:3008
9 xul.dll js::RunScript js/src/vm/Interpreter.cpp:422

This looks like the kind of OOM we were discussing today in IPC bug triage.

See Also: → https://bugzilla.mozilla.org/show_bug.cgi?id=1539498

I added this OOM check in bug 1539261. Before that patch we'd probably have just failed in some other more horrible way.

Regressed by: 1539261

Setting fix-optional for 68 based on comment 2 and low volume.

status-firefox68: affected → fix-optional

The priority flag is not set for this bug.
:jld, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(jld)

These all seem to be caused by IndexedDB invoking the copy constructor; it would need to be changed to std::move any structures containing SerializedStructuredCloneBuffer.

Blocks: 1539498
Component: IPC → DOM: IndexedDB
Flags: needinfo?(jld)
See Also: https://bugzilla.mozilla.org/show_bug.cgi?id=1539498 →
Blocks: 1541370
Priority: -- → P2

there are also a couple of other crash signatures new in 68 with the same MOZ_CRASH Reason:

  • [@ static class mozilla::dom::indexedDB::ObjectStoreAddPutParams& const mozilla::dom::indexedDB::ObjectStoreAddPutParams::operator=]
  • [@ mozilla::SerializedStructuredCloneBuffer::operator=]
  • [@ mozilla::dom::indexedDB::ObjectStoreAddPutParams::operator=]

not sure if i should add those here or open new bugs for them...

(In reply to [:philipp] from comment #6)

there are also a couple of other crash signatures new in 68 with the same MOZ_CRASH Reason:

  • [@ static class mozilla::dom::indexedDB::ObjectStoreAddPutParams& const mozilla::dom::indexedDB::ObjectStoreAddPutParams::operator=]
  • [@ mozilla::SerializedStructuredCloneBuffer::operator=]
  • [@ mozilla::dom::indexedDB::ObjectStoreAddPutParams::operator=]

not sure if i should add those here or open new bugs for them...

The crashes all seem to be related to indexedDB::ObjectStoreAddPutParams copy construction/assignment, including the ones with the too-general SerializedStructuredCloneBuffer::operator= signature, so they belong to this bug.

Crash Signature: [@ mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer] → [@ mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer] [@ static class mozilla::dom::indexedDB::ObjectStoreAddPutParams& const mozilla::dom::indexedDB::ObjectStoreAddPutParams::operator=] [@ mozilla::SerializedStructuredCloneBuffe…
Summary: Crash in [@ mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer] → Crash in [@ mozilla::SerializedStructuredCloneBuffer::SerializedStructuredCloneBuffer] due to OOM copying indexedDB::ObjectStoreAddPutParams

Is this just a random OOM, that happens to fail here often because we are cloning potentially big objects?

Flags: needinfo?(sgiesecke)
Flags: needinfo?(jvarga)

Yeah, looks like a big object. This can be mitigated in future when we change the implementation to compress data in the child before it's sent to the parent.

Flags: needinfo?(jvarga)
Assignee: nobody → sgiesecke
Status: NEW → ASSIGNED
Flags: needinfo?(sgiesecke)
Pushed by sgiesecke@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7014228a3cca Avoid copying large RequestParams sub-object in ObjectStoreAddOrPutRequestOp. r=dom-workers-and-storage-reviewers,janv

Backed out changeset 7014228a3cca (Bug 1545196) for causing build bustages at ActorsParent.cpp

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=7014228a3ccabee5d2c3e072ed4b36651844f210

Log failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=286138334&repo=autoland&lineNumber=40246

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=22c3548439448650935700df5a7b813e2f537f3f

[task 2020-01-23T13:02:55.611Z] 13:02:55     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB'
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -  /builds/worker/fetches/clang/bin/clang++ -std=gnu++17 -o ActorsParent.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/indexedDB -I/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/storage -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/third_party/sqlite3/src -I/builds/worker/workspace/build/src/xpcom/build -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O2 -fno-omit-frame-pointer -funwind-tables -Werror -Wno-error=shadow  -MD -MP -MF .deps/ActorsParent.o.pp   /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp
[task 2020-01-23T13:02:55.614Z] 13:02:55    ERROR -  /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:14961:23: error: binding value of type 'const mozilla::dom::indexedDB::PBackgroundIDBTransactionParent::RequestParams' (aka 'const mozilla::dom::indexedDB::RequestParams') to reference to type 'mozilla::dom::indexedDB::RequestParams' drops 'const' qualifier
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -    return AllocRequest(aParams, IsSameProcessActor());
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -                        ^~~~~~~
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -  /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:14661:21: note: passing argument to parameter 'aParams' here
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -      RequestParams&& aParams, bool aTrustParams) {
[task 2020-01-23T13:02:55.614Z] 13:02:55     INFO -                      ^
[task 2020-01-23T13:02:55.615Z] 13:02:55    ERROR -  /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:15594:23: error: binding value of type 'const mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent::RequestParams' (aka 'const mozilla::dom::indexedDB::RequestParams') to reference to type 'mozilla::dom::indexedDB::RequestParams' drops 'const' qualifier
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -    return AllocRequest(aParams, IsSameProcessActor());
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -                        ^~~~~~~
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -  /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:14661:21: note: passing argument to parameter 'aParams' here
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -      RequestParams&& aParams, bool aTrustParams) {
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -                      ^
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -  2 errors generated.
[task 2020-01-23T13:02:55.615Z] 13:02:55     INFO -  /builds/worker/workspace/build/src/config/rules.mk:744: recipe for target 'ActorsParent.o' failed
[task 2020-01-23T13:02:55.615Z] 13:02:55    ERROR -  make[4]: *** [ActorsParent.o] Error 1
[task 2020-01-23T13:02:55.616Z] 13:02:55     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB'
[task 2020-01-23T13:02:55.616Z] 13:02:55     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2020-01-23T13:02:55.618Z] 13:02:55     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/base'
[task 2020-01-23T13:02:55.619Z] 13:02:55     INFO -  dom/plugins/base/Unified_cpp_dom_plugins_base0.o
[task 2020-01-23T13:02:55.619Z] 13:02:55     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/base'
[task 2020-01-23T13:02:55.836Z] 13:02:55     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB/test/gtest'
[task 2020-01-23T13:02:55.838Z] 13:02:55     INFO -  /builds/worker/fetches/clang/bin/clang++ -std=gnu++17 -o Unified_cpp_test_gtest0.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/indexedDB/test/gtest -I/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB/test/gtest -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/indexedDB -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O2 -fno-omit-frame-pointer -funwind-tables -Werror  -MD -MP -MF .deps/Unified_cpp_test_gtest0.o.pp   Unified_cpp_test_gtest0.cpp
[task 2020-01-23T13:02:55.839Z] 13:02:55     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/indexedDB/test/gtest'
[task 2020-01-23T13:02:55.840Z] 13:02:55     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/workers/sharedworkers'
[task 2020-01-23T13:02:55.840Z] 13:02:55     INFO -  mkdir -p '.deps/'
Flags: needinfo?(sgiesecke)

Sorry, some required changes were apparently part of another patch stacked upon it, so I didn't notice that when building locally, and the bot attached to Phabricator wasn't working at the time I submitted it. Fixed it now and will reland.

Flags: needinfo?(sgiesecke)
Pushed by sgiesecke@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b404fd242e7f Avoid copying large RequestParams sub-object in ObjectStoreAddOrPutRequestOp. r=dom-workers-and-storage-reviewers,janv
Pushed by sgiesecke@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/15e8d9547568 Fix lifetime dependency issue in RemoteWorkerController. r=baku https://hg.mozilla.org/integration/autoland/rev/ccc2f1aad57a Avoid copying large RequestParams sub-object in ObjectStoreAddOrPutRequestOp. r=dom-workers-and-storage-reviewers,janv

Backed out 3 changesets (bug 1539498, bug 1545196) for build bustages failures in DOMTypes.h

/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/DOMTypes.h
/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ClientIPCTypes.h
/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h

Push that started the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=287169576&resultStatus=superseded%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=0f906da3634aaebe1267d030e31facb82feb2cfc

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287169576&repo=autoland&lineNumber=22644

Backout: https://hg.mozilla.org/integration/autoland/rev/3f81d87f402e1ffe15f31548b4ac03d01a66fec0

(In reply to Oana Pop-Rus from comment #18)

Backed out 3 changesets (bug 1539498, bug 1545196) for build bustages failures in DOMTypes.h

This is affecting gcc builds only. I have a fix for this.

Flags: needinfo?(sgiesecke)
Pushed by sgiesecke@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/62b6da6c3d53 Fix lifetime dependency issue in RemoteWorkerController. r=baku https://hg.mozilla.org/integration/autoland/rev/4c14eecf1683 Avoid copying large RequestParams sub-object in ObjectStoreAddOrPutRequestOp. r=dom-workers-and-storage-reviewers,janv

https://hg.mozilla.org/mozilla-central/rev/62b6da6c3d53
https://hg.mozilla.org/mozilla-central/rev/4c14eecf1683

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox74: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla74
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: