It was green when bug 1373562 landed (~3 weeks ago). https://treeherder.mozilla.org/#/jobs?repo=try&revision=752e75d36013a5cf9d0b1915305d6e2b084122b8&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=131235463 09:23:26 INFO - ================================== FAILURES =================================== 09:23:26 INFO - ________________________ XPCShellTestsTests.testChild _________________________ 09:23:26 INFO - self = <selftest.XPCShellTestsTests testMethod=testChild> 09:23:26 INFO - def testChild(self): 09:23:26 INFO - """ 09:23:26 INFO - Checks that calling do_load_child_test_harness without run_test_in_child 09:23:26 INFO - results in a usable test state. This test has a spurious failure when 09:23:26 INFO - run using |mach python-test|. See bug 1103226. 09:23:26 INFO - """ 09:23:26 INFO - self.writeFile("test_child_assertions.js", CHILD_HARNESS_SIMPLE) 09:23:26 INFO - self.writeManifest(["test_child_assertions.js"]) 09:23:26 INFO - > self.assertTestResult(True) 09:23:26 INFO - ..\testing\xpcshell\selftest.py:667: 09:23:26 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 09:23:26 INFO - ..\testing\xpcshell\selftest.py:524: in assertTestResult 09:23:26 INFO - """ % ("passed" if expected else "failed", self.log.getvalue())) 09:23:26 INFO - E AssertionError: Tests should have passed, log: 09:23:26 INFO - E ======== 09:23:26 INFO - E runxpcshelltests.py | using symbolizer at z:\build\build\src\obj-firefox\dist\bin\llvm-symbolizer.exe 09:23:26 INFO - E MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail. 09:23:26 INFO - E Running tests sequentially. 09:23:26 INFO - E SUITE-START | Running 1 tests 09:23:26 INFO - E TEST-START | test_child_assertions.js 09:23:26 WARNING - E TEST-UNEXPECTED-FAIL | test_child_assertions.js | xpcshell return code: -1073740940 09:23:26 INFO - E ======== 09:23:26 INFO - _____________________ XPCShellTestsTests.testChildMozinfo _____________________ 09:23:26 INFO - self = <selftest.XPCShellTestsTests testMethod=testChildMozinfo> 09:23:26 INFO - def testChildMozinfo(self): 09:23:26 INFO - """ 09:23:26 INFO - Check that mozinfo.json is loaded in child process 09:23:26 INFO - """ 09:23:26 INFO - self.writeFile("test_mozinfo.js", LOAD_MOZINFO) 09:23:26 INFO - self.writeFile("test_child_mozinfo.js", CHILD_MOZINFO) 09:23:26 INFO - self.writeManifest(["test_child_mozinfo.js"]) 09:23:26 INFO - > self.assertTestResult(True) 09:23:26 INFO - ..\testing\xpcshell\selftest.py:1362: 09:23:26 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 09:23:26 INFO - ..\testing\xpcshell\selftest.py:524: in assertTestResult 09:23:26 INFO - """ % ("passed" if expected else "failed", self.log.getvalue())) 09:23:26 INFO - E AssertionError: Tests should have passed, log: 09:23:26 INFO - E ======== 09:23:26 INFO - E runxpcshelltests.py | using symbolizer at z:\build\build\src\obj-firefox\dist\bin\llvm-symbolizer.exe 09:23:26 INFO - E MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail. 09:23:26 INFO - E Running tests sequentially. 09:23:26 INFO - E SUITE-START | Running 1 tests 09:23:26 INFO - E TEST-START | test_child_mozinfo.js 09:23:26 WARNING - E TEST-UNEXPECTED-FAIL | test_child_mozinfo.js | xpcshell return code: -1073740940 09:23:26 INFO - E ======== 09:23:26 INFO - ______________________ XPCShellTestsTests.testChildPass _______________________ 09:23:26 INFO - self = <selftest.XPCShellTestsTests testMethod=testChildPass> 09:23:26 INFO - def testChildPass(self): 09:23:26 INFO - """ 09:23:26 INFO - Check that a simple test running in a child process passes. 09:23:26 INFO - """ 09:23:26 INFO - self.writeFile("test_pass.js", SIMPLE_PASSING_TEST) 09:23:26 INFO - self.writeFile("test_child_pass.js", CHILD_TEST_PASSING) 09:23:26 INFO - self.writeManifest(["test_child_pass.js"]) 09:23:26 INFO - > self.assertTestResult(True, verbose=True) 09:23:26 INFO - ..\testing\xpcshell\selftest.py:611: 09:23:26 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 09:23:26 INFO - ..\testing\xpcshell\selftest.py:524: in assertTestResult 09:23:26 INFO - """ % ("passed" if expected else "failed", self.log.getvalue())) 09:23:26 INFO - E AssertionError: Tests should have passed, log: 09:23:26 INFO - E ======== 09:23:26 INFO - E runxpcshelltests.py | using symbolizer at z:\build\build\src\obj-firefox\dist\bin\llvm-symbolizer.exe 09:23:26 INFO - E MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail. 09:23:26 INFO - E Running tests sequentially. 09:23:26 INFO - E SUITE-START | Running 1 tests 09:23:26 INFO - E TEST-START | test_child_pass.js 09:23:26 INFO - E test_child_pass.js | full command: ['z:\\build\\build\\src\\obj-firefox\\dist\\bin\\xpcshell.exe', '-g', 'z:\\build\\build\\src\\obj-firefox\\dist\\bin', '-a', 'z:\\build\\build\\src\\obj-firefox\\dist\\bin', '-r', 'z:/build/build/src/obj-firefox/dist/bin/components/httpd.manifest', '-m', '-s', '-e', 'const _HEAD_JS_PATH = "z:/build/build/src/testing/xpcshell/head.js";', '-e', 'const _MOZINFO_JS_PATH = "c:\\\\users\\\\task_1505463087\\\\appdata\\\\local\\\\temp\\\\xpc-profile-rbrxdi\\\\mozinfo.json";', '-e', 'const _TESTING_MODULES_DIR = "z:\\\\build\\\\build\\\\src\\\\obj-firefox\\\\_tests\\\\modules\\\\";', '-f', 'z:\\build\\build\\src\\testing\\xpcshell\\head.js', '-e', 'const _SERVER_ADDR = "localhost"', '-e', 'const _HEAD_FILES = [];', '-e', 'const _JSDEBUGGER_PORT = 0;', '-e', 'const _TEST_FILE = ["c:/users/task_1505463087/appdata/local/temp/tmpgy7zlm/test_child_pass.js"];', '-e', 'const _TEST_NAME = "test_child_pass.js"', '-e', '_execute_test(); quit(0);'] 09:23:26 INFO - E test_child_pass.js | current directory: 'c:\\users\\task_1505463087\\appdata\\local\\temp\\tmpgy7zlm' 09:23:26 INFO - E test_child_pass.js | environment: ['MOZ_DEVELOPER_REPO_DIR=z:/build/build/src', 'XPCOM_DEBUG_BREAK=stack-and-abort', 'XPCSHELL_TEST_TEMP_DIR=c:\\users\\task_1505463087\\appdata\\local\\temp\\xpc-other-namgwo', 'MOZ_CRASHREPORTER=1', 'XPCSHELL_TEST_PROFILE_DIR=c:\\users\\task_1505463087\\appdata\\local\\temp\\xpc-profile-rbrxdi', 'MOZ_DISABLE_CONTENT_SANDBOX=1', 'ASAN_SYMBOLIZER_PATH=z:\\build\\build\\src\\obj-firefox\\dist\\bin\\llvm-symbolizer.exe', 'MOZ_DISABLE_NONLOCAL_CONNECTIONS=1', 'PATH=z:\\build\\build\\src\\obj-firefox\\_virtualenv\\Scripts;z:\\build\\build\\src\\clang\\bin;z:\\build\\build\\src\\vs2015u3\\VC\\bin\\amd64;z:\\build\\build\\src\\vs2015u3\\VC\\bin;z:\\build\\build\\src\\vs2015u3\\SDK\\bin\\x64;z:\\build\\build\\src\\vs2015u3\\VC\\redist\\x64\\Microsoft.VC140.CRT;z:\\build\\build\\src\\vs2015u3\\SDK\\Redist\\ucrt\\DLLs\\x64;z:\\build\\build\\src\\vs2015u3\\DIA SDK\\bin\\amd64;c:\\Program Files\\Mercurial;c:\\mozilla-build\\7zip;c:\\mozilla-build\\info-zip;c:\\mozilla-build\\kdiff3;c:\\mozilla-build\\moztools-x64\\bin;c:\\mozilla-build\\mozmake;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\msys\\local\\bin;c:\\mozilla-build\\nsis-3.01;c:\\mozilla-build\\nsis-3.0b3;c:\\mozilla-build\\nsis-2.46u;c:\\mozilla-build\\python;c:\\mozilla-build\\python\\Scripts;c:\\mozilla-build\\upx391w;c:\\mozilla-build\\wget;c:\\mozilla-build\\yasm;c:\\Windows\\system32;c:\\Windows;c:\\Windows\\System32\\Wbem;c:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;c:\\Program Files\\Amazon\\cfn-bootstrap\\;c:\\Program Files (x86)\\GNU\\GnuPG\\pub;c:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;c:\\mozilla-build\\python\\lib\\site-packages\\pywin32_system32;c:\\mozilla-build\\python\\lib\\site-packages\\pywin32_system32;c:\\mozilla-build\\python\\lib\\site-packages\\pywin32_system32;c:\\mozilla-build\\python\\lib\\site-packages\\pywin32_system32;z:\\build\\build\\src\\obj-firefox\\dist\\bin'] 09:23:26 INFO - E (xpcshell/head.js) | test MAIN run_test pending (1) 09:23:26 INFO - E (xpcshell/head.js) | test run_next_test 0 pending (2) 09:23:26 INFO - E (xpcshell/head.js) | test MAIN run_test finished (2) 09:23:26 INFO - E running event loop 09:23:26 INFO - E test_child_pass.js | Starting test_child_simple 09:23:26 INFO - E (xpcshell/head.js) | test test_child_simple pending (2) 09:23:26 INFO - E (xpcshell/head.js) | test run in child pending (3) 09:23:26 INFO - E (xpcshell/head.js) | test run_next_test 1 pending (4) 09:23:26 INFO - E (xpcshell/head.js) | test test_child_simple finished (4) 09:23:26 INFO - E (xpcshell/head.js) | test run_next_test 0 finished (3) 09:23:26 INFO - E (xpcshell/head.js) | test run_next_test 1 finished (2) 09:23:26 INFO - E CHILD-TEST-STARTED 09:23:26 INFO - E (xpcshell/head.js) | test MAIN run_test pending (1) 09:23:26 INFO - E TEST-PASS | test_child_pass.js | run_test - [run_test : 1] true == true 09:23:26 INFO - E (xpcshell/head.js) | test MAIN run_test finished (1) 09:23:26 INFO - E exiting test 09:23:26 INFO - E "CONSOLE_MESSAGE: (warn) [JavaScript Warning: "Use of nsIFile in content process is deprecated." {file: "z:/build/build/src/testing/xpcshell/head.js" line: 356}]" 09:23:26 INFO - E CHILD-TEST-COMPLETED 09:23:26 INFO - E "CONSOLE_MESSAGE: (warn) [JavaScript Warning: "Use of nsIFile in content process is deprecated." {file: "z:/build/build/src/testing/xpcshell/head.js" line: 356}]" 09:23:26 INFO - E (xpcshell/head.js) | test finished (1) 09:23:26 INFO - E exiting test 09:23:26 INFO - E PID 6072 | Unable to read VR Path Registry from Z:\\task_1505463087\\AppData\\Local\\openvr\\openvrpaths.vrpath 09:23:26 INFO - E PID 6072 | [Parent 6072, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 09:23:26 INFO - E PID 6072 | Unable to read VR Path Registry from Z:\\task_1505463087\\AppData\\Local\\openvr\\openvrpaths.vrpath 09:23:26 INFO - E PID 6072 | [Child 4968, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 09:23:26 INFO - E PID 6072 | [Child 4968, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 09:23:26 INFO - E PID 6072 | [Parent 6072, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 09:23:26 INFO - E PID 6072 | [Parent 6072, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346 09:23:26 WARNING - E TEST-UNEXPECTED-FAIL | test_child_pass.js | xpcshell return code: -1073740940

The stack of the exception: 0 Id: 3d64.c48 Suspend: 1 Teb: 00000058`f8f82000 Unfrozen # Child-SP RetAddr Call Site 00 00000058`f95fcce8 00007ffb`93d0e8cc ntdll!NtWaitForMultipleObjects+0x14 01 00000058`f95fccf0 00007ffb`93d0df81 ntdll!WerpWaitForCrashReporting+0xa8 02 00000058`f95fcd70 00007ffb`93d0d6dd ntdll!RtlReportExceptionHelper+0x381 03 00000058`f95fd2c0 00007ffb`93d277b9 ntdll!RtlReportException+0x9d 04 00000058`f95fd340 00007ffb`90c1c540 ntdll!RtlReportCriticalFailure$filt$0+0x33 05 00000058`f95fd370 00007ffb`93cd4fb6 ucrtbase!_C_specific_handler+0xa0 06 00000058`f95fd3e0 00007ffb`93cda07d ntdll!_GSHandlerCheck_SEH+0x6a 07 00000058`f95fd410 00007ffb`93c49c58 ntdll!RtlpExecuteHandlerForException+0xd 08 00000058`f95fd440 00007ffb`93c45851 ntdll!RtlDispatchException+0x368 09 00000058`f95fdb50 00007ffb`93d2775f ntdll!RtlRaiseException+0x311 0a 00000058`f95fe330 00007ffb`93d2df0a ntdll!RtlReportCriticalFailure+0x97 0b 00000058`f95fe440 00007ffb`93cd4f22 ntdll!RtlpHeapHandleError+0x12 0c 00000058`f95fe470 00007ffb`93d2dea2 ntdll!RtlpLogHeapFailure+0x96 0d 00000058`f95fe4a0 00007ffb`93c63f8f ntdll!RtlpAnalyzeHeapFailure+0x322 0e 00000058`f95fe500 00007ffb`93c611ed ntdll!RtlpFreeHeap+0x107f *** WARNING: Unable to verify checksum for w:\fx\mc\obj-asan\dist\bin\xul.dll 0f 00000058`f95fe770 00007ffb`534318ce ntdll!RtlFreeHeap+0x41d 10 (Inline Function) --------`-------- xul!alloc::heap::deallocate+0xb [C:\projects\rust\src\liballoc\heap.rs @ 127] 11 (Inline Function) --------`-------- xul!alloc::raw_vec::{{impl}}::drop+0x1c [C:\projects\rust\src\liballoc\raw_vec.rs @ 564] 12 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x1c [C:\projects\rust\src\libcore\ptr.rs @ 60] 13 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x49 [C:\projects\rust\src\libcore\ptr.rs @ 60] 14 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x49 [C:\projects\rust\src\libcore\ptr.rs @ 60] 15 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x49 [C:\projects\rust\src\libcore\ptr.rs @ 60] 16 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x6e [C:\projects\rust\src\libcore\ptr.rs @ 60] 17 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x6e [C:\projects\rust\src\libcore\ptr.rs @ 60] 18 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x6e [C:\projects\rust\src\libcore\ptr.rs @ 60] 19 00000058`f95fe830 00007ffb`53430683 xul!servo_arc::Arc<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>::drop_slow<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>+0x7e [W:\fx\mc\servo\components\servo_arc\lib.rs @ 250] 1a (Inline Function) --------`-------- xul!servo_arc::{{impl}}::drop+0x25 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 385] 1b (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x25 [C:\projects\rust\src\libcore\ptr.rs @ 60] 1c (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x25 [C:\projects\rust\src\libcore\ptr.rs @ 60] 1d (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x25 [C:\projects\rust\src\libcore\ptr.rs @ 60] 1e (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x25 [C:\projects\rust\src\libcore\ptr.rs @ 60] 1f 00000058`f95fe880 00007ffb`534f74d5 xul!servo_arc::Arc<style::stylesheets::stylesheet::StylesheetContents>::drop_slow<style::stylesheets::stylesheet::StylesheetContents>+0x33 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 250] 20 (Inline Function) --------`-------- xul!servo_arc::{{impl}}::drop+0x23 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 385] 21 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x23 [C:\projects\rust\src\libcore\ptr.rs @ 60] 22 (Inline Function) --------`-------- xul!servo_arc::{{impl}}::drop+0x2b [W:\fx\mc\servo\components\servo_arc\lib.rs @ 787] 23 (Inline Function) --------`-------- xul!core::ptr::drop_in_place+0x2b [C:\projects\rust\src\libcore\ptr.rs @ 60] 24 (Inline Function) --------`-------- xul!style::gecko_bindings::sugar::ownership::HasArcFFI::release+0x2b [W:\fx\mc\servo\components\style\gecko_bindings\sugar\ownership.rs @ 105] 25 00000058`f95fe8c0 00007ffb`4d99e4fd xul!style::gecko::arc_types::Servo_StyleSheetContents_Release+0x35 [W:\fx\mc\servo\components\style\gecko\arc_types.rs @ 45] 26 (Inline Function) --------`-------- xul!mozilla::RefPtrTraits<RawServoStyleSheetContents>::Release+0x5 [w:\fx\mc\obj-asan\dist\include\mozilla\ServoArcTypeList.h @ 10] 27 (Inline Function) --------`-------- xul!RefPtr<const RawServoStyleSheetContents>::ConstRemovingRefPtrTraits<const RawServoStyleSheetContents>::Release+0x5 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 408] 28 (Inline Function) --------`-------- xul!RefPtr<const RawServoStyleSheetContents>::~RefPtr+0x21 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 79] 29 00000058`f95fe900 00007ffb`4d98e38e xul!mozilla::ServoStyleSheetInner::~ServoStyleSheetInner+0x69 [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 107] 2a 00000058`f95fe940 00007ffb`4d97a0d6 xul!mozilla::ServoStyleSheetInner::~ServoStyleSheetInner+0x10 [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 105] 2b 00000058`f95fe980 00007ffb`4d9791db xul!mozilla::StyleSheetInfo::RemoveSheet+0x2c8 [w:\fx\mc\layout\style\StyleSheet.cpp @ 290] 2c 00000058`f95feab0 00007ffb`4d94a205 xul!mozilla::StyleSheet::LastRelease+0x10b [w:\fx\mc\layout\style\StyleSheet.cpp @ 85] 2d 00000058`f95feb00 00007ffb`497ab993 xul!mozilla::StyleSheet::Release+0x19f [w:\fx\mc\layout\style\StyleSheet.cpp @ 158] 2e (Inline Function) --------`-------- xul!nsCOMPtr_base::~nsCOMPtr_base+0x38 [w:\fx\mc\obj-asan\dist\include\nsCOMPtr.h @ 313] 2f (Inline Function) --------`-------- xul!nsTArrayElementTraits<nsCOMPtr<nsIDocShellTreeItem> >::Destruct+0x38 [w:\fx\mc\xpcom\ds\nsTArray.h @ 560] 30 (Inline Function) --------`-------- xul!nsTArray_Impl<nsCOMPtr<nsIDocShellTreeItem>,nsTArrayInfallibleAllocator>::DestructRange+0x62 [w:\fx\mc\xpcom\ds\nsTArray.h @ 2012] 31 00000058`f95fec10 00007ffb`4b9b63ec xul!nsTArray_Impl<nsCOMPtr<nsIDocShellTreeItem>,nsTArrayInfallibleAllocator>::RemoveElementsAt+0x7f [w:\fx\mc\xpcom\ds\nsTArray.h @ 1732] 32 (Inline Function) --------`-------- xul!nsTArray_Impl<nsCOMPtr<nsIStyleRule>,nsTArrayInfallibleAllocator>::Clear+0x28 [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 1738] 33 00000058`f95fec80 00007ffb`4ddc341d xul!nsTArray_Impl<nsCOMPtr<nsIStyleRule>,nsTArrayInfallibleAllocator>::~nsTArray_Impl+0x64 [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 884] 34 (Inline Function) --------`-------- xul!mozilla::Array<nsTArray<RefPtr<mozilla::StyleSheet> >,3>::~Array+0x13 [w:\fx\mc\obj-asan\dist\include\mozilla\Array.h @ 22] 35 00000058`f95fecc0 00007ffb`4ddc335f xul!nsStyleSheetService::~nsStyleSheetService+0x9d [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 49] 36 00000058`f95fed00 00007ffb`47073211 xul!nsStyleSheetService::Release+0x53 [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 51] 37 (Inline Function) --------`-------- xul!nsCOMPtr_base::assign_assuming_AddRef+0x61 [w:\fx\mc\xpcom\base\nsCOMPtr.h @ 355] 38 (Inline Function) --------`-------- xul!nsCOMPtr<nsISupports>::operator=+0x61 [w:\fx\mc\xpcom\base\nsCOMPtr.h @ 973] 39 00000058`f95fed40 00007ffb`4711b93a xul!nsComponentManagerImpl::FreeServices+0x24d [w:\fx\mc\xpcom\components\nsComponentManager.cpp @ 1113] 3a 00000058`f95fee80 00007ffb`4878d5da xul!mozilla::ShutdownXPCOM+0x541 [w:\fx\mc\xpcom\build\XPCOMInit.cpp @ 947] 3b 00000058`f95ff060 00007ff7`7227169f xul!XRE_XPCShellMain+0x2f6e [w:\fx\mc\js\xpconnect\src\XPCShellImpl.cpp @ 1413] 3c 00000058`f95ffb50 00007ff7`7227135d xpcshell!NS_internal_main+0x1a2 [w:\fx\mc\js\xpconnect\shell\xpcshell.cpp @ 68] 3d 00000058`f95ffc80 00007ff7`722fd505 xpcshell!wmain+0x35d [w:\fx\mc\toolkit\xre\nsWindowsWMain.cpp @ 115] 3e (Inline Function) --------`-------- xpcshell!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79] 3f 00000058`f95ffdc0 00007ffb`91252774 xpcshell!__scrt_common_main_seh+0x11d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253] 40 00000058`f95ffe00 00007ffb`93ca0d51 KERNEL32!BaseThreadInitThunk+0x14 41 00000058`f95ffe30 00000000`00000000 ntdll!RtlUserThreadStart+0x21

The stack above looks similar to the ones in bug 1385925 / bug 1399668.

(In reply to Mats Palmgren (:mats) from comment #2) > The stack above looks similar to the ones in bug 1385925 / bug 1399668. That's just because drop_in_place/drop_slow is the name of any destructor in Rust. So the only similarity with the bugs you linked to is that they're crashes in destructors. Ting-Yu, are you investigating this? Given that this only happens on clang-cl it's not immediately clear to me whether this affects release, but would be good to figure out in any case.

Yes, I'm trying to understand what goes wrong.

The crash also happens when run firefox normally.

Do you have any more details about why we're failing? Specifically, what condition led RtlpFreeHeap to call RtlpAnalyzeHeapFailure? Do you get any output on the debugger console? Also, if you launch the program under a debugger initially, the NT heap stuff will use a debug mode that may give more information. My first guess is that ASan redirected an allocation API but not the free, so we have an allocator mismatch -- but I'll wait for Ting to see if we have more diagnostics available. And if we don't figure it out by next week, this would be a great candidate for Microsoft's upcoming time-travel debugger.

Launch firefox.exe under windbg with verbose mode on, I got following: HEAP[firefox.exe]: Invalid address specified to RtlFreeHeap( 000001B4326A0000, 000011C032C6F960 ) Note I've seen "Invalid address specified to RtlFreeHeap", and "Critical error detected c0000374". 0:055> !address 000011C032C6F960 Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions... Usage: <unknown> Base Address: 000011c0`32b80000 End Address: 000011c0`32ce0000 Region Size: 00000000`00160000 ( 1.375 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 000011b4`32b80000 Allocation Protect: 00000001 PAGE_NOACCESS Content source: 1 (target), length: 6a0 0:055> !heap 000001B4326A0000 HEAPEXT: Unable to get address of ntdll!RtlpHeapInvalidBadAddress. Index Address Name Debugging options enabled 1: 1b4326a0000 Segment at 000001b4326a0000 to 000001b4326df000 (0003f000 bytes committed) Segment at 0000123439ad0000 to 0000123439bcf000 (000ff000 bytes committed) Segment at 000012343d3d0000 to 000012343d5cf000 (0001b000 bytes committed) 0:055> !heap 000011c032b80000 Index Address Name Debugging options enabled 0:055> !heap 000011b432b80000 Index Address Name Debugging options enabled 0:055> k # Child-SP RetAddr Call Site 00 00000005`425fd178 00007ffd`3f7f9a86 ntdll!RtlpBreakPointHeap+0x16 01 00000005`425fd180 00007ffd`3f7c127d ntdll!RtlpValidateHeapEntry+0x529b2 02 00000005`425fd1d0 00007ffd`3f7e95c9 ntdll!RtlDebugFreeHeap+0x1a1 03 00000005`425fd230 00007ffd`3f7611ed ntdll!RtlpFreeHeap+0x866b9 *** WARNING: Unable to verify checksum for W:\fx\mc\obj-asan\dist\bin\xul.dll 04 00000005`425fd4a0 00007ffd`001aeb9f ntdll!RtlFreeHeap+0x41d 05 00000005`425fd560 00007ffd`002031d6 xul!alloc::heap::deallocate+0x3f [C:\projects\rust\src\liballoc\heap.rs @ 128] 06 00000005`425fd5c0 00007ffd`0008b7c7 xul!alloc::raw_vec::{{impl}}::drop<style::stylesheets::CssRule>+0x86 [C:\projects\rust\src\liballoc\raw_vec.rs @ 559] 07 00000005`425fd630 00007ffd`000ac9c8 xul!core::ptr::drop_in_place<alloc::raw_vec::RawVec<style::stylesheets::CssRule>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 08 00000005`425fd670 00007ffd`00097137 xul!core::ptr::drop_in_place<collections::vec::Vec<style::stylesheets::CssRule>>+0x28 [C:\projects\rust\src\libcore\ptr.rs @ 60] 09 00000005`425fd6b0 00007ffd`000a2607 xul!core::ptr::drop_in_place<style::stylesheets::rule_list::CssRules>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 0a 00000005`425fd6f0 00007ffd`0009033f xul!core::ptr::drop_in_place<core::cell::UnsafeCell<style::stylesheets::rule_list::CssRules>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 0b 00000005`425fd730 00007ffd`000a236b xul!core::ptr::drop_in_place<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>+0x2f [C:\projects\rust\src\libcore\ptr.rs @ 60] 0c 00000005`425fd770 00007ffd`000b866a xul!core::ptr::drop_in_place<servo_arc::ArcInner<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>>+0x1b [C:\projects\rust\src\libcore\ptr.rs @ 60] 0d 00000005`425fd7b0 00007ffc`ffe2d649 xul!core::ptr::drop_in_place<alloc::boxed::Box<servo_arc::ArcInner<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>>>+0x1a [C:\projects\rust\src\libcore\ptr.rs @ 60] 0e 00000005`425fd7f0 00007ffd`001ac7af xul!servo_arc::Arc<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>::drop_slow<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>+0x39 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 252] 0f 00000005`425fd840 00007ffd`00098f97 xul!servo_arc::{{impl}}::drop<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>+0x7f [W:\fx\mc\servo\components\servo_arc\lib.rs @ 387] 10 00000005`425fd8b0 00007ffd`0009ba27 xul!core::ptr::drop_in_place<servo_arc::Arc<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] " 11 00000005`425fd8f0 00007ffd`0008d8db xul!core::ptr::drop_in_place<style::stylesheets::stylesheet::StylesheetContents>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 12 00000005`425fd930 00007ffd`000af57a xul!core::ptr::drop_in_place<servo_arc::ArcInner<style::stylesheets::stylesheet::StylesheetContents>>+0x1b [C:\projects\rust\src\libcore\ptr.rs @ 60] 13 00000005`425fd970 00007ffc`ffe2cd39 xul!core::ptr::drop_in_place<alloc::boxed::Box<servo_arc::ArcInner<style::stylesheets::stylesheet::StylesheetContents>>>+0x1a [C:\projects\rust\src\libcore\ptr.rs @ 60] 14 00000005`425fd9b0 00007ffd`001ad77f xul!servo_arc::Arc<style::stylesheets::stylesheet::StylesheetContents>::drop_slow<style::stylesheets::stylesheet::StylesheetContents>+0x39 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 252] 15 00000005`425fda00 00007ffd`00093e67 xul!servo_arc::{{impl}}::drop<style::stylesheets::stylesheet::StylesheetContents>+0x7f [W:\fx\mc\servo\components\servo_arc\lib.rs @ 387] 16 00000005`425fda70 00007ffd`0021ce21 xul!core::ptr::drop_in_place<servo_arc::Arc<style::stylesheets::stylesheet::StylesheetContents>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 17 00000005`425fdab0 00007ffd`000b8c67 xul!servo_arc::{{impl}}::drop<style::stylesheets::stylesheet::StylesheetContents>+0x51 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 788] 18 00000005`425fdb20 00007ffd`00478123 xul!core::ptr::drop_in_place<servo_arc::RawOffsetArc<style::stylesheets::stylesheet::StylesheetContents>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 19 00000005`425fdb60 00007ffd`005eef0f xul!style::gecko_bindings::sugar::ownership::HasArcFFI::release<style::stylesheets::stylesheet::StylesheetContents>+0x73 [W:\fx\mc\servo\components\style\gecko_bindings\sugar\ownership.rs @ 106] 1a 00000005`425fdbe0 00007ffc`eecf5df3 xul!style::gecko::arc_types::Servo_StyleSheetContents_Release+0x1f [W:\fx\mc\servo\components\style\gecko\arc_types.rs @ 45] 1b 00000005`425fdc20 00007ffc`eedb19c3 xul!mozilla::RefPtrTraits<RawServoStyleSheetContents>::Release+0x13 [w:\fx\mc\obj-asan\dist\include\mozilla\ServoArcTypeList.h @ 10] 1c 00000005`425fdc50 00007ffc`eed3ca80 xul!RefPtr<const RawServoStyleSheetContents>::ConstRemovingRefPtrTraits<const RawServoStyleSheetContents>::Release+0x13 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 409] 1d 00000005`425fdc80 00007ffc`eede4fd7 xul!RefPtr<const RawServoStyleSheetContents>::~RefPtr+0x80 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 81] 1e 00000005`425fdcc0 00007ffc`eedaf8bc xul!mozilla::ServoStyleSheetInner::~ServoStyleSheetInner+0x77 [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 107] 1f 00000005`425fdd00 00007ffc`eed93a67 xul!mozilla::ServoStyleSheetInner::~ServoStyleSheetInner+0x2c [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 105] 20 00000005`425fdd60 00007ffc`eed9171c xul!mozilla::StyleSheetInfo::RemoveSheet+0x337 [w:\fx\mc\layout\style\StyleSheet.cpp @ 290] 21 00000005`425fde90 00007ffc`eed40173 xul!mozilla::StyleSheet::LastRelease+0x12c [w:\fx\mc\layout\style\StyleSheet.cpp @ 85] 22 00000005`425fdf00 00007ffc`eed3fb52 xul!mozilla::StyleSheet::Release+0x5e3 [w:\fx\mc\layout\style\StyleSheet.cpp @ 158] 23 00000005`425fe120 00007ffc`dfc4e708 xul!mozilla::ServoStyleSheet::Release+0x22 [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 182] 24 00000005`425fe170 00007ffc`dfc4e673 xul!mozilla::RefPtrTraits<mozilla::StyleSheet>::Release+0x88 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 42] 25 00000005`425fe1d0 00007ffc`dfc32d50 xul!RefPtr<mozilla::StyleSheet>::ConstRemovingRefPtrTraits<mozilla::StyleSheet>::Release+0x13 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 399] 26 00000005`425fe200 00007ffc`dfc4e983 xul!RefPtr<mozilla::StyleSheet>::~RefPtr+0x80 [w:\fx\mc\obj-asan\dist\include\mozilla\RefPtr.h @ 81] 27 00000005`425fe240 00007ffc`dfc4e8fa xul!nsTArrayElementTraits<RefPtr<mozilla::StyleSheet> >::Destruct+0x13 [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 560] 28 00000005`425fe270 00007ffc`dfc4e82a xul!nsTArray_Impl<RefPtr<mozilla::StyleSheet>,nsTArrayInfallibleAllocator>::DestructRange+0x6a [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 2011] 29 00000005`425fe2d0 00007ffc`dfc4e7cd xul!nsTArray_Impl<RefPtr<mozilla::StyleSheet>,nsTArrayInfallibleAllocator>::RemoveElementsAt+0x4a [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 2060] 2a 00000005`425fe330 00007ffc`dfc4e770 xul!nsTArray_Impl<RefPtr<mozilla::StyleSheet>,nsTArrayInfallibleAllocator>::Clear+0x2d [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 1738] 2b 00000005`425fe370 00007ffc`dfc33263 xul!nsTArray_Impl<RefPtr<mozilla::StyleSheet>,nsTArrayInfallibleAllocator>::~nsTArray_Impl+0x30 [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 887] 2c 00000005`425fe3b0 00007ffc`ef597ea5 xul!nsTArray<RefPtr<mozilla::StyleSheet> >::~nsTArray+0x13 [w:\fx\mc\obj-asan\dist\include\nsTArray.h @ 2234] 2d 00000005`425fe3e0 00007ffc`ef598198 xul!mozilla::Array<nsTArray<RefPtr<mozilla::StyleSheet> >,3>::~Array+0x35 [w:\fx\mc\obj-asan\dist\include\mozilla\Array.h @ 22] 2e 00000005`425fe430 00007ffc`ef598021 xul!nsStyleSheetService::~nsStyleSheetService+0xd8 [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 49] 2f 00000005`425fe490 00007ffc`df685f77 xul!nsStyleSheetService::Release+0x91 [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 51] 30 00000005`425fe500 00007ffc`dfa2626b xul!nsCOMPtr_base::assign_assuming_AddRef+0x117 [w:\fx\mc\obj-asan\dist\include\nsCOMPtr.h @ 355] 31 00000005`425fe580 00007ffc`dfae7208 xul!nsCOMPtr<nsISupports>::operator=+0x2b [w:\fx\mc\obj-asan\dist\include\nsCOMPtr.h @ 974] 32 00000005`425fe5c0 00007ffc`dfc8b132 xul!nsComponentManagerImpl::FreeServices+0x228 [w:\fx\mc\xpcom\components\nsComponentManager.cpp @ 1116] 33 00000005`425fe700 00007ffc`dfc8a8b3 xul!mozilla::ShutdownXPCOM+0x872 [w:\fx\mc\xpcom\build\XPCOMInit.cpp @ 947] 34 00000005`425febc0 00007ffc`fadb5f96 xul!NS_ShutdownXPCOM+0x13 [w:\fx\mc\xpcom\build\XPCOMInit.cpp @ 822] 35 00000005`425febf0 00007ffc`e1cc9b01 xul!XRE_TermEmbedding+0x76 [w:\fx\mc\toolkit\xre\nsEmbedFunctions.cpp @ 224] 36 00000005`425fec40 00007ffc`ed5e253a xul!mozilla::ipc::ScopedXREEmbed::Stop+0x71 [w:\fx\mc\ipc\glue\ScopedXREEmbed.cpp @ 118] 37 00000005`425fec90 00007ffc`fadb78fa xul!mozilla::dom::ContentProcess::CleanUp+0x1a [w:\fx\mc\dom\ipc\ContentProcess.cpp @ 275] 38 00000005`425fecc0 00007ffc`faded77a xul!XRE_InitChildProcess+0x134a [w:\fx\mc\toolkit\xre\nsEmbedFunctions.cpp @ 710] *** WARNING: Unable to verify checksum for firefox.exe 39 00000005`425ff350 00007ff7`d04d0df0 xul!mozilla::BootstrapImpl::XRE_InitChildProcess+0x2a [w:\fx\mc\toolkit\xre\Bootstrap.cpp @ 65] 3a 00000005`425ff3a0 00007ff7`d04d0266 firefox!content_process_main+0x4f0 [w:\fx\mc\ipc\contentproc\plugin-container.cpp @ 63] 3b 00000005`425ff520 00007ff7`d04cf817 firefox!NS_internal_main+0x346 [w:\fx\mc\browser\app\nsBrowserApp.cpp @ 285] 3c 00000005`425ff760 00007ff7`d079343d firefox!wmain+0x5b7 [w:\fx\mc\toolkit\xre\nsWindowsWMain.cpp @ 115] 3d (Inline Function) --------`-------- firefox!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79] 3e 00000005`425ff920 00007ffd`3f692774 firefox!__scrt_common_main_seh+0x11d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253] 3f 00000005`425ff960 00007ffd`3f7a0d51 KERNEL32!BaseThreadInitThunk+0x14 40 00000005`425ff990 00000000`00000000 ntdll!RtlUserThreadStart+0x21

It seems the first pointer going to be released in xul!alloc::raw_vec::{{impl}}::drop is invalid, the stack of the raw_vec allocation (a different run but the stack is the same to comment 7): 3:059> !heap -p -a 0x125bbe02ffe0 address 0000125bbe02ffe0 found in _DPH_HEAP_ROOT @ 1dbae3b1000 in busy allocation ( DPH_HEAP_BLOCK: UserAddr UserSize - VirtAddr VirtSize) 125bbe3571a0: 125bbe02ffd0 28 - 125bbe02f000 2000 00007ffd3f7f3f7f ntdll!RtlDebugAllocateHeap+0x0000000000033237 00007ffd3f7e8294 ntdll!RtlpAllocateHeap+0x000000000008a7d4 00007ffd3f75b88a ntdll!RtlpAllocateHeapInternal+0x0000000000000a0a 00007ffd001aedcf xul!alloc::heap::allocate+0x000000000000004f [C:\projects\rust\src\liballoc\heap.rs @ 59] 00007ffd001aed30 xul!alloc::heap::exchange_malloc+0x0000000000000040 [C:\projects\rust\src\liballoc\heap.rs @ 154] 00007ffcffe1ede9 xul!servo_arc::Arc<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>::new<style::shared_lock::Locked<style::stylesheets::rule_list::CssRules>>+0x0000000000000139 [W:\fx\mc\servo\components\servo_arc\lib.rs @ 183] 00007ffd004a4698 xul!style::stylesheets::rule_list::CssRules::new+0x0000000000000098 [W:\fx\mc\servo\components\style\stylesheets\rule_list.rs @ 53] 00007ffcffa961c3 xul!style::stylesheets::stylesheet::StylesheetContents::from_str<geckoservo::error_reporter::ErrorReporter>+0x00000000000002d3 [W:\fx\mc\servo\components\style\stylesheets\stylesheet.rs @ 96] 00007ffcffd93849 xul!geckoservo::glue::Servo_StyleSheet_FromUTF8Bytes+0x00000000000003a9 [W:\fx\mc\servo\ports\geckolib\glue.rs @ 907] 00007ffceed411ba xul!mozilla::ServoStyleSheet::ParseSheet+0x00000000000003da [w:\fx\mc\layout\style\ServoStyleSheet.cpp @ 213] 00007ffceec61c76 xul!mozilla::css::Loader::ParseSheet+0x0000000000000826 [w:\fx\mc\layout\style\Loader.cpp @ 1646] 00007ffceed47d1b xul!mozilla::css::StreamLoader::OnStopRequest+0x0000000000000d7b [w:\fx\mc\layout\style\StreamLoader.cpp @ 129] 00007ffce6974801 xul!nsSyncLoadService::PushSyncStreamToListener+0x0000000000000dd1 [w:\fx\mc\dom\base\nsSyncLoadService.cpp @ 393] 00007ffceec576be xul!mozilla::css::Loader::LoadSheet+0x0000000000001e7e [w:\fx\mc\layout\style\Loader.cpp @ 1405] 00007ffceec72598 xul!mozilla::css::Loader::InternalLoadNonDocumentSheet+0x0000000000000c48 [w:\fx\mc\layout\style\Loader.cpp @ 2325] 00007ffceec718d1 xul!mozilla::css::Loader::LoadSheetSync+0x0000000000000141 [w:\fx\mc\layout\style\Loader.cpp @ 2187] 00007ffcef59bddf xul!LoadSheet+0x00000000000001bf [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 224] 00007ffcef599325 xul!nsStyleSheetService::LoadAndRegisterSheetInternal+0x0000000000000355 [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 262] 00007ffcef59ac03 xul!nsStyleSheetService::LoadAndRegisterSheet+0x0000000000000703 [w:\fx\mc\layout\base\nsStyleSheetService.cpp @ 176] 00007ffced5274c6 xul!mozilla::dom::ContentChild::RecvLoadAndRegisterSheet+0x0000000000000266 [w:\fx\mc\dom\ipc\ContentChild.cpp @ 2813] 00007ffce3105e97 xul!mozilla::dom::PContentChild::OnMessageReceived+0x00000000000160f7 [w:\fx\mc\obj-asan\ipc\ipdl\PContentChild.cpp @ 6809] 00007ffced535481 xul!mozilla::dom::ContentChild::OnMessageReceived+0x0000000000000201 [w:\fx\mc\dom\ipc\ContentChild.cpp @ 3694] 00007ffce1ca1eeb xul!mozilla::ipc::MessageChannel::DispatchAsyncMessage+0x000000000000049b [w:\fx\mc\ipc\glue\MessageChannel.cpp @ 2115] 00007ffce1c9cb83 xul!mozilla::ipc::MessageChannel::DispatchMessageW+0x00000000000005e3 [w:\fx\mc\ipc\glue\MessageChannel.cpp @ 2045] 00007ffce1c9ed24 xul!mozilla::ipc::MessageChannel::RunMessage+0x0000000000000444 [w:\fx\mc\ipc\glue\MessageChannel.cpp @ 1891] 00007ffce1c9fa24 xul!mozilla::ipc::MessageChannel::MessageTask::Run+0x0000000000000364 [w:\fx\mc\ipc\glue\MessageChannel.cpp @ 1924] 00007ffcdfbac197 xul!nsThread::ProcessNextEvent+0x0000000000001bd7 [w:\fx\mc\xpcom\threads\nsThread.cpp @ 1040] 00007ffcdfbb7017 xul!NS_ProcessNextEvent+0x00000000000001e7 [w:\fx\mc\xpcom\threads\nsThreadUtils.cpp @ 521] 00007ffce1cb0583 xul!mozilla::ipc::MessagePump::Run+0x0000000000000393 [w:\fx\mc\ipc\glue\MessagePump.cpp @ 97] 00007ffce1cb2777 xul!mozilla::ipc::MessagePumpForChildProcess::Run+0x0000000000000227 [w:\fx\mc\ipc\glue\MessagePump.cpp @ 302] 00007ffce1b4f8ed xul!MessageLoop::RunInternal+0x00000000000000cd [w:\fx\mc\ipc\chromium\src\base\message_loop.cc @ 327] 00007ffce1b4f48f xul!MessageLoop::RunHandler+0x00000000000000af [w:\fx\mc\ipc\chromium\src\base\message_loop.cc @ 320]

:dmajor is correct, it's mismatched malloc/free for the buffer of CssRules vector. malloc: 00 000000f4`777590c8 00007ffe`53e21394 ucrtbase!malloc 01 000000f4`777590d0 00007ffe`53dee6bd xul!fallible::try_double_vec<style::stylesheets::CssRule>+0x244 [W:\fx\mc\servo\components\fallible\lib.rs @ 70] 02 000000f4`777592e0 00007ffe`53d234ee xul!fallible::{{impl}}::try_push<style::stylesheets::CssRule>+0x7d [W:\fx\mc\servo\components\fallible\lib.rs @ 35] 03 000000f4`777593d0 00007ffe`53d24388 xul!style::stylesheets::stylesheet::Stylesheet::parse_rules<geckoservo::error_reporter::ErrorReporter>+0x4ae [W:\fx\mc\servo\components\style\stylesheets\stylesheet.rs @ 390] 04 000000f4`77759b10 00007ffe`53ef69b9 xul!style::stylesheets::stylesheet::StylesheetContents::from_str<geckoservo::error_reporter::ErrorReporter>+0x218 [W:\fx\mc\servo\components\style\stylesheets\stylesheet.rs @ 80] 05 000000f4`77759e50 00007ffe`48c3a33a xul!geckoservo::glue::Servo_StyleSheet_FromUTF8Bytes+0x3a9 [W:\fx\mc\servo\ports\geckolib\glue.rs @ 901] ... free: ... 0e 00000035`8affcdf0 00007ffe`a55011ed ntdll!RtlpFreeHeap+0x107f *** WARNING: Unable to verify checksum for w:\fx\mc\obj-asan\dist\bin\xul.dll 0f 00000035`8affd060 00007ffe`539c727f ntdll!RtlFreeHeap+0x41d 10 00000035`8affd120 00007ffe`53d73f96 xul!alloc::heap::deallocate+0x3f [C:\projects\rust\src\liballoc\heap.rs @ 128] 11 00000035`8affd180 00007ffe`53c93287 xul!alloc::raw_vec::{{impl}}::drop<regex::dfa::State>+0x86 [C:\projects\rust\src\liballoc\raw_vec.rs @ 559] 12 00000035`8affd1f0 00007ffe`5405aae8 xul!core::ptr::drop_in_place<alloc::raw_vec::RawVec<alloc::boxed::Box<str>>>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] 13 00000035`8affd230 00007ffe`5405f127 xul!core::ptr::drop_in_place<collections::vec::Vec<style::stylesheets::CssRule>>+0x28 [C:\projects\rust\src\libcore\ptr.rs @ 60] 14 00000035`8affd270 00007ffe`5405ea67 xul!core::ptr::drop_in_place<style::stylesheets::rule_list::CssRules>+0x17 [C:\projects\rust\src\libcore\ptr.rs @ 60] ...

The reason a normal build doesn't run into this is because the malloc/free goes to jemalloc. malloc: 00 000000e1`e33fc5a8 00007ffe`4c4aacee mozglue!Allocator<MozJemallocBase>::malloc [w:\fx\mc\memory\build\mozjemalloc.cpp @ 4671] 01 000000e1`e33fc5b0 00007ffe`4c315164 xul!fallible::try_double_vec<style::stylesheets::CssRule>+0x26e [W:\fx\mc\servo\components\fallible\lib.rs @ 70] 02 (Inline Function) --------`-------- xul!fallible::{{impl}}::try_push+0x62 03 000000e1`e33fc8c0 00007ffe`4c316dfc xul!style::stylesheets::stylesheet::Stylesheet::parse_rules<geckoservo::error_reporter::ErrorReporter>+0x524 [W:\fx\mc\servo\components\style\stylesheets\stylesheet.rs @ 393] 04 000000e1`e33fd0f0 00007ffe`4c6146f9 xul!style::stylesheets::stylesheet::StylesheetContents::from_str<geckoservo::error_reporter::ErrorReporter>+0x20c [W:\fx\mc\servo\components\style\stylesheets\stylesheet.rs @ 83] 05 000000e1`e33fd490 00007ffe`523ba104 xul!geckoservo::glue::Servo_StyleSheet_FromUTF8Bytes+0x3a9 [W:\fx\mc\servo\ports\geckolib\glue.rs @ 907] ... free: 00 000000e1`e33fdd68 00007ffe`8576de2c mozglue!je_free [w:\fx\mc\memory\build\malloc_decls.h @ 48] 01 000000e1`e33fdd70 00007ffe`4b5777bf mozglue!HeapFree+0x1c [w:\fx\mc\memory\mozalloc\winheap.cpp @ 71] 02 000000e1`e33fdda0 00007ffe`4b5cbdf6 xul!alloc::heap::deallocate+0x3f [C:\projects\rust\src\liballoc\heap.rs @ 128] 03 000000e1`e33fde00 00007ffe`4b454357 xul!alloc::raw_vec::{{impl}}::drop<style::stylesheets::CssRule>+0x86 [C:\projects\rust\src\liballoc\raw_vec.rs @ 559] ...

I assume this has happened because servo/components/fallible/lib.rs calls (C-land) malloc and realloc directly, rather than going through the official channels (libc::malloc/realloc). I propose to try using alloc() from servo/components/hashglobe/src/alloc.rs, and to add a realloc() function to that file too, which I can copy from liballoc_system (as the rest of it has been). Sound plausible? If the problem diagnosis is correct, then also we'll need to fix any copy/paste clones of fallible/lib.rs.

Bobby, does the patch look plausible to you? Given the difficulties we've had previously with these malloc calls, I'm a bit paranoid about it. ------ * adds hashglobe::alloc::realloc, as that was previously not implemented, copying and simplifying from liballoc_system. * routes malloc and realloc calls through hashglobe::alloc::, instead of doing it via direct 'extern "C"' calls. * removes the #[cfg(feature = "known_system_malloc")] markings in fallible/lib.rs (the fallible vec stuff) on the assumption that we no longer need to special-case these uses

Comment on attachment 8910768 [details] [diff] [review] A possible fix Manish did all the allocator munging, so I'll let him review.

[Tracking Requested - why for this release]: It's not yet clear whether this has any impact on non-ASAN builds, but requesting tracking so we can be sure to evaluate whether we should uplift the fix.

Comment on attachment 8910768 [details] [diff] [review] A possible fix Review of attachment 8910768 [details] [diff] [review]: ----------------------------------------------------------------- I'm not 100% clear if the simplification is correct; the original impl does different things for different alignments. It seems like this will mostly work too.

I would like to understand what's going on here and why we actually end up in two different system allocators. Julian, do you have an explanation?

Vec does not specify allocator for the RawVec |buf| [1], so the default allocator Heap is used [2]. When try_push(), try_double_vec() allocates a new buffer by either malloc or realloc [3], and replace the vector with a new one [4] which has the new buffer. At last, RawVec.drop() goes to Heap.deallocate() to free the buffer [5] and crash. Why not fix this by calling the RawVec's default allocator's alloc/realloc at [3]? [1] https://github.com/rust-lang/rust/blob/master/src/liballoc/vec.rs#L300 [2] https://github.com/rust-lang/rust/blob/master/src/liballoc/raw_vec.rs#L47 [3] http://searchfox.org/mozilla-central/source/servo/components/fallible/lib.rs#70,72 [4] http://searchfox.org/mozilla-central/source/servo/components/fallible/lib.rs#83 [5] https://github.com/rust-lang/rust/blob/master/src/liballoc/raw_vec.rs#L687

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #18) > Julian, do you have an explanation? No, I don't. dmajor speculated this .. > ASan redirected an allocation API but not the free, .. but I would guess something slightly different, namely that ASan redirected malloc/realloc/free calls from some DLLs in the process but not all. NI'ing Nathan; he knows about the Win64/ASan machinery IIUC. Nathan, do you have any insight into whether/how the Win64/ASan heap redirection can fail? Ting, to be clear: does this affect the "normal" Win64 builds? Or only Win64/ASan. (== is this is a short-term problem for '57) ?

(In reply to Julian Seward [:jseward] from comment #20) > Ting, to be clear: does this affect the "normal" Win64 builds? No. > Or only Win64/ASan. (== is this is a short-term problem for '57) ? Yes. I thought this can be reproduced if I add "ac_add_options --disable-jemalloc" to mozconfig because I assumed malloc and HeapFree are different allocators, but I was wrong. Original ucrtbase!malloc goes to _ucrtbase!_imp_HeapAlloc and RtlAllocateHeap in the end, so it does not have the crash like Win64/Asan. I'll check further for what's wrong with the ASan intercepted ucrtbase!malloc.

Now with Windows build fix. This seems to fix the Win64/ASan run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a9855d1d92e19321523a317d8ff00b3a148440e2&filter-tier=1&filter-tier=2&filter-tier=3

(In reply to Julian Seward [:jseward] from comment #20) > (In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #18) > > Julian, do you have an explanation? > > No, I don't. dmajor speculated this .. > > > ASan redirected an allocation API but not the free, > > .. but I would guess something slightly different, namely that ASan > redirected > malloc/realloc/free calls from some DLLs in the process but not all. > > NI'ing Nathan; he knows about the Win64/ASan machinery IIUC. Nathan, do you > have any insight into whether/how the Win64/ASan heap redirection can fail? ASan intercepts HeapFree only if it comes from ucrtbase: https://github.com/llvm-project/llvm-project-20170507/blob/135c4a0f43501987991610a289cf36758dbc5f8a/compiler-rt/lib/asan/asan_malloc_win.cc#L212 > Ting, to be clear: does this affect the "normal" Win64 builds? Or only > Win64/ASan. (== is this is a short-term problem for '57) ? Regardless, mixing allocators is a bad idea, and I would not be surprised if some AV or other injected software in the wild would fall over in the same way. I wouldn't use ASan as a reason not to fix for 57.

(In reply to Ting-Yu Chou [:ting] from comment #19) > Why not fix this by calling the RawVec's default allocator's alloc/realloc > at [3]? I too am curious about why not this. In the abstract, it sounds like the ideal solution, but maybe there's something particular about this code that I don't know.

(In reply to David Major [:dmajor] from comment #24) > (In reply to Ting-Yu Chou [:ting] from comment #19) > > Why not fix this by calling the RawVec's default allocator's alloc/realloc [I didn't know that Vecs/RawVecs carry their allocator along till just now. Thanks to dmajor for explaining.] That would indeed be a cleaner solution. But I can't get it to compile. I have a |vec: Vec| and I want to write vec.buf.alloc().alloc/realloc(..) but I ran into two problems: (1) How do I get the RawVec out of a Vec? .buf is private and I can't see any Vec:: method that will get it. (2) The compiler (1.20.0) complains that this is an unstable language feature: error: use of unstable library feature 'alloc': this library is unlikely to be stabilized in its current form or name (see issue #27783) error: use of unstable library feature 'allocator_api': the precise API and guarantees it provides may be tweaked slightly, especially to possibly take into account the types being stored to make room for a future tracing garbage collector (see issue #32838) So I am unclear how to proceed. Suggestions?

(In reply to Julian Seward [:jseward] from comment #20) > NI'ing Nathan; he knows about the Win64/ASan machinery IIUC. Nathan, do you > have any insight into whether/how the Win64/ASan heap redirection can fail? David has answered my question with far more insight that I could have; I would have just noted the alloc/free mismatches and declared that we should fix it. :)

(In reply to Julian Seward [:jseward] from comment #25) > (1) How do I get the RawVec out of a Vec? .buf is private and I can't see > any Vec:: method that will get it. > > (2) The compiler (1.20.0) complains that this is an unstable language > feature: > > error: use of unstable library feature 'alloc': this library is unlikely > to be stabilized in its current form or name (see issue #27783) > > error: use of unstable library feature 'allocator_api': the precise API > and guarantees it provides may be tweaked slightly, especially to > possibly take into account the types being stored to make room for a > future tracing garbage collector (see issue #32838) > > So I am unclear how to proceed. Suggestions? Yeah, you can't poke at these internals in stable rust. We'll need to mimic whatever it does a la HashGlobe.

(In reply to David Major [:dmajor] from comment #23) > ASan intercepts HeapFree only if it comes from ucrtbase: Both malloc/realloc redirect to ASan intercepted version, but deallocate does not. Step into alloc::heap::deallocate I got stack: xul!alloc::heap::deallocate xul!alloc_system::__rust_deallocate xul!HeapFree xul!_imp_HeapFree kernel32!HeapFreeStub ntdll!RtlFreeHeap :dmajor, do you have any ideas about how the signature xul!HeapFree is created? I don't see the error message of failed intercepting HeapFree when I set ASAN_OPTIONS:verbosity=2. So I wonder the way ASan intercepting HeapFree is sufficient for us. Note xul!alloc::heap::allocate does not redirect to ASan heap either. It seems to me at least we need to fix the mismatched allocator in try_push and raw_vec::drop. And then try to understand why ASan does not intercept rust's liballoc underlying implementation.

(In reply to Ting-Yu Chou [:ting] from comment #28) > :dmajor, do you have any ideas about how the signature xul!HeapFree is > created? I don't see the error message of failed intercepting HeapFree when > I set ASAN_OPTIONS:verbosity=2. So I wonder the way ASan intercepting > HeapFree is sufficient for us. Note xul!alloc::heap::allocate does not > redirect to ASan heap either. In general, ASan does not try to intercept kernel32!Heap* operations -- its DLL interceptor does not touch the first bytes in those functions. ASan only patches the pointers to HeapFree etc. that are in ucrtbase's import table, so when ucrtbase calls HeapFree, the call is redirected to ASan, but other callers of HeapFree are unchanged. > It seems to me at least we need to fix the mismatched allocator in try_push > and raw_vec::drop. And then try to understand why ASan does not intercept > rust's liballoc underlying implementation. I think we only need to fix the allocator mismatch. As far as I can tell, ASan's behavior regarding the Heap APIs is deliberate. As long as we're paired correctly (either malloc+free both go to ASan, or HeapAlloc+HeapFree both avoid ASan) the rest of this issue should disappear.

NI Julian on the status of fixing the allocator mismatch.

<dmajor> bholley: I think we should fix it for 57

and to be clear, I think something along the lines of the first patch (routing realloc through the hashglobe stuff) is what we want. I haven't reviewed the patch in detail though.

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #32) > and to be clear, I think something along the lines of the first patch > (routing realloc through the hashglobe stuff) is what we want. I looked at this some more. For the patch proposed in comment 22, here are the allocation chains for both hashglobe::hash_map and FallibleVec::try_push (which is the cause of the current problem): All of the following in servo/components/hashglobe/src: hash_map::try_resize -> Rawtable::new -> RawTable::try_new_uninitialized -> alloc::alloc -> libc::malloc // #[cfg(any(unix, target_os = "redox"))] -> HeapAlloc // #[cfg(windows)] All of the following in servo/components/fallible/src: try_push -> try_double_vec, try_double_small_vec -> alloc::alloc (from hashglobe) -> libc::malloc // #[cfg(any(unix, target_os = "redox"))] -> HeapAlloc // #[cfg(windows)] So, with the patch, at least FallibleVec uses the same allocator path as HashGlobe now. And from the try run in comment 22, it appears not to fail any more. Now, that doesn't actually solve the problem properly, because what we really need to do is allocate using the allocator from std::Vec, since we'll be deallocating with that allocator. But I don't see a way to get the allocator out of std::Vec. So, short of cloning std::Vec in the same way that hashglobe has cloned std:: stuff, I don't see anything else we can do here. But it could be I misunderstand the constraints/opportunities. Manish, does the above seem correct to you?

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #31) > <dmajor> bholley: I think we should fix it for 57 I agree. I'd rather fix it now rather than wait till after 57 and hence risk destabilising 58+.

> really need to do is allocate using the allocator from std::Vec, As far as I can tell, the allocator used by std::vec is the same one we use in hashglobe::alloc. The <Heap as Alloc> calls down to whatever allocator rustc has decided to use; and in the case of staticlibs this allocator is the system allocator (which we patch). hashglobe::alloc is a clone of the system allocator code.

(In reply to Manish Goregaokar [:manishearth] from comment #35) > > really need to do is allocate using the allocator from std::Vec, > > As far as I can tell, the allocator used by std::vec is the same one we use > in hashglobe::alloc. Thanks for investigating. On that basis, I propose to go ahead with the comment 22 patch, if there are no objections.

(In reply to Julian Seward [:jseward] from comment #36) > (In reply to Manish Goregaokar [:manishearth] from comment #35) > > > really need to do is allocate using the allocator from std::Vec, > > > > As far as I can tell, the allocator used by std::vec is the same one we use > > in hashglobe::alloc. > > Thanks for investigating. On that basis, I propose to go ahead with > the comment 22 patch, if there are no objections. Yes, that generally seems like the right approach, though it should be reviewed carefully. Let's get this in.

win64-asan treeherder results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=82e10284d84a3bda62346ed6d869abf053c2f357&filter-tier=1&filter-tier=2&filter-tier=3 everything-else treeherder results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=36a636ebe9f04b3a53e71adf5862df867bc4f916&filter-tier=1&filter-tier=2&filter-tier=3

This looks fine on try, but fails '(cd servo && ./mach test-unit)' when built for OSX. It doesn't fail similarly on Linux. It fails on every run, but I get two different crash stacks (see attachment). Sometimes the last test successfully completed is keyframes::test_missing_property_in_initial_keyframe and sometimes it is media_queries::test_mq_empty. Does anyone have suggestions on how to proceed? I am out of my depth here. Do the tests run in a fixed order, or are they run non-deterministically by a pool of threads? If the latter, is there a way to make them run in a fixed order? From looking at the stacks, and given that which of the two failure stacks we get seems to be random, I'm inclined to think there's a race somewhere, somehow relating to thread-specific data. One of the stacks has these frames frame #5: 0x00000001009e660a style_tests-d63eda4d3c16b9a3`je_tsd_cleanup_wrapper(arg=0x0000000101e6e300) + 26 at tsd.h:688 [opt] frame #6: 0x00007fffa5643355 libsystem_pthread.dylib`_pthread_tsd_cleanup + 470 frame #7: 0x00007fffa56430d9 libsystem_pthread.dylib`_pthread_exit + 152 (viz, some kind of tsd cleanup at the end of a thread's life), whereas the other stacks have this frame #33: 0x000000010088069b style_tests-d63eda4d3c16b9a3`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:361 [opt] frame #34: 0x000000010088069b style_tests-d63eda4d3c16b9a3`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::thread::{{impl}}::spawn::{{closure}}<closure,()> + 77 at mod.rs:393 [opt] frame #35: 0x000000010088064e style_tests-d63eda4d3c16b9a3`alloc::boxed::{{impl}}::call_box<(),closure> + 46 at boxed.rs:728 [opt] frame #36: 0x00000001009b632c style_tests-d63eda4d3c16b9a3`std::sys::imp::thread::{{impl}}::new::thread_start [inlined] alloc::boxed::{{impl}}::call_once<(),()> + 140 at boxed.rs:738 [opt] frame #37: 0x00000001009b6329 style_tests-d63eda4d3c16b9a3`std::sys::imp::thread::{{impl}}::new::thread_start [inlined] std::sys_common::thread::start_thread + 123 at thread.rs:24 [opt] frame #38: 0x00000001009b62ae style_tests-d63eda4d3c16b9a3`std::sys::imp::thread::{{impl}}::new::thread_start + 14 at thread.rs:90 [opt] frame #39: 0x00007fffa564193b libsystem_pthread.dylib`_pthread_body + 180 frame #40: 0x00007fffa5641887 libsystem_pthread.dylib`_pthread_start + 286 which, given the use of call_once(), seems startup-ish (multiple threads all try to initialise something, and call_once() ensures only one succeeds).

Per jsd's comments on irc, using RUST_TEST_THREADS=1 makes it crash only with the second stack (the shorter one) in comment 39's attachment.

Merged: https://github.com/servo/servo/pull/18661

Is this something we need to consider for Beta uplift or can it ride the 58 train?

Comment on attachment 8911129 [details] [diff] [review] bug1400754-stylo-Win64Asan-2.diff Approval Request Comment [Feature/Bug causing the regression]: Win64-Asan builds will segfault in Stylo without this patch. [User impact if declined]: None, because we are not giving out Win64-Asan builds to users currently. [Is this code covered by automated tests?]: Yes: ./mach try -b o -p win64-asan -u all -t none [Has the fix been verified in Nightly?]: Yes [Needs manual test from QE? If yes, steps to reproduce]: No [List of other uplifts needed for the feature/fix]: None [Is the change risky?]: No [Why is the change risky/not risky?]: Because it only affects Win64-Asan builds. It doesn't affect "normal" builds. [String changes made/needed]: None Additional comments: 1. This is really a followup fix for the main work on adding fallible OOM handling to Stylo (bug 1395064). Per comments above, it's not actually *necessary* for production builds (AFAIK). However, the Stylo infallible allocation machinery is somewhat fragile due to the need to not mismatch allocators. So I would prefer that this gets merged for 57 and baked/verified now, rather than waiting for and potentially destabilising 58. 2. What actually landed (comment 42) contains a last-minute fix relative to this attachment (8911129) and so it is what landed, that needs to be uplifted.

> [User impact if declined]: > None, because we are not giving out Win64-Asan builds to users currently. If I were a release manager, I would immediately decline the patch on this basis. :-) So I'd like to amend this impact statement. Not taking this patch means that 57 will have an allocator mismatch, which runs the risk of subtle and hard-to-debug crashes or memory corruptions, especially when mixed with injected software that tries to hook APIs. This is the reason why I've been pushing for a fix in 57.

Rebased for m-beta.

(In reply to David Major [:dmajor] from comment #45) > So I'd like to amend this impact statement. > Not taking this patch means that 57 will have an allocator mismatch, [..] Yes, agreed. That is a much better impact statement.

Comment on attachment 8915212 [details] [diff] [review] bug1400754-stylo-Win64Asan-8-for-beta.diff Approval Request Comment As described in comments 44, 45 and 47.

Note to release manager(s): we would prefer this to bake on the trunk for a few days. So, if it is approved (which I hope it will be), then can it be deferred to b7 rather than b6 ? I understand b6 will be built tomorrow.

Tracked, let's wait for data from Nightly58 before making a decision on this one, deferred to b7/b8.

Comment on attachment 8915212 [details] [diff] [review] bug1400754-stylo-Win64Asan-8-for-beta.diff Stylo related, Beta57+