In current nightlies on (at least) win32 and win64, we're missing debug info for some panic-ish rust functions. https://bugzilla.mozilla.org/show_bug.cgi?id=1448868#c25 saw this happening for `abort()`. https://bugzilla.mozilla.org/show_bug.cgi?id=1458161#c14 saw this happening for the newly-added `<GeckoHeap as Alloc>::oom()`. I wonder if these functions being divergent means the compiler assumes their debug info doesn't matter? Just a wild guess.

In bug 1456150 acrichto noted that we were missing unwind info for some of those functions, and he landed a fix to the Rust compiler to change that: https://github.com/rust-lang/rust/pull/50263 I'm not sure if that would also change debug symbols or not.

Oh nice! So we could test that by building with nightly rustc? (I don't particularly wanna volunteer, since the in the past, nightly rustc would constantly give me new warnings-as-errors and such.)

Just to see, I grabbed the minidump from this crash report which I got from one of the search queries in bug 1458161 comment 12: https://crash-stats.mozilla.com/report/index/7520ad35-f4d9-4cce-9b14-ab8020180504 crash-stats shows: 0 mozglue.dll mozalloc_abort(char const* const) memory/mozalloc/mozalloc_abort.cpp:34 1 mozglue.dll mozalloc_handle_oom(unsigned int) memory/mozalloc/mozalloc_oom.cpp:54 2 xul.dll xul.dll@0x5fa9f1 3 xul.dll static struct webrender::prim_store::PrimitiveIndex webrender::prim_store::PrimitiveStore::add_primitive(struct euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel>*, struct euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel>*, bool, union core::option::Option<webrender::freelist::FreeListHandle<webrender::clip::ClipStoreMarker>>, union core::option::Option<(u64, u16)>, union webrender::prim_store::PrimitiveContainer) gfx/webrender/src/prim_store.rs:1292 <...> MSVC shows: > mozglue.dll!mozalloc_abort(const char * const msg) Line 34 C++ mozglue.dll!mozalloc_handle_oom(unsigned int size) Line 54 C++ xul.dll!_encoder_encode_from_utf16_without_replacement() Unknown xul.dll!webrender::prim_store::PrimitiveStore::add_primitive(euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel> * self, euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel> * local_rect, bool local_clip_rect, core::option::Option<webrender::freelist::FreeListHandle<webrender::clip::ClipStoreMarker>> clip_sources, core::option::Option<(u64, u16)> tag, webrender::prim_store::PrimitiveContainer container) Line 1292 Unknown xul.dll!webrender::display_list_flattener::DisplayListFlattener::create_primitive(webrender_api::display_item::PrimitiveInfo<webrender_api::units::LayoutPixel> * self, alloc::vec::Vec<webrender::clip::ClipSource> info, webrender::prim_store::PrimitiveContainer clip_sources) Line 879 Unknown xul.dll!webrender::display_list_flattener::DisplayListFlattener::add_primitive(webrender::prim_store::ScrollNodeAndClipChain self, webrender_api::display_item::PrimitiveInfo<webrender_api::units::LayoutPixel> * clip_and_scroll, alloc::vec::Vec<webrender::clip::ClipSource> info, webrender::prim_store::PrimitiveContainer clip_sources) Line 964 Unknown xul.dll!webrender::display_list_flattener::DisplayListFlattener::flatten_item(webrender_api::display_list::DisplayItemRef self, webrender_api::api::PipelineId item, euclid::vector::TypedVector2D<f32, webrender_api::units::LayoutPixel> pipeline_id) Line 686 Unknown xul.dll!webrender::display_list_flattener::DisplayListFlattener::flatten_root(webrender::scene::ScenePipeline * self, euclid::size::TypedSize2D<f32, webrender_api::units::LayoutPixel> * pipeline) Line 342 Unknown xul.dll!webrender::display_list_flattener::DisplayListFlattener::create_frame_builder(webrender::scene::Scene * old_builder, webrender::clip_scroll_tree::ClipScrollTree * scene, alloc::arc::Arc<std::sync::rwlock::RwLock<std::collections::hash::map::HashMap<webrender_api::font::FontInstanceKey, webrender::glyph_rasterizer::FontInstance, core::hash::BuildHasherDefault<fxhash::FxHasher>>>> clip_scroll_tree, std::collections::hash::map::HashMap<webrender_api::image::ImageKey, webrender::resource_cache::ImageTiling, core::hash::BuildHasherDefault<fxhash::FxHasher>> font_instances, webrender::render_backend::DocumentView * tiled_image_map, std::collections::hash::set::HashSet<webrender_api::api::PipelineId, core::hash::BuildHasherDefault<fxhash::FxHasher>> * view, webrender::frame_builder::FrameBuilderConfig * output_pipelines, webrender::scene::Scene * frame_builder_config) Line 254 Unknown xul.dll!std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()>(closure f) Line 133 Unknown xul.dll!alloc::boxed::{{impl}}::call_box<(),closure>(closure * self, ...) Line 815 Unknown xul.dll!std::sys::windows::thread::{{impl}}::new::thread_start() Line 54 Unknown kernel32.dll!@BaseThreadInitThunk@12() Unknown mozglue.dll!patched_BaseThreadInitThunk(int aIsInitialThread, void * aStartAddress, void * aThreadParam) Line 858 C++ ntdll.dll!__RtlUserThreadStart() Unknown ntdll.dll!__RtlUserThreadStart@8() Unknown

Oh, wait, I accidentally opened it in MSVC 2015, but WinDBG gives a way nicer stack: 00 0d7fdab8 734e85d1 mozglue!mozalloc_abort(char * msg = 0x0d7fdad0 "out of memory: 0x000000003A000000 bytes requested")+0x36 [z:\build\build\src\memory\mozalloc\mozalloc_abort.cpp @ 34] 01 0d7fdb08 5e0aa9f2 mozglue!mozalloc_handle_oom(unsigned int size = <Value unavailable error>)+0x61 [z:\build\build\src\memory\mozalloc\mozalloc_oom.cpp @ 54] 02 0d7fdb68 5dffc117 xul!core::char_private::check+0x102 03 (Inline) -------- xul!alloc::vec::Vec<webrender::prim_store::BrushPrimitive>::push+0x46f [C:\projects\rust\src\liballoc\vec.rs @ 971] 04 0d7fdddc 5dffae29 xul!webrender::prim_store::PrimitiveStore::add_primitive(struct webrender::prim_store::PrimitiveStore * self = <Value unavailable error>, struct euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel> * local_rect = <Value unavailable error>, struct euclid::rect::TypedRect<f32, webrender_api::units::LayoutPixel> * local_clip_rect = <Value unavailable error>, union core::option::Option<webrender::freelist::FreeListHandle<webrender::clip::ClipStoreMarker>> clip_sources = <Value unavailable error>, union core::option::Option<(u64, u16)> tag = <Value unavailable error>, union webrender::prim_store::PrimitiveContainer container = <Value unavailable error>)+0xc67 [z:\build\build\src\gfx\webrender\src\prim_store.rs @ 1292] 05 0d7fded8 5dffcf86 xul!webrender::display_list_flattener::DisplayListFlattener::create_primitive(struct webrender::display_list_flattener::DisplayListFlattener * self = <Value unavailable error>, struct webrender_api::display_item::PrimitiveInfo<webrender_api::units::LayoutPixel> * info = <Value unavailable error>, struct alloc::vec::Vec<webrender::clip::ClipSource> clip_sources = <Value unavailable error>, union webrender::prim_store::PrimitiveContainer container = <Value unavailable error>)+0x129 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 879] 06 0d7fe128 5dff40a0 xul!webrender::display_list_flattener::DisplayListFlattener::add_primitive(struct webrender::display_list_flattener::DisplayListFlattener * self = <Value unavailable error>, struct webrender::prim_store::ScrollNodeAndClipChain clip_and_scroll = struct webrender::prim_store::ScrollNodeAndClipChain, struct webrender_api::display_item::PrimitiveInfo<webrender_api::units::LayoutPixel> * info = <Value unavailable error>, struct alloc::vec::Vec<webrender::clip::ClipSource> clip_sources = <Value unavailable error>, union webrender::prim_store::PrimitiveContainer container = <Value unavailable error>)+0xc16 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 964] 07 (Inline) -------- xul!webrender::display_list_flattener::DisplayListFlattener::add_yuv_image+0x1b28 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 2260] 08 0d7fead8 5dfef081 xul!webrender::display_list_flattener::DisplayListFlattener::flatten_item(struct webrender::display_list_flattener::DisplayListFlattener * self = <Value unavailable error>, struct webrender_api::display_list::DisplayItemRef item = <Value unavailable error>, struct webrender_api::api::PipelineId pipeline_id = <Value unavailable error>, struct euclid::vector::TypedVector2D<f32, webrender_api::units::LayoutPixel> reference_frame_relative_offset = <Value unavailable error>)+0x32a0 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 686] 09 (Inline) -------- xul!webrender::display_list_flattener::DisplayListFlattener::flatten_items+0xc7 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 391] 0a 0d7feeac 5dfea08a xul!webrender::display_list_flattener::DisplayListFlattener::flatten_root(struct webrender::display_list_flattener::DisplayListFlattener * self = <Value unavailable error>, struct webrender::scene::ScenePipeline * pipeline = <Value unavailable error>, struct euclid::size::TypedSize2D<f32, webrender_api::units::LayoutPixel> * frame_size = <Value unavailable error>)+0x581 [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 342] 0b 0d7ff298 5e008414 xul!webrender::display_list_flattener::DisplayListFlattener::create_frame_builder(struct webrender::frame_builder::FrameBuilder old_builder = <Value unavailable error>, struct webrender::scene::Scene * scene = <Value unavailable error>, struct webrender::clip_scroll_tree::ClipScrollTree * clip_scroll_tree = <Value unavailable error>, struct alloc::arc::Arc<std::sync::rwlock::RwLock<std::collections::hash::map::HashMap<webrender_api::font::FontInstanceKey, webrender::glyph_rasterizer::FontInstance, core::hash::BuildHasherDefault<fxhash::FxHasher>>>> font_instances = <Value unavailable error>, struct std::collections::hash::map::HashMap<webrender_api::image::ImageKey, webrender::resource_cache::ImageTiling, core::hash::BuildHasherDefault<fxhash::FxHasher>> tiled_image_map = <Value unavailable error>, struct webrender::render_backend::DocumentView * view = <Value unavailable error>, struct std::collections::hash::set::HashSet<webrender_api::api::PipelineId, core::hash::BuildHasherDefault<fxhash::FxHasher>> * output_pipelines = <Value unavailable error>, struct webrender::frame_builder::FrameBuilderConfig * frame_builder_config = <Value unavailable error>, struct webrender::scene::Scene * new_scene = <Value unavailable error>)+0x10ea [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 254] 0c (Inline) -------- xul!webrender::display_list_flattener::build_scene+0x3be [z:\build\build\src\gfx\webrender\src\display_list_flattener.rs @ 2270] 0d (Inline) -------- xul!webrender::scene_builder::{{impl}}::process_message::{{closure}}+0x3da [z:\build\build\src\gfx\webrender\src\scene_builder.rs @ 141] 0e (Inline) -------- xul!core::option::Option<webrender::scene_builder::SceneRequest>::map+0x444 [C:\projects\rust\src\libcore\option.rs @ 404] 0f (Inline) -------- xul!webrender::scene_builder::SceneBuilder::process_message+0x552 [z:\build\build\src\gfx\webrender\src\scene_builder.rs @ 140] 10 (Inline) -------- xul!webrender::scene_builder::SceneBuilder::run+0x13b2 [z:\build\build\src\gfx\webrender\src\scene_builder.rs @ 107] 11 (Inline) -------- xul!webrender::renderer::{{impl}}::new::{{closure}}+0x1456 [z:\build\build\src\gfx\webrender\src\renderer.rs @ 1676] 12 0d7ffaa4 5e006eb0 xul!std::sys_common::backtrace::__rust_begin_short_backtrace<closure,(struct closure f = <Value unavailable error>)+0x14d4 [C:\projects\rust\src\libstd\sys_common\backtrace.rs @ 133] 13 (Inline) -------- xul!std::thread::{{impl}}::spawn::{{closure}}::{{closure}}+0x54 [C:\projects\rust\src\libstd\thread\mod.rs @ 406] 14 (Inline) -------- xul!std::panic::{{impl}}::call_once+0x54 [C:\projects\rust\src\libstd\panic.rs @ 300] 15 (Inline) -------- xul!std::panicking::try::do_call+0x54 [C:\projects\rust\src\libstd\panicking.rs @ 480] 16 (Inline) -------- xul!panic_abort::__rust_maybe_catch_panic+0x54 [C:\projects\rust\src\libpanic_abort\lib.rs @ 38] 17 (Inline) -------- xul!std::panicking::try+0x54 [C:\projects\rust\src\libstd\panicking.rs @ 459] 18 (Inline) -------- xul!std::panic::catch_unwind+0x54 [C:\projects\rust\src\libstd\panic.rs @ 365] 19 (Inline) -------- xul!std::thread::{{impl}}::spawn::{{closure}}+0x85 [C:\projects\rust\src\libstd\thread\mod.rs @ 405] 1a 0d7ffb30 5e090ac2 xul!alloc::boxed::{{impl}}::call_box<(struct closure * self = 0x00400000)+0xe0 [C:\projects\rust\src\liballoc\boxed.rs @ 815] 1b (Inline) -------- xul!alloc::boxed::{{impl}}::call_once+0x5 [C:\projects\rust\src\liballoc\boxed.rs @ 825] 1c (Inline) -------- xul!std::sys_common::thread::start_thread+0x5b [C:\projects\rust\src\libstd\sys_common\thread.rs @ 24] 1d 0d7ffb4c 75727c04 xul!std::sys::windows::thread::{{impl}}::new::thread_start(void)+0x62 [C:\projects\rust\src\libstd\sys\windows\thread.rs @ 54] 1e 0d7ffb60 734de82d kernel32!BaseThreadInitThunk+0x24 1f 0d7ffb7c 77c6ad8f mozglue!patched_BaseThreadInitThunk(int aIsInitialThread = 0n4194304, void * aStartAddress = 0x1ea00000, void * aThreadParam = 0x0f4773b0)+0x3d [z:\build\build\src\mozglue\build\windowsdllblocklist.cpp @ 858] 20 0d7ffbc4 77c6ad5a ntdll!__RtlUserThreadStart+0x2f 21 0d7ffbd4 00000000 ntdll!_RtlUserThreadStart+0x1b

...that's not very readable, let's try this: https://gist.github.com/luser/173e3df9e845379aa4ff2e8d13b341dd Looking at it again, I note that the third frame that WinDBG prints is: 02 0d7fdb68 5dffc117 xul!core::char_private::check+0x102 which seems pretty bogus.

(In reply to David Major [:dmajor] from comment #2) > Oh nice! So we could test that by building with nightly rustc? (I don't > particularly wanna volunteer, since the in the past, nightly rustc would > constantly give me new warnings-as-errors and such.) I had a try push using nightly rust in bug 1456150: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0c74764b3782049da42762da8a90fbb12f0edcc8 I rebased it atop glandium's OOM handler changes and it broke because I have a hack in there to replace the rust `intentional_panic` function with a call to `std::alloc::oom()` and I screwed things up rebasing it atop glandium's changes: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a94d67b738da6a1a3ccbf63e3f95b68e7b4be19b The first one built OK so you ought to be able to use that to check? Alternately you could drop the .rs change from the patch and push it to try again (this might be nice only because now you can get the symbols on the symbol server).

Ted was that a crash report containing code compiled by a rustc which contained https://github.com/rust-lang/rust/pull/50263? In that did that PR for sure not fix the issue?

Alex: no, that crash report is from a nightly Firefox, so it's still using Rust 1.24 (or possibly 1.25 if it was in the timeframe where that was landed before it got backed out).

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5) > Looking at it again, I note that the third frame that WinDBG prints is: > 02 0d7fdb68 5dffc117 xul!core::char_private::check+0x102 > > which seems pretty bogus. Yeah, `char_private::check` is the same thing that I see when the debug info is bogus. I got that in both bug 1448868 bug 1458161.

move to new Rust/C++ component when created

char_private::check just happens to be the function preceding the one without debug info. Its size is 0xe7, so 0x102 is clearly after it.

(In reply to Alex Crichton [:acrichto] from comment #7) > Ted was that a crash report containing code compiled by a rustc which > contained https://github.com/rust-lang/rust/pull/50263? In that did that PR > for sure not fix the issue? Currently nightly doesn't generate debug info on any platform for the rust_oom symbol, but that might be because it's not an allocator function anymore, it's a language item.

I'm going to file an upstream issue with a reproducer on linux.

Hum, no, actually, while __rust_oom had missing debug info in < 1.27, rust_oom (symbol name changed) *does* have debug info in current nightly.

David, can you check this build? https://treeherder.mozilla.org/#/jobs?repo=try&revision=747ff339383de4785819b1af6360cc1a4455acea (last rust nightly + bug 1465709)

That build still has the bogus char_private::check name: xul!alloc::raw_vec::RawVec<rsdparsa::SdpLine, alloc::heap::Heap>::double<rsdparsa::SdpLine,alloc::heap::Heap>+0x8b [C:\projects\rust\src\liballoc\raw_vec.rs @ 326]: 326 00000001`8043858b 48c745f808000000 mov qword ptr [rbp-8],8 326 00000001`80438593 488d4de8 lea rcx,[rbp-18h] 326 00000001`80438597 e894471d00 call xul!core::char_private::check+0x130 (00000001`8060cd30) 326 00000001`8043859c 0f0b ud2 0:000> u 01`8060cd30 xul!core::char_private::check+0x130: 00000001`8060cd30 4881ec88000000 sub rsp,88h 00000001`8060cd37 4889c8 mov rax,rcx 00000001`8060cd3a 488b10 mov rdx,qword ptr [rax] 00000001`8060cd3d 488b4808 mov rcx,qword ptr [rax+8] 00000001`8060cd41 4885d2 test rdx,rdx 00000001`8060cd44 7507 jne xul!core::char_private::check+0x14d (00000001`8060cd4d) 00000001`8060cd46 e851ea6e02 call xul!GeckoHandleOOM (00000001`82cfb79c) 00000001`8060cd4b 0f0b ud2

(In reply to Alex Crichton [:acrichto] from comment #7) > Ted was that a crash report containing code compiled by a rustc which > contained https://github.com/rust-lang/rust/pull/50263? In that did that PR > for sure not fix the issue? So the answer to that question is now clearly no.

Sorry for the delay, but bummer! I'm unfortunately not entirely sure how to investigate this myself then :( Is there perhaps a smaller example which shows the same bad behavior which I can poke around?