Filed by: apavel [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=214148465&repo=autoland https://queue.taskcluster.net/v1/task/AHryWODGSz618DNAoHgHdA/runs/1/artifacts/public/logs/live_backing.log 16:52:41 INFO - closing window: 49392123905 16:52:41 INFO - closing window: 51539607553 16:52:42 INFO - closing window: 6442451084 16:52:42 INFO - closing window: 38654705668 16:52:42 INFO - closing window: 40802189316 16:52:42 INFO - closing window: 42949672964 16:52:42 INFO - closing window: 45097156613 16:52:42 INFO - closing window: 47244640260 16:52:42 INFO - closing window: 49392123908 16:52:42 INFO - closing window: 51539607556 16:52:42 INFO - closing window: 6442451087 16:52:42 INFO - closing window: 38654705671 16:52:42 INFO - closing window: 40802189333 16:52:42 INFO - closing window: 42949672967 16:52:42 INFO - closing window: 45097156618 16:52:42 INFO - closing window: 47244640309 16:52:42 INFO - closing window: 49392123912 16:52:42 INFO - closing window: 51539607559 16:52:42 INFO - closing window: 6442451090 16:52:43 INFO - closing window: 38654705688 16:52:43 INFO - closing window: 40802189336 16:52:43 INFO - closing window: 42949672970 16:52:43 INFO - closing window: 45097156637 16:52:43 INFO - closing window: 47244640316 16:52:43 INFO - navigating to about:blank 16:52:43 INFO - navigated to about:blank 16:52:43 INFO - starting checkpoint TabsClosedExtraProcesses... 16:52:44 INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedExtraProcesses-2.json.gz 16:52:44 INFO - closing preloaded browser 16:52:44 INFO - starting checkpoint TabsClosed... 16:52:44 INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosed-2.json.gz 16:53:14 INFO - starting checkpoint TabsClosedSettled... 16:53:14 INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedSettled-2.json.gz 16:53:14 INFO - starting gc... 16:53:15 INFO - gc done! 16:53:15 INFO - starting checkpoint TabsClosedForceGC... 16:53:15 INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedForceGC-2.json.gz 16:53:15 INFO - setting results 16:53:15 INFO - tearing down! 16:53:15 INFO - tearing down webservers! 16:53:40 INFO - processing data in Z:\task_1543333681\build\tests\results! 16:53:45 INFO - PERFHERDER_DATA: {"framework": {"name": "awsy"}, "suites": [{"extraOptions": ["stylo"], "name": "Resident Memory", "lowerIsBetter": true, "value": 588044763.2862101, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 334770176}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 293023744}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 1466859520}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 1456590848}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 990027776}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 472899584}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 472961024}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 439386112}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 412315648}]}, {"extraOptions": ["stylo"], "name": "Explicit Memory", "lowerIsBetter": true, "value": 373957347.8124979, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 217794071}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 166168087}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 1136501248}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 1114325504}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 754332160}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 277144064}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 277500416}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 244990464}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 219554304}]}, {"extraOptions": ["stylo"], "name": "Heap Unclassified", "lowerIsBetter": true, "value": 77751137.45692353, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 46842352}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 32905520}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 242183872}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 238676768}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 121183200}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 58635184}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 58318976}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 54671568}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 51442944}]}, {"extraOptions": ["stylo"], "name": "JS", "lowerIsBetter": true, "value": 116079035.9332693, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 104560400}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 62133840}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 255522128}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 242813408}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 209321888}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 92966432}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 93576464}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 75565376}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 68985408}]}, {"extraOptions": ["stylo"], "name": "Images", "lowerIsBetter": true, "value": 7259779.581239595, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 1034192}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 1034192}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 78198272}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 67746528}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 32901056}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 6656944}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 6656944}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 3179200}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 2132784}]}]} 16:53:45 INFO - Perfherder data written to Z:\task_1543333681\build\tests\results\perfherder_data.json 16:53:45 INFO - done tearing down! 16:53:45 INFO - TEST-PASS | awsy\test_memory_usage.py TestMemoryUsage.test_open_tabs | took 3569640ms 16:53:45 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cBAzpjbJQouxYUy_fpQ6iQ/artifacts/public/build/target.crashreporter-symbols.zip 16:53:50 INFO - mozcrash Copy/paste: Z:\task_1543333681\build\win32-minidump_stackwalk.exe c:\users\task_1543333681\appdata\local\temp\tmpbq0yvh.profile\minidumps\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp c:\users\task_1543333681\appdata\local\temp\tmpb2olby 16:53:52 INFO - mozcrash Saved minidump as Z:\task_1543333681\build\blobber_upload_dir\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp 16:53:52 INFO - mozcrash Saved app info as Z:\task_1543333681\build\blobber_upload_dir\9667aee6-f2da-4e97-8a41-de5a3d6f851d.extra 16:53:52 INFO - PROCESS-CRASH | marionette.py | application crashed [@ xul.dll + 0x3b066a] 16:53:52 INFO - Crash dump filename: c:\users\task_1543333681\appdata\local\temp\tmpbq0yvh.profile\minidumps\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp 16:53:52 INFO - Operating system: Windows NT 16:53:52 INFO - 10.0.15063 16:53:52 INFO - CPU: amd64 16:53:52 INFO - family 6 model 79 stepping 1 16:53:52 INFO - 16 CPUs 16:53:52 INFO - 16:53:52 INFO - GPU: UNKNOWN 16:53:52 INFO - 16:53:52 INFO - Crash reason: EXCEPTION_BREAKPOINT 16:53:52 INFO - Crash address: 0x7ff81fb1066a 16:53:52 INFO - Assertion: Unknown assertion type 0x00000000 16:53:52 INFO - Process uptime: 623 seconds 16:53:52 INFO - 16:53:52 INFO - Thread 33 (crashed) 16:53:52 INFO - 0 xul.dll + 0x3b066a 16:53:52 INFO - rax = 0x00007ff83e97fa88 rdx = 0x0000007013b38e92 16:53:52 INFO - rcx = 0x00000000000003e8 rbx = 0x00007ff824228e6d 16:53:52 INFO - rsi = 0x0000000000000011 rdi = 0x00000000000003e8 16:53:52 INFO - rbp = 0x0000007013b39438 rsp = 0x0000007013b38e10 16:53:52 INFO - r8 = 0x0000007013b38e92 r9 = 0x0000000000000000 16:53:52 INFO - r10 = 0x0000007013b38850 r11 = 0x0000000000000011 16:53:52 INFO - r12 = 0x00007ff82422a690 r13 = 0x00007ff824223d30 16:53:52 INFO - r14 = 0x0000007013b396a0 r15 = 0x0000000000000001 16:53:52 INFO - rip = 0x00007ff81fb1066a 16:53:52 INFO - Found by: given as instruction pointer in context 16:53:52 INFO - 1 0x21c4b96b5a0 16:53:52 INFO - rbp = 0x0000007013b39538 rsp = 0x0000007013b39448 16:53:52 INFO - rip = 0x0000021c4b96b5a0 16:53:52 INFO - Found by: previous frame's frame pointer 16:53:52 INFO - 2 0x1 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39548 16:53:52 INFO - rip = 0x0000000000000001 16:53:52 INFO - Found by: previous frame's frame pointer 16:53:52 INFO - 3 mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 2988 + 0x9] 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39580 16:53:52 INFO - rip = 0x00007ff83e951b71 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 4 mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 4192 + 0x23] 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b395f0 16:53:52 INFO - rip = 0x00007ff83e952886 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 5 mozglue.dll!Allocator<MozJemallocBase>::malloc(unsigned __int64) [malloc_decls.h:ad28397dfa43589ea5bd814391c26386c5bf751f : 37 + 0x18] 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39640 16:53:52 INFO - rip = 0x00007ff83e94e28d 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 6 mozglue.dll!arena_t::RallocSmallOrLarge(void *,unsigned __int64,unsigned __int64) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 3660 + 0x17] 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39730 16:53:52 INFO - rip = 0x00007ff83e94d850 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 7 mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 2988 + 0x9] 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39740 16:53:52 INFO - rip = 0x00007ff83e951b71 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 8 ntdll.dll!RtlWakeAllConditionVariable + 0x61 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39770 16:53:52 INFO - rip = 0x00007ff8527960f1 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 9 0x21c48d12490 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b397a0 16:53:52 INFO - rip = 0x0000021c48d12490 16:53:52 INFO - Found by: call frame info 16:53:52 INFO - 10 KERNELBASE.dll!Normalization__LoadTables + 0x94 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39818 16:53:52 INFO - rip = 0x00007ff84ed6e910 16:53:52 INFO - Found by: stack scanning 16:53:52 INFO - 11 0x7ff800000000 16:53:52 INFO - rbp = 0x00007ff824229800 rsp = 0x0000007013b39848 16:53:52 INFO - rip = 0x00007ff800000000 16:53:52 INFO - Found by: call frame info

This occurred when bug 1506546 landed: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Cx64%2Cquantumrender%2Copt%2Care%2Cwe%2Cslim%2Cyet%2Ctests%2Cby%2Ctaskcluster%2Cwith%2Ce10s%2Ctest-windows10-64-qr%2Fopt-awsy-e10s%2Csy-e10s%28sy%29&fromchange=93fccab4f951f154e61e87081ceae912d16db025&tochange=6b2aceb0979a49dbcce80db1b59ef8238bd0d2b8 Alexandre can you please take a look?

I'm more suspicious of bug 1509974, which landed two commits earlier. I did some more retriggers to check.

Confirmed with retriggers that this was bug 1509974. I'm looking at this with glandium. Stack is garbage but with some minidump analysis he's confirmed it's a rust panic.

I've tried triggering the same job with debug builds to see if we do a better job of either walking the stack or printing the panic message. https://treeherder.mozilla.org/#/jobs?repo=try&revision=246bc97e2a5926e4bd0c43bde3ea562e93df262f

The panic message is "no entry found for key", which corresponds to a hashmap[foo] operations. Likely source file option.rs.

The crashing thread name is "WRSceneBuilder#2" on the crash I looked at manually.

This might be the location that panics: xul.dll!static void webrender::prim_store::PrimitiveStore::prepare_interned_prim_for_render(struct webrender::prim_store::PrimitiveInstance *, struct webrender::frame_builder::PrimitiveContext *, struct webrender::frame_builder::PictureContext *, struct webrender::frame_builder::FrameBuildingContext *, struct webrender::frame_builder::FrameBuildingState *, struct webrender::render_backend::FrameResources *, struct webrender::prim_store::PrimitiveScratchBuffer *) [https://hg.mozilla.org/try/annotate/c77072391f95afd0c07fbb9b0c92028a6cf44f3d/gfx/wr/webrender/src/prim_store.rs#l3469 + 130]

So, the good news is that by running minidump_stackwalk manually on one of the crashes, I actually *do* a stack trace that looks good: Thread 33 (crashed) 0 xul.dll!static void MOZ_CrashOOL(const char *, int, const char *) [Assertions.h:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 311 + 0x0] rax = 0x00007ffb30fdfa88 rdx = 0x0000008e4d638c52 rcx = 0x00000000000003e8 rbx = 0x00007ffb14da6e6d rsi = 0x0000000000000011 rdi = 0x00000000000003e8 rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d638bd0 r8 = 0x0000008e4d638c52 r9 = 0x0000000000000000 r10 = 0x0000008e4d638610 r11 = 0x0000000000000011 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb1069073a Found by: given as instruction pointer in context 1 xul.dll!GeckoCrashOOL [nsAppRunner.cpp:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 5350 + 0xa] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d638c00 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb137405de Found by: call frame info 2 xul.dll!static void gkrust_shared::panic_hook(struct core::panic::PanicInfo *) [lib.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 234 + 0x7] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d638c30 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13b5019b Found by: call frame info 3 xul.dll!static void core::ops::function::Fn::call<fn(core::panic::PanicInfo*),(core::panic::PanicInfo*)>( * *, struct core::panic::PanicInfo *) [function.rs : 78 + 0xc] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639080 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13b500ec Found by: call frame info 4 xul.dll!static void std::panicking::rust_panic_with_hook() [panicking.rs : 481 + 0x6] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d6390b0 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb14206df1 Found by: call frame info 5 xul.dll!static void std::panicking::continue_panic_fmt() [panicking.rs : 391 + 0x16] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d6391c0 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb142096e1 Found by: call frame info 6 xul.dll!static void std::panicking::rust_begin_panic() [panicking.rs : 326 + 0x5] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639220 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb14209689 Found by: call frame info 7 xul.dll!static void core::panicking::panic_fmt() [panicking.rs : 77 + 0x5] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639250 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb1422656c Found by: call frame info 8 xul.dll!static void core::option::expect_failed() [option.rs : 1000 + 0x11] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d6392b0 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb1421c562 Found by: call frame info 9 xul.dll!static union core::option::Option<webrender_bindings::moz2d_renderer::Job> core::iter::{{impl}}::next<webrender_bindings::moz2d_renderer::Job,core::slice::Iter<webrender_api::image::BlobImageParams>,closure>(struct core::iter::Map<core::slice::Iter<webrender_api::image::BlobImageParams>, closure> *) [mod.rs : 1394 + 0xb8] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639330 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13cc49d9 Found by: call frame info 10 xul.dll!static struct alloc::vec::Vec<(webrender_api::image::BlobImageRequest, core::result::Result<webrender_api::image::RasterizedBlobImage, webrender_api::image::BlobImageError>)> webrender_bindings::moz2d_renderer::{{impl}}::rasterize(struct webrender_bindings::moz2d_renderer::Moz2dBlobRasterizer *, struct slice<webrender_api::image::BlobImageParams>*, bool) [moz2d_renderer.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 474 + 0x2f] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639440 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13cc43b5 Found by: call frame info 11 xul.dll!static struct webrender::tiling::Frame webrender::frame_builder::FrameBuilder::build(struct webrender::resource_cache::ResourceCache *, struct webrender::gpu_cache::GpuCache *, struct webrender::render_backend::FrameStamp, struct webrender::clip_scroll_tree::ClipScrollTree *, struct std::collections::hash::map::HashMap<webrender_api::api::PipelineId, alloc::sync::Arc<webrender::scene::ScenePipeline>, core::hash::BuildHasherDefault<fxhash::FxHasher>> *, struct euclid::scale::TypedScale<f32, webrender_api::units::WorldPixel, webrender_api::units::DevicePixel>, char, struct euclid::point::TypedPoint2D<f32, webrender_api::units::WorldPixel>, struct webrender::profiler::TextureCacheProfileCounters *, struct webrender::profiler::GpuCacheProfileCounters *, struct webrender::scene::SceneProperties *, struct webrender::render_backend::FrameResources *, struct webrender::prim_store::PrimitiveScratchBuffer *) [frame_builder.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 375 + 0x2645] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d639600 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13da46c0 Found by: call frame info 12 xul.dll!static struct webrender::internal_types::RenderedDocument webrender::render_backend::Document::build_frame(struct webrender::resource_cache::ResourceCache *, struct webrender::gpu_cache::GpuCache *, struct webrender::profiler::ResourceProfileCounters *) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 418 + 0x4a] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63a840 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d82f2c Found by: call frame info 13 xul.dll!static void webrender::render_backend::RenderBackend::update_document(struct webrender_api::api::DocumentId, struct alloc::vec::Vec<webrender_api::api::ResourceUpdate>, union core::option::Option<webrender::scene_builder::DocumentResourceUpdates>, struct alloc::vec::Vec<webrender_api::api::FrameMsg>, struct alloc::vec::Vec<webrender_api::api::NotificationRequest>, bool, bool, unsigned int *, struct webrender::profiler::BackendProfileCounters *, bool) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 1219 + 0x23] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63a960 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d5dc68 Found by: call frame info 14 xul.dll!static bool webrender::render_backend::RenderBackend::process_api_msg(union webrender_api::api::ApiMsg, struct webrender::profiler::BackendProfileCounters *, unsigned int *) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 1032 + 0x13d] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63bce0 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d695f7 Found by: call frame info 15 xul.dll!static void webrender::render_backend::RenderBackend::run(struct webrender::profiler::BackendProfileCounters) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 807 + 0xa1] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63d820 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d4ce22 Found by: call frame info 16 xul.dll!static void std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()>(struct closure) [backtrace.rs : 136 + 0xdf7] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63e560 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d49447 Found by: call frame info 17 xul.dll!static void alloc::boxed::{{impl}}::call_box<(),closure>(struct closure *, <NoType>) [boxed.rs : 646 + 0x5b] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63f470 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb13d4856d Found by: call frame info 18 xul.dll!static void std::sys::windows::thread::{{impl}}::new::thread_start() [thread.rs : 56 + 0x64] rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63fb90 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb1420f20d Found by: call frame info 19 kernel32.dll + 0x12774 rbx = 0x00007ffb14da6e6d rbp = 0x0000008e4d6391f8 rsp = 0x0000008e4d63fbd0 r12 = 0x00007ffb14da8690 r13 = 0x00007ffb14da1d30 r14 = 0x0000008e4d639460 r15 = 0x0000000000000001 rip = 0x00007ffb48122774 Found by: call frame info This suggests the AWSY harness is not getting or using the crash symbols correctly.

I also got stacks by forcing AWSY to run with a debug build [1]. According to aosmond, this should be fixed by bug 1492241. Thanks for helping track this down glandium! [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=246bc97e2a5926e4bd0c43bde3ea562e93df262f

Looks like this just hit central, doing some AWSY retriggers: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=qr%2Csy&selectedJob=215770799&revision=c6d5f17c0c813ce6b7d7cf959aa5377bbe249652

Thanks for looking into this Bobby, sorry i did not continue with retriggers and you had to.

Looks fixed \o/ (In reply to Andreea Pavel [:apavel] from comment #14) > Thanks for looking into this Bobby, sorry i did not continue with retriggers > and you had to. No worries!

Bobby, this is not fixed: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&group_state=expanded&fromchange=82c890b2e1ec940404e5789044d2bcc1b9ad2cd9&selectedJob=215854334 Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215854334&repo=autoland&lineNumber=2289 00:32:09 INFO - TEST-PASS | awsy\test_memory_usage.py TestMemoryUsage.test_open_tabs | took 3488060ms 00:32:09 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/CqGiHo1bTL-eXGWRAnPpzA/artifacts/public/build/target.crashreporter-symbols.zip 00:32:13 INFO - mozcrash Copy/paste: Z:\task_1544138790\build\win32-minidump_stackwalk.exe c:\users\task_1544138790\appdata\local\temp\tmp_xqtyh.profile\minidumps\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp c:\users\task_1544138790\appdata\local\temp\tmpbp4tbd 00:32:14 INFO - mozcrash Saved minidump as Z:\task_1544138790\build\blobber_upload_dir\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp 00:32:14 INFO - mozcrash Saved app info as Z:\task_1544138790\build\blobber_upload_dir\3108ad40-f986-4b97-a90d-34359f7b54c3.extra 00:32:14 INFO - PROCESS-CRASH | marionette.py | application crashed [@ xul.dll + 0x3b196a] 00:32:14 INFO - Crash dump filename: c:\users\task_1544138790\appdata\local\temp\tmp_xqtyh.profile\minidumps\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp 00:32:14 INFO - Operating system: Windows NT 00:32:14 INFO - 10.0.15063 00:32:14 INFO - CPU: amd64 00:32:14 INFO - family 6 model 79 stepping 1 00:32:14 INFO - 16 CPUs 00:32:14 INFO - 00:32:14 INFO - GPU: UNKNOWN 00:32:14 INFO - 00:32:14 INFO - Crash reason: EXCEPTION_BREAKPOINT 00:32:14 INFO - Crash address: 0x7ffbfc30196a 00:32:14 INFO - Assertion: Unknown assertion type 0x00000000 00:32:14 INFO - Process uptime: 3428 seconds 00:32:14 INFO - 00:32:14 INFO - Thread 31 (crashed) 00:32:14 INFO - 0 xul.dll + 0x3b196a 00:32:14 INFO - rax = 0x00007ffc216cfa88 rdx = 0x000000ada41b7e32 00:32:14 INFO - rcx = 0x00000000000003e8 rbx = 0x00007ffc00b1c91d 00:32:14 INFO - rsi = 0x0000000000000011 rdi = 0x00000000000003e8 00:32:14 INFO - rbp = 0x000000ada41b83d8 rsp = 0x000000ada41b7db0 00:32:14 INFO - r8 = 0x000000ada41b7e32 r9 = 0x0000000000000000 00:32:14 INFO - r10 = 0x000000ada41b77f0 r11 = 0x0000000000000011 00:32:14 INFO - r12 = 0x00007ffc00b1e140 r13 = 0x00007ffc00b177e0 00:32:14 INFO - r14 = 0x000000ada41b8640 r15 = 0x0000000000000001 00:32:14 INFO - rip = 0x00007ffbfc30196a 00:32:14 INFO - Found by: given as instruction pointer in context 00:32:14 INFO - 1 0x23b1fafd7a0 00:32:14 INFO - rbp = 0x000000ada41b84d8 rsp = 0x000000ada41b83e8 00:32:14 INFO - rip = 0x0000023b1fafd7a0 00:32:14 INFO - Found by: previous frame's frame pointer 00:32:14 INFO - 2 0x1 00:32:14 INFO - rbp = 0x00007ffc00b1d2b0 rsp = 0x000000ada41b84e8 00:32:14 INFO - rip = 0x0000000000000001 00:32:14 INFO - Found by: previous frame's frame pointer 00:32:14 INFO - 3 mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:3347058499c706d9651ad1cfa3aaae3c1d45bbe7 : 2804 + 0x9] 00:32:14 INFO - rbp = 0x00007ffc00b1d2b0 rsp = 0x000000ada41b8520 00:32:14 INFO - rip = 0x00007ffc216a1b71 00:32:14 INFO - Found by: stack scanning 00:32:14 INFO - 4 mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:3347058499c706d9651ad1cfa3aaae3c1d45bbe7 : 3935 + 0x23] 00:32:14 INFO - rbp = 0x00007ffc00b1d2b0 rsp = 0x000000ada41b8590 00:32:14 INFO - rip = 0x00007ffc216a2846 00:32:14 INFO - Found by: stack scanning

Doing another try push to run AWSY on a debug build to get stacks: https://treeherder.mozilla.org/#/jobs?repo=try&revision=93270b5cea3ead7db5d71ce4967679ab3d8cbf69

(In reply to Bobby Holley (:bholley) from comment #17) > Doing another try push to run AWSY on a debug build to get stacks: > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=93270b5cea3ead7db5d71ce4967679ab3d8cbf69 Andrew, see above. Looks like we're still crashing even with the fix from bug 1512156.

Sounds like https://github.com/servo/webrender/pull/3398 might help here.

(In reply to Bobby Holley (:bholley) from comment #20) > Sounds like https://github.com/servo/webrender/pull/3398 might help here. Yes -- sorry meant to follow up.

(In reply to Bobby Holley (:bholley) from comment #20) > Sounds like https://github.com/servo/webrender/pull/3398 might help here. So far so good, I don't see any failures after [1]. Andreea, can you double-check and close this bug if so? [1] https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=sy&revision=5d24ac329431289a8966eabc3325228dfd7f2c4d

This looks fixed based on the TH intermittent-failures listings. Feel free to reopen if it comes back.