Intermittent tier 2 mozrunner-startup | application crashed [@ mozilla::gl::GLContext::BeforeGLCall(char const*) const] - WPT on Geckoview crashes in /webvtt/rendering/cues-with-video/processing-model/
Categories
(Core :: Graphics: CanvasWebGL, defect, P1)
Tracking
()
People
(Reporter: intermittent-bug-filer, Assigned: impossibus)
References
(Blocks 2 open bugs)
Details
(Keywords: crash, intermittent-failure, regression, Whiteboard: [stockwell disabled])
Crash Data
Attachments
(1 file)
(deleted),
text/x-phabricator-request
|
Details |
Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=247486347&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/OFlPiuOSSx-TO1ej7GXetQ/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/OFlPiuOSSx-TO1ej7GXetQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
[task 2019-05-21T06:11:09.256Z] 06:11:09 INFO - TEST-START | /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_completely_move_up.html
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - Traceback (most recent call last):
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/base.py", line 192, in run_test
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - result = self.do_test(test)
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 808, in do_test
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - result = self.implementation.run_test(test)
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 881, in run_test
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - "height": 600})["value"]
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - m._handle_socket_failure()
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 650, in _handle_socket_failure
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - reraise(exc, val, tb)
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 26, in _
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - return func(args, kwargs)
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 590, in _send_message
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - msg = self.client.request(name, params)
[task 2019-05-21T06:11:10.541Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 273, in request
[task 2019-05-21T06:11:10.542Z] 06:11:10 WARNING - return self.receive()
[task 2019-05-21T06:11:10.542Z] 06:11:10 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 159, in receive
[task 2019-05-21T06:11:10.542Z] 06:11:10 WARNING - raise socket.error("No data received over socket")
[task 2019-05-21T06:11:10.542Z] 06:11:10 WARNING - error: No data received over socket
[task 2019-05-21T06:11:10.542Z] 06:11:10 WARNING -
[task 2019-05-21T06:11:10.960Z] 06:11:10 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpKGitND/456aef76-f2e8-265b-24b6-b5551516936a.dmp /builds/worker/workspace/build/symbols
[task 2019-05-21T06:11:15.367Z] 06:11:15 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/456aef76-f2e8-265b-24b6-b5551516936a.dmp
[task 2019-05-21T06:11:15.367Z] 06:11:15 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/456aef76-f2e8-265b-24b6-b5551516936a.extra
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - PROCESS-CRASH | mozrunner-startup | application crashed [@ mozilla::gl::GLContext::BeforeGLCall(char const) const]
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Crash dump filename: /tmp/tmpKGitND/456aef76-f2e8-265b-24b6-b5551516936a.dmp
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Operating system: Android
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - 0.0.0 Linux 3.10.0+ #1 PREEMPT Thu Jan 5 00:46:30 UTC 2017 x86_64
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - CPU: amd64
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - family 6 model 2 stepping 3
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - 1 CPU
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO -
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - GPU: UNKNOWN
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO -
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Crash address: 0x0
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Process uptime: not available
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO -
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - Thread 29 (crashed)
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - 0 libxul.so!mozilla::gl::GLContext::BeforeGLCall(char const) const [GLContext.h:ca560ff55451aafb3dae3f679d09206b120b38eb : 662 + 0x29]
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - rax = 0x00007ef160662151 rdx = 0x0000000000000004
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - rcx = 0x00007ef163aada88 rbx = 0x00007ef152df9000
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - rsi = 0x00007ef152ffe440 rdi = 0x000000000000001b
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - rbp = 0x00007ef152ffeb10 rsp = 0x00007ef152ffeaf0
[task 2019-05-21T06:11:15.533Z] 06:11:15 INFO - r8 = 0x0000000000000000 r9 = 0x00007ef18020b090
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r10 = 0x0000000000000022 r11 = 0x0000000000000246
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r12 = 0x0000000000000303 r13 = 0x0000000000000001
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r14 = 0x00007ef16067adbb r15 = 0x0000000000000001
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rip = 0x00007ef15c2dbf44
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - Found by: given as instruction pointer in context
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - 1 libxul.so!mozilla::gl::GLContext::fBlendFuncSeparate(unsigned int, unsigned int, unsigned int, unsigned int) [GLContext.h:ca560ff55451aafb3dae3f679d09206b120b38eb : 844 + 0x24]
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rbx = 0x00007ef152df9000 rbp = 0x00007ef152ffeb70
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rsp = 0x00007ef152ffeb20 r12 = 0x0000000000000303
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r13 = 0x0000000000000001 r14 = 0x00007ef152ffeb28
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r15 = 0x0000000000000001 rip = 0x00007ef15c329dc4
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - 2 libxul.so!mozilla::layers::CompositorOGL::BeginFrame(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>) [CompositorOGL.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 792 + 0x20]
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rbx = 0x00007ef153f66c70 rbp = 0x00007ef152ffec00
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rsp = 0x00007ef152ffeb80 r12 = 0x0000000000000320
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r13 = 0x00007ef153f66bb0 r14 = 0x0000000000000470
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r15 = 0x00007ef152ffec80 rip = 0x00007ef15c32b591
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - 3 libxul.so!mozilla::layers::LayerManagerComposite::Render(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&) [LayerManagerComposite.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 970 + 0x32]
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rbx = 0x00007ef152ffec80 rbp = 0x00007ef152ffeda0
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rsp = 0x00007ef152ffec10 r12 = 0x00007ef152823260
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r13 = 0x00007ef152ffede0 r14 = 0x00007ef1528233b8
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r15 = 0x00007ef152ffee20 rip = 0x00007ef15c40adc8
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - 4 libxul.so!mozilla::layers::LayerManagerComposite::UpdateAndRender() [LayerManagerComposite.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 598 + 0xb]
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rbx = 0x00007ef152823260 rbp = 0x00007ef152ffee60
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rsp = 0x00007ef152ffedb0 r12 = 0x00007ef152ffedc8
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r13 = 0x00007ef152ee7400 r14 = 0x00007ef152ffee20
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - r15 = 0x00007ef152823298 rip = 0x00007ef15c40a89f
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - 5 libxul.so!mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags) [LayerManagerComposite.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 521 + 0x8]
[task 2019-05-21T06:11:15.534Z] 06:11:15 INFO - rbx = 0x00007ef152823260 rbp = 0x00007ef152ffee90
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152ffee70 r12 = 0x0000000000000000
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152ee7400 r14 = 0x00007ef152ffeec8
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r15 = 0x0000000000000000 rip = 0x00007ef15c40a5e5
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - 6 libxul.so!mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) [CompositorBridgeParent.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 1013 + 0x1d]
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rbx = 0x0000067feb3dd466 rbp = 0x00007ef152ffef80
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152ffeea0 r12 = 0x00007ef152ee7588
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152ee7400 r14 = 0x0000000000000000
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r15 = 0x00007ef152ee7578 rip = 0x00007ef15c420c94
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - 7 libxul.so!mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) [CompositorVsyncScheduler.cpp:ca560ff55451aafb3dae3f679d09206b120b38eb : 249 + 0x11]
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rbx = 0x00007ef155b2cb20 rbp = 0x00007ef152ffefd0
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152ffef90 r12 = 0x00007ef155b2cb78
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152fff088 r14 = 0x000000000000006d
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r15 = 0x00007ef152ffefa0 rip = 0x00007ef15c4287ad
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - 8 libxul.so!mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler*, void (mozilla::layers::CompositorVsyncScheduler::)(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp), true, (mozilla::RunnableKind)1, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp>::Run() [nsThreadUtils.h:ca560ff55451aafb3dae3f679d09206b120b38eb : 1174 + 0x11]
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152ffefe0
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152ffefe0 r12 = 0x00007ef152fff2c0
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152fff088 r14 = 0x00007ef152fff050
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r15 = 0x00007ef152ffeff8 rip = 0x00007ef15c435c0f
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - 9 libxul.so!MessageLoop::RunTask(already_AddRefed<nsIRunnable>) [message_loop.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 442 + 0x11]
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff040
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152ffeff0 r12 = 0x00007ef152fff2c0
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152fff088 r14 = 0x00007ef152fff050
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r15 = 0x00007ef152ffeff8 rip = 0x00007ef15bcebf2e
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - 10 libxul.so!MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask&&) [message_loop.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 450 + 0x13]
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff070
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - rsp = 0x00007ef152fff050 r12 = 0x00007ef152fff2c0
[task 2019-05-21T06:11:15.535Z] 06:11:15 INFO - r13 = 0x00007ef152fff088 r14 = 0x00007ef152fff050
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00000000000000aa rip = 0x00007ef15bcec403
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - 11 libxul.so!MessageLoop::DoWork() [message_loop.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 523 + 0xb]
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff0d0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rsp = 0x00007ef152fff080 r12 = 0x00007ef152fff2c0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r13 = 0x00007ef152fff088 r14 = 0x00007ef152fff240
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00000000000000aa rip = 0x00007ef15bcec5b5
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - 12 libxul.so!base::MessagePumpDefault::Run(base::MessagePump::Delegate) [message_pump_default.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 35 + 0x9]
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff170
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rsp = 0x00007ef152fff0e0 r12 = 0x00007ef1535ca150
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r13 = 0x00000000cf8a2c01 r14 = 0x00007ef1535ca130
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00007ef152fff108 rip = 0x00007ef15bced05f
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - 13 libxul.so!MessageLoop::RunInternal() [message_loop.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 315 + 0x17]
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff1b0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rsp = 0x00007ef152fff180 r12 = 0x00007ef152fff228
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r13 = 0x00007ef15bcf0ea8 r14 = 0x00007ef152fff1c0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00007ef152fff230 rip = 0x00007ef15bcebd7a
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - 14 libxul.so!MessageLoop::Run() [message_loop.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 290 + 0x8]
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rbx = 0x00007ef152fff230 rbp = 0x00007ef152fff1f0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rsp = 0x00007ef152fff1c0 r12 = 0x00007ef152fff228
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r13 = 0x00007ef15bcf0ea8 r14 = 0x00007ef152fff1c0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00007ef152fff230 rip = 0x00007ef15bcebce4
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - 15 libxul.so!base::Thread::ThreadMain() [thread.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 192 + 0x8]
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rbx = 0x00007ef1535c9180 rbp = 0x00007ef152fff3d0
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - rsp = 0x00007ef152fff200 r12 = 0x00007ef152fff228
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r13 = 0x00007ef15bcf0ea8 r14 = 0x00007ef1535c91a8
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - r15 = 0x00007ef152fff230 rip = 0x00007ef15bcf6c9b
[task 2019-05-21T06:11:15.536Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 16 libxul.so!ThreadFunc(void*) [platform_thread_posix.cc:ca560ff55451aafb3dae3f679d09206b120b38eb : 40 + 0x6]
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbx = 0x00007ef152fff4d0 rbp = 0x00007ef152fff3e0
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rsp = 0x00007ef152fff3e0 r12 = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - r13 = 0x00007ef15bcf0ea8 r14 = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - r15 = 0x00007ef152fff4e8 rip = 0x00007ef15bcf0eb2
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 17 libc.so + 0x89772
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbx = 0x00007ef152fff4d0 rbp = 0x00000000000050aa
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rsp = 0x00007ef152fff3f0 r12 = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - r13 = 0x00007ef15bcf0ea8 r14 = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - r15 = 0x00007ef152fff4e8 rip = 0x00007ef17d8d0772
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 18 libxul.so!std::__ndk1::__vector_base<MessageLoop::PendingTask, std::__ndk1::allocator<MessageLoop::PendingTask> >::__destruct_at_end(MessageLoop::PendingTask*) [vector:ca560ff55451aafb3dae3f679d09206b120b38eb : 415 + 0xb]
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rsp = 0x00007ef152fff428 rip = 0x00007ef15bcf0ea8
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: stack scanning
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 19 0x7ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbx = 0x00007ef1643ff450 rbp = 0x00007ef15bcf0ea8
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rsp = 0x00007ef152fff430 r14 = 0x00000000000050aa
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - r15 = 0x0000000000000002 rip = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: call frame info
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 20 libc.so + 0x299ec
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbp = 0x00007ef15bcf0ea8 rsp = 0x00007ef152fff440
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rip = 0x00007ef17d8709ec
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: stack scanning
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 21 libc.so + 0x896c0
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbp = 0x00007ef15bcf0ea8 rsp = 0x00007ef152fff448
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rip = 0x00007ef17d8d06c0
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - Found by: stack scanning
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - 22 libc.so + 0x1ca66
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rbp = 0x00007ef15bcf0ea8 rsp = 0x00007ef152fff450
[task 2019-05-21T06:11:15.537Z] 06:11:15 INFO - rip = 0x00007ef17d863a66
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - Found by: stack scanning
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - 23 libxul.so!std::__ndk1::__vector_base<MessageLoop::PendingTask, std::__ndk1::allocator<MessageLoop::PendingTask> >::__destruct_at_end(MessageLoop::PendingTask*) [vector:ca560ff55451aafb3dae3f679d09206b120b38eb : 415 + 0xb]
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - rbp = 0x00007ef15bcf0ea8 rsp = 0x00007ef152fff4b8
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - rip = 0x00007ef15bcf0ea8
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - Found by: stack scanning
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - 24 0x7ef1535c9180
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - rbx = 0x00007ef1643fc118 rbp = 0x00007ef15bcf0ea8
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - rsp = 0x00007ef152fff4c0 r14 = 0x0000000000000000
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - r15 = 0x0000000000000000 rip = 0x00007ef1535c9180
[task 2019-05-21T06:11:15.538Z] 06:11:15 INFO - Found by: call frame info
Comment hidden (Intermittent Failures Robot) |
Comment 2•5 years ago
|
||
Debug android wpt tests were enabled on that merge.
Comment 4•5 years ago
|
||
There are 44 total failures in the last 4 days on android-em-7-0-x86_64 debug
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248068353&repo=mozilla-central&lineNumber=3474
[task 2019-05-23T22:38:09.578Z] 22:38:09 INFO - TEST-START | /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/italic_object/italic_white-space_pre-wrap_wrapped.html
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - Traceback (most recent call last):
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/base.py", line 192, in run_test
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - result = self.do_test(test)
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 808, in do_test
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - result = self.implementation.run_test(test)
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 881, in run_test
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - "height": 600})["value"]
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - m._handle_socket_failure()
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 650, in _handle_socket_failure
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - reraise(exc, val, tb)
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 26, in _
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - return func(args, kwargs)
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 590, in _send_message
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - msg = self.client.request(name, params)
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 273, in request
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - return self.receive()
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 159, in receive
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - raise socket.error("No data received over socket")
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING - error: No data received over socket
[task 2019-05-23T22:38:09.809Z] 22:38:09 WARNING -
[task 2019-05-23T22:38:10.250Z] 22:38:10 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpox_HGk/0c96df4f-d6a1-c560-4a75-76309c98020f.dmp /builds/worker/workspace/build/symbols
[task 2019-05-23T22:38:17.452Z] 22:38:17 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/0c96df4f-d6a1-c560-4a75-76309c98020f.dmp
[task 2019-05-23T22:38:17.452Z] 22:38:17 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/0c96df4f-d6a1-c560-4a75-76309c98020f.extra
[task 2019-05-23T22:38:17.622Z] 22:38:17 INFO - PROCESS-CRASH | mozrunner-startup | application crashed [@ mozilla::gl::GLContext::BeforeGLCall(char const) const]
[task 2019-05-23T22:38:17.622Z] 22:38:17 INFO - Crash dump filename: /tmp/tmpox_HGk/0c96df4f-d6a1-c560-4a75-76309c98020f.dmp
[task 2019-05-23T22:38:17.622Z] 22:38:17 INFO - Operating system: Android
[task 2019-05-23T22:38:17.622Z] 22:38:17 INFO - 0.0.0 Linux 3.10.0+ #1 PREEMPT Thu Jan 5 00:46:30 UTC 2017 x86_64
[task 2019-05-23T22:38:17.622Z] 22:38:17 INFO - CPU: amd64
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - family 6 model 2 stepping 3
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - 1 CPU
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO -
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - GPU: UNKNOWN
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO -
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Crash address: 0x0
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Process uptime: not available
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO -
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Thread 29 (crashed)
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - 0 libxul.so!mozilla::gl::GLContext::BeforeGLCall(char const) const [GLContext.h:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 662 + 0x29]
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rax = 0x00007d27cb65a25e rdx = 0x0000000000000004
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rcx = 0x00007d27cea8da88 rbx = 0x00007d27bdfeb000
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rsi = 0x00007d27be67c440 rdi = 0x000000000000001b
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rbp = 0x00007d27be67cb10 rsp = 0x00007d27be67caf0
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r8 = 0x0000000000000000 r9 = 0x00007d27eb225090
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r10 = 0x0000000000000022 r11 = 0x0000000000000246
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r12 = 0x0000000000000303 r13 = 0x0000000000000001
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r14 = 0x00007d27cb672ec8 r15 = 0x0000000000000001
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rip = 0x00007d27c72c04f6
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Found by: given as instruction pointer in context
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - 1 libxul.so!mozilla::gl::GLContext::fBlendFuncSeparate(unsigned int, unsigned int, unsigned int, unsigned int) [GLContext.h:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 844 + 0x24]
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rbx = 0x00007d27bdfeb000 rbp = 0x00007d27be67cb70
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - rsp = 0x00007d27be67cb20 r12 = 0x0000000000000303
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r13 = 0x0000000000000001 r14 = 0x00007d27be67cb28
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - r15 = 0x0000000000000001 rip = 0x00007d27c730e3a2
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - Found by: call frame info
[task 2019-05-23T22:38:17.623Z] 22:38:17 INFO - 2 libxul.so!mozilla::layers::CompositorOGL::BeginFrame(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>) [CompositorOGL.cpp:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 792 + 0x20]
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rbx = 0x00007d27bef32c70 rbp = 0x00007d27be67cc00
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rsp = 0x00007d27be67cb80 r12 = 0x0000000000000320
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r13 = 0x00007d27bef32bb0 r14 = 0x0000000000000470
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r15 = 0x00007d27be67cc80 rip = 0x00007d27c730fb6f
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - Found by: call frame info
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - 3 libxul.so!mozilla::layers::LayerManagerComposite::Render(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&) [LayerManagerComposite.cpp:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 970 + 0x32]
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rbx = 0x00007d27be67cc80 rbp = 0x00007d27be67cda0
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rsp = 0x00007d27be67cc10 r12 = 0x00007d27beb0a260
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r13 = 0x00007d27be67cde0 r14 = 0x00007d27beb0a3b8
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r15 = 0x00007d27be67ce20 rip = 0x00007d27c73ef238
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - Found by: call frame info
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - 4 libxul.so!mozilla::layers::LayerManagerComposite::UpdateAndRender() [LayerManagerComposite.cpp:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 598 + 0xb]
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rbx = 0x00007d27beb0a260 rbp = 0x00007d27be67ce60
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rsp = 0x00007d27be67cdb0 r12 = 0x00007d27be67cdc8
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r13 = 0x00007d27bde8a000 r14 = 0x00007d27be67ce20
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r15 = 0x00007d27beb0a298 rip = 0x00007d27c73eed0f
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - Found by: call frame info
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - 5 libxul.so!mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags) [LayerManagerComposite.cpp:9a89e2b0fa9d7e6393deb47452e3954d4c5adf66 : 521 + 0x8]
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rbx = 0x00007d27beb0a260 rbp = 0x00007d27be67ce90
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - rsp = 0x00007d27be67ce70 r12 = 0x0000000000000000
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r13 = 0x00007d27bde8a000 r14 = 0x00007d27be67cec8
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - r15 = 0x0000000000000000 rip = 0x00007d27c73eea55
[task 2019-05-23T22:38:17.624Z] 22:38:17 INFO - Found by: call frame info
Wes are you looking into this?
I've turned these jobs back to tier-3 so sheriffs won't see these jobs at all. I'll try to find someone to look into the crash, but that someone probably isn't me.
Comment 6•5 years ago
|
||
Oh, okay, thank you. Was not sure if there is an update that's why I ni you on these bugs.
Although looking now, the patch to change them back to tier-3 didn't seem to have an effect, so I'll need to look at it more thoroughly tomorrow. Thanks for the ping, though!
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•5 years ago
|
Comment 13•5 years ago
|
||
There is one failure in the last 7 days and 81 failures in the last 30. Removing the stockwell tag.
Updated•5 years ago
|
Assignee | ||
Comment 14•5 years ago
|
||
Here are the wpt reftests associated with this crash:
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/italic_object/italic_text-decoration_line-through.html
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/color_hsla.html
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/font_shorthand.html
- (intermittent?) /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/underline_object/underline_font_shorthand.html
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/italic_object/italic_outline_properties.html
- /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_completely_move_up.html
- /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_up.html
- /webvtt/rendering/cues-with-video/processing-model/2_cues_overlapping_partially_move_down.html
- /webvtt/rendering/cues-with-video/processing-model/enable_controls_reposition.html
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/bold_object/bold_animation_with_timestamp.html
- /webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/bold_object/bold_background_properties.html
I'm going to update the expectation metadata on these tests to green up the test suite.
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 16•5 years ago
|
||
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 19•5 years ago
|
||
Pushed by mjzffr@gmail.com: https://hg.mozilla.org/integration/autoland/rev/8c6d523bdb3b Update wpt metadata for webvtt tests that crash. r=alwu
Comment 20•5 years ago
|
||
bugherder |
Updated•5 years ago
|
Comment 21•5 years ago
|
||
This is still occurring on the merge with the fix https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=252977715&resultStatus=testfailed%2Cbusted%2Cexception
Comment hidden (Intermittent Failures Robot) |
Comment 23•5 years ago
|
||
The above modifications in comment 19 only disabled some tests so I'm setting this as [stockwell disabled] and adding the leave open tag cause it's still happening, like https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252994956&repo=autoland&lineNumber=7973. Maja, when you get the chance please take a look over this. Thank you.
Comment hidden (Intermittent Failures Robot) |
Updated•5 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 26•5 years ago
|
||
Pushed by mjzffr@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/253c999a2452 Disable more webvtt web-platform-tests for intermittent crashes; (no-review, test-annotation-only)
Assignee | ||
Updated•5 years ago
|
Comment 27•5 years ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Comment 29•5 years ago
|
||
Pushed by mjzffr@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/192af6793377 Disable more tests under webvtt/rendering/cues-with-video/processing-model; a=testonly
Updated•5 years ago
|
Updated•5 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 32•5 years ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Description
•