Filed by: dluca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=285110322&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZBp5A99QR_edb4U3-aIiBA/runs/0/artifacts/public/logs/live_backing.log

 WARNING -  {standard input}:121041: Warning: end of file not at end of a line; newline inserted
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  {standard input}:121197: Error: unknown pseudo-op: `.lv'
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  g++: internal compiler error: Killed (program cc1plus)
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  Please submit a full bug report,
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  with preprocessed source if appropriate.
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  See <https://gcc.gnu.org/bugs/> for instructions.
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  /builds/worker/workspace/build/src/config/rules.mk:744: recipe for target 'Unified_cpp_dom_base2.o' failed
[task 2020-01-15T23:16:17.483Z] 23:16:17    ERROR -  make[4]: *** [Unified_cpp_dom_base2.o] Error 4
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/base'
[task 2020-01-15T23:16:17.483Z] 23:16:17     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  gfx/angle/targets/translator/FindFunction.o
[task 2020-01-15T23:16:17.485Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.702Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.704Z] 23:16:17     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o WrapSwitchStatementsInBlocks.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2  -MD -MP -MF .deps/WrapSwitchStatementsInBlocks.o.pp   /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_ops/WrapSwitchStatementsInBlocks.cpp
[task 2020-01-15T23:16:17.704Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:17.745Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.746Z] 23:16:17     INFO -  mkdir -p '.deps/'
[task 2020-01-15T23:16:17.746Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  dom/filesystem/compat/Unified_cpp_filesystem_compat0.o
[task 2020-01-15T23:16:17.762Z] 23:16:17     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/filesystem/compat'
[task 2020-01-15T23:16:20.267Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.276Z] 23:16:20     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o FindFunction.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -D__NDK_FPABI__= -DANGLE_SKIP_DXGI_1_2_CHECK -DANGLE_ENABLE_KEYEDMUTEX -DANGLE_ENABLE_ESSL -DANGLE_ENABLE_GLSL -DANGLE_ENABLE_HLSL '-DCR_CLANG_REVISION="357692-1"' -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DNOMINMAX -DNTDDI_VERSION=NTDDI_WIN10_RS2 -DUNICODE -DWINVER=0x0A00 -D_ATL_NO_OPENGL -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SECURE_ATL -D_UNICODE -I/builds/worker/workspace/build/src/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/angle/checkout/out/gen/angle -I/builds/worker/workspace/build/src/gfx/angle/checkout/src -I/builds/worker/workspace/build/src/gfx/angle/checkout/src/common/third_party/base -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -msse2  -MD -MP -MF .deps/FindFunction.o.pp   /builds/worker/workspace/build/src/gfx/angle/checkout/src/compiler/translator/tree_util/FindFunction.cpp
[task 2020-01-15T23:16:20.276Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.280Z] 23:16:20     INFO -  gfx/angle/targets/translator/FindMain.o
[task 2020-01-15T23:16:20.281Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.573Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.576Z] 23:16:20     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -std=gnu++17 -o Unified_cpp_dom_canvas4.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DDEBUG=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/canvas -I/builds/worker/workspace/build/src/obj-firefox/dom/canvas -I/builds/worker/workspace/build/src/js/xpconnect/wrappers -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/html -I/builds/worker/workspace/build/src/dom/svg -I/builds/worker/workspace/build/src/dom/workers -I/builds/worker/workspace/build/src/dom/xul -I/builds/worker/workspace/build/src/gfx/angle/checkout/include -I/builds/worker/workspace/build/src/gfx/gl -I/builds/worker/workspace/build/src/image -I/builds/worker/workspace/build/src/js/xpconnect/src -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/style -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/media/libyuv/libyuv/include -I/builds/worker/workspace/build/src/gfx/skia -I/builds/worker/workspace/build/src/gfx/skia/skia -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -Wformat-overflow=2 -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -Os -fno-omit-frame-pointer -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/widget/gtk/compat-gtk3 -pthread -I/usr/include/gtk-3.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/gtk-3.0/unix-print -Wno-error=shadow -Wno-missing-braces -Wno-error=unused-result  -MD -MP -MF .deps/Unified_cpp_dom_canvas4.o.pp   Unified_cpp_dom_canvas4.cpp
[task 2020-01-15T23:16:20.577Z] 23:16:20     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/canvas'
[task 2020-01-15T23:16:20.593Z] 23:16:20     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/angle/targets/translator'
[task 2020-01-15T23:16:20.593Z] 23:16:20     INFO -  gfx/angle/targets/translator/FindSymbolNode.o

Can any of you guys please take a look at this bug?

We switched from AWS workers to GCP workers for these builds, which is probably relevant, but it seems like it may also have something to do with the toolchains.

Quoting myself from slack: "if it's intermittent, I'd shrug it off"

Seems memory is too tight for the number of cores on those workers. Filed bug 1609595 to adjust at the build system level and bug 1609596 to get a better picture.

I pretty much confirmed comment 4 in bug 1609595. Since only some gcc-based builds are affected, I suggest we set MOZ_PARALLEL_BUILD=24 on the affected builds. But we should avoid doing it while we're still on AWS...

I believe it is just a matter of increasing the RAM size in GCP. I will work on it today.

Update: I now see it happens only for toolchain builds, am I right? So maybe it is better to adjust the builds as :glandium suggested instead of making the instances more expensive.

This is to fix OOM in GCP build instances.

Depends on D62103

https://hg.mozilla.org/mozilla-central/rev/2ad798238dcc

In GCP we have the double the number of core compared to AWS
counterparts, but we use the same amount of memory. Request the builds
to be less parallel to avoid OOM.

These kind of failures are still happening even after the latest patch: https://treeherder.mozilla.org/logviewer.html#?job_id=288470457&repo=autoland
Th link: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=288470457&searchStr=Linux%2Cx64%2Cdebug%2Cbuild-linux64-fuzzing%2Fdebug%2C%28Bf%29&revision=174d5be06caa95bdc01153cdd8cff5152f406b6f

We use 16 cores in EC2 with 32GB, but in GCE we have 32 cores. We increase
the available memory to get the same proportion of memory/core.

This patch increases the available RAM in GCE instances.

https://hg.mozilla.org/mozilla-central/rev/aa01bf29eae7

https://hg.mozilla.org/releases/mozilla-beta/rev/8452a22f6249

https://hg.mozilla.org/releases/mozilla-esr68/rev/f8c580e22738

https://hg.mozilla.org/releases/mozilla-release/rev/c0693d2fff71