User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

I downloaded latest nightly sources and compiled them on armv7a-linux

Actual results:

I got this bunch of errors while linking:

37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_ComplexBitReverse'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/complex_bit_reverse_arm.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_FilterARFastQ12'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/filter_ar_fast_q12_armv7.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/spl_sqrt_floor_arm.o: in function WebRtcSpl_SqrtFloor': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_SqrtFloor'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/Unified_c_spl_sqrt_floor_gn0.o (symbol from plugin):(.text+0x0): first defined here
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -landroid_support
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -lunwind
37:10.62 collect2: error: ld returned 1 exit status

Expected results:

I expect the linking to work.

the compressed build log is attached, the error can be found right at the bottom.

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

tt_1, thanks for the report. This appears to be a build that is not in our normal Tier-1/2/3 builds. What machine are you building on?

unless I'm misstaken, it is a tier-3 build: armv7a-linux

I'm using cross compiler, ain't nobody can follow nightly branch with a rpi2 native compile.

It's pretty obvious that the build script believes it is compiling for armv7-android, but I'm uncertain how to bend the scripts in the correct direction.

Current workaround is to go with --disable-webrtc

If you can send me the .mozconfig file you're using to cross-compile on linux, I can probably get those build configs added without too much trouble.

Thanks for the offer, I will upload both native and from cross-compile tomorrow.

I can't keep my promise of providing you with the native mozconfig as well, but I believe only MOZ_OBJDIR, x-includess and --host switch would differ. If you don't find the information you seek, please ping me again any time.

Set release status flags based on info from the regressing bug 1654112

I believe https://hg.mozilla.org/mozilla-central/rev/dd3d80e100cea5b557ca3bfebf1872b0701ee3ca introduced the regression.

There are at least two incorrect assumptions in these build configs:

• That CONFIG["OS_TARGET"] == "Linux" implies CONFIG["CPU_ARCH"] != "arm". This explains the multiple definitions, because e.g. both complex_bit_reverse.c and complex_bit_reverse_arm.S are being built.

• That CONFIG["CPU_ARCH"] == "arm" implies CONFIG["OS_TARGET"] == "Android". This explains the cannot find -landroid_support errors.

This breaks building Ubuntu packages, as well as the snap package, for armhf (armv7).

I gave a go at the instructions in dom/media/webrtc/third_party_build/gn-configs/README.md on hardware running armhf, but step 6 fails when running gclient config https://github.com/mozilla/libwebrtc because depot_tools wants to bootstrap its environment by downloading pre-built artifacts from Google's servers, but those don't appear to exist for armhf. I'm not familiar enough with these tools, this will require further investigation.

For reference, here is the output:

$ ~/moz-libwebrtc# gclient config https://github.com/mozilla/libwebrtc
Errors:
  failed to resolve infra/3pp/tools/git/linux-armv6l@version:2.28.0.chromium.6 (line 27): no such tag
  failed to resolve infra/3pp/tools/cpython/linux-armv6l@version:2.7.17.chromium.22 (line 21): no such tag
  failed to resolve infra/3pp/tools/cpython3/linux-armv6l@version:3.8.0.chromium.8 (line 24): no such tag
~/depot_tools/bootstrap_python3: line 32: bootstrap-3.8.0.chromium.8_bin/python3/bin/python3: No such file or directory
cat: ~/depot_tools/python3_bin_reldir.txt: No such file or directory
[E2021-12-21T17:56:44.609360Z 1059 0 annotate.go:241] original error: no such file or directory
[E2021-12-21T17:56:44.609561Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.609624Z 1059 0 annotate.go:241] goroutine 1:
[E2021-12-21T17:56:44.609710Z 1059 0 annotate.go:241] #0 go.chromium.org/luci/vpython/python/interpreter.go:120 - python.(*Interpreter).GetVersion()
[E2021-12-21T17:56:44.609796Z 1059 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/config.go:286 - venv.(*Config).resolvePythonInterpreter()
[E2021-12-21T17:56:44.609854Z 1059 0 annotate.go:241]   reason: failed to determine Python version for: ~/depot_tools//python3
[E2021-12-21T17:56:44.609909Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610003Z 1059 0 annotate.go:241] #2 go.chromium.org/luci/vpython/venv/config.go:186 - venv.(*Config).makeEnv()
[E2021-12-21T17:56:44.610088Z 1059 0 annotate.go:241]   reason: failed to resolve system Python interpreter
[E2021-12-21T17:56:44.610156Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610209Z 1059 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:150 - venv.With()
[E2021-12-21T17:56:44.610264Z 1059 0 annotate.go:241]   reason: failed to initialize empty probe environment
[E2021-12-21T17:56:44.610318Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610372Z 1059 0 annotate.go:241] #4 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2021-12-21T17:56:44.610429Z 1059 0 annotate.go:241] #5 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2021-12-21T17:56:44.610513Z 1059 0 annotate.go:241] #6 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2021-12-21T17:56:44.610577Z 1059 0 annotate.go:241] #7 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2021-12-21T17:56:44.610638Z 1059 0 annotate.go:241] #8 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2021-12-21T17:56:44.610696Z 1059 0 annotate.go:241] #9 vpython/main.go:109 - main.mainImpl()
[E2021-12-21T17:56:44.610749Z 1059 0 annotate.go:241] #10 vpython/main.go:115 - main.main()
[E2021-12-21T17:56:44.610805Z 1059 0 annotate.go:241] #11 runtime/proc.go:204 - runtime.main()
[E2021-12-21T17:56:44.610862Z 1059 0 annotate.go:241] #12 runtime/asm_arm.s:857 - runtime.goexit()

There was an update to webrtc in the nightly branch, did it help with this bug? I can't test it on the spot, as I need to update my rust toolchain to a llvm-13 based one first.

Adding arm alone breaks ppc64el.

I updated to llvm-13 toolchain in the meantime, so will give the patch a try.

Can see some stray include dir, is this correct?

@@ -9069,6 +9451,7 @@
],
"include_dirs": [
"//",

•            "/tmp/central/obj-arm-unknown-linux-gnueabihf/third_party/libwebrtc/gn-output/gen/",
             "//third_party/abseil-cpp/"
         ],
         "libs": [],
  

Damn, forgot to re-run the fixer script on those jsons.

Ok, I saw that you push forced the patch. Its seems to work just fine with my cross-gcc setup, but have no idea how it behaves at runtime. Can you please kindly test that on an actuall armv7a device?

I successfully rebuilt the firefox snap (96.0-2) on armhf with the following patches applied, and it seems to behave correctly at runtime, including WebRTC features (tested on a RPi4 running Ubuntu 20.04 armhf, with https://mozilla.github.io/webrtc-landing/gum_test.html):

• https://phabricator.services.mozilla.com/D133409
• https://phabricator.services.mozilla.com/D117104
• https://phabricator.services.mozilla.com/D134435
• https://hg.mozilla.org/mozilla-central/rev/3607c4937e84
• https://phabricator.services.mozilla.com/D134738

(the first 4 patches are pre-requisites for the last one to apply cleanly).

I have confirmed that D134738 fixes building (and running) Firefox 97.0 on armhf. What is missing to move forward with this and get the patch merged?

The patch that's up for review at D134738 doesn't apply cleanly on nightly any longer (it was applying until yesterday). Is there anything that we can do to help with moving this forward?

Mike, do the patches linked to bug 1773223 render D134738 unnecessary? Or will it need refreshing once all the former have landed?
If so, is it reasonable to expect an updated patch before the soft code freeze for 103.0 ?

The patch not applying any longer on the 102 branch means we cannot build Ubuntu packages for armhf, which is (for Ubuntu) a supported architecture.

Backed out for causing build bustages.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=aFofmcymQfuaUogDoGLSTw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=85f0cccbd2a8b007005aa155058ad9007323d83c

Failure logs: https://treeherder.mozilla.org/logviewer?job_id=381951166&repo=autoland

Backed out for causing multiple failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/1bff014a2d90deeec6c821a7f1ee2a2fe3014bcb

Push with failures

Failure log mda

Failure log bc

Failure log wpt

Failure log crashtest

Ideally, the code would handle things in a more general way that doesn't
require manually dealing with these lists, but this would require more
testing than there is time left before 102 releases.

While here, remove HOST_CPU_ARCH, which is always the same and thus
never appears in a condition.

This changes none of the generated moz.builds for the current
configuration (but changes the outcome when adding new configurations)

https://hg.mozilla.org/mozilla-central/rev/4d109e7487ef
https://hg.mozilla.org/mozilla-central/rev/faf794de9900

Since nightly and release are affected, beta will likely be affected too.
$For more information, please visit auto_nag documentation.

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Broken builds for tier-3 downstreams.
• User impact if declined:
• Fix Landed on Version: 103
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): NPOTB

The patch landed in nightly and beta is affected.
:glandium, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox102 to wontfix.

For more information, please visit auto_nag documentation.

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

Approved for 102.1.0esr.

https://hg.mozilla.org/releases/mozilla-esr102/rev/251c39631403
https://hg.mozilla.org/releases/mozilla-esr102/rev/3447ce3987c3