Improve how webrtc moz.builds are generated
Categories
(Firefox Build System :: General, task)
Tracking
(firefox-esr102 fixed, firefox102 wontfix, firefox103 fixed)
People
(Reporter: glandium, Assigned: glandium)
References
Details
Attachments
(11 files)
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
(deleted),
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr102+
|
Details |
Assignee | ||
Comment 1•2 years ago
|
||
Michael, how realistic would it be to update third_party/libwebrtc/build to a recent revision? After the part in https://phabricator.services.mozilla.com/D148619, I currently have awful hacks that would most probably not be necessary if we were up-to-date.
Comment 2•2 years ago
|
||
tl;dr It is possible the current libwebrtc code would work with a mismatched more modern build directory version, but a decent amount of work to preserve tweaks for different machines we've added, and generating the build/json files across all our supported platforms to confirm everything still builds and functions.
I will likely be updating that directory during the fast-foward work, but it will be either 1) when something requires it or 2) at the very end. So far advancing the code forward and not the build directory has worked.
The long answer is that without trying it first, it is a complete unknown. The current mechanism for updating that directory is:
cd dom/media/webrtc/third_party_build
python3 vendor-libwebrtc.py \
--from-googlesource \
--commit ac22062b465485d3e1196ae9c506b3617cc16fb5 \
build
There is a caveat in that the current version ac22062b465485d3e1196ae9c506b3617cc16fb5
had some changes made before committing in moz-central (587c40771588). I don't know the history on that, but that is what I found during research before starting the fast-forward.
There is a fair number of patches (15) in moz-central on top of our vendored version, which would have to be accounted for during the update as well.
Bug 1748017 - remove source filter section to match upstream. r=bwc
Bug 1746140 - Add SPARC defines to third_party/libwebrtc/build/build_config.h r=mjf DONTBUILD
Bug 1654448 - P2 - readd partial support for BSD to webrtc build;r=mjf
Bug 1654112 - deconflate the target and host architectures in libwebrtc build files; r=mjf
Bug 1654112 - alter current cpu for win arm; r=mjf
Bug 1654112 - do not copy VS dlls durring webrtc gn generate; r=mjf
Bug 1654112 - Mirror Bug 1719115 - Add riscv64 defines to build/build_config.h. r=ng
Bug 1654112 - don't use libatomic on linux builds. r=ng
Bug 1654112 - Revert NTDDI to version known by win-2012 build machines; r=mjf
Bug 1654112 - Do not set _DEBUG explicitly on Windows; r=mjf
Bug 1654112 - don't look for windows tools whose values will be ignored anyway; r=dminor
Bug 1654112 - Add check for mozilla build that was in the previous import, restore changes; r=dminor
Bug 1654112 - Add the last committime file from WebRTC for GN; r=dminor
Bug 1654112 - Get OS X build working. r=ng,firefox-build-system-reviewers,glandium
Bug 1654112 - Tweak upstream gn files for Firefox build. r=ng
Assignee | ||
Comment 3•2 years ago
|
||
They don't appear necessary when generating the gn-derived moz.build
files.
Assignee | ||
Comment 4•2 years ago
|
||
- Because we don't have a native arm64 mac one, alias the x86_64 one.
- Because we always compress with zstd, don't pretend the main script
has any power on the compression (which was wrong for Windows).
Assignee | ||
Comment 5•2 years ago
|
||
The current script requires to be run on 4 different host platforms each
of which would handle a subset of a total of 32 mozconfigs. That is not
sustainable, and there are already missing configs that break tier-3
platforms.
This replaces the current setup with one that handles all platforms in
one go, although we still keep the internal sequence of GcConfigGen ->
fixup_json -> GnMozbuildWriter.
The downside is that because this relies on the upstream webrtc build
system supporting cross-compilation, and that it actively rejects some
configurations, we need some local hacks to make it work on Linux and
Mac, but for now, we have to leave out Windows, which requires more
work.
For some reason, that removes some duplicated include directories in the
json files, which moves things a little in one moz.build file.
We also remove the mozconfigs we don't use anymore.
Assignee | ||
Comment 7•2 years ago
|
||
The path is given to the function, there is no need to get it from
config.substs.
Assignee | ||
Comment 8•2 years ago
|
||
Also replace uses of config objects, which are only used for topsrcdir.
Assignee | ||
Comment 9•2 years ago
|
||
As we're shortly going to stop producing the intermediate json files,
we want the fixups to happen in the GN processor.
Ideally, we'd move them all, but cleaning up -isysroot is more involved,
while we won't need it once we don't use intermediate json files, so we
leave the -isysroot cleanup in fixup_json.py for now.
While here, gn_out["targets"][target_fullname]
doesn't need to be set
on every iteration of the loop.
Assignee | ||
Comment 10•2 years ago
|
||
Assignee | ||
Updated•2 years ago
|
Comment 11•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/cceb8e24ffb5 Remove instructions about libwebrtc and depot_tools. r=mjf https://hg.mozilla.org/integration/autoland/rev/8281fa0ead0e Turn gn into local toolchains. r=firefox-build-system-reviewers,ahochheiden
Comment 12•2 years ago
|
||
i just stumbled on this bug, and i just want to say i love this ! working on the openbsd configs in bugs #1760484 and #1747862 was.. hard, cumbersome, and ugly.
With all those changes, how will it work for tier3 platforms ? should i start testing all this on OpenBSD/amd64 & OpenBSD/arm64 ?
EDIT: i looked at https://phabricator.services.mozilla.com/D149205#change-4Y4VwJH0s4DG and i think we should be fine, will try to test that in the coming weeks on both platforms.
Comment 13•2 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/cceb8e24ffb5
https://hg.mozilla.org/mozilla-central/rev/8281fa0ead0e
Updated•2 years ago
|
Assignee | ||
Comment 14•2 years ago
|
||
This is redundant with the build system setting it in
toolchain.configure.
Assignee | ||
Comment 15•2 years ago
|
||
Comment 16•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/f349d4ad0088 Generate webrtc moz.builds for all platforms at once. r=mjf,firefox-build-system-reviewers,ahochheiden https://hg.mozilla.org/integration/autoland/rev/264adb57bb3c Remove variable expansion in the GN processor. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/fe0095ad6576 Use the GN binary path passed into generate_gn_config. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/05e201f82758 Remove unused parameters. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/c73d516d7d7f Move some json fixups into the GN processor. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/5e1c0d1d4018 Make the GN processor an independent script. r=firefox-build-system-reviewers,mjf,ahochheiden https://hg.mozilla.org/integration/autoland/rev/7eccf21b04f2 Make the definition of MOZ_X11 independent of the OS. r=firefox-build-system-reviewers,ahochheiden https://hg.mozilla.org/integration/autoland/rev/2d0053357489 Filter-out _FORTIFY_SOURCE in GN processing. r=firefox-build-system-reviewers,andi
Assignee | ||
Comment 17•2 years ago
|
||
Comment 18•2 years ago
|
||
Pushed by csabou@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2faea2e0af45 Remove now irrelevant gn processor tests.
Comment 19•2 years ago
|
||
The above fix was backed out because of the initial land build bustages: https://hg.mozilla.org/integration/autoland/rev/03c4ec9a58394b5e4cadf3e0837fd5c3639ec952
Failure logs: https://treeherder.mozilla.org/logviewer?job_id=381951166&repo=autoland
https://treeherder.mozilla.org/logviewer?job_id=381951209&repo=autoland
Comment 20•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/2b263cd76492 Generate webrtc moz.builds for all platforms at once. r=mjf,firefox-build-system-reviewers,ahochheiden https://hg.mozilla.org/integration/autoland/rev/4e1b0ff08342 Remove variable expansion in the GN processor. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/254ca4aa47d4 Use the GN binary path passed into generate_gn_config. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/f4b41046456f Remove unused parameters. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/9c92bd454718 Move some json fixups into the GN processor. r=firefox-build-system-reviewers,andi
Assignee | ||
Updated•2 years ago
|
Comment 21•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/11f3375a4a70 Make the GN processor an independent script. r=firefox-build-system-reviewers,mjf,ahochheiden https://hg.mozilla.org/integration/autoland/rev/f24f3dfa49aa Make the definition of MOZ_X11 independent of the OS. r=firefox-build-system-reviewers,ahochheiden https://hg.mozilla.org/integration/autoland/rev/fd9540114bfd Filter-out _FORTIFY_SOURCE in GN processing. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/e759ac4fb646 Remove now irrelevant gn processor tests.
Comment 22•2 years ago
|
||
Backed out for causing multiple failures
Backout link: https://hg.mozilla.org/integration/autoland/rev/1bff014a2d90deeec6c821a7f1ee2a2fe3014bcb
Assignee | ||
Updated•2 years ago
|
Comment 23•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/455d5db0d4bd Make the GN processor an independent script. r=firefox-build-system-reviewers,mjf,ahochheiden https://hg.mozilla.org/integration/autoland/rev/b43b6b2840f2 Make the definition of MOZ_X11 independent of the OS. r=firefox-build-system-reviewers,ahochheiden https://hg.mozilla.org/integration/autoland/rev/c4940360a297 Filter-out _FORTIFY_SOURCE in GN processing. r=firefox-build-system-reviewers,andi https://hg.mozilla.org/integration/autoland/rev/89d88f4d596f Remove now irrelevant gn processor tests.
Comment 24•2 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/455d5db0d4bd
https://hg.mozilla.org/mozilla-central/rev/b43b6b2840f2
https://hg.mozilla.org/mozilla-central/rev/c4940360a297
https://hg.mozilla.org/mozilla-central/rev/89d88f4d596f
![]() |
||
Comment 25•2 years ago
|
||
Generate webrtc moz.builds for all platforms at once. r=mjf,firefox-build-system-reviewers,ahochheiden
https://hg.mozilla.org/integration/autoland/rev/2b263cd7649205796467a480e2d96e061f58490c
https://hg.mozilla.org/mozilla-central/rev/2b263cd76492
Remove variable expansion in the GN processor. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/4e1b0ff08342dd2da9a0271d191b4fb902d99676
https://hg.mozilla.org/mozilla-central/rev/4e1b0ff08342
Use the GN binary path passed into generate_gn_config. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/254ca4aa47d473b2908075cf63ff194a346c5ea9
https://hg.mozilla.org/mozilla-central/rev/254ca4aa47d4
Remove unused parameters. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/f4b41046456f22d69c1ebbc60eabd038cece572d
https://hg.mozilla.org/mozilla-central/rev/f4b41046456f
Move some json fixups into the GN processor. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/9c92bd4547185202992c48a52938627197a3d856
https://hg.mozilla.org/mozilla-central/rev/9c92bd454718
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 26•2 years ago
|
||
Comment on attachment 9281137 [details]
Bug 1773223 - Remove instructions about libwebrtc and depot_tools.
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: Broken builds for tier-3 downstreams. The series of patch also makes it more tractable to fix such issues for those that would not be fixed by this and the current followups.
- User impact if declined:
- Fix Landed on Version:
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): NPOTB
Assignee | ||
Updated•2 years ago
|
Updated•2 years ago
|
Comment 27•2 years ago
|
||
Comment on attachment 9281137 [details]
Bug 1773223 - Remove instructions about libwebrtc and depot_tools.
Approved for 102.1esr.
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Comment 28•2 years ago
|
||
All the backouts and re-landings of these 3 bugs seem to have made these patches basically un-graftable to esr102 at this point. Can you please rebase them for uplift? A link to the Try push of the stack will suffice once ready.
Assignee | ||
Comment 29•2 years ago
|
||
https://treeherder.mozilla.org/jobs?repo=try&revision=c36fad309b9f6f657cb40a78895960455584c26b
(FWIW, I had no conflicts while cherry-picking these from central)
Comment 30•2 years ago
|
||
Thanks, I guess git had an easier time working through it all than hg did. I had issues starting with rev 455d5db0d4bd.
Comment 31•2 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-esr102/rev/b9c7acca406b
https://hg.mozilla.org/releases/mozilla-esr102/rev/9b949354279f
https://hg.mozilla.org/releases/mozilla-esr102/rev/c488b0a0f467
https://hg.mozilla.org/releases/mozilla-esr102/rev/d39ffca4512b
https://hg.mozilla.org/releases/mozilla-esr102/rev/90e23ee132cb
https://hg.mozilla.org/releases/mozilla-esr102/rev/881f5b001637
https://hg.mozilla.org/releases/mozilla-esr102/rev/b36429704d1d
https://hg.mozilla.org/releases/mozilla-esr102/rev/165116521e5e
https://hg.mozilla.org/releases/mozilla-esr102/rev/f55a1ba87145
https://hg.mozilla.org/releases/mozilla-esr102/rev/ed9ead1a1c7d
https://hg.mozilla.org/releases/mozilla-esr102/rev/98240d769927
Description
•