Use xdg camera portal and pipewire for webcam access
Categories
(Core :: WebRTC, enhancement)
Tracking
()
Tracking | Status | |
---|---|---|
firefox116 | --- | fixed |
People
(Reporter: rmader, Assigned: jgrulich)
References
(Blocks 5 open bugs)
Details
Attachments
(4 files, 7 obsolete files)
Accessing webcams through the portal[1] and pipewire has several advantages:
- tighter permission control for flatpaks (unsandboxed clients will get permission to use the camera without user feedback)
- multiplexing / multiple applications can use the webcam at the same time
- potentially simplifications in the code as it works more similarly to screen sharing
- dmabuf import for easier hardware encoding (?)
Context: at the recent GUADEC 2021 there was a BoF about this topic (with members of the wlroots community around as well) and it was agreed to push for making the webcam portal the default. FF should be on board, as for many users it's the main webcam user. There is a gstreamer demo client[2], but I haven't come across an example for a ffmpeg user yet. (edit: a GTK4 demo using the camera portal[4]).
As of today (Aug. 10. 2021), there's no upstream pipewire support for video capturing in WebRTC, so any work here should be upstreamed if possible.
1: https://github.com/flatpak/xdg-desktop-portal/blob/master/data/org.freedesktop.portal.Camera.xml
2: https://gitlab.gnome.org/-/snippets/762
3: https://webrtc.googlesource.com/src/+/refs/heads/main/modules/video_capture/linux/
4: https://gitlab.gnome.org/matthiasc/video-play
Development happens in https://github.com/rmader/gecko-dev/tree/bug1724900
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Comment 1•3 years ago
|
||
We'll need to support V4L2 as well as Pipewire.
Reporter | ||
Updated•3 years ago
|
Updated•3 years ago
|
Reporter | ||
Comment 2•3 years ago
|
||
TODO: fix (generated) build files
Depends on D122236
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Comment 3•3 years ago
|
||
Michael, in order to properly test this I'd love to have an easy way to build flatpak builds. Unfortunately this doesn't appear to work via try runs yet[1] and bhearsum from #Firefox CI said it's unlikely to change any time soon. Do you have any suggestion how to best test flatpak related changes? Do you just build things locally? If so, how? Thanks and regards.
1: https://treeherder.mozilla.org/jobs?repo=try&revision=43223b2fc9eea4937e586b2cb0c66b30b16039e1
Comment 4•3 years ago
|
||
Robert, unfortunately I have exactly zero experience with flatpack builds. You might try the #build to ask about build issues.
Reporter | ||
Comment 5•3 years ago
|
||
(In reply to Michael Froman [:mjf] from comment #4)
Robert, unfortunately I have exactly zero experience with flatpack builds. You might try the #build to ask about build issues.
Oh, sorry, that makes sense. You are the triage owner of WebRTC, not Flatpak - confused that, sorry!
Reporter | ||
Comment 6•3 years ago
|
||
Mitchell, redirecting the question to you: in order to properly test this I'd love to have an easy way to build flatpak builds. Unfortunately this doesn't appear to work via try runs yet[1] and bhearsum from #Firefox CI said it's unlikely to change any time soon. Do you have any suggestion how to best test flatpak related changes? Do you just build things locally? If so, how? Thanks and regards.
1: https://treeherder.mozilla.org/jobs?repo=try&revision=43223b2fc9eea4937e586b2cb0c66b30b16039e1
Edit: also opened bug 1726251 to track try builds.
Comment 7•3 years ago
|
||
That's a great question, and unfortunately I haven't worked with Firefox + Flatpak yet.
I'm going to redirect the NI to glandium, he's probably familiar with the area.
Also, FWIW, :mtabara did the RelEng work for setting up Flatpak releases, so he might have a workflow for testing with Flatpak builds, but no guarantees.
Comment 8•3 years ago
|
||
I have no knowledge about the build environment for flatpak, although I think those are just repacks.
Comment 9•3 years ago
|
||
No good way to test this on try indeed. I build them locally two years ago when I added the automation so I'll have to dig up some notes. I'm a bit swamped right now, but I'll do my best to paste it soon. Leaving the NI open until then.
Reporter | ||
Comment 10•3 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)
No good way to test this on try indeed. I build them locally two years ago when I added the automation so I'll have to dig up some notes. I'm a bit swamped right now, but I'll do my best to paste it soon. Leaving the NI open until then.
That would be great, thanks. I'll also give it a go (in bug 1726251), maybe I can get the try build to run.
Comment 11•3 years ago
|
||
Dug a bit into this today. If I wanted to hack locally on flatpaks today, here's what I'd do. I'm fairly certain there are optimisations here and there that can be applied, but roughly this is the logic:
-
In order to generate any flatpaks, we need a good docker image with the pre-requisites. That's https://searchfox.org/mozilla-central/source/taskcluster/docker/firefox-flatpak/Dockerfile file. Looking at the most recent dot release from yesterday, https://firefox-ci-tc.services.mozilla.com/tasks/UdPTilx5Sw2Zk2yhyffkFw#artifacts is the image is being used. In the artifacts there, there's an
image.tar.zst
within here. Download image, unzst it and load the image in docker, creating a container off of it. -
All the logic generating the flatpaks lies in the
run.me
script (from here). That file will be presend in the docker container but is expecting some environment variables. Again, looking at the most recent task in our lattest dot release, I see this task. One can download those env vars and tweak them accordingly to point to an existing beta instead for example.
"LANG": "C.UTF-8",
"LC_ALL": "C.UTF-8",
"VERSION": "91.0.2",
"BUILD_NUMBER": "1",
"CANDIDATES_DIR": "https://net-mozaws-prod-delivery-firefox.s3.amazonaws.com/pub/firefox/candidates",
"FLATPAK_BRANCH": "stable",
"MOZ_AUTOMATION": "1",
"L10N_CHANGESETS": "https://hg.mozilla.org/releases/mozilla-release/raw-file/1aa232202866276c3ee7a2748b2bb291bf4fe6fa/browser/locales/l10n-changesets.json",
- Tweaking these locally to match whatever beta/release version we want to test, I'd throw them into
pass
locally so that I can load them easier in Docker. (e.g. a file calledlocal-prod
) would look like:
VERSION=92.0b1
BUILD_NUMBER=1
...
-
Create the docker container feeding in the env vars e.g.
docker run -ti $(pass show local-prod | grep -v ^# | grep -v '^$' | sed 's/^/-e /') image_name /bin/bash
-
Tweak
run.me
to your preference and run it. The script is downloading the existing tar.gz for the version/build_no specified. Then downloads all langpacks, performs some sanity check and then starts packaging the flatpak, placing it at the end in the right path for Taskcluster to find it. One can uncomment various commands to speed things up (not download all locales for examples but instead only one or two?, skip checksums verification, etc).
If you mount your locale volume somewhere in the docker container, you can extract the local flatpak build at the end to play with it locally on your host machine.
Hope this helps.
Reporter | ||
Comment 12•3 years ago
|
||
@Mihai: thanks a lot, will try that.
@topic: just got my first (entirely green) frame with the Pipewire implementation on https://www.webrtc-experiment.com/RecordRTC/ \o/
An insight gained today (as already said in bug 1658900 comment 2): Pipewire appears to prefer DMAbuf for buffer sharing. That supports point 4 of comment 0, meaning that VAAPI video encoding for camera streams could end up being easier than thought.
Reporter | ||
Comment 13•3 years ago
|
||
Right now this is just enough to get my own camera running - but
it works \o/
Depends on D122470
Reporter | ||
Updated•3 years ago
|
Assignee | ||
Comment 14•3 years ago
|
||
Hi Robert, do you plan to upstream your work so all the browsers can benefit from this change? Do you need any help? I can see it has lots of similarities to screensharing code, maybe once you are finished I can help you to upstream this into WebRTC and we can try to de-duplicate the code.
Reporter | ||
Comment 15•3 years ago
|
||
(In reply to grulja from comment #14)
Hi Robert, do you plan to upstream your work so all the browsers can benefit from this change? Do you need any help? I can see it has lots of similarities to screensharing code, maybe once you are finished I can help you to upstream this into WebRTC and we can try to de-duplicate the code.
Hi Jan! I definitely plan to upstream this code once I get back to finish it. However, as I currently can't quite say when that will be (some weeks probably), please feel free to move it upstream / start from scratch / work on the issue! Just let me/us know here ;)
Assignee | ||
Comment 16•3 years ago
|
||
Hi Robert,
I started slowly working on this. As a first step, I moved implementation for xdg-desktop-portal into a separate class and added support for the Camera portal and opened a WIP review for WebRTC (you should be in CC). We will probably want to move the implementation somewhere else since the webcam implementation is a different kind of module in WebRTC, but I will do that once I have some code for the camera support.
WebRTC review: https://webrtc-review.googlesource.com/c/src/+/236845/
Reporter | ||
Comment 17•3 years ago
|
||
(In reply to grulja from comment #16)
...
Thanks Jan! In case you didn't see: I think it would be great if we help push https://github.com/flatpak/xdg-desktop-portal/pull/659 over the line so we have a common interface for both webcam and microphone access. Unfortunately I'll not be able to work on all this before January :/
Updated•3 years ago
|
Reporter | ||
Comment 18•2 years ago
|
||
Here's some upstream work going on: https://chromium-review.googlesource.com/c/chromium/src/+/3308882
Comment hidden (off-topic) |
Comment 20•2 years ago
|
||
Sorry, there was a problem with the detection of inactive users. I'm reverting the change.
Reporter | ||
Comment 21•2 years ago
|
||
Looks like all webrtc bits have landed to implement this: https://webrtc-review.googlesource.com/c/src/+/261620 and https://webrtc-review.googlesource.com/c/src/+/264553.
Here's the chromium specific part again: https://chromium-review.googlesource.com/c/chromium/src/+/3308882 - something roughly like that is probably what we'll need to do in FF.
Assignee | ||
Comment 22•2 years ago
|
||
(In reply to Robert Mader [:rmader] from comment #21)
Looks like all webrtc bits have landed to implement this: https://webrtc-review.googlesource.com/c/src/+/261620 and https://webrtc-review.googlesource.com/c/src/+/264553.
Here's the chromium specific part again: https://chromium-review.googlesource.com/c/chromium/src/+/3308882 - something roughly like that is probably what we'll need to do in FF.
For the record, I'm trying to backport these patches with intention to look what is missing on FF side and implement it if there is anything needed. The work is here: https://github.com/grulja/gecko-dev/tree/pipewire-camera.
Reporter | ||
Comment 23•2 years ago
|
||
Uh, awesome! Taking the freedom to assign you then :)
One thing still missing in webrtc is rotation support, which is needed on most phones (the cameras are usually mounted 90 deg rotated) - I hope to have a look into that soonish (upstream).
Assignee | ||
Comment 24•2 years ago
|
||
Adds a PipeWire based camera support that was recently merged into
WebRTC. This should be an experimental feature for now and therefore it
is kept behind a config option.
Assignee | ||
Comment 25•2 years ago
|
||
This is a simple backport of an WebRTC upstream change.
Upstream commit: 4beafa38d546ab6c0bb423c12762f0c4568aa5ce
Depends on D176622
Assignee | ||
Comment 26•2 years ago
|
||
This is a simple backport of an WebRTC upstream change.
Upstream commit: f0be3bee1fc17d96354ce3c9bf457673b82e7d2d
Assignee | ||
Comment 27•2 years ago
|
||
Adds a PipeWire based camera support that was recently merged into
WebRTC. This should be an experimental feature for now and therefore it
is kept behind a config option.
Depends on D176624
Assignee | ||
Comment 28•2 years ago
|
||
This is a simple backport of an WebRTC upstream change.
Upstream commit: 4beafa38d546ab6c0bb423c12762f0c4568aa5ce
Depends on D176625
Updated•2 years ago
|
Updated•2 years ago
|
Assignee | ||
Comment 29•2 years ago
|
||
With PipeWire addition to video capture we need to include PipeWire into
the build system and add some new functions we will be dlopening.
Depends on D176624
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Assignee | ||
Comment 30•1 year ago
|
||
This is a simple backport of an WebRTC upstream change.
Note: This change has not been merged yet.
WebRTC change: https://webrtc-review.googlesource.com/c/src/+/305222
Depends on D176648
Assignee | ||
Comment 31•1 year ago
|
||
Generate libwebrtc moz.build files after PipeWire camera backports
Depends on D178937
Comment 32•1 year ago
|
||
What's the current status of this? Anything else that needs to be done, like bug testing? If Flatpak bundles are available anywhere, I'll definitely be down to see how well it works.
Assignee | ||
Comment 33•1 year ago
|
||
(In reply to Dallas Strouse [:oro] from comment #32)
What's the current status of this? Anything else that needs to be done, like bug testing? If Flatpak bundles are available anywhere, I'll definitely be down to see how well it works.
We are still working on this change. It is basically complete, but there are just some minor issues I need to address before it is good enough to be merged.
Updated•1 year ago
|
Updated•1 year ago
|
Comment 34•1 year ago
|
||
Comment 35•1 year ago
|
||
Backed out for causing crashes on getUserMedia-audio
- backout: https://hg.mozilla.org/integration/autoland/rev/08cb4ca984b3aabd7881a042a28bdae29c0a5095
- push: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=cwwksra9QA651eWRqsDZXQ.0&revision=b94558d76bd2943ba1063d7fade51c4d54d0db35&group_state=expanded
- failure log: https://treeherder.mozilla.org/logviewer?job_id=419400764&repo=autoland&lineNumber=11480
Comment 36•1 year ago
|
||
Also failing on Mingw: /builds/worker/checkouts/gecko/ipc/glue/BackgroundParentImpl.cpp:1434:3: error: use of undeclared identifier 'resolver'
with failure log
Comment 37•1 year ago
|
||
Comment 38•1 year ago
|
||
bugherder |
Updated•1 year ago
|
Updated•1 year ago
|
Reporter | ||
Updated•1 year ago
|
Reporter | ||
Updated•1 year ago
|
Description
•