Open Bug 1286945 Opened 8 years ago Updated 1 year ago

Offer downscaled resolutions and decimated framerates in getUserMedia.

Categories

(Core :: WebRTC: Audio/Video, defect, P3)

defect

Tracking

()

REOPENED
Tracking Status
firefox50 --- affected

People

(Reporter: jib, Assigned: pehrsons)

References

(Blocks 3 open bugs)

Details

(Whiteboard: [jitsi-meet])

Attachments

(2 obsolete files)

After making constraints respect concurrent gUM access (bug 1213397), and seeing the issues with averaging and other attempts at resolving conflicts (e.g. bug 1213517 comment 152), plus the fact that we seem to mix FPS with maxFPS in our implementations (bug 1273734), it's becoming clear that we need to consider offering up downscaled resolutions and decimated framerates. This would be similar to what Chrome does, though we need to answer whether we wish to crop or always preserve aspect (Chrome crops).
Depends on: 1213517
Rank: 25
Priority: -- → P2
My preference would be to work the same as chrome, i.e. cropping, which I think is preferable to getting black bars. The typical use case is that the site requests a 4:3 aspect ratio resolution, and the webcam is widescreen. In this case it is preferable to get the cropped 4:3 image rather than one with black bars at the top and bottom.
E.g. Jitsi could ask for { video: { width: 1280, height: 720 }, audio: true } and it should work well for their needs, defaulting to 1280x720 or the closest to it. It would even prompt Firefox users once for cam+mic instead of twice for each like they do now. [1] https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#Parameters
Assignee: nobody → mchiang
These patches are only WIP. I would need more time to verify these patches on different platform and scenarios. I am also working on decimated framerate part.
Comment on attachment 8892951 [details] Bug 1286945 - support getusermedia continuous constraint by down scaling. https://reviewboard.mozilla.org/r/163962/#review169276 ::: dom/media/systemservices/CamerasChild.h:187 (Diff revision 1) > const int capture_id, webrtc::VideoCaptureCapability& capability, > + webrtc::VideoCaptureCapability& constraint, We want to separate the two different concepts: 1) the configuration we set to the hardware (capability) 2) the constraint user sets (constraint) For example, user may request a resolution 1000x700, which is not supported by the camera hardware. We then config camera with the capability 1280x720. Then downscale the output frame to the constraint 1000x700.
Attachment #8892950 - Attachment is obsolete: true
Attachment #8892950 - Flags: review?(jib)
Attachment #8892951 - Attachment is obsolete: true
Attachment #8892951 - Flags: review?(jib)
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
Current front-runner on behavior is to only downscale when we'd otherwise fail with OverconstrainedError, as described in bug 1388667 comment 10.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE

Closed wrong bug, sorry.

Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Whiteboard: [jitsi-meet]

Is there any progress on this? I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?
Thanks!

I would also like an update of some sort. Either this or add support for the aspectRatio constraint. I'm currently developing a webapp for taking profile photos That I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox. I also think this is the reason Slack refuses to support firefox for video calls.

(In reply to Sam Switzer from comment #15)

I want to work with as many cameras as possible in their best provided resolution, and it's sad how firefox forces every camera to 640x480 no matter what getusermedia says it's capable of. I've got it working pulling native resolution in all other browsers but Firefox.

Hi Sam. Firefox should enumerate all native resolutions of your camera just fine, as you can see here: https://jsfiddle.net/jib1/4p5gqm0o/show

If you are having problems pulling a specific native resolution with that link, please provide info about you camera. Thanks!

(In reply to fburka from comment #14)

I still have compressed 4:3 instead of 16:9 with Firefox 74. As there is much more need of WebRTC these quarantine days it would be awesome if Firefox could deliver beautiful video streams with the correct aspect ratio :-) I tried Jitsi, WebEx, talky.io, ... If I try meet.google.com I see the right aspect ratio before joining a room and the wrong 4:3 after joining - so there might be a workaround existing?

Hi fburka, as you can see in comment 16, Firefox supports all available native modes, and you can't get more "beautiful" than that, as far as quality.

Instead, this bug is about "compressing" higher resolutions down to emulate missing lower resolution modes, like Chrome does, even chopping off the bottom of e.g. 640x480 to produce 640x360 to emulate 16:9.

Unfortunately, the choice of camera resolution rests with the site, so it's hard to know why 4:3 was chosen for you. Many cameras have low-resolution 4:3 modes and high-resolution 16:9 modes. Likely, the site is choosing a lower resolution on purpose, either for perceived network bandwidth reasons or in the case of hangouts/meet, site bugs like bug 1630089 comment 4.

When a low resolution is chosen by the site, you'll likely see it as 4:3 in Firefox (the whole camera frame) vs a 16:9 (cropped native 4:3 camera frame) in Chrome (you're actually seeing more bits in Firefox!)

As to workarounds (for sites, not users):

  1. Sites could easily use CSS at playback time on receivers if parity and the 16:9 aesthetic is important (at minor bandwidth cost)
  2. If bandwidth is a concern, there are ways to downscale a high-res 16:9 mode in peer connection, using scaleResolutionDownBy but most sites don't seem to bother.
  3. File a bug on hangouts/meet to not look at unrelated data like navigator.hardwareConcurrency to determine HD vs VGA (& ★ this one)

We hope to get to this soon, as we realize it's become a web compat problem, as most sites don't seem interested in handling native modes directly.

Excited for this feature, thank you for identifying the root cause of 1645876. Is there an ETA for this or a way I could contribute? We are looking to use this when flash retires this year.

Does anyone know of workaround? Our video applications would really like to support firefox.

FWIW, I've noticed macOS-specific behavior:
On MacBook Air 13-inch 2020 build running macOS 10.15.5 and visiting https://test.webrtc.org/
FF81: 320x240 not available
Chrome86: 320x240 is available

while in comparison on Dell Inspiron 14-inch model 7490 with builtin webcam (labelled "Integrated Webcam") running Windows 10 Home visiting the same site...
FF81 and Chrome86: 320x240 is available

The spec was changed to require this 4 days ago.

Priority: P3 → P2

For what it's worth, the lack of frame rate decimation caused a web compat problem I ran into a few days ago that was threatening to derail remote school in Firefox for a bunch of students. See https://twitter.com/bytingtheapple/status/1338901952168513537 and preceding thread. Clever put in a workaround, but it would be good to not force people to do that...

For those looking for a workaround using MediaRecorder, used ffmpeg wasm for desktop support. https://ffmpegwasm.github.io/

Blocks: 1724078

The bug assignee didn't login in Bugzilla in the last 7 months.
:jib, could you have a look please?
For more information, please visit auto_nag documentation.

Assignee: bonchiang → nobody
Flags: needinfo?(jib)
Assignee: nobody → jib
Flags: needinfo?(jib)
Severity: normal → S4
Priority: P2 → P3

The severity field for this bug is relatively low, S4. However, the bug has 9 duplicates.
:jib, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Flags: needinfo?(jib)
Severity: S4 → S3
Flags: needinfo?(jib)
Assignee: jib → apehrson
Rank: 25

Hi,
any update on this being one of the last pieces together with this https://bugzilla.mozilla.org/show_bug.cgi?id=1531461 to get FaceTime working?
Thanks

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: