31 crashes from 7+ Linux installations, first affected Nightly is 96.0a1 20211123094249.

Regression from bug 1698778? Pushlog for first Nightly build ID in crash reports

Crash report: https://crash-stats.mozilla.org/report/index/9b435d5e-aa97-4810-a008-30f200211125

Reason: SIGSYS

Top 5 frames of crashing thread:

0 libpthread.so.0 pthread_setaffinity_np@@GLIBC_2.3.4 nptl/nptl/pthread_setaffinity.c:38
1 libgallium_dri.so util_queue_thread_func src/util/u_queue.c:269
2 libgallium_dri.so impl_thrd_routine include/c11/threads_posix.h:87
3 libpthread.so.0 start_thread nptl/nptl/pthread_create.c:463
4 libc.so.6 __GI___clone 

The Bugbug bot thinks this bug should belong to the 'Core::Security: Process Sandboxing' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

More signatures

It looks like those are from using VA-API in the RDD process, in which case it's not exactly a “regression” — the feature is still preffed off, but now it's in a state where it mostly works and can be tested simply by going into about:config.

More Linux rdd SIGSYS 0x00000000000000cb signatures.

These syscalls operate on a given thread ID, or 0 to indicate the
calling thread; the seccomp-bpf policy can allow the fixed value 0, and
a SIGSYS trap handler can check for the current thread's ID and remap it
to 0, but we don't have a safe way to check if it's another thread of
the current process (which is the policy we'd like to have). If they
weren't filtered at all, they could act on any thread of any process
owned by the same user, which is not ideal.

Currently the GMP process policy allows them for the calling thread
and treats them as unexpected syscalls otherwise (crashing by default
on Nightly, else ENOSYS), while the content process policy allows them
without filtering (bug 1413313) and we need also them in the RDD process
(this bug). The direction we're going to take is to allow them on the
current thread and fail with EPERM otherwise.

This patch, therefore, changes the default behavior to EPERM and moves
the code into the policy superclass so that other process types can share it.

As discussed in the previous patch, we'd like to allow these syscalls for
any thread of the calling process, but technical limitations mean that we
have to restrict them to the calling thread (or not at all). Therefore,
they are allowed for the calling thread, and EPERM is returned otherwise.

This is a slight problem for Mesa, which in some cases changes a
thread's own scheduler attributes and in other places tries to modify
other threads, but at least if we allow this safe subset it's possible
that a future Mesa version could work within that.

The impact of denying the other-thread case of these syscalls should be
limited to slightly reduced performance via suboptimal scheduling.

https://hg.mozilla.org/mozilla-central/rev/93992f9b7523
https://hg.mozilla.org/mozilla-central/rev/6db01471ba22

The patch landed in nightly and beta is affected.
:jld, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.