Just FYI.
We introduced TRRServiceChannel, which can be used OMT for fetching the DNS data. Since LoadInfo is main thread only (because of this bug), we also introduce another TRRLoadInfo for TRRServiceChannel. TRRLoadInfo only implements a few interface of nsILoadInfo, since most of members in LoadInfo are not thread-safe for now.
When this bug is fixed, we can try to make LoadInfo thread-safe and ditch TRRLoadInfo.

I looked into this a bit again recently, and here are a few of my current thoughts:

Mutable State

There are a few pieces of mutable state (after Init()) on principal objects, which makes them not fully immutable. In order to get this working sooner-rather-than-later, we probably can't eliminate all of these pieces of mutable state, but we can at least make them threadsafe:

• For nsIContentSecurityPolicy on ExpandedPrincipal it should be possible to instead store a DataMutex<nsMainThreadPtrHandle<nsIContentSecurityPolicy>> which would avoid most of the issues there. It wouldn't be settable from OMT, but things like cloning the principal with different OriginAttributes would be fine. This would let us avoid waiting for CSP to be moved off of the type.
• The nsCOMPtr<nsIURI> mDomain on ContentPrincipal could be also guarded by a DataMutex<...> without too much difficulty, with bool mHasExplicitDomain changed to a std::atomic<bool>. However, that could potentially have a performance impact. I think the impact could be minimized by checking the mHasExplicitDomain bool before even trying to acquire the mutex to make the checks more efficient. Alternatively, we could make the field unguarded and main-thread accessible only, but that feels less obviously safe and means some methods won't be able to work OMT.
• For WeakPtr<WebExtensionPolicy> on ContentPrincipal, it is a lazily initialized field which is created whenever AddonPolicy() is called. I think one of the easier ways to handle it would be to have a reference counted type using NS_INLINE_DECL_THREADSAFE_REFCOUNTING_WITH_DELETE_ON_MAIN_THREAD which contains the WeakPtr<WebExtensionPolicy> and store a pointer to that instead (perhaps behind a mutex to be safe, even if the field is only ever touched on the main thread or in the destructor, it can of course be optimized if needed)

In general, I think these pieces of mutable state are workable. Many of them could be optimized further if they end up being performance issues, but I think that wrapping specific fields in DataMutex and marking the other fields explicitly as const is probably our best bet for getting an initial threadsafe principal type.

Reference Counting

The reference counting for principal types is special, as the refcount field is inherited from JSPrincipals. However, this field is already atomic (https://searchfox.org/mozilla-central/rev/e3c34d34cc7dce56df377f05183ad379ac1de123/js/public/Principals.h#25), so making reference counting atomic for principals should only require removing the assertions from nsJSPrincipals.cpp (https://searchfox.org/mozilla-central/rev/e3c34d34cc7dce56df377f05183ad379ac1de123/caps/nsJSPrincipals.cpp#27,36)

Init Methods

Currently, principals are generally created with a default constructor and then initialized by calling an init method of some form, which ends in calling BasePrincipal::FinishInit. The principal isn't usable until this method is called. This is a fairly simple change in the majority of cases, as the distance between the constructor & init calls aren't very wide, and the Init methods are generally not exposed on nsIPrincipal. The default constructors will need to be removed and replaced with explicit constructors which take arguments similar to the init methods (although some may need to be made private so that the more ergonomic init API can be exposed as a static ::Create method or similar).

The main issue here is likely to be the remaining uses of principals through the component system, which can all be dealt with in one way or another:

1. Instances which create a null principal with "@mozilla.org/nullprincipal;1" can be switched to use Services.scriptSecurityManager in JS or invoking NullPrincipal::CreateWithoutOriginAttributes in C++.
2. The System Principal can remain a singleton but with the init method removed, and the relevant initialization instead happening in the constructor. It can still be fetched using the service manager
3. Other service types are not accessed directly as components through the service manager, so should be able to have their contract IDs removed (though not their CIDs because of nsISerializable).

nsISerializable

The last issue with the component manager is that every nsIPrincipal implements nsISerializable. There is no longer support for writing out new serialized forms of these types with nsISerializable::Write() (they're all stubbed out now), but they can still be read with ::Read(). This will probably need to be solved in a similar way to how it was for nsIURI, where the CID is changed to map to a different object (e.g. ContentPrincipal::Deserializer) which is non-threadsafe & implements nsISerialzable, and when QI-ed to nsIPrincipal will return the constructed principal from the read call.

The only type which won't need this change is SystemPrincipal whose Read method is already a no-op, so it can continue to directly implement the interface.

Determining which methods are Threadsafe

Once all of these are changed, I think that the core principal object will be mostly threadsafe at rest, though many methods will still be main-thread only. We'll need to go through and check which ones are using main-thread-only services etc. From a quick scan, many methods will not work off-main-thread, and will need to be documented as such.

• Many methods access AddonPolicy() (even if just to null-check it) which is main-thread-only due to WebExtensionPolicy being cycle-collected. These methods won't work off-main-thread and some important ones may need to be changed.
• Many methods use various global components which either are not or may not be fully threadsafe, such as: nsIIOService/nsINetUtil, nsIEffectiveTLDService, mozIThirdPartyUtil, BlobURLProtocolHandler, nsScriptSecurityManager, NSPR, and the various nsIProtocolHandler implementations.
• Finally some methods will just never work of-main-thread as they intentionally call into other components for specific purposes, such as ContentPrincipal::SetDomain which calls into the JSAPI to update compartment wrappers etc.

These include some pretty important methods, like ContentPrincipal::GenerateOriginNoSuffixFromURI, which might make a threadsafe nsIPrincipal somewhat unusable off-main-thread. I imagine that at a minimum we'd want to support the relevant APIs for creating principals.

If we do end up doing this change, we'll probably want to start by annotating basically every method as being main-thread-only, adding a MOZ_ASSERT(NS_IsMainThread()) to it, and then iteratively remove these assertions as we determine that a specific method is actually safe to call off-main-thread.

I think this might be one of the more invasive changes which are required for
making nsIPrincipal threadsafe, as it requires somewhat fundamentally changing
how we hold a reference to the extension policy in ContentPrincipal. I chose to
use an explicit tearoff type here rather than something else as some properties
of AddonPolicy may need to be used off-main-thread as well in the future, and
this provides a better starting point for changes like that, as specific fields
can be duplicated onto the tearoff.

The actual weak reference needs to be threadsafe so that when the nsIPrincipal
is destroyed, it can drop the WeakPtr on the correct thread.

nsIContentSecurityPolicy is not threadsafe refcounted, so in order to make it
safe to potentially drop the last reference to this type off-main-thread, we
need to store it within a nsMainThreadPtrHandle.

Depends on D118762

This is a way to make the mutable state in nsIPrincipal explicitly threadafe.
In some cases, it would be possible to use more specialized data structures or
otherwise avoid mutexes, but for simplicity it seems better to avoid that
unless mutex overhead appears in profiles.

Depends on D118763

This marks various methods which appear to not be comptible with being used
off-main-thread. This is mostly because they either use particular global
services which are not threadsafe (such as nsScriptSecurityManager,
mozIThirdPartyUtil and nsIEffectiveTLDService). This excludes some fairly
core methods such as CreateContentPrincipal, Subsumes, and GetSiteOrigin,
however I'm hopeful that this can be improved in the future as these methods
are needed and are migrated to work off-main-thread.

Depends on D118764

The refcount field is already atomic and provided by JSPrincipals, and
this change allows these types to be manipulated from off-main-thread.

Depends on D118765

I've attached my early-draft patches which I wrote a while ago for getting nsIPrincipal to be threadsafe without touching any of the services it uses. I don't currently have any plans for landing these as-is, and I think there are probably some other issues which we'd want to clean up first, but this might work as a framework for someone who comes after this.

I have WIP patches for this building on top of bug 1793463, so taking this bug.

This method isn't actually safe to run off-main-thread yet, as nsIPrincipal
manipulation isn't being made threadsafe until a later part, however it can be
called off-main-thread when creating a new content principal with a Blob URL
spec.

Depends on D162806

This patch only makes the very basics of nsIPrincipal manipulation threadsafe,
such as reference counting, and some trivial methods. The more complex methods
will be made threadsafe in following parts.

Depends on D163031

This is used in various places in principals, as well as in the implementation
of nsScriptSecurityManager::SecurityCompareURIs which is also now threadsafe
after this change.

Depends on D163032

The script security manager is not a threadsafe service, but the method just
calls SecurityCompareURIs (which was made threadsafe in part 3). Switch to
calling that directly.

Depends on D163033

This is required because the script security manager which currently owns the
singleton is main-thread only. This change still ties the lifecycle of the
static to that service, but also makes it generally available from any thread.

Depends on D163034

This is required for deserializing nsIPrincipal instances from PrincipalInfo to
be threadsafe, as setting domain with SetDomain() is only safe on the main
thread, due to it enumerating and updating JS wrappers.

Depends on D163035

This makes some previously hidden prefs visible. This is required in order to
perform sone WebExtension security checks off-main-thread.

Depends on D163036

This migrates the restricted domain preference handling to be performed on the
ExtensionPolicyService, wrapping it in an RWLock, making it threadsafe to
interact with.

Depends on D163037

This requires migrating some members from WebExtensionPolicy to
WebExtensionPolicyCore. The mHostPermissions member could not be fully
transferred, as the WebIDL reflector needs to be cached for
WebExtensionPolicy.allowedOrigins, however the threadsafe core is shared.

Depends on D163038

This changes out almost all places AddonPolicy() is used within nsIPrincipal
implementations, replacing it with the threadsafe AddonPolicyCore() method
and WebExtensionPolicyCore type.

Depends on D163039

After the previous changes, the majority of methods on nsIPrincipal are now
threadsafe. This patch documents which methods are still bound to the main
thread, and adds thread assertions to them to avoid potential misuse.

Depends on D163040

These are far from comprehensive, and act mostly as smoke-tests that the
principal code is usable from off-main-thread.

Depends on D163041

https://hg.mozilla.org/mozilla-central/rev/36d6e57846bc
https://hg.mozilla.org/mozilla-central/rev/3dd1fdedf5c3
https://hg.mozilla.org/mozilla-central/rev/7848b5bda5b7
https://hg.mozilla.org/mozilla-central/rev/df9f872a917f
https://hg.mozilla.org/mozilla-central/rev/1f782374c701
https://hg.mozilla.org/mozilla-central/rev/88367e46d4b7
https://hg.mozilla.org/mozilla-central/rev/d3b6d72aac53
https://hg.mozilla.org/mozilla-central/rev/988661e2839f
https://hg.mozilla.org/mozilla-central/rev/edbbfa877dba
https://hg.mozilla.org/mozilla-central/rev/2653dc41f551
https://hg.mozilla.org/mozilla-central/rev/fe7977c19c33
https://hg.mozilla.org/mozilla-central/rev/945dea6ff55a