Struct types are represented at runtime as either a OutlineTypedObject or an InlineTypedObject depending on whether the struct type can fit entirely inside a JSObject allocation [1].

When this is combined with prefix subtyping (a struct subtype may have more fields, possibly making it outline), we don't know at any given struct accessor whether the object was allocated inline/outline and have to do a branch. This complicates codegen and is slow.

There seem to be three options here in roughly increasing complexity:

1. Add a data pointer in InlineTypedObject that points to it's inline memory, making it representation compatible with OutlineTypedObject. Every accessor can than dereference this pointer to get the either inline/outline memory.
2. Always allocate as many fields as we can inline, before appending a footer word that points to the remainder fields in malloc memory. Every accessor is for a static field and can tell whether it can get it inline or if it has to find the footer word to find the malloc'ed area.
3. Add a 'large object area' and remove the JSObject size limit (at least for struct objects) so that struct objects always have data inline. I've been told this is very difficult.

A discarded idea was to use a shared implementation limit on the number of struct fields to guarantee data is always inlined. V8 uses a limit of 999 fields. With v128 types, this is worst case ~15KiB, and likely far beyond what we'd want to support.

I believe that (2) is the sweet spot in complexity and runtime performance. It has the added benefit that struct and array objects will use different classes, and we can start to specialize them.

[1] https://searchfox.org/mozilla-central/rev/d3c2f51d89c3ca008ff0cb5a057e77ccd973443e/js/src/vm/JSObject.h#550

The key idea with this bug is to have an 'inline area' for fields on the struct that can fit within MaxInlineBytes, and an 'outline area' for the rest of the fields. When a struct access is compiled, it checks whether the offset of the field is < MaxInlineBytes and if so reads from the inline area, or else it loads the pointer to the outline area and performs the load.

Here are the steps as I understand it for implementing this:

1. Rename InlineTypedObject to WasmStructObject
2. Rename InlineTypedObject::data_ to inlineData_;
3. Add uint8_t* outlineData_; before uint8_t inlineData_[0];
4. Reduce MaxInlineBytes to account for outlineData_ pointer size
5. Modify js::InlineTypedObject::allocKindForRttValue to truncate nbytes to at mostMaxInlineBytes

• This is because any nbytes over MaxInlineBytes will be in the outline area, and don't change what AllocKind the WasmStructObject needs

6. Modify InlineTypedObject::create

• If nbytes > MaxInlineBytes, then allocate extra storage using js_malloc and store pointer in outlineData_
• Otherwise, store nullptr in outlineData_

7. Add InlineTypedObject::finalize

• Checks if outlineData_ is not null and calls js_free if it is

8. Replace uint8_t* TypedObject::typedMem() with `uint8_t* TypedObject::typedMem(size_t offset) which will return address in inline or outline area depending on if offset > MaxInlineBytes
9. Update TypedObject::visitReferences,loadValue,initDefault for new typedMem
10. Update baseline struct.get/set to load from outline data only if field.offset > MaxInlineBytes

• This removes need for SymbolicAddress::InlineTypedObject and branches on it

I should also add, that after this we can rename not only InlineTypedObject to WasmStructObject, but we should also rename OutlineTypedObject to WasmArrayObject. We should also rename WasmTypedObject.h/cpp to WasmGcObject.h/cpp.

WIP patch (pretty close to reviewable). Note, depends on 3 as-yet
unlanded patches of bug 1774840.

This refactors the wasm struct representation to use a hybrid
in-line/out-of-line representation and does other cleanups. Changes:

• Rename InlineTypedObject (and related names) to WasmStructObject.

• Rename OutlineTypedObject (and related names) to WasmArrayObject.

• WasmStructObject: add in-line and out-of-line areas and padding to ensure
  the inline area is 8-aligned.

• WasmStructObject_MaxInlineBytes: remove implicit assumptions that the size
  of the struct prior to the first data field is equal to sizeof(TypedObject),
  that being the parent class.

• WasmStructObject and class StructLayout: add logic, comments and
  assertions to deal with the in-line/out-of-line split.

• WasmStructObject: new functions fieldOffsetToAreaAndOffset,
  fieldOffsetToAddress

• Baseline compiler cases for emitStruct*: have been updated accordingly -- a
  significant simplification.

  • emitGcStructSet has changed slightly, so as to take areaBase (a RegPtr)
    and areaOffset (a fixed offset). The caller must set those up correctly
    for the access; it is not for emitGcStructSet itself to decide whether
    this is an in-line or out-of-line access.

• InlineTypedObject::create (a.k.a. WasmStructObject::create) has been
  manually inlined into its caller, to reduce the complexity of initialisation
  code.

• New class RttValue::PropOffset used for carrying data between
  RttValue::lookupProperty to + TypedObject::loadValue. May be removed in bug
  1785776 rework.

• SymbolicAddress::WasmStructObjectClass (was ::InlineTypedObjectClass)
  is now unnecessary, hence is deleted.

The logic and code for OutlineTypedObject (a.k.a. WasmArrayObject) is
unchanged. The above refactoring does however make it possible to do some
cleanups to the array code and to reduce the complexity and overall
manual-RTTI-style of class TypedObject. These are tracked in followup bug
1785776.

https://hg.mozilla.org/mozilla-central/rev/b1c9c4967426