Steps to reproduce

1. With devtools.custom-formatters and devtools.custom-formatters.enabled set to true
2. Open the console in this page
3. Evaluate the following snippet

 window.devtoolsFormatters = [{
     header: (obj, config) => {
       if (obj.hasOwnProperty("customFormatHeader")) {
         localStorage.setItem("test", 12);
         return [
           "span",
           {
           },
           "custom formatted header",
         ];
       }
       return null;
     },
   }
 ];

 ({
   customFormatHeader: true
 })

Expected results

The regular object is printed (Object { customFormatHeader: true }) on the console, as the header function is effectful (calling localStorage.setItem)

Actual results

The custom formatted version is printed (custom formatted header), meaning the side-effect-free debugger failed to bail.

We're missing something here.
The side-effect-free debugger has a onNativeCall hook that should handle those native calls (https://searchfox.org/mozilla-central/rev/17aeb39742eba71e0936ae44a51a54197100166d/devtools/server/actors/webconsole/eval-with-debugger.js#397-416)
But the doc states that

SpiderMonkey only calls onNativeCall hooks when execution is inside a debugger evaluation associated with the debugger that has the onNativeCall hook. Such evaluation methods include Debugger.Object.executeInGlobal, Debugger.Frame.eval, and associated methods.

(See https://github.com/mozilla/gecko-dev/blob/43cb6eca1c3069d46d589d52ab4949257e630d19/js/src/doc/Debugger/Debugger.md#onnativecallcallee-reason)

The issue is that we're using DebuggerObject#call to call the custom formatter hook functions (https://searchfox.org/mozilla-central/rev/17aeb39742eba71e0936ae44a51a54197100166d/devtools/server/actors/object.js#791), and so the onNativeCall hook is not triggered, which makes our side-effect-free setup quite inefficient.

We would need to either:

• trigger onNativeCall hooks when calling DebuggerObject#call/DebuggerObject#apply
• or find a way to call the custom formatters via executeInGlobal

Depends on D165140

https://hg.mozilla.org/mozilla-central/rev/679b28b3e286

Reproduced on Firefox 109.0.1(build ID: 20230127170202) on macOS 12.
Verified as fixed on Firefox 110.0b8(build ID: 20230202190127) and Nightly 111.0a1(build ID: 20230203091639) on macOS 12, Windows 10, Ubuntu 22.