I just noticed that the main process of my Firefox Nightly has a really high memory consumption of 4GB. A huge portion of that is actually related to DevTools. Here the relevant excerpt from the about:memory report:

│  │  │  ├────677,906,976 B (20.46%) -- realm([System Principal], DevTools (Module loader))
│  │  │  │    ├──674,217,736 B (20.35%) -- classes
│  │  │  │    │  ├──354,692,720 B (10.71%) -- class(Object)/objects
│  │  │  │    │  │  ├──201,589,744 B (06.09%) ── gc-heap [5]
│  │  │  │    │  │  └──153,102,976 B (04.62%) -- malloc-heap
│  │  │  │    │  │     ├──153,081,216 B (04.62%) ── slots [5]
│  │  │  │    │  │     └───────21,760 B (00.00%) ── elements/normal [5]
│  │  │  │    │  ├──100,053,608 B (03.02%) -- class(Map)/objects
│  │  │  │    │  │  ├───94,753,936 B (02.86%) ── malloc-heap/misc [5]
│  │  │  │    │  │  └────5,299,672 B (00.16%) ── gc-heap [5]
│  │  │  │    │  ├───87,296,072 B (02.64%) -- class(Set)/objects
│  │  │  │    │  │   ├──66,233,968 B (02.00%) ── malloc-heap/misc [4]
│  │  │  │    │  │   └──21,062,104 B (00.64%) ── gc-heap [4]
│  │  │  │    │  ├───54,346,800 B (01.64%) -- class(Array)/objects
│  │  │  │    │  │   ├──41,863,424 B (01.26%) ── gc-heap [5]
│  │  │  │    │  │   └──12,483,376 B (00.38%) -- malloc-heap
│  │  │  │    │  │      ├──12,479,616 B (00.38%) ── elements/normal [5]
│  │  │  │    │  │      └───────3,760 B (00.00%) ── slots [5]
│  │  │  │    │  ├───40,238,576 B (01.21%) -- class(Function)/objects
│  │  │  │    │  │   ├──40,161,520 B (01.21%) ── gc-heap [5]
│  │  │  │    │  │   └──────77,056 B (00.00%) ── malloc-heap/slots [5]
│  │  │  │    │  ├───21,241,480 B (00.64%) -- class(LexicalEnvironment)/objects
│  │  │  │    │  │   ├──21,184,456 B (00.64%) ── gc-heap [4]
│  │  │  │    │  │   └──────57,024 B (00.00%) ── malloc-heap/slots [4]
│  │  │  │    │  ├───15,694,672 B (00.47%) -- class(Call)/objects
│  │  │  │    │  │   ├──15,684,176 B (00.47%) ── gc-heap [5]
│  │  │  │    │  │   └──────10,496 B (00.00%) ── malloc-heap/slots
│  │  │  │    │  ├──────239,024 B (00.01%) -- class(<non-notable classes>)/objects
│  │  │  │    │  │      ├──150,608 B (00.00%) ── gc-heap [5]
│  │  │  │    │  │      └───88,416 B (00.00%) -- malloc-heap
│  │  │  │    │  │          ├──41,024 B (00.00%) ── slots [5]
│  │  │  │    │  │          ├──36,272 B (00.00%) ── misc [5]
│  │  │  │    │  │          └──11,120 B (00.00%) ── global-data [5]
│  │  │  │    │  ├──────131,392 B (00.00%) -- class(Source)/objects
│  │  │  │    │  │      ├──131,264 B (00.00%) ── gc-heap
│  │  │  │    │  │      └──────128 B (00.00%) ── malloc-heap/slots
│  │  │  │    │  ├──────101,472 B (00.00%) ── class(Proxy)/objects/gc-heap [3]
│  │  │  │    │  ├───────70,584 B (00.00%) -- class(XPCWrappedNative_NoHelper)/objects
│  │  │  │    │  │       ├──38,784 B (00.00%) ── malloc-heap/slots
│  │  │  │    │  │       └──31,800 B (00.00%) ── gc-heap
│  │  │  │    │  ├───────42,336 B (00.00%) -- class(WeakSet)/objects
│  │  │  │    │  │       ├──42,176 B (00.00%) ── malloc-heap/misc
│  │  │  │    │  │       └─────160 B (00.00%) ── gc-heap
│  │  │  │    │  ├───────27,280 B (00.00%) ── class(SavedFrame)/objects/gc-heap
│  │  │  │    │  ├───────21,840 B (00.00%) ── class(ScriptSource)/objects/gc-heap
│  │  │  │    │  └───────19,880 B (00.00%) ── class(With)/objects/gc-heap
│  │  │  │    ├────2,648,464 B (00.08%) -- scripts
│  │  │  │    │    ├──1,399,904 B (00.04%) ── malloc-heap/data [5]
│  │  │  │    │    └──1,248,560 B (00.04%) ── gc-heap [5]
│  │  │  │    ├──────567,584 B (00.02%) -- baseline
│  │  │  │    │      ├──544,768 B (00.02%) ── fallback-stubs [3]
│  │  │  │    │      └───22,816 B (00.00%) ── data
│  │  │  │    ├──────230,224 B (00.01%) ── jit-scripts
│  │  │  │    ├──────189,536 B (00.01%) ── saved-stacks-set [4]
│  │  │  │    ├───────28,960 B (00.00%) ── non-syntactic-lexical-scopes-table [3]
│  │  │  │    └───────24,472 B (00.00%) ── sundries/malloc-heap [5]

Note that I do not have any browser toolbox open right now, but I had yesterday when I was trying to remote debugging on Android. I also used about:debugging in parallel so not sure if one of these components could have caused that.

Nevertheless without DevTools being open we shouldn't have this memory acquired anymore. Also minimizing memory doesn't help and the memory usage stays high.

Interesting is that when I restarted Firefox by applying an update the MBT got opened again. So maybe even with it closed we seem to keep some code running which causes it to re-spawn during an update?

This is a known issue. We fixed major leak last year against the regular toolbox, but we haven't had time to look at the browser console/toolbox yet.

https://searchfox.org/mozilla-central/source/devtools/client/framework/test/allocations/browser_allocations_browser_console.js
Already reports some leaks when closing the browser console:
The browser-console test leaked 25554 objects (52595 with missing allocation site) in the parent process

It would be nice to start from here as these allocation script are easily actionable.
Then, we might try to write such allocation test, but against a browser toolbox. But this might be hard to track the DevToolsServer of the MBT.