Thank you for helping make Firefox better. If you are reporting a defect, please complete the following:

What were you doing?

1. (Optional; Not sure if doing this is required to make this reproduce) Be on a profile with lots of tabs open (in my case it's 730, but the vast majority of them are on-demand)
2. Open the Browser Toolbox, do nothing with it. Let it sit in the background and continue your normal browsing session
3. Wait for Firefox's processes to bubble up to something around 6GB~ of RAM usage (the longer you wait the bigger it'll get, and the hang will also get longer)
4. Close the Browser Toolbox

What happened?

Firefox is unresponsive. It takes a while until the processes cool down to normal RAM usage, and then Firefox is responsive again.

What should have happened?

Slightly more normal RAM usage during the time when the Browser Toolbox is open, and no hung Firefox after closing it.

Anything else we should know?

Profile:
https://share.firefox.dev/31Hrodb

According to the profile, this seems to be related to pool cleanup.
This might have been improved by Bug 1684965. Pool::getParent was known to be slow and was significantly improved by this bug.

Itiel: sorry for the late ping, but can you test again with a recent Nightly and see if it performs better?

(In reply to Julian Descottes [:jdescottes] from comment #1)

According to the profile, this seems to be related to pool cleanup.
This might have been improved by Bug 1684965. Pool::getParent was known to be slow and was significantly improved by this bug.

Itiel: sorry for the late ping, but can you test again with a recent Nightly and see if it performs better?

Not in the slightest. Memory usage got to 8.2GB and Nightly was too much sluggish that I couldn't even press the Profiler button.
Nightly then hang for ~5 minutes (after closing the toolbox), with the memory usage going up to 8.7GB, then it went back down to a more sane 2GB and the UI became responsive again.
Let me know if you'd like to get a fresh profile and I'll try again.