In bug 1603839, I recently reported an error state caused by the exhaustion of X clients resources. I just found a way to consistently reproduce this issue based on a description on Reddit. In short, the error condition that I described (exhaustion of X client resources) can easily be triggered within one browser instance by allowing an unlimited number of processes.
This is a deterministic and consistent way to reproduce the problem, but don't mistake this for the issue being rare. When the user is running many GUI applications, this issue can already be triggered with the default config only.

The general problem is that every content process of Firefox wants a piece of X - it expects XOpenDisplay to always return a value. Due to the relatively low maximum number of clients, the condition can be reached relatively easily.

Is it really necessary for every content process to become a client of the X server?

STR:

1. Launch Firefox (from the terminal, to see stderr)
2. Visit about:config and set dom.ipc.processCount to -1 .
3. Run the following snippet to create multiple tabs that are hosted in their own process.

for (let url of ['http://example.com/', 'http://example.org/', 'http://example.net/']) {
 for (let userContextId = 0; userContextId < 40; ++userContextId) {
  windowRoot.ownerGlobal.gBrowser.addTab(url, {userContextId, triggeringPrincipal: Services.scriptSecurityManager.getSystemPrincipal()})
 }
}

(note: I'm creating container tabs in non-existing containers. That doesn't matter for the STR, but if it ever becomes relevant, set privacy.userContext.enabled=true and privacy.userContext.ui.enabled=true, and add new containers in about:preferences#containers)

4. If step 3 doesn't cause issues, add more different domains to the list of URLs at the first line (e.g. change "http" to "https"), or increase < 40 to a higher number to force more containers/processes.
5. Click on the last tab.
6. Click on the terminal/console from step 1 to read the output from the Firefox process.

Expected:

• Step 5: The tab should have been loaded.
• Step 6: No error messages.

Actual:

• Step 5: The tab appears to have been crashed. Or it is blank. Or Firefox displays the about:restartrequired page ("Sorry. We just need to do one small thing to keep going").
• Step 6: The following message is displayed ("Maximum number of clients reached" means x11 clients)

Maximum number of clients reachedMaximum number of clients reachedUnable to init server: Could not connect: Connection refused

(/tmp/firefox/firefox-bin:695087): Gtk-WARNING **: 16:11:13.496: cannot open display: :0

There are many more errors, all consequences of the inability to get a valid display:

• step 6: assertion failure in https://searchfox.org/mozilla-central/rev/dc4560dcaafd79375b9411fdbbaaebb0a59a93ac/gfx/thebes/gfxPlatformGtk.cpp#100

Maximum number of clients reachedAssertion failure: mCompositorDisplay (Failed to create compositor display!), at /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatformGtk.cpp:100

• step 6 (debug build, symbolized stack trace from segfault, (gdb) p $rdi shows that the first argument is 0, i.e. a null pointer). Previously an assertion would have been triggered (bug 1603839), but now we happily pass a null pointer to XGetWindowAttributes, which is apparently not allowed.

#0  0x00007f6efeb8e91b in XGetWindowAttributes () at /usr/lib/libX11.so.6
#1  0x00007f6efb04e143 in mozilla::widget::GtkCompositorWidget::GtkCompositorWidget(mozilla::widget::GtkCompositorWidgetInitData const&, mozilla::layers::CompositorOptions const&, nsWindow*) () at /tmp/fx78/firefox/libxul.so
#2  0x00007f6ef85d6e4a in mozilla::widget::CompositorWidgetParent::CompositorWidgetParent(mozilla::widget::CompositorWidgetInitData const&, mozilla::layers::CompositorOptions const&) () at /tmp/fx78/firefox/libxul.so
#3  0x00007f6ef7127149 in mozilla::layers::CompositorBridgeParent::AllocPCompositorWidgetParent(mozilla::widget::CompositorWidgetInitData const&) () at /tmp/fx78/firefox/libxul.so
#4  0x00007f6efa8f71b1 in mozilla::layers::PCompositorBridgeParent::OnMessageReceived(IPC::Message const&) () at /tmp/fx78/firefox/libxul.so
#5  0x00007f6efa8fc064 in mozilla::layers::PCompositorManagerParent::OnMessageReceived(IPC::Message const&) () at /tmp/fx78/firefox/libxul.so
#6  0x00007f6ef9e7e75a in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) () at /tmp/fx78/firefox/libxul.so
#7  0x00007f6ef9e7f8c9 in mozilla::ipc::MessageChannel::MessageTask::Run() () at /tmp/fx78/firefox/libxul.so
#8  0x00007f6ef9e72e4b in base::MessagePumpDefault::Run(base::MessagePump::Delegate*) () at /tmp/fx78/firefox/libxul.so
#9  0x00007f6efa895a4f in MessageLoop::Run() () at /tmp/fx78/firefox/libxul.so
#10 0x00007f6efa89c5c0 in base::Thread::ThreadMain() () at /tmp/fx78/firefox/libxul.so
#11 0x00007f6efa8996ea in ThreadFunc(void*) () at /tmp/fx78/firefox/libxul.so
#12 0x00007f6f0109546f in start_thread () at /usr/lib/libpthread.so.0
#13 0x00007f6f00c773d3 in clone () at /usr/lib/libc.so.6

This is not a recent regression. I have experienced this problem for a long while, and am still able to reproduce the issue on today's Nightly.

See also bug 1129492, where this will probably be (eventually) fixed.

mCompositorDisplay was removed in Bug 1657315.