User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

1. Start firefox (multiprocess windows are enabled, remote processes=3)
2. Run: ps -eaf | grep -i firefox
3. Run: lsof -p <process ids> | grep -i font | wc -l
   1188

Actual results:

The above result of 1188 regular font files open for a single browser window (297 per child/container by default) is horribly multiplied when running with more remote processes (the default being 8; 17 has been observed with automatic performance management, which is >5k files open).

With the osx default of kern.maxfiles=12288 nearly half of the file descriptors on the system are being consumed just for fonts in Firefox (obviously other files are also in use, but the fonts are egregious).

Further, some unusual font paths seem to have been searched (*'d below):
*/Library/Application Support/Apple/Fonts/Language/Support/
/Library/Fonts/
/System/Library/Fonts
~/Library/Fonts/
/System/Library/Assets/com_apple_MobileAsset_Font3/

Expected results:

Load only the small set of required fonts, rather than every font (and language) that could be found.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

lsof not only lists open file descriptors, but various other things that open or map file contents. For me, the fourth column in the lsof output for each line that matches "font" is "txt", which indicates a memory mapping. I don't have any file descriptor numbers mentioned in that column, so none of the fonts are contributing to file descriptor usage.

Does that match what's happening on your machine?

(Actually I'm not really sure why it's "txt" and not "mem". I guess CoreText must do something weird?)

Yes -- lsof is handy that way ;D
I've included a couple of lines here by way of sample output:

plugin-co 2432 cat txt REG 1,5 6020 76044 /Library/Application Support/Apple/Fonts/Language Support/NotoSansShavian-Regular.ttf
plugin-co 2432 cat txt REG 1,5 698896 111264944 /usr/lib/dyld

Checking via Apple's 'Activity Monitor' for open files and ports, the files shown as open are consistent with lsof's output, suggesting that whatever else is going on, MacOS thinks the files are, indeed, open.

Doing the crudest form of test, the number of files shown as open on the system goes up when firefox is started, and goes down when firefox is stopped. Modifying kern.maxfiles to permit a larger number of files causes the error to go away.

lsof's man page claims that 'txt' followed by a space means 'program text (code and data), mode unknown and no lock'. REG says it's a standard file -- none of this really helps when it comes to understanding what MacOS believes to be the case.

Experimenting a bit:

$ defaults read loginwindow SystemVersionStampAsString
10.15.4

$ sysctl kern.maxfiles
kern.maxfiles: 49152

$ for x in `seq 0 4999` ; do echo $x >$x ; done

$ cat filetest.c
#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <unistd.h>

int main() {
  open("0", O_RDONLY);  // open and leak this one fd
  for (int x = 0; x < 5000; x++) {
    char name[8];
    snprintf(name, 8, "%d", x);
    int fd = open(name, O_RDONLY);
    if (fd < 0) {
      printf("failed opening file %d\n", x);
      sleep(600);
      return 0;
    }
    void* p = mmap(NULL, 1, PROT_READ, MAP_SHARED, fd, 0);
    if (!p || p == MAP_FAILED) {
      printf("failed mapping file %d\n", x);
      sleep(600);
      return 0;
    }
    close(fd);  // close each fd after mmaping its contents
  }
  printf("successfully mapped all files\n");
  sleep(600);
  return 0;
}

$ make filetest && for x in `seq 0 9` ; do ( ./filetest & ) ; done

$ lsof -c filetest | wc -l
   50071

$ lsof -c filetest | grep /4999
filetest 42001  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42003  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42005  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42007  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42009  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42011  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42013  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42015  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42017  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999
filetest 42019  cam  txt    REG    1,5         5             7780441 /private/tmp/files/4999

$ lsof -c filetest | grep /0
filetest 42001  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42001  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42003  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42003  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42005  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42005  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42007  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42007  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42009  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42009  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42011  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42011  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42013  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42013  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42015  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42015  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42017  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42017  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0
filetest 42019  cam  txt    REG    1,5         2             7775440 /private/tmp/files/0
filetest 42019  cam    3r   REG    1,5         2             7775440 /private/tmp/files/0

So it looks like mmaped files show up as "txt" on macOS, and open file descriptors show their FD number. All of these are listed in Activity Monitor's "Open Files and Ports" section too. But given the number of mappings I was able to make, it looks like it's not consuming whatever kern.maxfiles is counting.

When your system is reporting that it's running out of file descriptors, what does lsof -Ff | grep '^f\d' | wc -l tell you? That should be the total number of file descriptors open across all processes.

FWIW on my system, with a running Firefox Nightly that has 13 plugin-container processes, that command gives me 4658. After quitting Firefox, it drops down to 3893.

+needinfo Nika, in case there's anything to worry about for Fission here, and Jed, who might have more insight into resource usage on macOS.

$ lsof -Ff | grep '^f\d' | wc -l
3013
$ lsof -Ff | grep -v '^f\d' | wc -l
8883

Out of the total 11896* open files, 6564 are from Firefox -- 1081 are open file descriptors, 5462 are txt, and the remaining 21 are cwd. To be hitting maxfiles it seems pretty likely that macOS is counting more than open file descriptors.[0]

*Yes, kern.maxfiles=12288 -- but in order to have any success at all at running commands, I need to be a titch under that ;D
[0] Oddly, kern.maxfilesperproc=10240, so no issues there

I'd be curious to know if you're seeing tons and tons of fonts loaded, FD, mmap or whatever, though. The sheer number I'm seeing pulled in is odd/impressive in its own right.

I do. My guess (and Jonathan can correct me) is it's because we iterate through all of the fonts available on the system shortly after a process starts up:

https://searchfox.org/mozilla-central/rev/5e4d4827aa005d031580d2d17a01bae1af138b2e/gfx/thebes/gfxMacPlatformFontList.mm#959

When Jonathan's shared memory font list is enabled (set the gfx.e10s.font-list.shared pref to true), then my content processes only show the fonts they happened to use mapped into memory. In general, it's no problem to have all of these files mapped into memory; the 64 bit address space is pretty big after all.

So I'm curious if you set this pref and restart, whether you still run into the same resource issues. If yes, then it must be that something about the way CoreText has these files open that consumes the kern.max_files value. If no, then perhaps the fonts are a red herring, and it's some other files the content processes have open. (Or it might not be Firefox at all.)

Also I notice that you are on macOS 10.12, so that's another difference from the environment I'm testing in. (And CoreText probably has had many changes since then.)

Looking at this, there's definitely a change for the better - the main Firefox process is still consuming All The Fonts, but the containers only seem to be pulling in a subset.

$ lsof -Ff | grep '^f\d' | wc -l
3120 (was: 3013)
$ lsof -Ff | grep -v '^f\d' | wc -l
6050 (was: 8883)

Out of the total 9170 (prev: 11896) open files, 4619 (6564 prev) are from Firefox -- 1331 (1081 prev) are open file descriptors, 3267 (5462 prev) txt, and the remaining 21 (21 prev) are cwd. Note that this is across 4 instances of Firefox and 17 plugin containers; the previous numbers are for 3 instances of Firefox and 14 plugin containers.

So my question then is, does that help avoid the resource exhaustion you were experiencing?

(In reply to Cameron McCormack (:heycam) from comment #6)

+needinfo Nika, in case there's anything to worry about for Fission here, and Jed, who might have more insight into resource usage on macOS.

If we have shared memory fonts enabled everywhere I don't think there should be a problem with Fission, as we're primarily worried about real memory exhaustion, and shared memory theoretically shouldn't exhaust that.

I suppose this could exascerbate issues with FD exhaustion due to too many IPC channels and shared memory FDs floating around? I know we've been looking into issues around this with high process counts on Linux, and I suppose macOS could be impacted as well. We might be able to cut down on the number of FDs in use by migrating to using mach ports for IPC/shmem, but that's a fairly large project which we ideally don't want to do during the Fission timeframe.

I don't have much to add, other than that I'm surprised that MacOS can handle today's workloads with a 12k systemwide limit on fds and memory mappings combined, and I wonder if there are ways to bypass it that their dynamic library loader is using. (I think they have a globally shared submap with all of the common libraries, for example.)

Dup'ing to bug 701661, as this was actually reported long ago.