AllocGCChunk in jsgcchunk.cpp has the same problem as reported on jemalloc in bug 527356 / bug 470217. (Looks like the algorithm was copied from jemalloc in bug 553812). It makes assumptions about the return values of consecutive calls to mmap, which are invalid assumptions in certain cases (particularly Linux users with grsecurity kernels), leading to an infinite loop.

I don't know if this is an ideal solution but it reportedly fixes the hang in this case.

(In reply to comment #1) > Created attachment 491381 [details] [diff] [review] > patch Are you sure that this work on Mac? The code does not use mmap but rather vm_allocate.

I hadn't tested it on Mac, but I've tried it briefly now. If I force the first two MapPages(FindChunkStart(p), GC_CHUNK_SIZE) calls in AllocGCChunk to fail, so that it gets to run the code in this patch, then it appears to work - it doesn't crash or give assertion failures when unmapping the partial region. (But I don't know enough about memory management to claim that this is a robust fix.)

(In reply to comment #3) > I hadn't tested it on Mac, but I've tried it briefly now. I think for now we need to ifdef the code to explude MAC. When we get reports about it been broken, then we can reconsider it.

(Oops, forgot to follow up on this.) I changed it to exclude all platforms except the ones that use the mmap-based functions in the "defined(XP_UNIX) || defined(XP_BEOS)" path. Also fixed a missing null check on the new MapPages call.

See https://bugzilla.mozilla.org/show_bug.cgi?id=470217 and https://bugzilla.mozilla.org/show_bug.cgi?id=527356 On my machine, - vanilla SpiderMonkey 1.8.5 (https://developer.mozilla.org/en-US/docs/SpiderMonkey/1.8.5) has the infinite loop. - patched SpiderMonkey 1.8.5 does not have the infinite loop. - vanilla SpiderMonkey 24 does not have the infinite loop. So maybe this bug can be closed ?

I can confirm that this is now fixed. SpiderMonkey is compatible with PaX ASLR (RANDMMAP) since version 24. It's no longer a problem in either Firefox or Thunderbird.