+++ This bug was initially created as a clone of Bug #557545 +++ 1. We need to have tests that cause malloc() and friends to return null in as many circumstances as possible, to detect exploitable bugs in the handling (or lack of handling) of allocation failures. 2. We need to have tests for the behavior of the browser in genuine out-of-memory situations, so that we can find exploitable bugs that only occur when third-party or operating-system code fails to allocate memory, using an allocator (like VirtualAlloc, or one internal to the library) that we may not be able to modify as we would for #1. When I was working on bug 557545, I remember stumbling once or twice across (non-reproducible) DEP violations, which indicates to me that there is potentially a serious problem in this area.

Brandon: seems like something for your team to fit in somewhere.

Benjamin Otte suggested a brilliant trick: on every malloc, fork and return null in the child. http://www.squarefree.com/2011/09/01/lessons-from-js-engine-bugs/#comment-12324 I'd rather see bug 611123 fixed than try to test the current mess, though.

Hm, the crash-tester.c script is not available anymore on his website but to keep the brainstorming process up, I will attach a dtrace script. Although it will only find "double free" errors but dtrace might also be used for this kind of purpose. Execute it like this: dtrace -qs free.d -p <pid>

I had another idea which involves LLCov instrumentation on all locations where malloc is used. Once the fuzzer can see those locations, it can run our tests first, record all mallocs touched, then use gcparam (is that the right function in the shell?), to cause artificial OOMs at every single place in a linear way (just reducing till all malloc points have failed once).

Actually my last comment was referring to the JS engine, but I realized now that this bug has a wider scope. First of all, are we talking about an OS/Arch specific problem here or just the more general problem of failing to check if allocation succeeded? And secondly, which functions are we talking about, system malloc? Also, I'm trying to imagine which OOM specific failure could lead to e.g. DEP violations as comment 0 suggested. Not checking malloc's returned pointer will only null deref. Of course, an error handling path for OOM condition could contain a regular bug that is only triggered for this path, is that what you meant?

A eax[0+offset] for example would make a null ptr bug exploitable. Exploiting zero sized heap allocations is probably also relevant to this topic: http://goo.gl/B9hvJ

(In reply to Christoph Diehl [:cdiehl] from comment #7) > A eax[0+offset] for example would make a null ptr bug exploitable. Only if offset is really large, which is usually not the case. > Exploiting zero sized heap allocations is probably also relevant to this > topic: http://goo.gl/B9hvJ I'll take a look at this :)

I believe it depends on the subject. There were some issues in WebGL exactly like that. Also, most researchers stop examining a crash when they see ECX(0x0)+45 and often don't even know that the offset is controllable.

I wrote a patch that modifies moz_malloc, such that it will obtain a call trace when called, and search the call trace for a specific caller location. If that location is found, the function simulates OOM by returning NULL. It can also output the call traces (with configurable length). What we could do with this is: 1) Run a test with call trace output (with length e.g. 2-3) 2) Force OOM at every recorded point and observe crash The problem is how to triage crashes, as I don't know what our expectation here is. I assume NULL crashes should be ok. What about near-NULL? And how far would we allow that?

I talked to khuey about the strategy here, and in general we should file all bugs that crash due to fallible allocation and do not use mozalloc_abort. That will also make this kind of testing a lot easier.