This is a followup to bug 1164164. Current WEbAPIs for the img and canvas elements do not provide a reliable way for a web app to create a thumbnail for an image. If image memory is low, a call to CanvasRenderingContext2D.drawImage() can fail to decode the source image, and not actually draw anything to the canvas, resulting in a blank thumbnail. In older versions of gecko this would at least cause an exception, but that apparently violated standards, so now drawImage() failures are silent and web apps don't even know when they have produced invalid thumbnails. This can be awkwardly worked around by writing some random pixels into the canvas, then drawing the image on top (assuming no transparency) and then reading the pixel data back to verify that the random data has changed. If it hasn't changed, then drawImage probably failed. This is a very awkward workaround with poor performance charactersistics. And without any control over gecko's image cache, it is not at all clear that a web app has any way of knowing when to retry the thumbnail creation. So I think this needs to be fixed in the platform. At a minimum we need a way for gecko to throw an error when an image fails to decode because of lack of memory. We need to be able to distinguish the case of an image that does not decode because it is corrupt from an image that does not decode because we are temporarily out of memory. Ideally, though, I'd like a new API that allows me to decode images outside of the contstraints of gecko's image cache and its memory constraints. For example, if I could pass an encoded blob to the canvas drawImage() call, it could decode into the canvas buffer that is already allocated, eliminating the possibility of failure because of low image memory. (Since drawImage() is a synchronous call, doing exactly what I propose is probably out of the question, but that is the idea... Canvas has the API for encoding images. I'd like it to have an API for decoding them as well, so that images can be decoded into a canvas in a way that is unaffected by the amount of memory being used by gecko's decoded image cache.)

[Blocking Requested - why for this release]: In 2.2 there was an exception when thumbnails weren't generated, but in master when this occurs, we just get a black square. It is rare but does occur on low memory devices. We need to fix this somehow. Needinfo Seth so he knows I've filed this bug. I think this is something we'll discuss at the Whistler workweek.

(In reply to David Flanagan [:djf] from comment #0) > Ideally, though, I'd like a new API that allows me to decode images outside > of the contstraints of gecko's image cache and its memory constraints. I don't see that happening, because the memory constraints exist for a good reason - to avoid OOM. Furthermore, I'm not convinced it's the right way to solve the problem. My perspective is that we need a finer-grained notion of locking images, so that higher priority images (such as an image that is being used for a synchronous call like canvas.drawImage) take precedence over lower priority images. > I'd like it to have an API for decoding them as well, so that images can be > decoded into a canvas in a way that is unaffected by the amount of memory > being used by gecko's decoded image cache.) The thing is, in general we might have to copy the canvas buffer to decode into it (for example, it might be in the wrong format, or it might be in GPU memory), which would require allocating just as much memory as if we decoded the image separately. This might help in some cases but I don't think it's something we can rely upon as a solution.

Hey David, This is a 3.0 that got to 2.5 list. Can you please confirm if this still is required for 2.5 and block accordingly. Thanks

I still really want this platform feature. And I believe that it is still possible for Gallery to fail to create thumbnails on low-memory devices. I nominated it as a blocker 3 months ago. It doesn't seem to have gotten any traction yet, so it would be pretty late in the game to make it a blocker now. Basically, I don't feel like my call to make, Mahe. Seth: is my fear about thumbnail creation still founded, or will an attempt to draw an off-screen image into a canvas always evict unlocked images from memory so that it can complete successfully? I don't want to make a fuss about this if it is just a theoretical concern, but it it can still happen, then I at least need to get an exception thrown when it does happen

(In reply to David Flanagan [:djf] from comment #4) > Seth: is my fear about thumbnail creation still founded, or will an attempt > to draw an off-screen image into a canvas always evict unlocked images from > memory so that it can complete successfully? I don't want to make a fuss > about this if it is just a theoretical concern, but it it can still happen, > then I at least need to get an exception thrown when it does happen Your fear is founded. =) I think the danger is lower now, and it's getting lower all the time as we improve memory management of images (*major* improvements have landed in 43) but it's impossible to guarantee that canvas.drawImage() will never fail. I completely agree that we need to throw an exception in this case, and I'm planning to ship that in 44.

Thanks Seth and David. 2.5 is on 44, so hopefully this gets rolled out too. Based on the comments, I will block for 2.5 but with P3 priority, as danger is pretty low.

Mahe and I spoke about this. I told him that other than Seth's work on image memory use reduction, we're not going to have a non-canvas-based thumbnailing API in 44 (2.5) so I'm removing the blocker status here.