When creating textures prodecurally using canvas 2d, it's not always desirable to use the entire area of the canvas as a texture. In cases where multiple textures of different sizes are created reusing the same canvas 2d as a scratch space, having a fast, on-GPU, way to copy only parts of that canvas into a new texture would be great. Here's a concrete use case: in Shumway, we're working towards a system where we use a scratch 2d canvas for drawing individual shapes which are then used as textures in a WebGL-based compositor. Hence, we don't create all our textures during some startup phase, but have to be able to constantly create new ones at runtime. Right now, the only way to do that while using a single scratch canvas is using getImageData, forcing a roundtrip to main memory.

Does framebufferTexture2d+copyTexImage2d not satisfy this?

Wouldn't this require uploading the entire canvas 2d as a texture before using copyTexImage2d?

Ah, canvas2d->texture. If canvas2d is GPU accelerated, then this can be optimized away, though we don't do this yet. Is there an advantage to using the same canvas2d surface to render multiple chunks? (Versus resizing the canvas2d before rendering each chunk you want to upload)

Mm. That also requires us to be able to somehow hook up the Canvas 2D backing texture to the WebGL context. Not impossible, but requires breaking through some boundaries. I suppose the C2D texture will be exported via EGLImage or similar anyway, so we can bring it in. It sounds like what you want is TexImage2D taking a DOM element, but specifying a rectangle to upload instead of taking the whole thing. It's a bit of a double-edged sword though... if the DOM element is not backend by a GL texture that we can import, then we'll end up doing even more work to copy over (since we'd have to copy the rectangle out first). Though -- what Jeff suggested with FramebufferTexture2D seems to be a better approach -- being able to take a <canvas> and directly binding it as a texture. So what you really want (I think) is a variant of EGLImageTargetTexture2DOES / EGLImageTargetRenderbufferStorageOES -- where you can just treat a canvas 2d directly as a texture, no copying needed. (Or use RenderbufferStorage and CopyTexImage2D to copy if you want.) We could spec this out such that binding a canvas forces a "resolve" of the canvas texture, flushing any pending drawing to the texture before binding it. Would certainly be interesting.

(In reply to Jeff Gilbert [:jgilbert] from comment #3) > Ah, canvas2d->texture. If canvas2d is GPU accelerated, then this can be > optimized away, though we don't do this yet. > > Is there an advantage to using the same canvas2d surface to render multiple > chunks? (Versus resizing the canvas2d before rendering each chunk you want > to upload) IIUC, canvas2d resizing should be avoided at all costs, as it has significant overhead. (In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #4) > Though -- what Jeff suggested with FramebufferTexture2D seems to be a better > approach -- being able to take a <canvas> and directly binding it as a > texture. So what you really want (I think) is a variant of > EGLImageTargetTexture2DOES / EGLImageTargetRenderbufferStorageOES -- where > you can just treat a canvas 2d directly as a texture, no copying needed. > (Or use RenderbufferStorage and CopyTexImage2D to copy if you want.) > > We could spec this out such that binding a canvas forces a "resolve" of the > canvas texture, flushing any pending drawing to the texture before binding > it. Would certainly be interesting. We'd need the copying, as we want to reuse the canvas2d for creating other textures, but this approach sounds great to me. mbx, do you agree?

(In reply to Till Schneidereit [:till] from comment #5) > We'd need the copying, as we want to reuse the canvas2d for creating other > textures, but this approach sounds great to me. mbx, do you agree? Or you just create another canvas2D, if you can assume that creating a 2D canvas is equivalent to creating a texture, right?

The goal here is to paint tiles using canvas2D and cache them in texture atlases on the GPU. So, one canvas2D as a scratch pad and several large texture atlases on the GPU side.

I think the best path forward for this is still getting TexImage2D(canvas) calls accelerated properly. There's two levels of this: 1. Turn TexImage2D(canvas) into a GPU-side blit. 2. Implement copy-on-write mechanics for TexImage2D(canvas). This way, for your usecase, you can call TexImage2D(canvas), bind that to an FB, and use CopyTexImage to copy it to a texture (or CopyTexSubImage for atlasing), with no extra copy needed. Step one is much easier, but only gets you half the way there, but still miles better than now.

I filed bug 980749 to track this.

Shumway want to use subrects of a scratch canvas as textures for WebGL compositing. Michael says this depends on general canvas to WebGL copying (bug 729385). This is a high priority if we want to enable Shumway's WebGL compositor.

I upload my patch queue for completeness here. (Since all of them are in review/WIP status and they have some dependency) The patch queue contains patches for several related bugs: 1. They fixes skiaGL on linux (Bug 1057554) 2. They uses skiaGL's ability to handle alpha division in GPU (Bug 1001069) 3. They avoid download-than-upload pattern for this bug while applicable - If non-alpha-mul'ed version needed, it just upload the ImageData::GetData, otherwise it blit the ImageData->GetTexture() to target texture - Full canvas upload case should very similar where we can get the texId by DrawTarget::GetNativeSurface API (Bug 729385) They should enhance all kinds of TexImage2D type (TexSubImage2D missed) texture upload at the same time (crop/y-flip/rbswap/alpha {de}mul in GPU blit). It shows some enhancement in http://www.billbaxter.com/slowCanvasToTex and the patch for bug 1001069 shows near 5~8x enhancement in http://jsperf.com/cwahlers-getimagedata/2 while use skiaGL. To make it better, I think I will need some help from someone who knows WebIDL/VM/JIT, to avoid GetImageData readback from GPU while possible. I will start to create one simple test page for webgl.texImage2D(ImageData) API to test/debug my patch first, then make the patches reviewable. Anyone wants to try these patches should apply the patch in following order to avoid conflict: test_skia.patch test_webgl.patch test_carry_texture.patch test_imagedata_webgl.patch

(In reply to Chiajung Hung [:chiajung] from comment #11) > Created attachment 8495132 [details] > WIP > > I upload my patch queue for completeness here. (Since all of them are in > review/WIP status and they have some dependency) > > The patch queue contains patches for several related bugs: > 1. They fixes skiaGL on linux (Bug 1057554) > 2. They uses skiaGL's ability to handle alpha division in GPU (Bug 1001069) > 3. They avoid download-than-upload pattern for this bug while applicable > - If non-alpha-mul'ed version needed, it just upload the > ImageData::GetData, otherwise it blit the ImageData->GetTexture() to target > texture > - Full canvas upload case should very similar where we can get the texId > by DrawTarget::GetNativeSurface API (Bug 729385) > > They should enhance all kinds of TexImage2D type (TexSubImage2D missed) > texture upload at the same time (crop/y-flip/rbswap/alpha {de}mul in GPU > blit). It shows some enhancement in > http://www.billbaxter.com/slowCanvasToTex and the patch for bug 1001069 > shows near 5~8x enhancement in http://jsperf.com/cwahlers-getimagedata/2 > while use skiaGL. > > To make it better, I think I will need some help from someone who knows > WebIDL/VM/JIT, to avoid GetImageData readback from GPU while possible. > > I will start to create one simple test page for webgl.texImage2D(ImageData) > API to test/debug my patch first, then make the patches reviewable. > > Anyone wants to try these patches should apply the patch in following order > to avoid conflict: > test_skia.patch > test_webgl.patch > test_carry_texture.patch > test_imagedata_webgl.patch If these patches add an accelerated path for GetImageData and upload of ImageData, I really do not think they are what we want.

A better approach might be to special-case binding a HTMLCanvasElement.. the canvas already tracks its dirty regions (especially for simple operations). We can track that and re-upload as needed on bind?

(In reply to Chiajung Hung [:chiajung] from comment #11) > Created attachment 8495132 [details] > WIP > > I upload my patch queue for completeness here. (Since all of them are in > review/WIP status and they have some dependency) > This sounds interesting but I think you'd get a lot more feedback if you attached the individual patches. That way we can comment on individual files/lines, etc. I don't think I understand the point of the getImageData/putImageData changes. You want to avoid a readback when it's the whole canvas that was putImageData()ed? That seems like it could be good, but it's a separate problem from this bug, no? > - If non-alpha-mul'ed version needed, it just upload the > ImageData::GetData, otherwise it blit the ImageData->GetTexture() to target > texture This sounds promising.

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) from comment #14) > (In reply to Chiajung Hung [:chiajung] from comment #11) > > Created attachment 8495132 [details] > > WIP > > > > I upload my patch queue for completeness here. (Since all of them are in > > review/WIP status and they have some dependency) > > > > This sounds interesting but I think you'd get a lot more feedback if you > attached the individual patches. That way we can comment on individual > files/lines, etc. Sure. Most of them need merge/split to fit individual bugs. I will do that once I briefly test most combination of affected part. > > I don't think I understand the point of the getImageData/putImageData > changes. You want to avoid a readback when it's the whole canvas that was > putImageData()ed? That seems like it could be good, but it's a separate > problem from this bug, no? No, if it is a whole canvas, the only choices would be (1) use the texture directly(shared context/EGLImage) (2) blit to target to fix Yflip/alpha. The case is: ============================[1]===================================== var imgData = 2d.getImageData(cropRect); <-- We download the image from GPU here gl.texImage2D(imgData); ============================[1]===================================== If we can download the image from GPU only when imgData.data is accessed, we can avoid a malloc and a GPU DMA at least for case[1]. For example, ============================[2]===================================== var imgData = 2d.getImageData(cropRect); var pixel = imgData.data; <-- Download here is preferred for (var i = 0; i < pixel.length; i++) if (i % 4) pixel[i] = 255 - pixel[i]; //Inverse the color gl.texImage2D(imgData); ============================[2]===================================== The idea is: Download those data if you really need them in CPU! Why we download the data if we really don't care them anyway? (case [1]) Note: Download while getImageData is the easiest way to implement the API, which avoid any problem like: ============================[3]===================================== var imgData = 2d.getImageData(crop); 2d.drawXXXX; var pixel = imgData.data; ============================[3]===================================== In such case, we would need handle DrawTarget dirty in ImageData class if we delay the "pixel resolution(alpha div+download)". The really problem which makes me give up this way is because ImageData::WrapObject enumerate all field and calls to ImageData::GetData as soon as CanvasRenderingContext2D::GetImageData return'ed. That way, I can't postpone the "pixel resolution" until first Javascript read. Another similar idea is to use GrallocBuffer/PBO as the ImageData::mData's memory arena. Which can avoid explicit download at all(GPU sync is still implied), but there maybe some security concerns. > > > - If non-alpha-mul'ed version needed, it just upload the > > ImageData::GetData, otherwise it blit the ImageData->GetTexture() to target > > texture > > This sounds promising.