When BasicCompositor/OMTC is used in combination with the GTK backend, various thread-unsafe GTK calls are made on the compositor thread inside nsWindow::StartRemoteDrawing and nsWindow::EndRemoteDrawing.

This is a multi-pronged assault on thread-unsafety in BasicCompositor, nsShmImage, and nsWindow. Some small changes inside BasicCompositor to ensure it is passing the same region/size info it internally uses throughout the entire composing process to the widget backend, so that they better agree on what size draw target they are working with. Significant changes to nsShmImage to both get rid of gdk calls (replacing them with raw X calls) and working directly with DrawTargets instead of gfxASurfaces. In nsWindow, getting rid of the obsolete GetThebesSurface() interface and replacing it with a more succinct GetDrawTarget(), as well making sure to get the underlying X drawable information when the window is created, so that we don't need to unsafely query this information from GTK when compositing is occurring.

Comment on attachment 8661448 [details] [diff] [review] Remove GTK calls from compositor thread Review of attachment 8661448 [details] [diff] [review]: ----------------------------------------------------------------- ::: widget/gtk/nsWindow.cpp @@ +6338,5 @@ > + if (!dt) { > + RefPtr<gfxXlibSurface> surf = new gfxXlibSurface(mXDisplay, mXWindow, mXVisual, size); > + if (!surf->CairoStatus()) { > + dt = gfxPlatform::GetPlatform()->CreateDrawTargetForSurface(surf.get(), surf->GetSize()); > + } Can we just create the Cairo draw target around the mXDisplay, mXWindow, mXVisual, size instead of also creating a thebes surface?

(In reply to Jeff Muizelaar [:jrmuizel] from comment #2) > Comment on attachment 8661448 [details] [diff] [review] > Remove GTK calls from compositor thread > > Review of attachment 8661448 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: widget/gtk/nsWindow.cpp > @@ +6338,5 @@ > > + if (!dt) { > > + RefPtr<gfxXlibSurface> surf = new gfxXlibSurface(mXDisplay, mXWindow, mXVisual, size); > > + if (!surf->CairoStatus()) { > > + dt = gfxPlatform::GetPlatform()->CreateDrawTargetForSurface(surf.get(), surf->GetSize()); > > + } > > Can we just create the Cairo draw target around the mXDisplay, mXWindow, > mXVisual, size instead of also creating a thebes surface? The problem is that the only other way we can do that is by directly using cairo_xlib_surface_create. If used within the widget/gtk code that will cause us to use system cairo to do it. With gfxXlibSurface we will get a tree cairo xlib surface, which is much safer for now.

Try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=dc02e6dc5cad