Found by Christoph Diehl's "Faulty" fuzzer, see bug 777067 This is similar to bug 968833, but now for the PCompositableTransaction protocol.

Similar to bug 968833, I suggest sec-crit here. While the present faulty session gives a nice assert failure, that's because it was only calling a virtual method that turned out to be asserting on the current object type. Looking at CompositableTransactionParent.cpp there are places that static_cast between layer types based on untrusted inputs exactly like bug 968833.

To give just one example: http://hg.mozilla.org/mozilla-central/file/a96affc08b1c/gfx/layers/ipc/CompositableTransactionParent.cpp#l158 case CompositableOperation::TOpPaintTextureRegion: { MOZ_LAYERS_LOG(("[ParentSide] Paint ThebesLayer")); const OpPaintTextureRegion& op = aEdit.get_OpPaintTextureRegion(); CompositableParent* compositableParent = static_cast<CompositableParent*>(op.compositableParent()); CompositableHost* compositable = compositableParent->GetCompositableHost(); ThebesLayerComposite* thebes = static_cast<ThebesLayerComposite*>(compositable->GetLayer());

Interesting bit in this Faulty log: [Faulty] pickle field {int} of value: 8 changed to: 7 In CompositableType enum, 8 is COMPOSITABLE_IMAGE, 7 is BUFFER_TILED.

hit an avatar of this in bug 971262. Patch coming up.

Assigning to bjacob because he has a patch.

Comment on attachment 8374559 [details] [diff] [review] fix-PCompositableTransaction-casts [Security approval request comment] How easily could an exploit be constructed based on the patch? Not easily. To someone with good understanding of our codebase (please send your resumes) the patch gives away that passing forged messages to PCompositorTransaction can result in unsafe memory accesses. It remains to figure how exactly to forge such messages (not hard, since it was found by fuzzing generic pickles) and how to actually exploit that (harder). Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem? No. Which older supported branches are affected by this flaw? All of them. Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be? No backport ready, but wouldn't be difficult. How likely is this patch to cause regressions; how much testing does it need? Not very risky, but not trivial.

Comment on attachment 8374559 [details] [diff] [review] fix-PCompositableTransaction-casts sec-approval=dveditz

landed on central https://hg.mozilla.org/mozilla-central/rev/3bfe9190b72d

v1.1 will be EOL on Monday, so it seems highly unlikely that this is going to be uplifted there :)

If we were going to backport these to 1.2 and 1.3, that time was a long time ago.

Marking [qa-] for desktop QA verification. FxOS QA may choose to verify at a later date.

Applied cleanly to SeaMonkey 2.26.1 (Gecko 29-based) https://hg.mozilla.org/releases/mozilla-release/rev/304350ccb7ac