This might be related to MathML's weird behavior of changing the DOM when a <mo> contains leading/trailing whitespace.

Nightly: bp-9468a162-f725-4319-9675-97a142120703

On Windows 7: bp-557eedc1-5ceb-4458-a1f3-2ee462120704.

getSelection().toString() runs FlushPendingNotifications, and nsMathMLTokenFrame::Init calls nsIContent->SetText("", aNotify = false) but it seems that nsNodeUtils::CharacterDataChanged() is required to update the range and that is only called when aNotify is set. I wonder whether nsIContentSerializer::AppendText() (implemented in nsPlainTextSerializer) should sanity-check its arguments, even though the core problem is layout changing the DOM during frame construction.

Wallpapering nsIContentSerializer::AppendText() to sanity check arguments gets us as far as trying to repaint the selection. Assertion failure: startOffset <= startParent->Length() && endOffset <= endParent->Length(), at /home/karl/moz/dev/content/base/src/nsContentIterator.cpp:1203 #5 0x00007f8b8af00651 in nsContentSubtreeIterator::Init (this=0x4532cc0, aRange=0x3d1a380) at /home/karl/moz/dev/content/base/src/nsContentIterator.cpp:1202 #6 0x00007f8b8ad062d2 in mozilla::Selection::selectFrames (this=0x39b72a0, aPresContext=0x3018c20, aRange=0x3d1a380, aSelect=true) at /home/karl/moz/dev/layout/generic/nsSelection.cpp:4026 #7 0x00007f8b8ad06cbe in mozilla::Selection::Repaint (this=0x39b72a0, aPresContext=0x3018c20) at /home/karl/moz/dev/layout/generic/nsSelection.cpp:4193 #8 0x00007f8b8acff529 in nsFrameSelection::RepaintSelection (this=0x3b049a0, aType=1) at /home/karl/moz/dev/layout/generic/nsSelection.cpp:1752 #9 0x00007f8b8ac0a4d0 in PresShell::RepaintSelection (this=0x40f3a20, aType=1) at /home/karl/moz/dev/layout/base/nsPresShell.cpp:1561 #10 0x00007f8b8abcccee in nsDocViewerFocusListener::HandleEvent (this=0x3b38780, aEvent=0x4532c60) at /home/karl/moz/dev/layout/base/nsDocumentViewer.cpp:3529

My patch in bug 785720 fixes this crash.