On closer looking, it seems like it's simpler if CharBuffer is just always UTF-16, which isn't different from today. That lets us avoid certain mess like having to have UTF-8 parsing of regular expressions, for example.

Comment on attachment 8988052 [details] [diff] [review] Patch Review of attachment 8988052 [details] [diff] [review]: ----------------------------------------------------------------- ::: js/src/frontend/TokenStream.h @@ +329,5 @@ > // WARNING: TokenStreamPosition assumes that the only GC things a Token > // includes are atoms. DON'T ADD NON-ATOM GC THING POINTERS HERE > // UNLESS YOU ADD ADDITIONAL ROOTING TO THAT CLASS. > > + /** The type of this token. */ (off topic) are we switching to this "/** ... */" comment style, instead of "//" or maybe "///" ? I thought we were switching from "/* ... */" to "//" at least. @@ +1067,5 @@ > }; > > +class TokenStreamCharsShared > +{ > + using CharBuffer = Vector<char16_t, 32>; might be nice to add some comment that explains why this is char16_t, instead of template with CharT.

Pushed by jwalden@mit.edu: https://hg.mozilla.org/integration/mozilla-inbound/rev/e20c1d6852e4 Move TokenStreamCharsBase::tokenbuf and related functions into a new TokenStreamCharsShared base class, make the character type always char16_t, and rename it/them to 'charBuffer' for clarity. r=arai