While working on removing the string used to store lists of fonts in nsFont (bug 280443), I realized that the fontlist manipulation code in nsMathMLChar.cpp would probably be better implemented in gfx. Not sure exactly where the right division line is yet but the swizzling of fontlists within nsMathMLChar::StretchInternal is probably way too complex for what it's trying to do. For example, running the roughly 300 mathml reftests, the SetFontFamily method is called 132,000 times (!!!). I know we're not so concerned about MathML speed but the complexity of this code impedes progress on gfx text features so I think it would be worthwhile to restructure this a bit. There's a discussion on bug 947654 as to what the precise font fallback behavior should be for MathML elements. I think this bug should cover the work in gfx to support whatever behavior it is that MathML code needs. Right now my hunch is that the right line is "given the fontlist for this element, give me the first available math-table font that supports this character" and "does this font support this set of characters". I don't think the actual fallback for stretchy characters (e.g. replace char X with parts chars A B C) should live in gfx.

Now that the mstyle stuff has been cleaned up, I believe the stretchy code is currently the least efficient part of MathML and could be optimized. After the changes to bug 663740, I wonder whether: - SetFontFamily is actually still needed (perhaps only for the generic Unicode table?) - We could replace EnumerateFamilies with just checking the list of fonts in the gfxFontGroup (again, I'm not sure about the generic Unicode table) I didn't found time recently to read in details the discussion about the fallback, John's patches and to think about that in general, though.

(In reply to Frédéric Wang (:fredw) from comment #1) > After the changes to bug 663740, I wonder whether: > - SetFontFamily is actually still needed (perhaps only for the generic > Unicode table?) > - We could replace EnumerateFamilies with just checking the list of fonts in > the gfxFontGroup (again, I'm not sure about the generic Unicode table) More precisely and as I recall, before bug 663740 nsMathMLChar had a nsRenderingContext and SetFontFamily was supposed to set the current font-family on that rendering context, and guess that the font exists on success. Also, EnumerateFamilies was needed because we worked with font-family / nsFont instead of gfxFont*. Now I suspect that we could essentially browse the list of fonts in the gfxFontGroup, obtained from the current font-family on the <mo>. However, I'm not sure about the edge cases with fallback fonts and generic font-family...

(In reply to Frédéric Wang (:fredw) from comment #2) > Now I suspect that we could essentially browse the list of fonts in the > gfxFontGroup, obtained from the current font-family on the <mo>. However, > I'm not sure about the edge cases with fallback fonts and generic > font-family... This is a bad idea, the abstraction needs to be one level up. MathML code should not be doing random-access on the fontlist, it should be querying gfxFontGroup to select a font that's best for supporting a given character or set of characters. With downloadable fonts, managing the fontlist is tricky because the state can change as fonts are pulled in. MathML code really shouldn't have to know or understand that sort of thing.

I think what we need to figure out is (1) what are the parameters used to pick a font for stretchy chars and (2) what does this code need to use it for. Whatever the various methods used in nsMathMLChar.cpp are for, it looks to be doing it *very* inefficiently, even for a single stretchy character, there are lots of textruns created, lots of fonts being experimented with *repeatedly* and lots of calls to SetFontFamily. It really seems like we should be able to structure this such that it's a one-pass process over <fontlist> + <math fallbacks> and avoid shaping the same textrun with the same font repeatedly. To view textruns generated by MathML code: export NSPR_LOG_MODULES=textrun:5

Just to provide a quick sample of the inefficiencies with the MathMLChar code, take the example below, from the existing opentype-stretchy.html reftest: <math displaystyle="true"> <mrow><mo>&#x2A1B;</mo></mrow> </math> This causes a *huge* number of textruns to be created. Some rebuilding of textruns can be written down as due to the use of a downloadable font, but not the amount here. Here are the textruns built *just* for the content above. First column is the count of the number of times a line appeared in the log: 15 (textrun) fontgroup: [Cambria Math] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN 15 (textrun) fontgroup: [STIXIntegralsD] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN 15 (textrun) fontgroup: [XITS Math] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN 15 (textrun) fontgroup: [stretchy,serif] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN 5 (textrun) fontgroup: [stretchy] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN 3 (textrun) fontgroup: [stretchy] default: serif lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [−] ENDTEXTRUN 2 (textrun) fontgroup: [stretchy] default: serif lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size: 20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN So the simple string [⨛] is shaped with Cambria Math, STIXIntegralsD, XITS Math and the downloadable font 15 times for *each* font!!!

Regarding the text runs, IIRC they are used to: - get a glyph index from a character. There was some functions like GetGlyphID but they were not implemented on all platforms. Moreover, for bug 945183 we need to get the glyph ID of the mirrored glyph after application of the Unicode mirroring + ltrm feature and the later didn't seem possible without shaping with a textrun. - measure a glyph (given its glyph index or char unicode value) on all platforms. Karl said this could finally become possible with the Azure project, but suggested to use gfxTextRun for now. - paint the glyph. That's possible without a textrun but since we store them on the nsMathMLChar class, we just reuse that. - handle both glyph by index & character. This is mostly because we handle old fonts and Unicode-based constructions. I think we could move to glyph index only once we can rely exclusively on MATH fonts. Note that if the user has math fonts installed with the appropriate glyphs, the textruns will only be created for the first font (and perhaps again when web fonts arrive). Here the open-stretchy testcase intentionally uses an exotic glyph (integral with a top bar) to be sure that it is not available in installed math fonts, so the code is particularly inefficient since all the fonts are tested. I believe the old code was even worse because we had to call all the TryVariants/TryParts routines for the various fonts everywhere in nsMathMLChar (both measuring phase and painting phase). Now this is only done during the textrun creation phase. But of course, I'd really like to see this improved...

Shaping is designed to be efficient for multiple instances of the same word on a page, so inefficiency from multiple text should be coming from multiple allocations and copies rather than repeated font work. Some of the logic in nsMathMLChar is more complex than it needs to be. Making the code as simple as possible while still doing the right thing is more important than performance here. Welcome performance improvements often come with simplification too.

(In reply to Karl Tomlinson (needinfo?:karlt) from comment #7) > Shaping is designed to be efficient for multiple instances of the same word > on a page, so inefficiency from multiple text should be coming from multiple > allocations and copies rather than repeated font work. > > Some of the logic in nsMathMLChar is more complex than it needs to be. > Making the code as simple as possible while still doing the right thing is > more important than performance here. Welcome performance improvements > often come with simplification too. I think we're on the same page here, I think this code needs to be simpler. The fact that *so* many identical textruns are being recreated is to me more an indication that something needs to be thought through a bit better. The complexity here makes changing code in other parts of Gecko *much* harder, which is definitely what I experienced working on bug 280443. For this bug, as described in the third paragraph of the description, I think code that's sniffing through a list of fonts should live in gfx. Both for efficiency and code simplicity. Plus the loading behavior associated with downloadable fonts means that iterating over a list of fonts is not a simple task that should be exposed through some form of iterator-like API.

As I said 947654 comment 43, one of the issue is that the nsMathMLChar code is really just doing some kind of text shaping so it should really mostly belong to gfx (and perhaps other parts like e.g. text-shadow & background selection support should try to be in nsMathMLmoFrame and/or shared with nsTextFrame). Since it currently lives in layout/ we try or want to use gfx API or reinvent the wheel via textrun manipulations. It's possible that if we move most of the code to gfx, such APIs won't really be necessary... I believe the first step would be to remove support for old fonts + implementing the Unicode fallback differently (perhaps by creating a fake OpenType MATH table or nsOpenTypeGlyphTable instance). This would already simplify the code considerably, avoid the need for external *.properties files, and help migration to gfx. Currently, we handle both the old and new fonts, which sometimes do not behave exactly the same. I wonder how long we want to wait before removing support for these old fonts? (There is another problem that the MathML code is maintained by volunteers and so does not evolve as fast as other parts of Gecko. It was broken 8 years ago until Karl repaired it and hadn't evolved much until I found time to finally work on the OpenType MATH support recently. I suspect it would have been even harder to switch to arrays of font-family before the recent changes, because the font families were use everywhere in both measuring and painting phase. So seeing the glass half full, it was not as difficult as it could have been to do bug 280443 for MathML)

FYI, some people at MathUI said that recent versions of Firefox are slower to display MathML for very large pages. I wonder if that has something to do with bug 663740.

(In reply to Frédéric Wang (:fredw) from comment #10) > FYI, some people at MathUI said that recent versions of Firefox are slower > to display MathML for very large pages. I wonder if that has something to do > with bug 663740. This should be relatively easy to test. Can you come up with a set of pages/content to test with? Given the structure of the existing code I think there's probably ample room for perf improvements. :)

IIUC, this happened for large pages generated with LaTeXML, so checking the XHTML papers of ArXiv would do: http://arxmliv.kwarc.info/ I haven't had time to check exactly, but just wanted to raise the issue to motivate further improvements and before I forget.

Testing in bug 1043358 shows a significant decrease in rendering time when the stretch performing <mo> elements are replaced with <mtext> elements.

(In reply to James Kitchener from comment #13) > Testing in bug 1043358 shows a significant decrease in rendering time when > the stretch performing <mo> elements are replaced with <mtext> elements. In that context, is there any testing I can do the help?

Some numbers: bug 1043358 comment 46 bug 1061349 comment 4

In bug 1139709, after the improvement patches there I observed the testcase spending >50% of its load time in nsMathMLChar::GetMaxWidth (mostly in StretchInternal).

@Karl: I started implementing the access to math values as well as the shaping of math stretchy operator in Harfbuzz, see https://github.com/behdad/harfbuzz/pull/241. Can you please check the proposed API (hb-ot-layout.h and hb-ot-shape.h) and indicate if you think that would be enough to replace nsMathMLChar completely? Thanks!

Is there a separate API for largeop? nsMathMLChar.h has some other flags also. LARGER and SMALLER might be the only ones that are really necessary. Perhaps they can be replaced with scaling when drawing the buffer, assuming a new buffer is used for the character? Would be nice to pass an enum or flag, instead of 0/1, for horizontal/vertical, but that's something for hb developers to consider. hb_direction_t is too different to overload I guess.

(In reply to Karl Tomlinson (ni?:karlt) from comment #18) > Is there a separate API for largeop? I initially proposed one, but Khaled pointed out that we can just ask normal stretching to the HB_OT_MATH_CONSTANT_DISPLAY_OPERATOR_MIN_HEIGHT size (or any additional workaround we have for integral etc). Getting the "max orthogonal advance" is not needed for large operator. > LARGER and SMALLER might be the only ones that are really necessary. Perhaps > they can be replaced with scaling when drawing the buffer, assuming a new > buffer is used for the character? I think the MathML spec asks LARGER, which is how I currently implemented it. I don't know exactly in which situation SMALLER is needed but that should not be hard to implement. hb-ot-shape-math-stretchy reuses the same buffer. > > Would be nice to pass an enum or flag, instead of 0/1, for > horizontal/vertical, > but that's something for hb developers to consider. hb_direction_t is too > different to overload I guess. Yes, I had the same thought. Other points to discuss are: - Do we want the Unicode table in Harfbuzz or keep doing it ourselves or just remove it? - Will that be enough to implement the font selection (this bug)? Maybe we want to avoid mixing stretchy operators from different fonts and so always use the font of the base glyph and hence stop at the first successful call to hb-ot-shape-math-stretchy? Or otherwise, we'll have to call hb-ot-shape-math-stretchy several times and select the best size which may not really improve performance here.

(In reply to Frédéric Wang (:fredw) from comment #19) > I think the MathML spec asks LARGER, which is how I currently implemented > it. I don't know exactly in which situation SMALLER is needed but that > should not be hard to implement. With a quick look through Gecko, it seems normal or nearer are usually used, with larger for roots, and smaller only for the maximum width. Would be interesting to know what other rendering engines do re nearest or larger. Should nested parentheses always get larger? > Other points to discuss are: > - Do we want the Unicode table in Harfbuzz or keep doing it ourselves or > just remove it? If we can expect a better MATH font to be available, then we can remove Unicode table support, but I suspect we still have many systems without a better MATH font. If there are some MATH fonts on the system, then it is less clear whether to use the Unicode table with the author specified font or to look for a MATH font. The MATH font will presumably give more size options. Some experimentation, may answer this question. It seems the Unicode table fallback could be done in Harfbuzz (though I'm not familiar with the details of implementation there), but if it is eventually going to be phased out, then there's probably not much point adding it to harfbuzz. > - Will that be enough to implement the font selection (this bug)? Maybe we > want to avoid mixing stretchy operators from different fonts and so always > use the font of the base glyph and hence stop at the first successful call > to hb-ot-shape-math-stretchy? Or otherwise, we'll have to call > hb-ot-shape-math-stretchy several times and select the best size which may > not really improve performance here. I don't know exactly what the proposal was for this bug. Harfbuzz support provides the stretching, but not font selection. I'd guess always using the font of the base glyph would be a bit restrictive as the base glyph may be in a font that does not have a MATH table. Stopping at the first successful call to hb-ot-shape-math-stretchy seems reasonable, but, if it provides Unicode table fallback, finding a MATH font may be better. I don't think it is necessary to try several MATH fonts. It won't be practical to call hb-ot-shape-math-stretchy, or even check for a MATH table, on every font for fallback, so this will need to be restricted to the set of author-specified fonts + some known MATH fonts. If searching for a font to shape with the Unicode table, then I guess calling hb-ot-shape-math-stretchy on each font will be slow. The optimal way would be to search for a font that supports all the Unicode characters required and then shape.

(In reply to Karl Tomlinson (ni?:karlt) from comment #20) Regarding the target size, the MathML spec says that "operators should stretchy to cover the target size" which means NS_STRETCH_LARGER (which is actually larger or equal). This is how it is implemented in WebKit. For the preferred width, WebKit takes the maximum of all possible constructions for the current font for vertical operators (we do not know the target size) and simply the target size for vertical operators. The MATH table only describes how to extend delimiter until "that is enough to achieve the size goal" and does not have equivalent parameters, so I'm not sure if they really specify a rule. The TeX book in the chapter 17 mentioned in NS_STRETCH_NEARER comment describes stretching of |\left| and~|\right| which are equivalent to stretching of symmetric fences in MathML. However, they allow the size to be smaller: "It is usually best not to cover the formula completely, however, but just to come close; so \TeX\ allows you to specify two parameters [...] The minimum delimiter size is taken to be at least $y\cdot f/1000$, and at least $y-\delta$. Appendix~B sets $f=901$ and $\delta=5\pt$". Appendix G rule 19 describes the same rule. Note that with the TeX definition, NS_STRETCH_NEARER is actually NS_STRETCH_LARGER with a smaller size so it might not be needed to have a separate parameter for that. More generally, I think all the parameter could be implemented by specifying a minsize and maxsize (which exists in MathML): - NS_STRETCH_LARGER: minsize = target size, maxsize = infity - NS_STRETCH_NORMAL: minsize = target size - 10%, maxsize = target size + 10% - NS_STRETCH_NEARER: minsize = max(target size - 10%, target size - 5pt), maxsize = infity (or target size + something if we do not follow TeX rules) - NS_STRETCH_SMALLER: minsize = 0, maxsize = target size What do you think?

(In reply to Frédéric Wang (:fredw) from comment #21) > - NS_STRETCH_SMALLER: minsize = 0, maxsize = target size Well that's probably not very good for that case. However, I wonder whether we really need this for the preferred max width. Also, I suspect NS_STRETCH_NORMAL and NS_STRETCH_NEARER could just be interpreted as NS_STRETCH_LARGER with a smaller target size. In the same font, the size variants should in theory grow in the order they are tested and finally we can try the assembly. The "+ something" seems only useful if we are testing different fonts where we want to get the best match over all the fonts. But I'm not sure it's really worth it if we move to the situation where we try only the first MATH font.

So I think the rule of thumb is that a font with a MATH table will give the best math rendering in general not only for stretchy operators but also for unicode coverage, math values, font features etc So we want to try and move to that situation. Doing math layout with Unicode-only font will only be useful for basic equations or for people with low expectation on rendering quality. Ideally, more fonts with a MATH table should be designed to accompany popular text fonts and should be available by default on systems. Given this and your answer, I propose to do the following: 1) We DOM/CSS context is - the base Unicode character (+ its CSS direction) - the font-family specified by author + some known fallback MATH fonts. - a target size 2) We do the following attempts to select a font from the list: a) We try and find a MATH font with a given glyph for the base character. b) If there is a Unicode construction for that base character, find the first font that has glyphs for all the Unicode characters involved in the construction. c) Otherwise, find the first font with a glyph for the base character. For a), the proposed Harfbuzz API allows to check whether a font has a MATH table (just a non-null pointer check) but not whether it has a MathGlyphConstruction for some glyph. This can be added if necessary but that will probably make the font selection slower. Also, using different MATH fonts can make rendering style inconsistent with constructions coming from a different than the base size. For b), we can probably start simplifying our Unicode fallback by only using glyph assembly (not size variants) and finding constructions by binary search in a hardcoded table rather than loading the *.properties file. This is what I started to do in Harfbuzz https://github.com/behdad/harfbuzz/pull/241/files#diff-db2d1e0cef191d004e282d35a21f44aeR215 but as you said it's probably not worth doing it if that's going to go away and that would allow to simplify/improve the Harfbuzz proposal. 3) Once we have a font then a) If we have a MATH font, use Harfbuzz API to do hb_ot_shape_math_stretchy_max_orthogonal_advance (for preferred width computation) and hb_ot_shape_math_stretchy (for selecting the construction when we know the target size and doing the shaping). b) If we have a font with Unicode construction, do the same with our own implementation. We only have limited glyph and no overlap so that can be simpler. c) Otherwise just use the base character (maybe with the scale fallback). As I previously say, assuming that the constructions in a given MATH font have increasing sizes and with the simplification of using only glyph assembly for the Unicode fallback the stretching in a) and b) can be reduced to a "NS_STRETCH_LARGER".

Comment 23 sounds sensible for stretchy characters. I'm not sure whether you were intending that for other characters also, but preferring MATH fonts for over author-specified fonts for regular characters would be quite a departure from the normal CSS font selection preferring author-specified fonts and then looking for a language-supporting font in fallback. Perhaps that is not a bad departure. I often wish Firefox would use my preferred fonts instead of author-specified fonts. I'm not sure what to think about minsize/target/maxsize. It corresponds with minsize/maxsize in MathML, but we don't want hb_ot_shape_math_stretchy to fail when there is nothing between minsize and maxsize. We want the nearest size available. What might differ in different situations is what is considered "nearest". But if this is always "larger or equal to", then there is no need to provide other variations. A target size alone is sufficient. The difficulty with "larger or equal to" requirements is that often "larger or roughly equal to" is what is really meant, and then "roughly" becomes vague and really depends on how much larger the other option is. I think we actually always have a preferred/target size. My personal opinion is that finding the nearest glyph and scaling to the preferred size is best, but I know we have different preferences with scaling. "nearest" may be in a geometric / log scale sense where 2/3 and 3/2 are equidistant from 1. I'm happy to leave the choice to you.

(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #24) > Comment 23 sounds sensible for stretchy characters. > > I'm not sure whether you were intending that for other characters also No, the present bug entry is only for stretchy operators. Although ideally people will use MATH fonts for the math text too so that math layout constants and rendering style makes sense. > I'm not sure what to think about minsize/target/maxsize. > What might differ in different situations is what is considered "nearest". Since there can be several definitions of "nearest", I believe the best for the Harfbuzz API would probably be to pass a function pointer (maybe NULL) hb_bool_t isNearer(hb_position_t a, hb_position_b) so that Harfbuzz can find the nearest option among the possible output sizes. One of my problem was that we have two different kinds of stretching: either use a size variant or use a glyph assembly, and my choice of "larger" was probably influenced by the latter. For the size variant, it is obvious how to find the "nearest" size from a list. For glyph assembly, the MATH spec defines an iterative process to progressively grow the size that would allow to do that to: > To ensure that the width/height is distributed equally and the symmetry of the shape is preserved, following steps can be used by math handling client. > > Assemble all parts by overlapping connectors by maximum amount, and removing all extenders. This gives the smallest possible result. > Determine how much extra width/height can be distributed into all connections between neighboring parts. If that is enough to achieve the size goal, extend each connection equally by changing overlaps of connectors to finish the job. > If all connections have been extended to minimum overlap and further growth is needed, add one of each extender, and repeat the process from the first step. However, I adding extender one by one did not sound efficient so I directly implemented the glyph assembly stretching using some direct computation to find the repetition of extenders and overlap that gives the best upper approximation of target size: https://github.com/fred-wang/harfbuzz/blob/MATH-2/src/hb-ot-shape-math.cc#L303 It was less clear how to make the algorithm more general, but now I think this calculation can be followed by some decrement of repeatCountExt and increase of connectorOverlap to find a better result with respect to some definition of "nearest". I could also calculate a lower bound for repeatCountExt so that I have an interval [repeatCountExt-, repeatCountExt+] to test with a binary search...

(In reply to Frédéric Wang (:fredw) from comment #25) > Since there can be several definitions of "nearest", I believe the best for > the Harfbuzz API would probably be to pass a function pointer (maybe NULL) > > hb_bool_t isNearer(hb_position_t a, hb_position_b) > > so that Harfbuzz can find the nearest option among the possible output sizes. So thinking again to this, something like IsSizeOK is not needed with the proposal of comment 23 because once we have a font selected we always try and find the best match for the request. So we only want IsSizeBetter which essentially has three cases: least upper bound (LARGER, LARGEOP, INTEGRAL), greatest lower bound (SMALLER) or nearest (NEARER and NORMAL). So I guess Harfbuzz should just receive an enum parameter with these three cases (+ maybe a special flag to distinguish LINEAR or LOG scale). > It was less clear how to make the algorithm more general, but now I think > this calculation can be followed by some decrement of repeatCountExt and > increase of connectorOverlap to find a better result with respect to some > definition of "nearest". I could also calculate a lower bound for > repeatCountExt so that I have an interval [repeatCountExt-, repeatCountExt+] > to test with a binary search... Thinking again to this, there are actually only two repeatCountExt to check so that should be fine.

(In reply to Frédéric Wang (:fredw) from comment #26) > (In reply to Frédéric Wang (:fredw) from comment #25) > > Since there can be several definitions of "nearest", I believe the best for > > the Harfbuzz API would probably be to pass a function pointer (maybe NULL) > > > > hb_bool_t isNearer(hb_position_t a, hb_position_b) > > > > so that Harfbuzz can find the nearest option among the possible output sizes. > > So thinking again to this, something like IsSizeOK is not needed with the > proposal of comment 23 because once we have a font selected we always try > and find the best match for the request. > > So we only want IsSizeBetter which essentially has three cases: least upper > bound (LARGER, LARGEOP, INTEGRAL), greatest lower bound (SMALLER) or nearest > (NEARER and NORMAL). So I guess Harfbuzz should just receive an enum > parameter with these three cases (+ maybe a special flag to distinguish > LINEAR or LOG scale). I agree harfbuzz doesn't need IsSizeOK. I don't have a strong opinion re the choice between enum and function. The function is versatile. It is less convenient to use than an enum, but I guess the default (NULL) is usually appropriate, so that's not a problem. If using that approach, it needs a target size or closure parameter, to know which of a or b is nearer.

(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #27) > I don't have a strong opinion re the choice between enum and function. > The function is versatile. It is less convenient to use than an enum, but I > guess the default (NULL) is usually appropriate, so that's not a problem. I think the enum will be more convenient for the implementation of the glyph assembly stretching since you can basically try to approximate the target size and then adjust a bit the number of parts / overlap to get the best size around that first approximation. Whereas a IsSizeBetter function would be too general and would not provide a logical order to test the size. I didn't check what we do, though. Note that Harfbuzz public API is in C so actually we can not have optional parameters and all HarfBuzz users will have to explicitly specify a function or an enum. But actually do we really need all these stretchy options or is it just a legacy of old implementation that we can get rid of? In the past, I never really tried to think about what we do and just tried to preserve the current behavior... I mean, as I see the MathML/MATH spec seems to only require the LARGER and there is only this TeXBook suggestion to reduce a bit the target size for fences. The NEARER and NORMAL allowing an interval around the target size seems specific to our implementation (and an extension of the TeXBook suggestion). At least I didn't use that when I implemented stretching in WebKit and Khaled thinks this is not needed by other math engines. Preferred width computation seems specific to web engines (or at least engines aware of CSS linebreaking). But I'm not clear about the use of SMALLER for that. If we choose to overestimate the width (as we do elsewhere in the MathML code) then for vertical stretching, we basically only want the maximum width for all size variants & pieces of the assembly and for horizontal stretching, we can just use LARGER on the target size (WebKit actually uses the target size itself). > If using that approach, it needs a target size or closure parameter, to know > which of a or b is nearer. Yes, that was obviously a mistake on my side.

For stretching, Gecko uses only nearest and larger really. NEARER only makes a difference if other fonts are going to be considered. Gecko usually uses nearest (or similar). If an expression includes a mixture of different fences ||{}[](), then these may have slight differences in height. The larger (or equal) approach could lead to the outer fences being considerably larger than the inner fences. This sounds undesirable in textstyle at least IMO. Maybe displaystyle too. I'm not sure. If an implementation wants an operator of exact size, then it will need to scale, and I guess the best glyph to scale is the nearest. Again this seems useful in textstyle at least. It seems that there is reason to think that clients may have different concepts of best glyph in different situations, but nearest and larger are probably enough to cover that. Variant glyphs are reasonably simple to handle I guess? Perhaps the choice here can be left to the client to determine. Would it make sense for harfbuzz to focus on APIs for glyph assembly and its minimum size? Am I overlooking complexities with variant glyphs that harfbuzz would hide because it knows the target size? SMALLER is for the case where the maximum size of the stretch is known, through a maxsize attribute I guess, and so the width of only smaller glyphs needs to be considered. I think it is not actually used right now because the maxsize attribute is not handled. SMALLER is only a stretch flag because of the implementation. It would make more sense to have a maxsize a parameter to hb_ot_shape_math_stretchy_max_orthogonal_advance(). But if variants are easy enough to query then perhaps harfbuzz should just provide an API for the size of the glyph assembly?

(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #29) > If an expression includes a mixture of different fences ||{}[](), then these > may have slight differences in height. The larger (or equal) approach could > lead to the outer fences being considerably larger than the inner fences. > This sounds undesirable in textstyle at least IMO. Maybe displaystyle too. > I'm not sure. I'm not sure I understand this, I see two cases: 1) <mrow><mo>[</mo><mo>(</mo><mo>{</mo> SOMECONTENT <mo>}</mo><mo>)</mo><mo>]</mo></mrow> 2) <mrow><mo>[</mo><mrow><mo>(</mo><mrow><mo>{</mo> SOMECONTENT <mo>}</mo></mrow><mo>)</mo></mrow><mo>]</mo></mrow> In 1), per the MathML spec the target size is the max height of the non-stretchy element in the <mrow>, that is SOMECONTENT (I'm not sure that we only consider non-stretchy elements btw). All the fences will then use the same target size. In 2), braces will use target size the height of SOMECONTENT, parenthesis the same + some delta, brackets parenthesis + some delta + some other delta. Indeed, that can lead to parenthesis being much bigger, but I'm guessing that a) the TeX rule of using reducing a bit the target size could help to reduce the increase. b) Fences in the same font provide similar size, so they won't really get bigger after the inner stretch that is some other delta = 0. Of course we would have to try... > If an implementation wants an operator of exact size, then it will need to > scale, and I guess the best glyph to scale is the nearest. Again this seems > useful in textstyle at least. I'm not really sure that considering the nearest upper instead of just the nearest is really problematic. And I think MATH-aware engines won't do the scale anyway. > Variant glyphs are reasonably simple to handle I guess? Perhaps the choice > here can be left to the client to determine. Would it make sense for > harfbuzz to focus on APIs for glyph assembly and its minimum size? I think so. Khaled already suggested to exclude the base size as well as other hacks we do for display operator size from the stretch API so it makes sense to only keep glyph assembly. > Am I overlooking complexities with variant glyphs that harfbuzz would hide > because it knows the target size? No, it's only a loop testing the size and finding the largest one. The only difference with our implementation is that it uses the AdvanceMeasurement value from MathGlyphVariantRecord Table which may lead to different values that the one provided by standard functions to retrieve glyph metrics (and hopefully closer to the designer's expectation). And the shaping is essentially a no-op. > SMALLER is for the case where the maximum size of the stretch is known, > through a maxsize attribute I guess, and so the width of only smaller glyphs > needs to be considered. I think it is not actually used right now because > the > maxsize attribute is not handled. SMALLER is only a stretch flag because of > the implementation. OK, maxsize is specific to the MathML implementation. But actually I'm not sure it's really used in practice so the gain we obtain for the preferred width implementation is not clear. > It would make more sense to have a maxsize a parameter > to > hb_ot_shape_math_stretchy_max_orthogonal_advance(). But if variants are easy > enough to query then perhaps harfbuzz should just provide an API for the size > of the glyph assembly? OK, we could just have hb_ot_shape_math_stretchy_max_orthogonal_advance for the glyph assembly. Khaled, what do you think about exposing: 1) variant count 2) variant id, variant advance (probably in the same function) 3) glyph assembly shaping 4) glyph assembly advance and italic correction (probably in the same function)

(In reply to Frédéric Wang (:fredw) from comment #30) > 3) glyph assembly shaping > 4) glyph assembly advance and italic correction (probably in the same > function) I meant "max_orthogonal_advance", but actually this is not really needed. We can just use 3) with a target size 0 and then the x_advance or y_advance of the last glyph in the output buffer will give what you want. Also, I believe I can easily add some boolean to 3) in order to check whether we get better result with the lower size o = o_min and r = r_min - 1 (using notations from http://www.maths-informatique-jeux.com/blog/frederic/?post/2016/04/16/OpenType-MATH-in-HarfBuzz)

So coming back to the initial idea without trying to rewrite the whole stretch code for now: Currently, StretchInternal browses the list of font family names and call StretchEnumContext::EnumCallback on it. For each name, we search whether the font has a MATH table, or a non-Unicode *.properties table, or fallback to the Unicode table. For a generic font name, we actually always use the Unicode table. After bug 1007090, I think we could just try and get the fm->GetThebesFontGroup()->GetFirstMathFont() and use it for an nsOpenTypeTable. Otherwise, just fallback to the Unicode table. The only difference is that the Unicode table will no longer be tested for each single font name but that's probably not a serious issue. I believe in that case, we no longer need to append the "font.name.serif.x-math" list, as it is done automatically during style resolution? Karl, WDYT?

(In reply to Frédéric Wang (:fredw) from comment #32) > After bug 1007090, I think we could just try and get the > fm->GetThebesFontGroup()->GetFirstMathFont() and use it for an > nsOpenTypeTable. I think we need a good reason to deviate from the CSS-specified font search. I can agree with arguments for some deviations, but not to this extent. One argument is that a font with MATH table support for stretching a particular character will be a better choice than an author-specified font without MATH table support. The Unicode table does not support size variants, only construction from parts, and so such a MATH font would be expected to support more sizes. I can therefore agree with the argument that it is reasonable to hope that the MATH font will provide more consistency over all sizes, and so can be preferred over a font with just Unicode table support. The same argument doesn't really apply while we still have mathfontSTIXGeneral.properties because that does have size variants. However, we don't really want to keep that, and are just keeping it for systems with no MATH fonts, and so I think we can even consider these old STIX fonts as just a fallback for when there are no MATH fonts. I'm not happy, however, with the assumption that only the first MATH font need be checked for supporting a particular character. CSS is designed to support different fonts providing different characters. A MATH font with support for stretching only a subset of characters seems reasonably likely, particularly with downloaded fonts that may be trimmed to reduce size. Thus far, comments in this bug have assumed that we need only consider the first MATH font with support for the character. It seems this should be support for stretching the character, not just rendering the character at standard size. There has been an assumption that support for stretching the character should mean that this font is used for all desired sizes. It seems reasonably likely that some MATH fonts may provide support for more sizes than others, and so perhaps the first font with support for the character at an /acceptable/ size would be best, but I think I can agree it is reasonable to experiment with the simpler option of just using the first that would support stretching the character. The other advantage of a MATH font is that it is likely to include information for better positioning of sub/super-scripts, but this benefit would be limited to certain situations. e.g. sums and integrals. > Otherwise, just fallback to the Unicode table. > The only difference is that the Unicode table will no longer be tested for > each single font name but that's probably not a serious issue. > Otherwise, just fallback to the Unicode table. The complication there, I think, is that some fonts don't have U+23AE INTEGRAL EXTENSION. Provided we have fonts in font.name.serif.x-math with support for U+23AE, that will avoid the problem of picking up problem fonts in fallback. Hopefully author-specified fonts would not have this problem. > I believe in that case, we no longer need to append the > "font.name.serif.x-math" list, as it is done automatically during style > resolution? I expect there is a way that fonts could be iterated without explicitly adding font.name.serif.x-math, but I don't know the details.