User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1) Gecko/20090616 Firefox/3.5 (.NET CLR 3.5.30729) Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1) Gecko/20090616 Firefox/3.5 (.NET CLR 3.5.30729) Arabic letters are disconnected in Windows Vista in web pages such as Gamil and Forums. However they get connected in the normal way if you press ENTER. In other times, letters are typed correctly and connected, but when one word is edited, the problem of "being disconnected" reappears and when pressing Enter it disappears. Reproducible: Always Steps to Reproduce: 1. My system locale of the non-unicode was set to Arabic. Everything was fine. 2. The minute I changed it back to English, the problem appears. 3. by changing System Locale of non-unicode programs to Arabic solves the problem. When it is set to English, the problem begins.

Same problem.. I tested Beta4 on Fedora 11 and it was fine but in my windows vista it was horrible.. and it appears when I edit the text in any kind of text boxes.

Simon, any idea? Ahmed, talia, is this a regression? If so, could you help in figuring out a regression range?

The problem begins when editing the text after writing

I see this on XP also.

It doesn't happen in 3.1 beta 3. I'll try to get a more useful regression window.

Good: http://hg.mozilla.org/mozilla-central/rev/b618ca270e9b (2009-04-18-04) Bad: http://hg.mozilla.org/mozilla-central/rev/4e1bda073737 (2009-04-20-04) http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b618ca270e9b&tochange=4e1bda073737 Bug 332655 is the most likely suspect.

With respect to whether this is serious enough to block, see dupe bug 500492 comment 2: |I should also add that this does NOT affect the text |displayed in Arabic pages, and after I submit the text it will be displayed |correctly. | |It is annoying though, because the words don't look right it is difficult to go |back and correct mistakes without getting really confused.

In my opinion, this bug is serious enough to make many people consider even switching to another browser or staying with FF3. Since, with this bug present, people using Firefox to write in Arabic or Persian will have real difficulties writing emails, comments, etc.

Amir: that's fine, we're not pushing this as a forced update, so people are welcome to make that decision. I agree that it's bad, but I don't know if we can hold ship of the browser for it. Is there a workaround? Does reloading the form help? Highlighting all text? Something like that?

Pasting a lengthy conversation here that we had in #l10n: <Pike> linostar: what's your opinion on the severity? <linostar> Pike: as far as I see, it only affect the appearence of the input, and doesn't affect neither the submitted text nor the text already displayed in the page. Besides, it occurs only when the system locale for non-unicode is set to English <linostar> However, I can't say it is not severe, since I know a lot of Arabic users use English as default locale for non-unicode <linostar> The behavior of disconnected letters is not acceptable for an ordinary Arabic user. I am afraid it is an important issue. <Pike> linostar: the question is, would 3.5.1 be early enough to fix that, or do we need to slip 3.5 into the next month? At least that's what I see it boiling down to <linostar> Pike: in concise words, I don't advice releasing 3.5.1 without fixing this issue. Firefox 3.5 will appear then as if it has lost its Arabic language support. <smontaguAFK> hmm, this thing about the system locale is puzzling <smontaguAFK> (drive-by comment, I'm not back) <smontaguAFK> btw, my system locale for non-unicode is Hebrew, so apparently s/English/not Arabic/ <linostar> smontaguAFK: it seems that it is likely a font problem (perhaps the font used for displaying Arabic ,when the system locale is set to non-Arabic, is of bad quality) <Pike> linostar: as it's a regression, it shouldn't depend on the fonts <Pike> it has something to do on how we chunk the frames for text layout <linostar> why is it related to the system locale? <Pike> linostar: the bug only appears if you just enter arabic text into existing non-arabic text, right? <Pike> linostar: good question, apparently puzzling smontaguAFK, too <linostar> Pike: yes <linostar> it appears that the Arabic letters inherit the disconnecting feature from the English characters surrounding them <Pike> linostar: yeah, that seems to map to the bug 332655 that regressed this <smontaguAFK> Pike: yes, but shaping is supposed to happen happily across text frame boundaries <smontaguAFK> though perhaps we should check that that hasn't regressed too :S <Pike> smontaguAFK: if you don't have time, is there someone else that knows enough about the internals to check that? I wonder how to get something helpful back to beltzner <smontaguAFK> http://www.submission.org/quran/reader/arabic/v-02A.htm looks OK <Pike> smontaguAFK, linostar: when I mark text in some of the submissions (or what they are), I see that the background jumps up and down in height between the ' ' and the arab text. Not for all though <Pike> I wonder if that could be related <Pike> linostar: do you think you could create a reftest? a page with two textfields, one dynamically changed which should look like the other but doesn't? I don't know if you'd even have to fake key events, though

(In reply to comment #13) > Pasting a lengthy conversation here that we had in #l10n: > > <Pike> linostar: what's your opinion on the severity? > <linostar> Pike: as far as I see, it only affect the appearence of the > input, and doesn't affect neither the submitted text nor the text already > displayed in the page. Besides, it occurs only when the system locale for > non-unicode is set to English > <linostar> However, I can't say it is not severe, since I know a lot of > Arabic users use English as default locale for non-unicode > <linostar> The behavior of disconnected letters is not acceptable for an > ordinary Arabic user. I am afraid it is an important issue. > <Pike> linostar: the question is, would 3.5.1 be early enough to fix > that, or do we need to slip 3.5 into the next month? At least that's what I see > it boiling down to > <linostar> Pike: in concise words, I don't advice releasing 3.5.1 > without fixing this issue. Firefox 3.5 will appear then as if it has lost its > Arabic language support. > <smontaguAFK> hmm, this thing about the system locale is puzzling > <smontaguAFK> (drive-by comment, I'm not back) > <smontaguAFK> btw, my system locale for non-unicode is Hebrew, so > apparently s/English/not Arabic/ > <linostar> smontaguAFK: it seems that it is likely a font problem > (perhaps the font used for displaying Arabic ,when the system locale is set to > non-Arabic, is of bad quality) > <Pike> linostar: as it's a regression, it shouldn't depend on the fonts > <Pike> it has something to do on how we chunk the frames for text layout > <linostar> why is it related to the system locale? > <Pike> linostar: the bug only appears if you just enter arabic text into > existing non-arabic text, right? > <Pike> linostar: good question, apparently puzzling smontaguAFK, too > <linostar> Pike: yes > <linostar> it appears that the Arabic letters inherit the disconnecting > feature from the English characters surrounding them > <Pike> linostar: yeah, that seems to map to the bug 332655 that > regressed this > <smontaguAFK> Pike: yes, but shaping is supposed to happen happily > across text frame boundaries > <smontaguAFK> though perhaps we should check that that hasn't regressed > too :S > <Pike> smontaguAFK: if you don't have time, is there someone else that > knows enough about the internals to check that? I wonder how to get something > helpful back to beltzner > <smontaguAFK> http://www.submission.org/quran/reader/arabic/v-02A.htm > looks OK > <Pike> smontaguAFK, linostar: when I mark text in some of the > submissions (or what they are), I see that the background jumps up and down in > height between the ' ' and the arab text. Not for all though > <Pike> I wonder if that could be related > <Pike> linostar: do you think you could create a reftest? a page with > two textfields, one dynamically changed which should look like the other but > doesn't? I don't know if you'd even have to fake key events, though My system locale for non-unicode is set to Arabic (Saudi Arabia) and I have this problem (Windows Vista SP2) !!

I do not see this kind of issue with Telugu. (My system locale is set to English.) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5

(In reply to comment #12) > Is there a workaround? Does reloading the form help? Highlighting all text? > Something like that? Yes. If I cut all the text I typed and paste it again it will be displayed correctly, until I try to change it again of course. (In reply to comment #13) There's no need for English text to make the problem appear. It's just more severe when the Arabic I write is surrounded by English text (all the characters become disconnected, as apposed to just some characters).

Unfortunately, this issue is OS-independant. It appears also on a Linux box. (In reply to comment #12) > Is there a workaround? Does reloading the form help? Highlighting all text? > Something like that? Apperently, there is no practical workaround, since any further alteration on the input text will disconnect the Arabic letters again.

This is a test case that illustrates the behavior of undesired disconnection of Arabic letters.

I've attached a video describing the problem.. The second text box is the right one.

I think we'll have to leave this for 3.5.1. Let's get a fix worked up and on trunk ASAP.

(In reply to comment #12) > Amir: that's fine, we're not pushing this as a forced update, so people are > welcome to make that decision. I agree that it's bad, but I don't know if we > can hold ship of the browser for it. > > Is there a workaround? Does reloading the form help? Highlighting all text? > Something like that? I also agree that this can not be a blocker for the whole browser, but it might be a good idea to at least warn people about it in Know Issues section and discourage them from updating.

(In reply to comment #15) > (In reply to comment #13) > > Pasting a lengthy conversation here that we had in #l10n: > > > > <Pike> linostar: what's your opinion on the severity? > > <linostar> Pike: as far as I see, it only affect the appearence of the > > input, and doesn't affect neither the submitted text nor the text already > > displayed in the page. Besides, it occurs only when the system locale for > > non-unicode is set to English > > <linostar> However, I can't say it is not severe, since I know a lot of > > Arabic users use English as default locale for non-unicode > > <linostar> The behavior of disconnected letters is not acceptable for an > > ordinary Arabic user. I am afraid it is an important issue. > > <Pike> linostar: the question is, would 3.5.1 be early enough to fix > > that, or do we need to slip 3.5 into the next month? At least that's what I see > > it boiling down to > > <linostar> Pike: in concise words, I don't advice releasing 3.5.1 > > without fixing this issue. Firefox 3.5 will appear then as if it has lost its > > Arabic language support. > > <smontaguAFK> hmm, this thing about the system locale is puzzling > > <smontaguAFK> (drive-by comment, I'm not back) > > <smontaguAFK> btw, my system locale for non-unicode is Hebrew, so > > apparently s/English/not Arabic/ > > <linostar> smontaguAFK: it seems that it is likely a font problem > > (perhaps the font used for displaying Arabic ,when the system locale is set to > > non-Arabic, is of bad quality) > > <Pike> linostar: as it's a regression, it shouldn't depend on the fonts > > <Pike> it has something to do on how we chunk the frames for text layout > > <linostar> why is it related to the system locale? > > <Pike> linostar: the bug only appears if you just enter arabic text into > > existing non-arabic text, right? > > <Pike> linostar: good question, apparently puzzling smontaguAFK, too > > <linostar> Pike: yes > > <linostar> it appears that the Arabic letters inherit the disconnecting > > feature from the English characters surrounding them > > <Pike> linostar: yeah, that seems to map to the bug 332655 that > > regressed this > > <smontaguAFK> Pike: yes, but shaping is supposed to happen happily > > across text frame boundaries > > <smontaguAFK> though perhaps we should check that that hasn't regressed > > too :S > > <Pike> smontaguAFK: if you don't have time, is there someone else that > > knows enough about the internals to check that? I wonder how to get something > > helpful back to beltzner > > <smontaguAFK> http://www.submission.org/quran/reader/arabic/v-02A.htm > > looks OK > > <Pike> smontaguAFK, linostar: when I mark text in some of the > > submissions (or what they are), I see that the background jumps up and down in > > height between the ' ' and the arab text. Not for all though > > <Pike> I wonder if that could be related > > <Pike> linostar: do you think you could create a reftest? a page with > > two textfields, one dynamically changed which should look like the other but > > doesn't? I don't know if you'd even have to fake key events, though > > My system locale for non-unicode is set to Arabic (Saudi Arabia) and I have > this problem (Windows Vista SP2) !! My language for non-Unicode programs is set to Persian and I still face this problem (on Vista SP1). Persian and Arabic use the same script and layout algorithm with mostly the same letters. So, if the problem is related to the locale, then it should not be noticed with the locale set to Persian. Are you sure that this is related to the locale at all?

UBUNTU and MAC are confirmed to have the problem.

The problem is in this file nsBidiPresUtils.cpp start debugging from line 390 the the file is in this directory "../layout/base/"

smontagu and jfkthame are better targets

Reproduced here on Mac OS X using both FF 3.5 RC and current Minefield trunk. The breakage occurs during at the character offset that *was* a bidi direction boundary before the editing action that modifies the text (keystroke, cut/paste operation, etc). This position becomes a textRun boundary and prevents shaping. The failure can be seen in LTR text as well, if bidi boundaries are present and textRun offsets shift. In the attached screenshot, the "ffi" ligature has broken in the edit field as a result of deleting four characters at the beginning of the line, which caused these characters to shift across the position that was originally a direction boundary. Obviously, with Latin ligatures the issue is much less obvious or critical, but if this had been, say, Indic text it could have been seriously garbled.

That's weird. BuildTextRunsScanner::ContinueTextRunAcrossFrames is the relevant code here. Are we possibly seeing an empty text frame with the opposite direction interrupting a run of text in the same direction?

We successfully shape across an opposite-direction frame that has no content, I think; try data:text/html,<p>off<span style="direction:rtl; unicode-bidi: bidi-override; padding: 10px;"></span>icial</p> (the padding is to make the presence of the empty frame clearly visible).

I've posted a screen-capture video showing the behavior at http://screencast.com/t/mByMUnSj Here, the text contains some Hindi (with reordering and a half-form consonant) and some English (with a ligature), followed by a Hebrew letter which causes a direction boundary. Inserting text at the beginning of the field "pushes" the English and then the Hindi across the original position of the direction boundary, and the shaping can be seen to break at this point.

After many trails, this bug affect these languages: Arabic, Hebrew, Persian, Hindu This is a very obvious bug .. it should block the release of 1.9.1 until it get fixed.

I agree, this isn't something you can put off by putting it in "known issues". It's very serious because it affects anyone and everyone who writes emails, blogs, comments or even search queries in 3 or 4 different languages spoken by hundreds of millions of people on all operating systems with no practical workaround. I think 3.5 should wait a bit until this is sorted out.

I haven't yet succeeded in finding a fix for this. It seems likely that the source of the problem is that we rebuild text runs frame by frame during bidi resolution, so that the embedding levels of the frames which we use to determine whether text runs can cross frame boundaries are not yet updated. The quickest way to fix it for 1.9.1 would be just to back out bug 332655.

OK what will be the problem if we back out bug 332655 ?

After sleeping on it, I realize that the first paragraph of comment 33 is nonsense, because we don't actually rebuild the text runs during bidi resolution: we clear them then and rebuild them later. The real problem is as follows: Using the following steps to reproduce: 1) Type "Hello world" in an input field. 2) Position the caret between "Hello " and " world" 3) Type 2 Arabic characters that should connect to one another, e.g. سو On typing the second character we enter bidi resolution with this frame tree: Text(0)@0x2bb4ad0[0,6,F] next=0x2d72bd4 next-continuation=0x2d72bd4 {60,0,1933,900} [state=80224610] SELECTED [content=0x1c517c00] sc=0x2b979cc pst=:-moz-non-element< "hello " > Text(0)@0x2d72bd4[6,1,F] next=0x2d72c1c prev-continuation=0x2bb4ad0 next-continuation=0x2d72c1c {1993,0,603,900} [state=c0024600] SELECTED [content=0x1c517c00] sc=0x2b979cc pst=:-moz-non-element< "\u0633" > Text(0)@0x2d72c1c[7,7,T] prev-continuation=0x2d72bd4 {2596,0,2179,900} [state=c0424600] SELECTED [content=0x1c517c00] sc=0x2b979cc pst=:-moz-non-element< "\u064a world" > Bidi resolution then splits the third text frame, giving: Text(0)@0x2bb4c84[0,6,F] next=0x2d64b68 next-continuation=0x2d64b68 {60,0,1933,900} [state=80224610] SELECTED [content=0x40d0df70] sc=0x2b813d0 pst=:-moz-non-element< "hello " > Text(0)@0x2d64b68[6,1,F] next=0x2d64bb0 prev-continuation=0x2bb4c84 next-continuation=0x2d64bb0 {1993,0,596,900} [state=c0024610] SELECTED [content=0x40d0df70] [overflow=-4,0,600,900] sc=0x2b813d0 pst=:-moz-non-element< "\u0634" > Text(0)@0x2d64bb0[7,1,F] next=0x2d650b8 prev-continuation=0x2d64b68 next-continuation=0x2d650b8 {2589,0,2179,900} [state=c0424600] SELECTED [content=0x40d0df70] sc=0x2b813d0 pst=:-moz-non-element< "\u0648" > Text(0)@0x2d650b8[8,6,T] prev-continuation=0x2d64bb0 {0,0,0,0} [state=00024602] SELECTED [content=0x40d0df70] sc=0x2b813d0 pst=:-moz-non-element< " world" > The second text frame with an Arabic character is a non-fluid continuation of the first one with the same directionality and the same content, and so cannot share a textrun with it (bug 393758 comment 1). I can think of two possible ways to fix this: either make a fluid continuation in this situation, or adjust the frame offsets so that both Arabic characters end up in the same frame. Either way, I wouldn't like to risk putting such a fix in to 1.9.1 this late in the game. I'll address the impact of backing out bug 332655 in a separate comment.

The blocking decision is final, but thanks for your input. This will block the release of 3.5.1 hard, though.

smontagu: thanks for the information; I do think we should probably err on the side of optimizing for non bidi over supporting full bidi, if that helps shed relative merit of a backout of bug 332655 Also: can we make sure to get litmus tests on this?

This takes the route of making a fluid continuation when the directional run crosses a text frame boundary. I also added some comments and tightened up the code a little.

Awesome! WFM in basic tests on OS X. I'll run a Windows build in a while, but as this is all in platform-independent code I expect it'll be fine there too.

Yes, this resolves the problem on Windows as well (tried it on Vista).

Tryserver builds at https://build.mozilla.org/tryserver-builds/smontagu@mozilla.com-try-cd9adc435209/

Awesome :) will it be released in 1.9.1.0.1 or we have to wait 1.9.1.1 !!

I did a quick test on Linux, and it is working for me so far.

I Hope To Fix This Issue Soon

http://hg.mozilla.org/mozilla-central/rev/c017517703ae (In reply to comment #43) > will it be released in 1.9.1.0.1 or we have to wait 1.9.1.1 !! I don't think a 1.9.1.0.1 is planned. 1.9.1.1 is scheduled as a quick turn-around release within the next few weeks.

Since i updated to Firefox 3.5 final on XP, in some cases, although i couldn't quite understand which cases exactly, Hebrew diacritics (niqqud) are not combined with letters, but appear after them in input fields. It happened on several different sites: wikipedia, ynet.co.il and live.mozilla.org.il I told it to Tomer and Simon on http://live.mozilla.org.il/ and they say that it may be related to this bug. Just in case, i am uploading a screenshot.

(In reply to comment #49) > Since i updated to Firefox 3.5 final on XP, in some cases, although i couldn't > quite understand which cases exactly, Hebrew diacritics (niqqud) are not > combined with letters, but appear after them in input fields. > > It happened on several different sites: wikipedia, ynet.co.il and > live.mozilla.org.il > > I told it to Tomer and Simon on http://live.mozilla.org.il/ and they say that > it may be related to this bug. It sounds like this may well be the same bug. Could you download Simon's tryserver build (see comment 42) and confirm whether the problem is fixed in that version?

Requesting approval for 1.9.1.1 after a week baking on trunk. I can confirm that this also fixes the (Windows-only) Hebrew diacritics issue in comment 49. I more or less had no choice but to include the patch for bug 496006 in the merge to branch, so I've added its reftest too, as well as the mochitest for this bug. For "how bad is it if we don't take this patch", I can't put it better than comment 32: "It's very serious because it affects anyone and everyone who writes emails, blogs, comments or even search queries in 3 or 4 different languages spoken by hundreds of millions of people on all operating systems with no practical workaround."

Comment on attachment 387391 [details] [diff] [review] Branch patch Approved for 1.9.1.1. a=ss for release-drivers. Simon: Please also mark bug 496006 as fixed1.9.1.1 since you took those changes in this patch.

Verified fix on Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.1pre) Gecko/20090714 Shiretoko/3.5.1pre Ubiquity/0.1.5 and Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1pre) Gecko/20090714 Shiretoko/3.5.1pre

I just want to thank all Mozilla developers here for their diligence and responsiveness toward fixing bugs. This bug has finally gone away for me with today's release of 3.5.1. :)

Is the bug not entirely gone? http://clip2net.com/clip/m20788/1248219203-clip-23kb.jpg Should be: http://clip2net.com/clip/m20788/1248219678-clip-24kb.jpg It's hard to recreate, I think if you have multiple lines and add something to a previous line it will sometimes happen.

Happens to English letters in Arabic context too! http://clip2net.com/clip/m20788/1248221237-clip-20kb.jpg Probably had something to do with the direction of the text. (LTR in the first pic, RTL in the third)

(In reply to comment #59) > Happens to English letters in Arabic context too! > > http://clip2net.com/clip/m20788/1248221237-clip-20kb.jpg > > Probably had something to do with the direction of the text. (LTR in the first > pic, RTL in the third) I got reversed English like that too, but I can't reproduce it, but what I can see common (from the screenshot) is that it is a RTL text editing area and English text get reversed when getting into new line.

Please file a new bug on the reversed letters. It may or may not be related to this one.

(In reply to comment #60) > I got reversed English like that too, but I can't reproduce it, but what I can > see common (from the screenshot) is that it is a RTL text editing area and > English text get reversed when getting into new line. Since nsider uploaded the screenshots to 3rd party website, the files have been expired and I'm not able see it. I am able to reproduce a similar issue with Hebrew text. Have you opened an issue for that already?