topcrash for 3.0b2 [@ MimeMultipart_parse_line ] earliest crash found is build 20090217025755 bp-b0cbe94d-cefe-4981-8565-30d0a2090305 "I was trying to look at an old meeting invite email that originated from a Lotus Notes client bp-ef99d6b3-7f15-44db-86b9-f053c2090307 0 thunderbird.exe MimeMultipart_parse_line mimemult.cpp:201 1 thunderbird.exe convert_and_send_buffer mimebuf.cpp:184 2 thunderbird.exe mime_LineBuffer mimebuf.cpp:272 3 thunderbird.exe MimeObject_parse_buffer mimeobj.cpp:275 4 thunderbird.exe MimeMultipartRelated_parse_eof mimemrel.cpp:1075 5 thunderbird.exe MimeMultipart_close_child mimemult.cpp:616 6 thunderbird.exe MimeMultipart_parse_line mimemult.cpp:185 7 thunderbird.exe convert_and_send_buffer mimebuf.cpp:184 8 thunderbird.exe mime_LineBuffer mimebuf.cpp:272 9 thunderbird.exe MimeObject_parse_buffer mimeobj.cpp:275 10 thunderbird.exe MimeMessage_parse_line mimemsg.cpp:232 11 thunderbird.exe convert_and_send_buffer mimebuf.cpp:184 12 thunderbird.exe mime_LineBuffer mimebuf.cpp:272 13 thunderbird.exe MimeObject_parse_buffer mimeobj.cpp:275 14 thunderbird.exe mime_display_stream_write mimemoz2.cpp:946 15 thunderbird.exe nsStreamConverter::OnDataAvailable nsStreamConverter.cpp:938 16 thunderbird.exe nsMailboxProtocol::ReadMessageResponse nsMailboxProtocol.cpp:593 17 thunderbird.exe nsMailboxProtocol::ProcessProtocolState nsMailboxProtocol.cpp:690 18 thunderbird.exe nsMsgProtocol::OnDataAvailable nsMsgProtocol.cpp:347 19 thunderbird.exe nsInputStreamPump::OnStateTransfer netwerk/base/src/nsInputStreamPump.cpp:508 20 thunderbird.exe nsInputStreamPump::OnInputStreamReady netwerk/base/src/nsInputStreamPump.cpp:398 21 xpcom_core.dll nsInputStreamReadyEvent::Run xpcom/io/nsStreamUtils.cpp:111 22 xpcom_core.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:510 23 xpcom_core.dll NS_ProcessNextEvent_P nsThreadUtils.cpp:227 24 thunderbird.exe nsBaseAppShell::Run widget/src/xpwidgets/nsBaseAppShell.cpp:170 25 thunderbird.exe nsAppStartup::Run toolkit/components/startup/src/nsAppStartup.cpp:192

Crashes at if (obj->options->state->partsToStrip.Length() > 0) http://hg.mozilla.org/comm-central/file/6fcaf32e4036/mailnews/mime/src/mimemult.cpp#l201 partsToStrip null?

Setting blocker ? to get attention.

partsToStrip is an object, not a pointer - it can't be null. Since the user isn't detaching/deleting attachments, it should have a length of 0.

much more likely that state is null - I'll put up a patch.

by inspection, this is my best guess as to what's crashing...

marking blocking+, since it's a topcrasher.

The best explanation I can come up with for what is happening here is that MimeMultipartRelated_parse_eof is getting called twice on a single instance. My patch at http://hg.mozilla.org/comm-central/rev/f0bdc29e9c1c changed the behavior so that if a second pass through happens, we will not fix-up body->parent or body->options to be equal to obj and obj->options, respectively. (The logic replaces relobj->headobj with the newly created node; on the second pass through, there relobj->headobj is no longer there to be replaced!) This would likely reslt in both the crash observed in this crasher, as well as this one in MimeInlineTextHTML_parse_eof that we have 5 of in 3.0b3pre: http://crash-stats.mozilla.com/report/index/03c18437-350c-4d31-94ab-d644d2090510 and this slightly different crash in the same place: http://crash-stats.mozilla.com/report/index/615903d3-9330-4b10-ae4e-129b02090510 In all cases it looks like the situation is caused by the multipart/related being embedded inside a multipart itself. Presumably mimemult.cpp has some scenario in which it does the call multiple times. I was unable to quickly cause gloda's test_mime_emitter.js to cause the problem but the stack traces aren't giving me much to go on and I haven't written a fuzzer. Hoping for bienvenu thoughts?

oh, the fact that it was a meeting invite made me suspect Lightning was involved, and I thought some of Lightning's mime use might end up with null state - that was just a vague recollection. Your theory is certainly possible, though nested mime multipart messages should be pretty common, so we'd need to figure out what's special about the crash messages. What we need is a message that causes the crash...

(In reply to comment #8) > Your theory is certainly possible, though nested mime multipart messages should > be pretty common, so we'd need to figure out what's special about the crash > messages. What we need is a message that causes the crash... So do we want to try the patch or try and get a test case?

I'd like to see if the patch makes the crashes go away from crash-stats - is there any more info in crash-stats about possible causes, especially if Lightning is involved? Generic libmime crashes are bad w/ gloda because gloda runs all messages through libmime, and I'm not sure that it backs off in the case of crashes indexing a particular message.

from last 3 months, 3 of 6 crashes with comments bp-da649106-7520-4be5-8c6a-3dc8c2090518 "...Every other invitation sent from Lotus Notes causes it." bp-3fc5525b-2a1a-4a47-93b8-eb6592090410 "Latest versions (from trunk) of TB3 and Lightning. Crashed when a particular invitation from Lotus Notes is displayed." bp-ef99d6b3-7f15-44db-86b9-f053c2090307 "Left computer. I went to sleep. Computer went to sleep. Woke up. Came back. Woke it up. Saw this." No longer topcrash for 3.0b2 - not even top 100. 27 crashes in 28 days However, #31 for 3.0b3pre. Not sure what can be deduced from that.

(In reply to comment #11) > from last 3 months, 3 of 6 crashes with comments > bp-da649106-7520-4be5-8c6a-3dc8c2090518 "...Every other invitation sent from > Lotus Notes causes it." Might be worth getting a note invite and attacjing it to this bug - do we know anybody at IBM we could ask this from ?

maybe post on the newsgroup and forums for someone with Lotus notes installed?

Will do.

This is what I received after I posted to the newsgroups. It did not crash my Thunderbird, but I don't use lightning.

I forwarded the info to my IBM contacts. Hopefully someone might take a look. I'm not at IBM anymore, so not much I can do.

@15 attachment id=381573 The attached ICS data is not compliant to the RFC: Sorry, your calendar could not be parsed. Error: Error was: Error at line 32: Illegal character in opaque part at index 4: CID: Cause: Caused by: Illegal character in opaque part at index 4: CID:<FFFF__=0ABBFF58DFF422698f9e8a93df938@us.ibm.com> Test-output with: http://severinghaus.org/projects/icv/ Same error has been seen before with previous LN 7.0 and we have had problems with ReminderFox also. Beside the non-compliance to RFC it seems to be unclear why TB (using LG) crashes. Any os/version/release available here?? CC to 'Fallen' also.

Comment on attachment 376713 [details] [diff] [review] proposed fix - protect against null options and options->state. agreed that our lack of forward progress on actually identifying the underlying problem justifies just landing the patch. However, I would request that at the very least we spin off a new bug if we mark this one fixed without actually addressing the underlying problem I believe exists.

I tried that ics file with lightning and it did not crash (w/o this patch). So we are still w/o a test case.

(In reply to comment #19) David I think testing that ICS data file with LG will just give an error -- as with the Validator (see #17). AFASIK both (V & LG) are based on the same sw-sources. So a test against this bug needs to receive a LN invitation directly into TB/LG. This is because of the MimeMultipart of such messages. From our ReminderFox iCal support I have an old msg which was send using LN7 and it seems to have the same error as the Validator shows. Not sure about the mimetype, but it's an S/MIME signed message with Content-Type: multipart/mixed What is the test environment required? Could give it a test, using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.21) Gecko/20090302 Thunderbird/2.0.0.21 --- but have to install LG 0.9.

Lightning displays it just fine, as near as I could tell. I attached the ics file and sent a message to myself - I could try sending it as a multipart alternative part, actually, which may be what Lotus Notes does (as opposed to multipart related).

multipart alternative didn't crash either. So, Ludovic, I think it would be good to see the whole message you got from Notes, instead of just the ics attachment.

> multipart alternative didn't crash either. So, Ludovic, I think it would be > good to see the whole message you got from Notes, instead of just the ics > attachment. Here you go.

Test w att id=381904: - the .eml format seems not to help here much, can open the eml as an external msg, read the text and ICS attachment/data, but seems not to op like a normal msg received via pop/imap - copied the source of the .eml to a local folder and needed to add a header line like: From - Tue Apr 08 13:45:35 2008 to get it working as a 'normal' msg. But is that the same behavior?? Maybe David can answer. - no problem opening that modifier msg from a TB folder also with the ICS attachment. LG sees the invitation and processes it normally, as ReminderFox does [But RmFx handles the ICS as an 'foreign' invitation -- not allowing to Accept/Decline etc, which lacks with LG0.9 -- separate bug!]. Resumee: not a real test case; tester(s) should receive an invitation DIRECTLY from a LN account Tested with: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.21) Gecko/20090302 Lightning/0.9 Thunderbird/2.0.0.21; ReminderFox 1.9.1

It did not crash for me when I received it, but I don't have lightning installed.

Let's just write a fuzzer. And by we, I mean ideally a volunteer with some spare cycles. Message generation library: http://mxr.mozilla.org/comm-central/source/mailnews/test/resources/messageGenerator.js Test code that uses it, demonstrating usage: http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/test/unit/test_mime_emitter.js The fuzzer doesn't actually need to verify the output of course. In fact, since part of the goal would be to gibberish-y structures, we arguably can't do that unless the fuzzer has a "legal" mode. For simplicity of initial development, I would propose having the fuzzer masquerade as a test that simply does not define any tests. If you do a "make SOLO_FILE=test_mime_fuzzer.js check-interactive" (in the objdir that corresponds to where the test lives, sourcew-wise) the make system takes care of the paths for you. Then at the prompt you could type "run_fuzzer(100)" or something instead of _execute_test() which would simply be a no-op.

Günter, are you saying you can reproduce this problem with a direct message, or that you can't reproduce it w/ a .eml file/local file message? I think Lightning streams the message when you click on it, so I don't think it distinguishes between a newly arrived message and an existing message, but if there's any doubt, copying an unread message like this to an imap inbox should fool anybody. I think a fuzzer is a fine idea, but we might have to know what's special about Lightning that's causing the crash in order to make the fuzzer exercise that permutation (e.g., the way Lightning streams a content part). Assuming that Lightning is involved at all.

(In reply to comment #27) > Günter, are you saying you can reproduce this problem with a direct message, or > that you can't reproduce it w/ a .eml file/local file message? Sorry if it's not clear. I never had a crash! Never had any problems with it! Not with the pure ICS file, not with the hacky imported .eml file/msg. Additionally I moved the "imported .eml file" from my "Local Folders" to an IMAP Inbox. Any action on it (display with folder pane, open the msg, add the invitation to any LG calendar [sqlite-, ics-type]) will NOT show any problems at all. So as asked before: what was the initial scenario the 'crasher' reported? OS, TB-, LG-version (my test env see #24 comment).

(In reply to comment #28) > So as asked before: what was the initial scenario the 'crasher' reported? > OS, TB-, LG-version (my test env see #24 comment). Upps, just noticed that info is included with http://crash-stats.mozilla.com/report/index/b0cbe94d-cefe-4981-8565-30d0a2090305 'Details': Mac OS X /10.5.6 9G55 So I just tested it on our MAC 10.5.6 (Mozilla/5.0 (Macintosh; U; Intel Mac OS X; de; rv:1.8.1.21) Gecko/20090302 Thunderbird/2.0.0.21) -- but without Lightning! ==>> also NO problem, no crash!

Test Invite from Lotus Notes 6.5 (ICS file) receive as attachment via webmail. No special settings were made to the meeting invite (no attachment, no "location", no "category"). Let me know if you want another ICS invite with other settings.

As a biany file this time.

Thank you for the ICS file, but we need the entire message that contains the ICS file if we are to have a hope of reproducing the crash. Also, (in general, to anyone providing attachments on this bug) please note whether you have actually produced a crash with the attachment. If you haven't, unless inspection of the MIME structure or ICS file shows obvious abnormalities from which we can extrapolate the likely problem, the attachment is unlikely to be of any use.

comment # 30 ..32 Peter send an invitation to me (to my 'testaccount' based on WIXxp, TB2.0.0.21/LG0.9). I had NO problem to receive it, opened, added the invitiation to the calendar, 'accepted' it with sending the reply msg. NO problem at all. Later today I will be able to test it on MAC and will report back. Will post the invitation msg as attachment to this bug. Günter

Invitation msg based on: Lotus Notes Release 6.5.5 November 30, 2005 Some names, id etc changed to respect privacy ;-)

Comment on attachment 376713 [details] [diff] [review] proposed fix - protect against null options and options->state. Okay, given lack of reproducible test case and its occasional appearance in crash-stats, I think its worth taking this and seeing what effects it has. sr=Standard8 subject to filing a bug as asuth suggests in comment 18.

Fix checked in. Bug 496886 filed for underlying issue.

re-opening...

I hate playing whack-a-mole with this crash, but it's the number one crash now, and we're not getting anywhere finding a message that exhibits this problem.

Comment on attachment 391734 [details] [diff] [review] more bullet-proofing Although it is top crash, I think it is just one user or one set of users - the windows version is exactly the same, and the additional dlls referenced under modules (e.g. RocketDock.dll) all appear to be the same (on a random straw poll of about 20 out of the 120 odd crashes listed). So I think someone has a testcase ;-) As we don't know who they are, we should definitely add this check for now.

yes, and judging from the short uptimes, they're usually crashing fairly close to startup - one possible cause is gloda trying to index the same message.

v.fixed - last crash is from build 20090731