This bug was filed from the Socorro interface and is report bp-406f94a7-397c-400a-9afb-e97a52150320. ============================================================= Got this on Windows XP 32-bit using Firefox 37 beta 7, build ID: 20150319212106. I was only able to reproduce this once so, unfortunately, I don't have the STR. Seems like only 32-bit Windows platforms are affected. Also, there is a pretty big growth for the crash rate in Firefox 37 beta: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=ReleaseData%28void*%2C+unsigned+int%29#tab-sigsummary

Why was this filed in plugins? It's pretty clearly not a plugin bug. It's hard to tell for sure, but this appears to be a double-free of a nsScriptError, and I wonder if any of the logic at http://hg.mozilla.org/releases/mozilla-beta/annotate/790546ceb89f/xpcom/base/nsConsoleService.cpp#l287 changed in a way that might be causing problems.

On nightly this crash appears to have started in low volume at or around 20141215. https://crash-stats.mozilla.com/search/?signature=%3DReleaseData%28void*%2C+unsigned+int%29&date=%3E2014-10-01&release_channel=nightly&product=Firefox&_facets=signature&_facets=build_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-build_id Nothing in the regression range https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5288b15d22de&tochange=f14dcd1c8c0b jumps out at me as an obvious candidate. There is a rewrite of ReplaceSubstring, but I don't see how that would cause this symptom. Nathan, do you have any ideas about this one?

(In reply to Benjamin Smedberg [:bsmedberg] from comment #2) > On nightly this crash appears to have started in low volume at or around > 20141215. > > https://crash-stats.mozilla.com/search/ > ?signature=%3DReleaseData%28void*%2C+unsigned+int%29&date=%3E2014-10- > 01&release_channel=nightly&product=Firefox&_facets=signature&_facets=build_id > &_columns=date&_columns=signature&_columns=product&_columns=version&_columns= > build_id&_columns=platform#facet-build_id > > Nothing in the regression range > https://hg.mozilla.org/mozilla-central/ > pushloghtml?fromchange=5288b15d22de&tochange=f14dcd1c8c0b jumps out at me as > an obvious candidate. There is a rewrite of ReplaceSubstring, but I don't > see how that would cause this symptom. > > Nathan, do you have any ideas about this one? I don't think it could be ReplaceSubstring--at least with that regression range--because the change for ReplaceSubstring that landed was backed out in that same range. (The actual fix for ReplaceSubstring didn't land until 15 December, so I guess it would be in the 2014-12-16 nightly?) Nothing else in that range seems obvious, either. Is it possible it could be the merge range prior to that and there was just insufficient volume for the 2014-12-12 nightly, or something like that? The stack for the report looks kinda bizarro. realloc to DOM performance bits to style sheet bits?

When I look at the last 28 days of data, it doesn't look to be rising strongly, but it's there consistently at least for the last few betas. On a query for b6 crashes, it's at #37 with 0.33% of total crashes.

[Tracking Requested - why for this release]: Somewhat high crash (#28, 0.49%) on 38 beta

So as far as I can tell, nsScriptError has an mMessage field whose data is null and flags are F_SHARED|F_TERMINATED. Does that violate some invariant? These string classes are hard to DXR for so I can't be sure. But F_SHARED+nullptr strikes me as bad. We've been going on a move constructor spree lately. Perhaps did this string get its data pointer moved away? Or, is this enough information to construct a repro? Maybe just do something that causes a script error and see what happens to mMessage?

(In reply to David Major [:dmajor] from comment #6) > So as far as I can tell, nsScriptError has an mMessage field whose data is > null and flags are F_SHARED|F_TERMINATED. > > Does that violate some invariant? These string classes are hard to DXR for > so I can't be sure. But F_SHARED+nullptr strikes me as bad. Comments suggest that this is explicitly OK: http://mxr.mozilla.org/mozilla-central/source/xpcom/string/nsTSubstring.h#1118 There's also several instances of assigning literal F_SHARED | F_TERMINATED flags. Not being able to DXR things suggests that there's a(nother) concrete reason for converting our string classes to use templates instead. Hooray! > We've been going on a move constructor spree lately. Perhaps did this string > get its data pointer moved away? Possible, though we haven't added move constructor support to our string classes yet. (There is a bug open to do that, however.) > Or, is this enough information to construct a repro? Maybe just do something > that causes a script error and see what happens to mMessage? Maybe....

Those comments say that having F_TERMINATED is allowed (and in fact required) when you have F_SHARED. That part is fine. The thing I'm wondering about is having mData==nullptr when you're F_SHARED. That I don't see anywhere in the comments.

(In reply to David Major [:dmajor] from comment #8) > Those comments say that having F_TERMINATED is allowed (and in fact > required) when you have F_SHARED. That part is fine. The thing I'm wondering > about is having mData==nullptr when you're F_SHARED. That I don't see > anywhere in the comments. Ah, sorry, my mistake; I interpreted the "nullptr" in "F_SHARED+nullptr" as a shorthand for "null-terminated", not that mData was nullptr in these cases. I think you're correct: F_SHARED with a nullptr mData shouldn't be happening. The question is then: how did we get to the point where we assigned mData nullptr? I haven't seen anything in the string code that suggests we can wind up at that point yet.

How are you determining what the flags are at this point? Groveling through the memory, which is...included with the minidump?

The flags were fortunately still sitting in a register.

Not tracking as it is not (yet?) a top crash but please resubmit if it starts to become more important

Something funny is going on. There are several files loaded that we don't usually see. These are Windows binaries so it's not necessarily malware/addons/AVs but it sure is suspicious. 75% (146/195) vs. 6% (2367/40996) AcLayers.dll 88% (171/195) vs. 26% (10495/40996) mpr.dll 79% (155/195) vs. 34% (14140/40996) winspool.drv 49% (95/195) vs. 8% (3215/40996) sfc.dll 49% (96/195) vs. 9% (3834/40996) sfc_os.dll And even though this is a main-process crash, an unusually high number of comments mention Flash crashes (so many that I don't think they're talking about other crashes in general).

From the crash signature [@ ReleaseData ], the affected versions are: - Nightly: 47.0a1, 45.0a1, 44.0a2 - Aurora: 46.0a2, 45.0a2 - Beta: 45.0b1, 45.0b2, 44.0b1, 44.0b2, 44.0b7, 44.0b9

This spikes on the e10s-beta46-noapz a/b experiment. Currently #18, with 137 crashes, 0.94%. It's 0.8% higher than the control group.

I found many reports have this call stack: Frame Module Signature Source 0 xul.dll ReleaseData(void*, unsigned int) xpcom/string/nsSubstring.cpp 1 xul.dll nsTArray_Impl<mozilla::net::nsHttpHeaderArray::nsEntry, nsTArrayInfallibleAllocator>::DestructRange(unsigned int, unsigned int) xpcom/glue/nsTArray.h 2 xul.dll nsTArray_Impl<mozilla::net::nsHttpHeaderArray::nsEntry, nsTArrayInfallibleAllocator>::ReplaceElementsAt<mozilla::net::nsHttpHeaderArray::nsEntry, nsTArrayInfallibleAllocator>(unsigned int, unsigned int, mozilla::net::nsHttpHeaderArray::nsEntry const*, unsigned int) xpcom/glue/nsTArray.h 3 xul.dll mozilla::net::HttpChannelChild::OnStartRequest(nsresult const&, mozilla::net::nsHttpResponseHead const&, bool const&, mozilla::net::nsHttpHeaderArray const&, bool const&, bool const&, unsigned int const&, nsCString const&, nsCString const&, mozilla::net::NetAddr const&, mozilla::net::NetAddr const&, unsigned int const&) netwerk/protocol/http/HttpChannelChild.cpp 4 xul.dll mozilla::net::HttpChannelChild::RecvOnStartRequest(nsresult const&, mozilla::net::nsHttpResponseHead const&, bool const&, mozilla::net::nsHttpHeaderArray const&, bool const&, bool const&, unsigned int const&, nsCString const&, nsCString const&, mozilla::net::NetAddr const&, mozilla::net::NetAddr const&, short const&, unsigned int const&) netwerk/protocol/http/HttpChannelChild.cpp

For crashes containing the signature [@ ReleaseData ], about 54.0% has the stack [0] ReleaseData [1] nsTArray_Impl<T>::DestructRange [2] nsTArray_Impl<T>::RemoveElementsAt [3] mozilla::net::nsHttpRequestHead::~nsHttpRequestHead [4] mozilla::net::HttpBaseChannel::~HttpBaseChannel about 11.0% has the stack [0] ReleaseData [1] nsTArray_Impl<T>::DestructRange [2] nsTArray_Impl<T>::RemoveElementsAt [3] mozilla::net::nsHttpResponseHead::~nsHttpResponseHead [4] mozilla::net::nsHttpResponseHead::`scalar deleting destructor'

Marking affected for 46, I don't think we need to track for 46 though. There only seem to be a few crashes here for 47. Please do nominate it for tracking in 47/48 if that changes significantly.

This crash is not an e10s release blocker, but we should fix it to reduce e10s' overall crash rate.

Looking just at 46.b5, which was our longest running beta build during the experiment, it's pretty clear these are all very early in startup. Search faceted on uptime: https://crash-stats.mozilla.com/search/?product=Firefox&signature=%3DReleaseData&version=46.0b5&_facets=signature&_facets=version&_facets=uptime&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-uptime As Kanru mentions in comment 17 most appear to be related to mozilla::net::HttpChannelChild::Release(). Jason, is there anyone on the netwerk team that might be able to look at this, give us some feedback?

Valentin, Can you have a look at this?

I'm currently looking at some of the crash reports, but considering comment 17 and the crash address this is most likely the same as bug 1247982.

This seems to be more or less the same as bug 1247982. I expect e10s makes it stand out more, but it's a problem on non-e10s too. The crashes in nsHttpRequest/ResponseHead seem to always crash at the same address, and to have a non-corrupted stack trace. There are also other crashes in ReleaseData with stack traces that are a bit more confusing.

Stats: https://crash-stats.mozilla.com/report/list?date=2016-04-27&range_unit=days&range_value=28&signature=ReleaseData Windows 7 65.90% 3538 Firefox 46.0b5 29.32% 1574 Comments: https://crash-stats.mozilla.com/report/list?date=2016-04-27&range_unit=days&range_value=28&signature=ReleaseData#tab-comments "I was on a MSN webpage about places to visit and it crashed." https://crash-stats.mozilla.com/report/index/f699c1fd-05b3-45a0-84da-3cd652160422

I don't see shutdownkill annotations in these reports.

bp-ad163f23-10ef-42b1-a89e-d11512160507 A nightly crash report (after patch from bug 1247982 had landed.)

Valentin, we're still seeing this in 49. Doesn't look like this was a dupe of bug 1247982. Is there someone on the netwerk team that can take a look?

This appears to be a startup crash - https://crash-stats.mozilla.com/search/?product=Firefox&signature=%3DReleaseData&dom_ipc_enabled=!__null__&_facets=signature&_facets=uptime&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-uptime

In release, the crash is spiking since 2016-05-05 (increased by ~125% until 2016-05-08) and is #21 in top-crashes for 46.0.1.

We suspect the remaining crashes will be fixed by bug 507571.

This is still happening in 50, apparently bug 507571 doesn't fix the remaining crashes.

One thing I noticed is F_VOID is not always set when mData points to sEmptyBuffer, for instance: https://dxr.mozilla.org/mozilla-central/rev/4c05938a64a7fde3ac2d7f4493aee1c5f2ad8a0a/xpcom/string/nsTSubstring.cpp#335-338

Do you have any other ideas what could cause this? If not, I'd like to add MOZ_CRASH() to nsHttpHeaderArray::SetHeader() and nsHttpHeaderArray::SetHeader_internal() when !value.mData, which seems are the only places can put null mData nsCString.

(In reply to Ting-Yu Chou [:ting] from comment #33) > Do you have any other ideas what could cause this? If not, I'd like to add > MOZ_CRASH() to nsHttpHeaderArray::SetHeader() and > nsHttpHeaderArray::SetHeader_internal() when !value.mData, which seems are > the only places can put null mData nsCString. Uh, is it possible that because nsEntry is a struct, mHeaders.AppendElement() doesn't actually call a constructor, so the value field might actually be uninitialized if we don't explicitly set it afterwards?

No, AppendElement() will go to nsTArrayEleemntTraits::Construct() to new a nsEntry and call its constructor which initializes the value and variety. https://dxr.mozilla.org/mozilla-central/rev/fef429fba4c64c5b9c0c823a6ab713edbbcd4220/xpcom/glue/nsTArray.h#1550

Crash volume for signature 'ReleaseData': - nightly (version 50): 105 crashes from 2016-06-06. - aurora (version 49): 343 crashes from 2016-06-07. - beta (version 48): 638 crashes from 2016-06-06. - release (version 47): 1560 crashes from 2016-05-31. - esr (version 45): 723 crashes from 2016-04-07. Crash volume on the last weeks: W. N-1 W. N-2 W. N-3 W. N-4 W. N-5 W. N-6 W. N-7 - nightly 24 6 15 8 16 9 20 - aurora 43 27 39 34 27 50 93 - beta 76 89 72 56 118 141 59 - release 218 198 214 195 202 212 212 - esr 8 17 10 16 6 36 44 Affected platform: Windows

These crashes are also related to ReleaseData(): https://crash-stats.mozilla.org/signature/?product=Firefox&signature=je_free%20%7C%20nsAString_internal%3A%3ASetCapacity&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1#reports Interestingly, on 49a2, the crashes all happen in accessing the chunk @0xc4800000, but the virtual address space is only 2GB. It looks like the crashes are from several installations, each of which presents repeated crash at the same address. Instead of catching the crash in nsHTTPHeaderArray, I think it would be more effective if we add the check to nsTSubstring by dereferencing nsTSubstring_CharT::mData whenever it is assigned a nonnull value to crash early during assignment rather than in freeing it.

Some reports came in with the annotations added in bug 1293501, for example bp-ca4aa30b-249f-4f31-b392-0ec232160822: HttpHeaderArray 06 00 00 00 06 00 00 00 c0 e5 fd 5c 00 00 00 00 16 00 00 00 05 00 00 00 01 00 00 00 08 be c8 5c e8 03 f1 0b 41 00 00 00 05 00 00 00 02 00 00 00 HttpHeaderArray[0].header Host Which is: HttpHeaderArray mLength=6 mCapacity=6 [0] header._val = 0x5cfde5c0 ("Host") value.mData = 0x0 value.mLength = 0x16 value.mFlags = 0x5 (F_SHARED | F_TERMINATED) variety = 0x1 (eVarietyRequestOverride) [1] header._val = 0x5cc8be08 value.mData = 0x0bf103e8 value.mLength = 0x41 value.mFlags = 0x5 (F_SHARED | F_TERMINATED) variety = 0x2 (eVarietyRequestDefault) I am double checking the code calling nsHttpRequestHeader::SetHeader(nsHttp::Host, ...).

By now, I see only two paths could alter the value of the header "Host", but none of them can leave the string has |mData| null and |mFlags| F_SHARED|F_TERMINATED: 1. nsHttpRequestHead::SetHeader This goes to nsTSubstring_CharT::Assign(const self_type&) [1]: a. If |aStr| is a shared one, |mData| will be assigned with |aStr.mData| and the following AddRef() call [2] will make sure |mData| is not null. b. Otherwise it goes to [3], which MutatePrep() calls nsStringBuffer::Alloc() [4] and the copy() [5] guarantee |mData| is not null. 2. nsHttpRequestHead::SetHeaders This goes to nsTarray_Impl::ReplaceElementsAt() [6], which AssignRangeAlgorithm calls nsEntry's copy constructor, then the default constructor nsTSubstring_CharT() and nsTSubstring_CharT::Assign() will be used to set |nsEntry.value.mData|, and based on 1 above, it will not be null. And based on comment 39, the |mData| of the value of the header "Host" is somehow changed to null after BaseChannel::Init() [7] set it and before HttpChannelChild::OnStartRequest() calls nsHttpRequestHead::SetHeaders() [8]. [1] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/string/nsTSubstring.cpp#394 [2] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/string/nsTSubstring.cpp#430 [3] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/string/nsTSubstring.cpp#332-333 [4] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/string/nsTSubstring.cpp#132-133 [5] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/string/nsTSubstring.cpp#352 [6] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/xpcom/glue/nsTArray.h#1919 [7] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/netwerk/protocol/http/HttpBaseChannel.cpp#193 [8] https://dxr.mozilla.org/mozilla-central/rev/052656fc513c05da969590ac5934abd67271a897/netwerk/protocol/http/HttpChannelChild.cpp#454

The current annotation reveals that |mData| of the string is not set to null by any of the nsHttpRequestHead API. The crash reports: bp-d8bfb305-f07a-46b5-bccf-757c32160901 bp-5a7c4dcb-aec1-4978-bf92-dff3d2160831 have InvalidHttpHeaderArray like: "in SetHeaders 06 00 00 00 06 00 00 00 60 87 9d 11 00 00 00 00 0f 00 00 00 05 00 00 00 01 00 00 00" which means |mData| is found null when enter SetHeaders().

So this means incidental data corruption occurs via another path, right?

One possibility, should be easy to rule out. Some unknown to me VisitHeader takes a reference to the value that it can corrupt. (const can easily be ignored.) https://dxr.mozilla.org/mozilla-central/rev/3ba5426a03b495b6417fffb872d42874edb80855/netwerk/protocol/http/nsHttpHeaderArray.cpp#302

(In reply to Valentin Gosu [:valentin] from comment #43) > So this means incidental data corruption occurs via another path, right? Yes, the check is basically done in ReentrantMonitorAutoEnter's constructor/destructor, and from the reports I listed in comment 42, |mData| is found null in the constructor when SetHeaders is called. (In reply to Jonathan Howard from comment #44) > One possibility, should be easy to rule out. Some unknown to me VisitHeader > takes a reference to the value that it can corrupt. (const can easily be > ignored.) Yes, for instance FetchEventRunnable keeps a reference to the value: https://dxr.mozilla.org/mozilla-central/rev/dbe4b47941c7b3d6298a0ead5e40dd828096c808/dom/workers/ServiceWorkerPrivate.cpp#1293 Just note there're some similar crashes, e.g., bp-fb68964d-2f29-4074-938c-006a12160901, which happens in ~nsFont() while destructing the array |mFontlist| of FontFamilyList. So this could be a more fundamental issue. I am thinking to use either debug registers or VirtualProtect to find out who corrupts it, however the signature will be changed...

the volume of the [@ ReleaseData] signature is growing since the firefox 49 & 50 releases: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=ReleaseData&date=>%3D2016-07-06T14%3A55%3A21.000Z#graphs

I no longer actively work on this, and I'd like to summary for the things I have so far. 1. In comment 39 only the first nsEntry's value.mData is 0x0, the other fields seems are valid, which doesn't seem to be a memory corruption to me. 2. I couldn't find a place that clear nsEntry.value.mData to null, see comment 36 and comment 40. 3. In bug 1293501 I tried: a. https://hg.mozilla.org/mozilla-central/rev/5661cf2d2b85 This checks the first nsEntry value.mData in ReentrantMonitorAutoEnter's ctor/dtor, and it shows mData is not set to null by any of nsHttpRequestHead API, see comment 42. b. https://hg.mozilla.org/mozilla-central/rev/2eebd44ff2e9 This uses VirtualProtect to protect the first page (4k) of the array (nsHttpHeaderArray.mHeaders) in ReentrantMonitorAutoEnter's ctor/dtor, I added padding to nsEntry so the protected page covers only the first nsEntry: |nsTArrayHeader|nsEntry.header|padding|nsEntry.value.mData| However I didn't see any suspicious crash reports with reason EXCEPTION_ACCESS_VIOLATION_WRITE. Two things to note here i) this patch causes jemalloc4 to crash, see bug 1315285, and ii) then I didn't see any crashes that it's the first nsEntry has value.mData null, see bug 1293501 comment 27 and 28. 4. This crash signature occurrs not only in nsHttpHeaderArray.mHeaders, but also FontFamilyList.mFontlist. Which makes me think this is something about nsTArray on Windows 32-bit builds. Hope this helps.

Mass wontfix for bugs affecting firefox 52.

I filed bug 1431167 to improve the signature here, so different string issues won't be bucketed together. That said, almost all of the crashes I saw still look very HTTP-related. I think the signature for most of them will end up as something like: ReleaseData | nsTArray_Impl<T>::DestructRange | nsTArray_Impl<T>::ReplaceElementsAt<T> | mozilla::net::nsHttpRequestHead::SetHeaders

Moving to p3 because no activity for at least 24 weeks.