Jeff Walden [:Waldo] Reporter
	Description • 18 years ago

Assertions in xpcshell tests are fatal; assertions in mochitest aren't.  There isn't a way to run a GUI test which should fail if an assertion fails, but since we treat assertion failures as bugs there should be.

Making mochitests fatal with assertions probably won't happen (yet -- I hold out hope for the future!), so this probably requires a new GUI test harness in which assertions *are* fatal.  Alternately, maybe mochitest can be tweaked to be run with assertions as fatal for specific tests (probably requires invoking it twice), or maybe the proper GUI can be opened from xpcshell with enough pain (most of which would be in a head_*.js file).

	Jesse Ruderman
	Comment 1 • 18 years ago

How about making a list of assertions that the mochitests are known to hit, and considering it a failure if any other assertion is hit?

	Jeff Walden [:Waldo] Reporter
	Comment 2 • 18 years ago

You could do that, but I suspect any such list would likely accumulate unnecessary entries over time as assertions are added/removed from the code being tested.  Is it really a good idea to make core code changes possibly also necessitate changes to the list of assertions?  I don't think it is.

	:Gavin Sharp [email: gavin@gavinsharp.com]
	Comment 3 • 17 years ago

From bug 370767 comment 0, by Nickolay:

Assertions are bad and there was talk about treating them the same way as
crashes. Right now it's a bit unclear what happens if something asserts. From
what I could figure out on IRC, they are ignored now.

In any case, it would be nice if 
a) they were not ignored
b) just a particular test was marked as failed and the testsuite continued to
run, since most assertions are not really -that- fatal.

This applies to all of our test harnesses ideally, in particular mochitest and
reftest.

The relevant snippet of #developers log:

<asqueella> sayrer: is the idea that mochitests fail on assertion failures?
<asqueella> if not, how does one go about testing lack of assertion in certain
cases?
<sayrer> asqueella: there's nothing set up for that
<asqueella> it's a pity
<asqueella> so it ignores assertions?
<sayrer> it doesn't operate at that level
<sayrer> tinderbox could watch for assertions if we wanted
<asqueella> well
<asqueella> yeah
<asqueella> that's what I'm talkign about I guess
<sayrer> talk to robcee then :)
<robcee> sayrer: I just run it from the command line on the test box and ran
through perfectly
<shaver> it wouldn't be too hard to let mochitests detect assertions
<shaver> add a static int assertCount;
<shaver> have NS_ASSERT increment it
<shaver> add two little functions to a service or whatever to clear and sample
the variable
<shaver> and robert is your mother's brother
<sayrer> patches welcome
<sayrer> asserts are the least of worries, really
<asqueella> why?
<sayrer> since the trunk is so horked 
<asqueella> they're supposed to be fatal
<asqueella> the state of trunk shouldn't affect the test harnesses :)
<sayrer> I'm not big on test harnesses that always fail
<sayrer> ymmv

	Boris Zbarsky [:bzbarsky]
	Comment 5 • 17 years ago

Are mochitests even run on debug builds?  That would be a first step to getting useful assertions...

	Jeff Walden [:Waldo] Reporter
	Comment 6 • 17 years ago

I don't think so, which is actually rather bad: xpcshell tests explicitly fail if they trip assertions (but of course only in debug builds).

	Rob Campbell [:rc] (:robcee)
	Comment 7 • 17 years ago

we have bug 372581 opened requesting debug builds. I think this is important and we should get some debug builds setup and running.

	(not currently active) Ted Mielczarek
	Comment 8 • 16 years ago

Everything left in Core:Testing is going to Testing:General. Filter on CleanOutCoreTesting to ignore.

	(not currently active) Ted Mielczarek
	Comment 9 • 16 years ago

Dupe of bug 472557 or bug 404077 (or the reverse?)

	Martijn Wargers (dead)
	Comment 10 • 10 years ago

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Dupe of bug 472557 or bug 404077 (or the reverse?)

Yes.