This test http://jsperf.com/matrix-test is 3 times faster in Chrome and 2 times faster in Opera. Result with my computer: * Firefox 4.0b8pre (20101018): 290,754 (3rd test) * Opera 10.63: 631,368 (3rd test) * Chrome 8.0.552.0 dev: 1,011,358 (3rd test)

Paul, do you understand the scoring setup used by jsperf? Alternately, can you create a version of this test outside their test harness? They have some "calibration" monkey business that makes it very hard to tell what their numbers actually mean; in particular I've had patches that sped up something by 2x have no effect on the score jsperf reports for the same thing.

Note that a version of the test outside their harness would also make it easier to profile....

I will do it

OK, on the attached test I see these results (on Mac 10.6): Chrome 8 dev: nb iteration 1000000 duration matrix0 1919 duration matrix1 792 duration matrix2 648 Opera 10.61: nb iteration 1000000 duration matrix0 3235 duration matrix1 1569 duration matrix2 701 Current m-c tip: nb iteration 1000000 duration matrix0 849 duration matrix1 574 duration matrix2 594 However on the same exact builds I can confirm that the jsperf.com test reports much higher numbers in Chrome and Opera. So what's going on here?

Oh, there's one obvious difference. The jsperf test is calling makeIdentity() once per iteration, while the attached testcase calls it once and then calls inverse() a bunch of times on it.

With that testcase, I have: Chrome 8: duration matrix0 2110 duration matrix1 1053 duration matrix2 899 Opera 10.61: duration matrix0 4352 duration matrix1 2310 duration matrix2 1222 Minefield: duration matrix0 1163 duration matrix1 1058 duration matrix2 1250 So now at least we're slower than Chrome on matrix2, presumably because of their better gc. But this _still_ totally doesn't match the jsperf results.

Yes, sounds like jsperf mess up the script. INVALID? Cedric, anything to add?

One other thing. For matrix2 for Chrome, say, jsperf reports 984,024 ops/s over here. The timing test above reports 899ms to do 1e6 ops. So that works out to 1,112,347 ops/s as measured locally. For Minefield, the relevant numbers are 231,740 ops/s on jsperf and 1250ms to do 1e6 ops; the latter works out to 800,000 ops/s. So somehow jsperf's "overhead" for Chrome is order of 10% while for us it's order of 4x. Can someone figure out exactly how jsperf generates those numbers?

And to be clear, we've had this stuff (jsperf.com testcases) pasted around a _lot_ recently. So I think we need to either fix whatever makes all jsperf tests slow in our impl or get jsperf fixed if the bug is on their end.

I sent mail to Mathias (the guy who created jsperf); will see what he says.

Yes but makeIdentity was the same for all test (so I removed from the test), I dont know what is going on the jsperf stuff. I have tried to increase by one 0 the number of iteration and result are really weird: my numbers: firefox beta6 nb iteration 1000000 duration matrix0 1096 duration matrix1 1311 duration matrix2 1185 minefield from yesterday nb iteration 1000000 duration matrix0 5727 duration matrix1 3210 duration matrix2 2467 chromium 8 from yesterday nb iteration 1000000 duration matrix0 1905 duration matrix1 839 duration matrix2 664

just to be clear the previous number was posted with 1000000 iterations, I dont know why but my minefield version seems slow. I will check I dont have a config that would have an effect on it

Here I see: Chrome 8: matrix0 445714 runs/s matrix1 807990 runs/s matrix2 858009 runs/s Opera 10.61: matrix0 231504 runs/s matrix1 346134 runs/s matrix2 695549 runs/s Minefield: matrix0 770645 runs/s matrix1 784818 runs/s matrix2 787869 runs/s

> I dont know why but my minefield version seems slow. Do you have Firebug installed? But again, I see the problem on jsperf on a build that's showing fast numbers with the testcases attached here. So there's a jsperf-related problem no matter what.

I would prefer to put date code outside the loop test. Could you have a test with the new script and report your number ?

Yes it was FireBug :) thank you for the suggestions

ok here my number with last script: chromium nb iteration 1000000 matrix0 Run/s 514933 matrix1 Run/s 1157407 matrix2 Run/s 1477104 minefield nb iteration 1000000 matrix0 Run/s 1474926 matrix1 Run/s 1650165 matrix2 Run/s 1633986

> Yes it was FireBug :) OK... now why? And why did you not realize it? ;) And I assume you still see the jsperf slowness even with firebug off? See discussion in bug 603426; if you have something that would be useful to add there, please do... It really sounds like the two remaining issues are jsperf stuff and bug 603426.

Well I gess I trusted too much jsperf :) Thank you for the discussion, I learnt more about testing preformance with js.

(In reply to comment #10) > Can someone figure out exactly how jsperf generates those numbers? Here’s how: http://github.com/mathiasbynens/benchmark.js/blob/master/benchmark.js#L178-185 Note that the benchmarking script used on jsPerf is based on JSLitmus, so that’s where most of the magic comes from. For the record, my test results on http://jsperf.com/matrix-test#run are as follows: Chrome 7.0.517.41 (latest stable release) matrix0: Looped 163,840 times in 0.6399508125 seconds = 256,020 operations per second matrix1: Looped 327,680 times in 0.5329508125 seconds = 614,841 operations per second matrix2: Looped 655,360 times in 0.7509508125 seconds = 872,707 operations per second Opera 10.63 (latest stable release) matrix0: Looped 163,840 times in 0.793016 seconds = 206,604 operations per second matrix1: Looped 163,840 times in 0.668876 seconds = 244,948 operations per second matrix2: Looped 327,680 times in 0.718876 seconds = 455,823 operations per second Firefox 3.6.10 (latest stable release) matrix0: Looped 10,240 times in 0.574079 seconds = 17,837 operations per second matrix1: Looped 40,960 times in 0.520079 seconds = 78,757 operations per second matrix2: Looped 81,920 times in 0.890079 seconds = 92,037 operations per second Firefox 4.0b8pre: matrix0: Looped 81,920 times in 0.618928625 seconds = 132,358 operations per second matrix1: Looped 163,840 times in 0.881928625 seconds = 185,775 operations per second matrix2: Looped 163,840 times in 0.716429 seconds = 228,690 operations per second That indeed differs from the test latest test case made by Cedric, which gives me the following results for Firefox 4.0b8pre: nb iteration 1,000,000 matrix0: Run/s 1,388,888 matrix1: Run/s 1,600,000 matrix2: Run/s 1,577,287 To be honest, I have no idea why there’s this much difference. It doesn’t make much sense — jsPerf uses a reverse while loop to run the tests; the non-jsPerf test case uses a for-loop, which is known to be *slower*. Even if for-loops would be faster in Minefield, this might make a tiny difference, but it shouldn’t be this big. Also, the number of iterations is higher in the non-jsPerf test case, which should only add to its accuracy. If no one spots any major bugs in the benchmark.js source code I linked above, I’ll get in touch with Robert (JSLitmus author / voodoo JS coder).

I attached a test case using a reverse while loop instead of a for-loop just to be sure — but the results seem to be about the same.

Chrome 7.0.517.41 (latest stable version): matrix0: 146,357 ops/sec matrix1: 145,319 ops/sec matrix2: 138,442 ops/sec Firefox 4.0b8pre: matrix0: 146,619 ops/sec matrix1: 145,319 ops/sec matrix2: 144,550 ops/sec

Interesting. Using the latest attached testcase I see chrome once again scoring much higher than minefield. Mathias, thanks for the code link; let me go stare at it.

OK. So I grabbed the code from the comment 24 and ran it locally. I also logged the function f, count and raw runtime in seconds that we're dealing with in _run, before any of the calibrations stuff is taken into account. I get log output something like this in Minefield: function () { inv2 = osg.Matrix2.inverse(i2); } 163840 0.682 and like this in Chrome: function () { inv2 = osg.Matrix2.inverse(i2); } 655360 0.521 If I try to time that directly, like so: function g(f, count) { try { var start = new Date(); var i = count; while (i--) f(); alert(new Date() - start); } catch (e) { } } g(function() { inv2 = osg.Matrix2.inverse(i2); }, 163840); I get 77ms in Minefield and 144ms in Chrome....

It's definitely something about the way the test ramps up its iteration counts. If I change comment out the two calibration tests and change INIT_COUNT from 10 to 100000, I get totally different numbers in Minefield, but the same ones in Chrome.

If I disable methodjit, the slowness goes away on the jslitmus testcase. Looks like for some reason the rampup causes us to blacklist the loop, but only when methodjist is enabled. And JM is slower than V8 on this code. I'll attach a shell testcase demonstrating this problem. Now... this is true for the jslitmus testcase. On jsperf.com, disabling methodjit leads to a significant slowdown... Adding a dependency on bug 580468 for the jslitmus issue, but _something_ else is going on with jsperf.com.

When run -j -m, the third number printed is much smaller than the first one; the first is about the same as both numbers when run -j, while the third is about the same as both numbers when run -m.

OK, with the jsperf code, but not with the jslitmus code (?) the calibrations stuff ends up mattering. If I turn off the running of the calibration functions, or even the running of just _one_ of the calibration functions, then things are fast.

Ah, no. I have to explicitly turn off this part to make things fast: new Test('Calibrating loop', 'cl', function(count) { while (count--) { } }), if I leave that in, even if I comment out the other calibration function, things are slow. If I comment it out _and_ disable methodjit (to work around the issue from comment 29), things are fast.

And the reason for that is: trace stopped: 8860: can only inc numbers Abort recording of tree file:///Users/bzbarsky/test.js:179@104 at file:///Users/bzbarsky/test.js:97@5: argdec. (test.js is the file linked to from comment 22). And in particular, |count| there is undefined. I get the above three times, which makes sense: that calibration is run over and over. And then we blacklist the loop at like 179. Which is the main benchmark loop.

So the reason the undefined doesn't happen in the original jslitmus is that their test-runner looks like this: if (this.loopArg) { // ... let it do the iteration itself f(count); } else { // ... otherwise do the iteration for it while (i--) f(); } while jsbench dropped the if condition... but the while(count--) test above matches the loopArg regexp, so was getting called with an actual argument in jslitmus, but not jsbench. Of course jsbench also doesn't use the result of the loop calibration.... ;)

Er, s/jsbench/jsperf/g in comment 33.

So there are three issues here, I think: 1) Should we attempt to trace inc() on undefined? 2) Why does JM blacklist the loop when ramping up, and will bug 580468 fix that? 3) Why is JM 3x slower than tracer here? dvander, dmandelin, want me to file those as separate bugs blocking this one?

I filed bug 605858 on item 1 from comment 35, and bug 605851 on item 3.

(In reply to comment #35) > 2) Why does JM blacklist the loop when ramping up, and will bug 580468 fix When running JM+TM in combination, a trace gets blacklisted if it's run for fewer than 8 iterations the first time it executes. This heuristic is made a little more friendly in bug 580468. We watch the first 10 executions of the trace and blacklist if it executes fewer than 200 total iterations (so an average of 20 per execution). Do you know if this will be sufficient to allow this loop to be traced?

> When running JM+TM in combination, a trace gets blacklisted if it's run for > fewer than 8 iterations the first time it executes. Ah, and in this case ramp-up starts with 10 iterations, so we end up running it on trace for only 2 iterations and blacklisting? The ramp-up here goes up by powers of 2 starting at 10, so 10 executions will get us way past 200 total iterations.

js> x = undefined js> x++ NaN js> x++ NaN Not really useful even in a benchmark. Is it intended? /be

NO, it's not intended. See comment 33.

(In reply to comment #40) > NO, it's not intended. See comment 33. I had trouble parsing that without reading the code (which I didn't cuz I am a lazy bastard ;-). Thanks, hope the code can be fixed. Mathias, is it all under your control? /be

(In reply to comment #41) > Thanks, hope the code can be fixed. Mathias, is it all under your control? Yes, it is. I just deployed this change: http://github.com/mathiasbynens/benchmark.js/commit/77696aa7310607b31562b608226b33107c9830ae That appears to fix the jsPerf/Firefox issues. I’ll see what I can do to reset all the Browserscope results.

For what it’s worth, I’ve just cleared the Browserscope results for all 2,768 jsPerf test cases.

I just have a tried on jsperf on my matrix test, and if I run the test many times, I can have very different result. I dont know if it's expected eg: first run 682,327 8% slower 749,206 fastest 745,796 0% slower second run 144,736 37% slower 204,037 12% slower 232,730 fastest third run 137,454 53% slower 208,718 29% slower 295,216 fastest I used firefox b8 the same version I reported this bug. Hopes it will help

Cedric, at this point you're just seeing

Er, seeing the effects of comment 35 item 2. The first run times there are the ones you should more or less expect in general once that issue is resolved.

Firefox 17 97,119 ±9.03% 41% slower 160,043 ±7.66% 1% slower 169,675 ±12.45% fastest Firefox 19 (20121022 Nightly) 67,074 ±2.18% 63% slower 167,460 ±4.96% 9% slower 184,023 ±4.68% fastest Chrome 22 215,892 ±2.41% 57% slower 491,895 ±1.37% fastest 488,784 ±1.15% 0.41% slower

Firefox 21 145,575 ±5.32% 46% slower 225,466 ±2.72% 15% slower 269,307 ±4.74% fastest Great improvement from earlier Ion development (My last reply was showing a regression on the first test).