Jeff Muizelaar [:jrmuizel] Reporter
	Description • 12 years ago

A large regression is clear on tcheckerboard. I don't really see the same on eideticker.

http://graphs.mozilla.org/graph.html#tests=[[175,63,20]]&sel=1340894474844,1341499274844&displayrange=7&datatype=running

	William Lachance (:wlach)
	Comment 1 • 12 years ago

(In reply to Jeff Muizelaar [:jrmuizel] from comment #0)
> A large regression is clear on tcheckerboard. I don't really see the same on
> eideticker.
> 
> http://graphs.mozilla.org/graph.html#tests=[[175,63,20]]&sel=1340894474844,
> 1341499274844&displayrange=7&datatype=running

tcheckerboard tests against a copy of timecube.com, so I guess that really means we should add a test against a copy of that to the eideticker suite.

If you look at another checkerboarding benchmark (tcheck2) which tests against CNN you see that there was no regression there either: 

http://graphs.mozilla.org/graph.html#tests=[[175,63,20],[201,63,20]]&sel=1340894474844,1341499274844&displayrange=7&datatype=running

	Jeff Muizelaar [:jrmuizel] Reporter
	Comment 2 • 12 years ago

I believe the same regression should be happening on the task.js page and it doesn't really show up there either.

	William Lachance (:wlach)
	Comment 3 • 12 years ago

(In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> I believe the same regression should be happening on the task.js page and it
> doesn't really show up there either.

Hmm, any ideas why that would be the case? I'm pretty sure there's no issue with the HDMI capture or analysis code.

	Jeff Muizelaar [:jrmuizel] Reporter
	Comment 4 • 12 years ago

(In reply to William Lachance (:wlach) from comment #3)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> > I believe the same regression should be happening on the task.js page and it
> > doesn't really show up there either.
> 
> Hmm, any ideas why that would be the case? I'm pretty sure there's no issue
> with the HDMI capture or analysis code.

It could be that presentation of the data just makes it hard to see changes. i.e. You can't just look at data from the last couple of days.

It also seems like the tests are not run often enough. There seems to be no data between 6/30 and 7/4 on cnn.com and reddit

	William Lachance (:wlach)
	Comment 5 • 12 years ago

(In reply to Jeff Muizelaar [:jrmuizel] from comment #4)
> (In reply to William Lachance (:wlach) from comment #3)
> > (In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> > > I believe the same regression should be happening on the task.js page and it
> > > doesn't really show up there either.
> > 
> > Hmm, any ideas why that would be the case? I'm pretty sure there's no issue
> > with the HDMI capture or analysis code.
> 
> It could be that presentation of the data just makes it hard to see changes.
> i.e. You can't just look at data from the last couple of days.

I just added support to zoom in on a particular region of an Eideticker graph on the dashboard, let me know if it helps. I'm not seeing much of a pattern on the dashboard, but as you say, we are missing some data.

> It also seems like the tests are not run often enough. There seems to be no
> data between 6/30 and 7/4 on cnn.com and reddit

There's still issues of stability with the test harness. I think I've nailed down at least a few issues recently, but some still remain. I now have a tool to run eideticker against a particular range of builds -- I'll try running that for taskjs to see if it caught the regression.

	William Lachance (:wlach)
	Comment 6 • 12 years ago

(In reply to William Lachance (:wlach) from comment #5)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #4)
> > (In reply to William Lachance (:wlach) from comment #3)
> > It also seems like the tests are not run often enough. There seems to be no
> > data between 6/30 and 7/4 on cnn.com and reddit
> 
> There's still issues of stability with the test harness. I think I've nailed
> down at least a few issues recently, but some still remain.

I just filed bug 771375 about the dashboard update script's flakiness. Hopefully when this is fixed the dashboard update process should be more consistent.

	William Lachance (:wlach)
	Comment 7 • 10 years ago

This bug is pretty old, and I don't think there's enough specific actionable information to go anywhere with it.