Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Description • 14 years ago

Ideally, we would modify the reftest harness so as to allow us to compare four renderings instead of two for each test T.  We would want to run the normal reftest process on T and and its reference T_r once for <browser> and again for <browser remote>.  This would yield renderings T, T remote, T_r, and T_r remote.  We could gather all four in parallel, in theory.  Then, for simplicity, assume we want T's reference T_r to be != to the test T.  We would check
  T   == T remote
  T_r == T_r remote
  T   != T_r

It would be nice to run this continuously on all platforms as extension of normal reftests.  It's unclear what perf overhead this would incur.

It's not totally obvious how the browser process would know that the content process was done painting.  However, Rob and I discussed this a bit at the work week, and it seems that we could probably just load the reftest script into the content process and re-use pretty much all the logic in there.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 1 • 14 years ago

(Hm, no blocking-fennec flag?)

IMHO this needs to block fennec-betaN.  We really really need to be continuously running tests of all the APIs we've added and of shadow-layer rendering.

	(not currently active) Ted Mielczarek
	Comment 2 • 14 years ago

I was going to suggest adding a --remote pref that ran reftest with all the drawing taking place in a content process, but that's not quite what you want, is it? You want to compare local rendering vs. remote rendering.

Is it overkill to run this as part of regular reftest? Do you expect that people hacking layout will expose bugs in remote rendering, or do you think that it's more likely to be other things, and reftest is just a convenient seed of "try drawing a lot of things"? It feels like it'd be odd to start adding tests to reftest for bugs in the remote rendering, if they don't really test bugs in our layout engine so much as our content process rendering.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 3 • 14 years ago

(In reply to comment #2)
> I was going to suggest adding a --remote pref that ran reftest with all the
> drawing taking place in a content process, but that's not quite what you want,
> is it? You want to compare local rendering vs. remote rendering.
> 

Right.

> Is it overkill to run this as part of regular reftest?

Possibly, experience would tell.

> Do you expect that
> people hacking layout will expose bugs in remote rendering, or do you think
> that it's more likely to be other things, and reftest is just a convenient seed
> of "try drawing a lot of things"?

I don't understand why it's "either or"; both can happen, there are lots of ways to break remote rendering.

> It feels like it'd be odd to start adding
> tests to reftest for bugs in the remote rendering, if they don't really test
> bugs in our layout engine so much as our content process rendering.

We do need reftests to check new APIs we've added for fennec with layout implementations that are different for local vs. remote browser (by necessity), regardless of whether we support comparing *all* reftests between local vs. remote.  But we use reftests to check HW-accelerated rendering, even though HW accel has nothing to do with layout, and it's been fabulously useful.  That's the idea I have in mind.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 4 • 14 years ago

Also FWIW I believe Rob proposed something similar for comparing SW rendering vs. HW-accelerated rendering, which would have been very cool, but unfortunately there are various GPU details that make that infeasible.

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 5 • 14 years ago

I'm not sure we actually need to compare <browser> against <browser remote> rendering. If we just run reftests and take snapshots of the chrome window containing <browser remote>, isn't that going to exercise the shadow layer system pretty well?

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 6 • 14 years ago

In other words, just run reftests using <browser remote> on mobile.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 7 • 14 years ago

(In reply to comment #5)
> I'm not sure we actually need to compare <browser> against <browser remote>
> rendering. If we just run reftests and take snapshots of the chrome window
> containing <browser remote>, isn't that going to exercise the shadow layer
> system pretty well?

Sure, but that leaves open the possibility for all sorts of glitches in remote=true that don't show up for remote=false.

To be clear, I filed this bug to cover making a system that allows me to compare remote=true to remote=false renderings, maybe re-using the reftest framework.  Whether or not we run this on tinderbox or for what platforms is a totally separate discussion.

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 8 • 14 years ago

(In reply to comment #7)
> (In reply to comment #5)
> > I'm not sure we actually need to compare <browser> against <browser remote>
> > rendering. If we just run reftests and take snapshots of the chrome window
> > containing <browser remote>, isn't that going to exercise the shadow layer
> > system pretty well?
> 
> Sure, but that leaves open the possibility for all sorts of glitches in
> remote=true that don't show up for remote=false.

So you're worried about bugs that affect the test and reference the same way on every single test, and therefore aren't detected at all?

That's already a potential problem with reftests, which we're not bothered by.

> To be clear, I filed this bug to cover making a system that allows me to
> compare remote=true to remote=false renderings, maybe re-using the reftest
> framework.  Whether or not we run this on tinderbox or for what platforms is a
> totally separate discussion.

OK.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 9 • 14 years ago

I'm worried about bugs that would affect a particular test+reference the same for remote but not local browser.  Clipping, snapping, etc. of shadow trees e.g.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 10 • 14 years ago

so we really want to just load the test and reference in a browser=true environment while having the core reftest extension running in chrome.  Then we can snapshot and compare in chromeland.

This would be reftest only?  What about jsreftests and crashtests?

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 11 • 14 years ago

(In reply to comment #10)
> so we really want to just load the test and reference in a browser=true
> environment while having the core reftest extension running in chrome.  Then we
> can snapshot and compare in chromeland.
> 

For <browser remote>, yes that's mostly true.  We'll need the reftest code that detects when the page load/animation has reached a snapshot-able state to live in the content process, though.

> This would be reftest only?  What about jsreftests and crashtests?

I don't know much about jsreftests and crashtests, but if they're not comparing renderings, then just running them with something like the --remote option Ted suggested in comment 2 should be enough.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 12 • 14 years ago

Filed bug 615386 on the reftest --remote-style framework mentioned in comment 2.