Closed Bug 493050 Opened 15 years ago Closed 12 years ago

Tgfx results are almost certainly bogus on at least some platforms

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: johnath, Assigned: jrmuizel)

References

Details

Taking Vista trunk as a for instance:

http://graphs-new.mozilla.org/#tests=[{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2267%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2268%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2269%22}]

TGfx has been at "1" for a long time.

Even looking at all TGfx, everywhere (link below) there seem to be some platforms reporting 20... somethings, but almost all of them are at 1.

If we think this is a valuable benchmark, then we need to understand what's happening and fix it.  But if it were that important, I'd want to hope someone would have noticed it not working very well.

If it's not important, we should stop running it, and reporting it, and get back some cycle time and graph server space.

CC'ng some GFX people.

All TGfx:

http://graphs-new.mozilla.org/#tests=[{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2251%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2252%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2253%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2255%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2256%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2257%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2259%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2260%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2261%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2263%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2264%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2265%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2267%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2268%22},{%22test%22:%2220%22,%22branch%22:%221%22,%22machine%22:%2269%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%226%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%227%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%228%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2210%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2211%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2213%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2214%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2215%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2216%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2218%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2219%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2220%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2221%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2222%22},{%22test%22:%2220%22,%22branch%22:%222%22,%22machine%22:%2223%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2228%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2230%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2232%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2233%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2234%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2236%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2237%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2238%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2240%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2241%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2242%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2244%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2245%22},{%22test%22:%2220%22,%22branch%22:%223%22,%22machine%22:%2246%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2277%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2278%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2279%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2280%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2281%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2282%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2283%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2284%22},{%22test%22:%2220%22,%22branch%22:%224%22,%22machine%22:%2285%22}]
If it's reporting 1's, that generally means that someone was using RDP to access the machine and left it without a real 'display'... but would have an impact on Tp3 numbers as well, so I'm not sure.
Component: Release Engineering: Talos → Release Engineering
RDP is never used to access these machines, they all auto-start VNC on reboot and use that.
Any reason not to reassign to Core::GFX:Thebes?
It doesn't have anything to do with thebes; someone just needs to look into the test and DOMWindowUtils::Redraw, to make sure that things are behaving as they should.  As of about 4 weeks ago Redraw() was doing the right thing, so I'm not sure what's going on.
Right, just wondering since I keep running into it in RelEng triage and it's not waiting for RelEng resources; it's waiting for GFX resources. I'll let it sit for now though.
Vlad - is this test bogus?  Can we just turn it off?

What information could I provide to help make that decision?
Assignee: nobody → anodelman
Re-assigning so that this can be properly triaged.
Assignee: anodelman → vladimir
Let's pretend that Jeff will want to work on this :)
Assignee: vladimir → jmuizelaar
(In reply to comment #9)
> Let's pretend that Jeff will want to work on this :)

No update in 10 months. Since then we've changed from running XP and Vista to now running XP and Win7. 

Jeff, do you want to keep this open for you to work on, or can we close?
(In reply to comment #10)
> (In reply to comment #9)
> > Let's pretend that Jeff will want to work on this :)
> 
> No update in 10 months. Since then we've changed from running XP and Vista to
> now running XP and Win7. 
> 
> Jeff, do you want to keep this open for you to work on, or can we close?

Pushing to Testing:Talos, for Jeff to decide if he needs it open anymore or not.
Component: Release Engineering → Talos
Product: mozilla.org → Testing
QA Contact: release → talos
Version: other → unspecified
Can we disable this test until is generates data that is

1) reliable
2) meaningful
3) actionable (like, a regression means that we do something)

I fear that right now we are wasting processing time of a test that is ignored.
The pages that are in Tgfx, especially the synthetic ones, should be providing useful data.  For example, there's one in there explicitly for border rendering, which would have provided info when D2D caused border rendering to slow down.

I don't know why the test is broken, but the core thing that the test does should be sound; if there's a bug in the way that it's implemented that causes timing to be wrong or something, we should fix that.  I don't have a problem with disabling it temporarily, but we shouldn't disable and forget; someone should actually fix the test.
I explained why the Tgfx numbers are bogus in bug 589396.

I think if we just use a setTimeout from the onload handler to kick off the redraw calls that would fix the problem. We would need to adjust the number of redraws we do to be a reasonable number now that the redraw calls actually do something though.
we do not run tgfx tests on talos and have removed it from our list of possible tests to run.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.