Ed Morley [:emorley] Reporter
	Description • 11 years ago

Several PGO talos runs on inbound are busted, eg:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=pgo.*talos&fromchange=eab74a7144c0&tochange=badfeeb91254

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=26006500&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=26008185&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=26008562&tree=Mozilla-Inbound

Done:
1) Retrigger on other trees to see if this was caused by a mozharness change.
2) Retrigger PGO on pushes inside the regression range above.

Inbound is closed.

	Ed Morley [:emorley] Reporter
	Comment 1 • 11 years ago

(In reply to Ed Morley [:edmorley UTC+1] from comment #0)
> Done:
> 1) Retrigger on other trees to see if this was caused by a mozharness change.

mozilla-central:
https://tbpl.mozilla.org/?jobname=pgo.*talos

= green

> 2) Retrigger PGO on pushes inside the regression range above.

These will take 3-4 hours.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 2 • 11 years ago

I noticed we don't run pgo talos on mozharness, it is run the old way.  I suspect this has some of the reasons for failure.

	Ed Morley [:emorley] Reporter
	Comment 3 • 11 years ago

Oh ha, good spot! :-)

	Armen [:armenzg]
	Comment 4 • 11 years ago

Adding jhopkins as buildduty to be on the loop.

I will look into this.

Talos PGO on central looks good as per retriggers from comment 1.

	Armen [:armenzg]
	Comment 5 • 11 years ago

I know I'm duplicating a block of code but I did not want to try to optimize it while I'm trying to fix the tree.

	Armen [:armenzg]
	Comment 6 • 11 years ago

Comment on attachment 784452 [details] [diff] [review]
[buildbotcustom] duplicate code to create talos ScriptFactory for PGO builders

Review of attachment 784452 [details] [diff] [review]:
-----------------------------------------------------------------

Landed as r=bustage as aki is on PTO and jhopkins could only agree with the plan more than being able to review it.
http://hg.mozilla.org/build/buildbotcustom/rev/00217a62c5bc

I've asked jyeo to clean the code after we get out of the hole rather than having duplicated code.

As soon as we reconfigure, we will trigger talos jobs and see after 15 minutes what results we get.

Let's hope this was the issue.

	Ed Morley [:emorley] Reporter
	Comment 7 • 11 years ago

Thanks Armen! :-)

	Armen [:armenzg]
	Comment 8 • 11 years ago

We have fixed all other talos jobs.
We still have to fix xperf pgo; It is still running the old way.

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Windows%207%2032-bit%20mozilla-inbound%20pgo%20talos%20xperf

TEST-UNEXPECTED-FAIL : xperf: File '{firefox}\omni.ja' was accessed and we were not expecting it.  DiskReadCount: 46, DiskWriteCount: 0, DiskReadBytes: 3014656, DiskWriteBytes: 0
TEST-UNEXPECTED-FAIL : xperf: File '{firefox}\browser\omni.ja' was accessed and we were not expecting it.  DiskReadCount: 28, DiskWriteCount: 0, DiskReadBytes: 1835008, DiskWriteBytes: 0
Return code: 1

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 9 • 11 years ago

is omni.ja only seen on PGO builds?  I haven't seen this in all my testing and the initial deployment of xperf.

	Armen [:armenzg]
	Comment 10 • 11 years ago

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #8)
> We still have to fix xperf pgo; It is still running the old way.
>
I don't know how I thought that it was still running the old code.

Do we want to close this bug and file another one for the xperf pgo issue?

	(No longer employed by Mozilla) Aaron Klotz
	Comment 11 • 11 years ago

On PGO builds we optimize the omni.ja such that we can readahead part of it when it it first opened. I suspect that, since those reads occur earlier in the browser execution, xperf sees this as a new read during startup. We should whitelist the omni.ja files.

	Armen [:armenzg]
	Comment 12 • 11 years ago

I wonder how TalosFactory was not letting us see this.

It seems that the issue is also on Mozilla-Central.
I've re-triggered all xperf pgo jobs back to last Friday.
https://tbpl.mozilla.org/?jobname=Windows%207%2032-bit%20mozilla-central%20pgo%20talos%20xperf&showall=1

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 14 • 11 years ago

landed on talos, will deploy tomorrow morning:
http://hg.mozilla.org/build/talos/rev/f3e761110635

	Ed Morley [:emorley] Reporter
	Comment 15 • 11 years ago

Original issue fixed, closing.