Justin Dolske [:Dolske] Assignee
	Description • 16 years ago

Last I checked, Tinderbox's log output was sending data line-by-line (ie, each line in a separate packet). This results in needless network and parser overhead.

I'm guessing that sticking a "$| = 0" somewhere would fix it. Basically the reverse of http://perl.plover.com/FAQs/Buffering.html

	cls
	Comment 1 • 16 years ago

Do you actually see a problem or do you just suspect that there's a problem? 

Per the referenced URL,  when you force the output to use buffering, you can wind up with output in the wrong order if you aren't careful.  Also, I'm wondering how much the web server is automatically buffering the output even though most of the cgi's set STDOUT to autoflush.

	Justin Dolske [:Dolske] Assignee
	Comment 2 • 16 years ago

(In reply to comment #1)
> Do you actually see a problem

Yes. I just double checked with etherreal, there's still a flood of packets with small TCP payloads, each of which contains exactly one line.

For example, when the build environment is dumped at the top of a log, each one of those short lines is a separate packet. In the middle of the mochitest results, each "*** 1234 INFO TEST-PASS blah" line is a separate packet.

Oddly, I did spot a few spells in my capture where it was getting bursts of 1368 byte payloads containing multiple lines. Probably just random buffering effects.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 3 • 16 years ago

justdave, would it be possible to try dolske's suggestion on dm-webtools04 ? Then we could compare logs pulled from there and tinderbox.m.o - although I can't figure out how to get a waterfall on 04 yet.

	Justin Dolske [:Dolske] Assignee
	Comment 4 • 15 years ago

(In reply to comment #0)

> I'm guessing that sticking a "$| = 0" somewhere would fix it.

Or, likely, just removing the $|=1 here:

http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/showlog.cgi#89

	Dave Miller [:justdave]
	Comment 5 • 15 years ago

ok, done.

	Justin Dolske [:Dolske] Assignee
	Comment 6 • 15 years ago

Yeah, this made grabbing a log about 2.5x faster.

Comparing a similar 9.5MB log on production vs stage...

Production sent 80,400 packets, net speed was 1.3MB/s.
Stage sent 11,500 packets, net speed was 3.4MB/s

I did notice that the output of some logs on the stage server was different than on production, guess there are other scripts (log parsing?) that are out of sync? FF seemed to load the stage page slower, that could be a result of the formatting difference.

	David Baron :dbaron:
	Comment 7 • 15 years ago

Any chance of getting this deployed in production (looks like it was only fixed on stage above)?

	dwitte@gmail.com
	Comment 8 • 15 years ago

Can we get this fixed on production? Anything to make log transfer faster would be... most useful.

	Reed Loden [:reed]
	Comment 9 • 15 years ago

Need a patch attached to this bug, reviewed, and committed...

	Nick Thomas [:nthomas] (UTC+12)
	Comment 11 • 15 years ago

cls, ping on the review.

	cls
	Comment 12 • 15 years ago

Comment on attachment 420999 [details] [diff] [review]
Patch v.1

rs=cls

	Reed Loden [:reed]
	Comment 13 • 15 years ago

Checking in showlog.cgi;
/cvsroot/mozilla/webtools/tinderbox/showlog.cgi,v  <--  showlog.cgi
new revision: 1.35; previous revision: 1.34
done

	David Baron :dbaron:
	Comment 14 • 15 years ago

What needs to happen to get this deployed?

	Justin Dolske [:Dolske] Assignee
	Comment 15 • 15 years ago

(Bug 559348 deployed this.)