bhearsum@mozilla.com (:bhearsum) Reporter
	Description • 14 years ago

We hit an issue recently where a test depended on an external website, and started failing strangely when that website had issues. To help weed out more of these sort of things we'd like to start logging all external requests made by buildslaves. That is:
bm-xserve*
moz2-*
linux-ix-*
w32-ix-*
mw32-ix-*
mv-moz2-linux-ix-*
try-*
n900-*
t-r3-w764-*
talos-r3-*
win32-*

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 1 • 14 years ago

Joduinn just told me that he wants everything logged, to potentially help us catch an infected machine, so please disregard the list of hosts in comment #0.

	Ravi Pina [:ravi]
	Comment 2 • 14 years ago

Enabled logging on fw1.mtv1, fw1.scl1, and fw1.sjc1 for all connections from build to public addresses.

I'll leave this open for tracking and will review data on a daily basis for several days.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 3 • 14 years ago

Is it possible to publish the data somewhere public, so that releng and developers can analyze too?

	matthew zeier [:mrz]
	Comment 4 • 14 years ago

Logging access is something infrasec handles.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 5 • 14 years ago

Any update?

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 6 • 14 years ago

(In reply to comment #4)
> Logging access is something infrasec handles.

IT setup logging of all inbound traffic in bug#602741, and gave us the info from those logs. Dont mind if its IT or InfraSec who gives us those logs - whatever is fastest WFM. 

We are looking for a list of which RelEng systems are attempting to connect to the outside world. We expect this to uncover what test suites are reaching out to external sites, so we can then file bugs with QA. It *might* uncover other surprises, but lets see.


(We need this in place before we can safely *block* access from build-vpn machines to the outside world.)

	Chris Lyon [:clyon]
	Comment 7 • 14 years ago

should be in server and not in security. I am OK with the logs.

	Ravi Pina [:ravi]
	Comment 8 • 14 years ago

I've provided Zandr logs from SCL1 which he is parsing through to get some useful data.

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 10 • 14 years ago

Thank you!

	bhearsum@mozilla.com (:bhearsum) Reporter
	Comment 11 • 14 years ago

What period of time were these captured over, out of curiosity?

	Zandr Milewski [:zandr]
	Comment 12 • 14 years ago

The clock on the firewall was messed up, but if my math is right, this is all but the first 9 minutes of 18 January 2011 (UTC). 

That's a coincidence, btw, what I got was the current log plus the previous 3 files, which rotate at 10MB.

	Zandr Milewski [:zandr]
	Comment 13 • 14 years ago

Speaking of clocks... I'll note again here that I pulled out NTP, since it appears that some machines are using pool.ntp.org. As a result, we hit port 123 on 250 unique IP addresses in 24hrs, of which <10 are the usual suspects at MS and Apple.

I'll file a bug to use an internal NTP server instead as soon as I have a chat with netops to figure out which one it should be.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 14 • 14 years ago

We use ntp1.build.mozilla.org for build machines but I bet the talos-r3 boxes never got configured to that level of detail.

	Mike Taylor [:bear]
	Comment 15 • 14 years ago

Comment on attachment 504925 [details]
prelim grep-fu on firewall logs

Port 5672 is for RabbitMQ and we will need to enable in/out access from our build RabbitMQ server to the main mozilla pulse server.  Currently our RMQ server is running on cruncher.

	matthew zeier [:mrz]
	Comment 16 • 13 years ago

Marking post-fx4 because turning on logging increase firewall load.  Don't want that right now.

	Zandr Milewski [:zandr]
	Comment 17 • 13 years ago

(In reply to comment #16)
> Marking post-fx4 because turning on logging increase firewall load.  Don't want
> that right now.

Did you turn it off? Otherwise, it's been on since comment 2.

	Ravi Pina [:ravi]
	Comment 18 • 13 years ago

I disabled logging on fw1.mtv1 for all sessions going from build.mtv1 -> *

	Zandr Milewski [:zandr]
	Comment 19 • 13 years ago

(In reply to comment #18)
> I disabled logging on fw1.mtv1 for all sessions going from build.mtv1 -> *

Yeah, this is mostly about fw1.scl1. (Talos (10.12.48.0/22) -> internet)

	matthew zeier [:mrz]
	Comment 20 • 13 years ago

Oh!  I forget we have more than one releng fw point now.

	Zandr Milewski [:zandr]
	Comment 21 • 13 years ago

More grep-fu on a recent log snapshot.

Per joduinn, this closes this bug, different bugs will be filed to fix tests that are making outbound connections.

Again, I've filtered out NTP.