Closed Bug 1607241 Opened 5 years ago Closed 5 years ago

Enable live logging for GCP level 1 builders

Categories

(Taskcluster :: Workers, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: coop, Assigned: tomprince)

References

Details

Attachments

(1 file)

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1597996#c4, we need live logging enabled in GCP for level 1 for parity with AWS.

Type: enhancement → defect

From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?

That's correct, and public IPs (which I think are already in place)

(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #2)

That's correct, and public IPs (which I think are already in place)

Yes, and we can check this pretty easily once the port change is made by checking the logs for any running, tier-3, GCP task.

Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.

(In reply to Kendall Libby [:fubar] (he/him) from comment #4)

Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.

I just checked, and GCP live logs are still broken:

https://firefox-ci-tc.services.mozilla.com/tasks/FCIPDkLoTSCUBIm2gIUGcw/runs/1/logs/live/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FFCIPDkLoTSCUBIm2gIUGcw%2Fruns%2F1%2Fartifacts%2Fpublic%2Flogs%2Flive.log

...complains about a "NetworkError when attempting to fetch resource." The raw log links wants points to:

https://gecko-1-b-linux-gcp-yolslfwvsk-sbkrjhceiiq.c.fxci-production-level1-workers.internal:32770/log/IF1rsIOsSwWOMmMaEye89Q

...which is not found.

(In reply to Kendall Libby [:fubar] (he/him) from comment #1)

From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?

fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.

Flags: needinfo?(klibby)

Sorry, I messed this up -- we need all ports 32768-65535 open.

Done.

Flags: needinfo?(klibby)

(In reply to Chris Cooper [:coop] pronoun: he from comment #6)

fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.

should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.

(In reply to Kendall Libby [:fubar] (he/him) from comment #8)

Done.

Still can see the live logs. From IRC, we think this is because we're missing the stateless DNS config for the GCP worker pools.

dustin: 11:14 AM https://gecko-3-b-linux-gcp-kizjakhdr2anv058tm17iq.c.fxci-production-level3-workers.internal:32770/log/nkKdtMnVTIWcSRhWg4j4ng that isn't a stateless-dns hostname
that would mean that the worker doesn't have the stateless DNS config
I think that's DNS_SERVER_SECRET, hm
coop 11:15 AM so we'll need new images?
dustin: 11:15 AM no, I think that just needs to get put in the worker-pool config
https://firefox-ci-tc.services.mozilla.com/secrets/worker-pool%3Agecko-3%2Fb-linux probably has that secret, but there's no equivalent gcp secret
(I can't see that secret)

Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)

(In reply to Kendall Libby [:fubar] (he/him) from comment #9)

(In reply to Chris Cooper [:coop] pronoun: he from comment #6)

fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.

should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.

Looks like we don't actually have this for level 3 builders right now, so no, let's not enable it for level 3 builders in GCP. Sorry for the churn.

(In reply to Chris Cooper [:coop] pronoun: he from comment #10)

Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)

Mid-air ate my NI requests for comment #10

Flags: needinfo?(mtabara)
Flags: needinfo?(mozilla)
Flags: needinfo?(bugspam.Callek)
Assignee: nobody → mozilla
Status: NEW → ASSIGNED
Flags: needinfo?(mtabara)

I've created the secrets (and the patch above changes our tools to do so by default).

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(mozilla)
Flags: needinfo?(bugspam.Callek)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: