Enable live logging for GCP level 1 builders
Categories
(Taskcluster :: Workers, defect)
Tracking
(Not tracked)
People
(Reporter: coop, Assigned: tomprince)
References
Details
Attachments
(1 file)
(deleted),
text/x-phabricator-request
|
Details |
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1597996#c4, we need live logging enabled in GCP for level 1 for parity with AWS.
Reporter | ||
Updated•5 years ago
|
Comment 1•5 years ago
|
||
From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?
Comment 2•5 years ago
|
||
That's correct, and public IPs (which I think are already in place)
Reporter | ||
Comment 3•5 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #2)
That's correct, and public IPs (which I think are already in place)
Yes, and we can check this pretty easily once the port change is made by checking the logs for any running, tier-3, GCP task.
Comment 4•5 years ago
|
||
Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.
Reporter | ||
Comment 5•5 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #4)
Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.
I just checked, and GCP live logs are still broken:
...complains about a "NetworkError when attempting to fetch resource." The raw log links wants points to:
...which is not found.
Reporter | ||
Comment 6•5 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #1)
From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
Comment 7•5 years ago
|
||
Sorry, I messed this up -- we need all ports 32768-65535 open.
Comment 9•5 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #6)
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.
Reporter | ||
Comment 10•5 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #8)
Done.
Still can see the live logs. From IRC, we think this is because we're missing the stateless DNS config for the GCP worker pools.
dustin: 11:14 AM https://gecko-3-b-linux-gcp-kizjakhdr2anv058tm17iq.c.fxci-production-level3-workers.internal:32770/log/nkKdtMnVTIWcSRhWg4j4ng that isn't a stateless-dns hostname
that would mean that the worker doesn't have the stateless DNS config
I think that's DNS_SERVER_SECRET, hm
coop 11:15 AM so we'll need new images?
dustin: 11:15 AM no, I think that just needs to get put in the worker-pool config
https://firefox-ci-tc.services.mozilla.com/secrets/worker-pool%3Agecko-3%2Fb-linux probably has that secret, but there's no equivalent gcp secret
(I can't see that secret)
Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)
Reporter | ||
Comment 11•5 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #9)
(In reply to Chris Cooper [:coop] pronoun: he from comment #6)
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.
Looks like we don't actually have this for level 3 builders right now, so no, let's not enable it for level 3 builders in GCP. Sorry for the churn.
Reporter | ||
Comment 12•5 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #10)
Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)
Mid-air ate my NI requests for comment #10
Assignee | ||
Comment 13•5 years ago
|
||
Updated•5 years ago
|
Updated•5 years ago
|
Assignee | ||
Comment 14•5 years ago
|
||
I've created the secrets (and the patch above changes our tools to do so by default).
Description
•