Fix Firefox CI's dependence on TASKCLUSTER_WORKER_GROUP
Categories
(Taskcluster :: General, enhancement)
Tracking
(Not tracked)
People
(Reporter: bstack, Assigned: wcosta)
References
Details
Attachments
(5 files, 1 obsolete file)
Providers in w-m should be able to set workergroup to whatever they want. Currently we make an assumption that it maps to an AZ which can't necessarily always be true.
Reporter | ||
Updated•5 years ago
|
Updated•5 years ago
|
Comment 1•5 years ago
|
||
My proposal here is to introduce $TASKCLUSTER_WORKER_LOCATION
of the format <cloud>/<details>
, available within tasks. The <details>
portion is specific to the cloud. For example, it would be AZ for Amazon and zone for Google. Hardware workers (standalone and static) will be able to define this string arbitrarily.
Taskcluster-worker-runner would be responsible for supplying this to workers. Workers will fall back to unknown/unknown
in a case where it is not supplied, so the env var is always set.
This will be documented in taskcluster-service docs along with other per-task variables, and with reference to tc-worker-runner for per-cloud definition of <details>
.
This is simple, but presents a user-visible change, so should probably be handled with an RFC.
Comment 2•5 years ago
|
||
Assignee | ||
Comment 4•5 years ago
|
||
Assignee | ||
Comment 5•5 years ago
|
||
This version contains the TASKCLUSTER_WORKER_LOCATION environment
variable.
Reporter | ||
Comment 6•5 years ago
|
||
Updated•5 years ago
|
Reporter | ||
Comment 7•5 years ago
|
||
Create a worker pool to test builds in gcp
Assignee | ||
Comment 8•5 years ago
|
||
Assignee | ||
Comment 9•5 years ago
|
||
With the migration from AWS to GCP, we also need to migrate sccache
buckets from S3 to Google Storage.
The problem is how we deal with regions, since there isn't an exact
correspondence on the region names between the two cloud providers.
To make the transition smoother, docker-worker (and soon generic-worker)
provides a new environment variable called TASKCLUSTER_WORKER_LOCATION,
with information about the cloud provider the task is running on. Using
this new variable, we configure sccache to use the corresponding storage
service of the cloud provider where the task runs.
The bucket names in Google Storage are shorter because GCS imposes a
limit of 30 characteres for the names.
Ref: https://github.com/taskcluster/taskcluster-rfcs/pull/148/files
Assignee | ||
Comment 10•5 years ago
|
||
Comment 11•5 years ago
|
||
Comment 12•5 years ago
|
||
bugherder |
Assignee | ||
Comment 13•5 years ago
|
||
:grenade sccache support via gcpCredentials is ready, if you intend to implement in OCC, the only thing it is needed is the TASKCLUSTER_WORKER_LOCATION environment variable. It is a json string in the format:
{
"cloud": "google|aws"
"region": "<region name>"
"availabilityZone": "<zone>"
}
Comment 14•5 years ago
|
||
when bug 1580430 is resolved, we should be able to spin up windows gcp workers again and i will add this env var
Comment 15•5 years ago
|
||
this variable is now being set on gcp windows workers.
https://papertrailapp.com/events?q=system%3Agamma%20program%3AOpenCloudConfig%20TASKCLUSTER_WORKER_LOCATION
Comment 16•5 years ago
|
||
we have windows builds running in gcp again. these instances have their TASKCLUSTER_WORKER_LOCATION
environment variable set as can be seen in papertrail or build logs. USE_SCCACHE
is also set to 1
however these builds are not making successful use of sccache and build logs do not contain an ===SCCACHE STATS===
section.
Description
•