Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra
Categories
(Developer Services :: Mercurial: hg.mozilla.org, task)
Tracking
(Not tracked)
People
(Reporter: glob, Unassigned)
References
Details
Attachments
(6 obsolete files)
Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra.
Comment 1•5 years ago
|
||
WIP DO NOT LAND
This revision adds support for uploading to Azure Blob Storage using their
Python SDK. Some assumptions are made at the moment regarding the method of
authentication (using SAS tokens), however this may change in the future.
Azure Blob Storage requires an alphanumeric name for containers that does not
contain spaces or dashes. The region is specified on the account level, and
thus all containers within an account belong to the same region. The
convention taken at this time is similar to that with S3 and GCP buckets,
and that is to include the region name in the container name.
Various pending TODOs.
Comment 2•5 years ago
|
||
WIP DO NOT LAND
Fetch an authentication token from the Microsoft Identity Platform. Use this
token to authenticate against the Azure REST API which is used to fetch the
service tags.
For more info see:
- https://docs.microsoft.com/en-us/rest/api/azure/
- https://docs.microsoft.com/en-us/azure/active-directory/develop/
Required environment variables:
- AZURE_SUBSCRIPTION_ID
- AZURE_APP_CLIENT_SECRET
- AZURE_APP_CLIENT_ID
- AZURE_APP_TENANT_ID
Various pending TODOs
Depends on D72750
Comment 3•5 years ago
|
||
Depends on D73387
Updated•5 years ago
|
Comment 4•5 years ago
|
||
WIP DO NOT LAND
Add support for serving clone bundles from Azure, by checking incoming IP
against the IP prefixes file fetched from Azure. Also added some test data.
TODO: add test, refactor some of the code.
Depends on D73387
Comment 5•5 years ago
|
||
WIP DO NOT LAND
Implement some Terraform code to provision a resource group, storage account,
and container for the bundles.
Depends on D73470
Comment 6•5 years ago
|
||
scripts: instead of manually calling the Azure auth API and fetching an auth
token, do this via the ClientSecretCredential object which is provided by the
azure-identity package.
hgmo: instead of using a SAS token to authenticate against Blob Storage,
use the ClientSecretCredential object directly with the storage account.
Depends on D74132
Updated•5 years ago
|
Updated•4 years ago
|
Updated•4 years ago
|
Updated•4 years ago
|
Updated•4 years ago
|
Comment 7•4 years ago
|
||
Abandoning this until further notice.
Comment 8•2 years ago
|
||
I think we should consider looking into this again. Now we have many level 3 Azure tasks, including some on the release pipeline. This came up because an Azure task in the release pipeline timed out after cloning for 1.5 hours and delayed the release (though it's unclear if it was actually taking that long to clone, or just got stuck somehow).
But either way, some Windows builds are now in Azure too, so this will also increase developer productivity.
Updated•2 years ago
|
Description
•