Open Bug 1632495 Opened 5 years ago Updated 2 years ago

Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra

Categories

(Developer Services :: Mercurial: hg.mozilla.org, task)

Production

Tracking

(Not tracked)

REOPENED

People

(Reporter: glob, Unassigned)

References

Details

Attachments

(6 obsolete files)

Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra.

WIP DO NOT LAND

This revision adds support for uploading to Azure Blob Storage using their
Python SDK. Some assumptions are made at the moment regarding the method of
authentication (using SAS tokens), however this may change in the future.

Azure Blob Storage requires an alphanumeric name for containers that does not
contain spaces or dashes. The region is specified on the account level, and
thus all containers within an account belong to the same region. The
convention taken at this time is similar to that with S3 and GCP buckets,
and that is to include the region name in the container name.

Various pending TODOs.

WIP DO NOT LAND

Fetch an authentication token from the Microsoft Identity Platform. Use this
token to authenticate against the Azure REST API which is used to fetch the
service tags.

For more info see:
- https://docs.microsoft.com/en-us/rest/api/azure/
- https://docs.microsoft.com/en-us/azure/active-directory/develop/

Required environment variables:
- AZURE_SUBSCRIPTION_ID
- AZURE_APP_CLIENT_SECRET
- AZURE_APP_CLIENT_ID
- AZURE_APP_TENANT_ID

Various pending TODOs

Depends on D72750

Attachment #9144902 - Attachment is obsolete: true

WIP DO NOT LAND

Add support for serving clone bundles from Azure, by checking incoming IP
against the IP prefixes file fetched from Azure. Also added some test data.

TODO: add test, refactor some of the code.

Depends on D73387

WIP DO NOT LAND

Implement some Terraform code to provision a resource group, storage account,
and container for the bundles.

Depends on D73470

scripts: instead of manually calling the Azure auth API and fetching an auth
token, do this via the ClientSecretCredential object which is provided by the
azure-identity package.

hgmo: instead of using a SAS token to authenticate against Blob Storage,
use the ClientSecretCredential object directly with the storage account.

Depends on D74132

Attachment #9146852 - Attachment is obsolete: true
Attachment #9146309 - Attachment is obsolete: true
Attachment #9145054 - Attachment is obsolete: true
Attachment #9144900 - Attachment is obsolete: true
Attachment #9143795 - Attachment is obsolete: true

Abandoning this until further notice.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX

I think we should consider looking into this again. Now we have many level 3 Azure tasks, including some on the release pipeline. This came up because an Azure task in the release pipeline timed out after cloning for 1.5 hours and delayed the release (though it's unclear if it was actually taking that long to clone, or just got stuck somehow).

But either way, some Windows builds are now in Azure too, so this will also increase developer productivity.

Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Assignee: zeid → nobody
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: