Closed Bug 1650386 Opened 4 years ago Closed 1 years ago

S3 TC - Purge all the "target.crashreporter-symbols-full.zip" older than 2 months

Categories

(Cloud Services :: Operations: Taskcluster, task)

task

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1651965

People

(Reporter: Sylvestre, Assigned: wezhou)

References

(Blocks 1 open bug)

Details

With catlee's query:
https://console.cloud.google.com/bigquery?project=task-inventory-test&folder=&organizationId=&j=bq:US:bquxjob_1262264a_173149aadd9&page=queryresults
we are correlating storage and usage of S3.

It seems that target.crashreporter-symbols-full.zip are almost never accessed.

I propose that we purge all artifact older than two months to have immediate save.

According to this bigquery, we are talking about 82T of data, which represents a significant cost.

https://console.cloud.google.com/bigquery?sq=302012580098:01484217c40a41fa8a2f31c52d0452ca

SELECT
  round(SUM(size_gb)/1024, 2) as size_tb
FROM (
  SELECT
    *,
    size/1024/1024/1024 AS size_gb
  FROM
    `inv_20200630.inventory`
  WHERE
    mtime <= '2020-05-01'
    AND REGEXP_CONTAINS(key, r"target.crashreporter-symbols-full.zip")
)

List of id is about 114 640 elements. results can be found here:

https://console.cloud.google.com/bigquery?sq=302012580098:d4a51fed0d3442c3a48c438e485946f3

I asked Gabriele and Glandium they are fine.
For releases, symbols are stored on the symbol server

Maybe we could be more aggressive for this particular artifact and expire it after a couple of weeks?

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: General → Crash Reporting
Product: Firefox → Toolkit

I don't think this belongs here, Marco where should this be handled?

Flags: needinfo?(mcastelluccio)

Maybe Firefox Build System::Task Configuration?

Component: Crash Reporting → Task Configuration
Flags: needinfo?(mcastelluccio)
Product: Toolkit → Firefox Build System

Since https://hg.mozilla.org/mozilla-central/rev/03b009ee6729 (from bug 1790453), we are expiring this artifact after 3 months from creation. We can probably be even more aggressive.

It's still worth doing this to reduce costs right now. In a year from now this would be useless as we'd have all such artifacts be expiring quickly anyway.

Discussed with :wezhou on Slack, he's going to look into this.

Assignee: nobody → wezhou
Status: NEW → ASSIGNED
Component: Task Configuration → Operations: Taskcluster
Product: Firefox Build System → Cloud Services

This was done (see bug 1651965 comment 47).

Status: ASSIGNED → RESOLVED
Closed: 1 years ago
Duplicate of bug: 1651965
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.