S3 TC - Purge all the "target.crashreporter-symbols-full.zip" older than 2 months
Categories
(Cloud Services :: Operations: Taskcluster, task)
Tracking
(Not tracked)
People
(Reporter: Sylvestre, Assigned: wezhou)
References
(Blocks 1 open bug)
Details
With catlee's query:
https://console.cloud.google.com/bigquery?project=task-inventory-test&folder=&organizationId=&j=bq:US:bquxjob_1262264a_173149aadd9&page=queryresults
we are correlating storage and usage of S3.
It seems that target.crashreporter-symbols-full.zip
are almost never accessed.
I propose that we purge all artifact older than two months to have immediate save.
According to this bigquery, we are talking about 82T of data, which represents a significant cost.
https://console.cloud.google.com/bigquery?sq=302012580098:01484217c40a41fa8a2f31c52d0452ca
SELECT
round(SUM(size_gb)/1024, 2) as size_tb
FROM (
SELECT
*,
size/1024/1024/1024 AS size_gb
FROM
`inv_20200630.inventory`
WHERE
mtime <= '2020-05-01'
AND REGEXP_CONTAINS(key, r"target.crashreporter-symbols-full.zip")
)
List of id is about 114 640 elements. results can be found here:
https://console.cloud.google.com/bigquery?sq=302012580098:d4a51fed0d3442c3a48c438e485946f3
Reporter | ||
Comment 1•4 years ago
|
||
I asked Gabriele and Glandium they are fine.
For releases, symbols are stored on the symbol server
Comment 2•4 years ago
|
||
Maybe we could be more aggressive for this particular artifact and expire it after a couple of weeks?
Comment 3•4 years ago
|
||
Bugbug thinks this bug should belong to this component, but please revert this change in case of error.
Comment 4•4 years ago
|
||
I don't think this belongs here, Marco where should this be handled?
Comment 5•4 years ago
|
||
Maybe Firefox Build System::Task Configuration?
Comment 6•2 years ago
|
||
Since https://hg.mozilla.org/mozilla-central/rev/03b009ee6729 (from bug 1790453), we are expiring this artifact after 3 months from creation. We can probably be even more aggressive.
It's still worth doing this to reduce costs right now. In a year from now this would be useless as we'd have all such artifacts be expiring quickly anyway.
Comment 7•2 years ago
|
||
Discussed with :wezhou on Slack, he's going to look into this.
Updated•1 years ago
|
Comment 8•1 years ago
|
||
This was done (see bug 1651965 comment 47).
Description
•