We are already doing this for try. We store files for 28 days:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/task.py#1809

We should do similar things for autoland and m-c. And maybe for m-b, m-r and m-esr

Context: we have storing a lot of data and it has a significant cost.

This needs to be more granular than reducing the retention per branch. For one, the per-push shippable builds are useful for mozregression, so those should, in fact, be kept longer, short of uploading them somewhere else (which may itself be desirable). Other artifacts from builds are potentially less interesting. Artifacts from other builds are also less interesting.

BTW, I'm not sure taskcluster allows an artifact expiry shorter than the expiry of the task itself. So it might not actually be possible to do per-artifact expiration.

Or even the other way around (artifact expiry longer than the expiry of the task itself)

Ok, I just checked, it seems they both work.

Correction, artifact expiration needs to be shorter or equal than the task expiration.

I think a good starting point is to preserve the current policy and keep build artifacts for 1 year, and change the policy for non-build artifacts to 3 months. This gives us significant savings without reducing what binaries are available for mozregression.

Backed out changeset 49da43027193 (bug 1649987) for gecko decision bustage

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&fromchange=49da43027193fc09fc35c53ef8397c8b99c00974&searchStr=gecko%2Cdecision&tochange=774f97c76ce3b7c0fa416e725cad27fb480c67e0&selectedTaskRun=KpUtcdWoSp-axaTjQYeSoQ.0

Backout link: https://hg.mozilla.org/integration/autoland/rev/774f97c76ce3b7c0fa416e725cad27fb480c67e0

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312837922&repo=autoland&lineNumber=317

...
[task 2020-08-12T15:14:20.537Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-08-12T15:14:20.537Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-08-12T15:14:20.537Z] Exception: In task.run 'fetch-wasm-misc':
[task 2020-08-12T15:14:20.537Z] extra keys not allowed @ data['artifacts'][0]['expires-after']
[task 2020-08-12T15:14:20.537Z] {'allow-ptrace': True,
[task 2020-08-12T15:14:20.537Z]  'artifacts': [{'expires-after': '1000 years',
[task 2020-08-12T15:14:20.537Z]                 'name': 'public',
[task 2020-08-12T15:14:20.537Z]                 'path': '/builds/worker/artifacts',
[task 2020-08-12T15:14:20.538Z]                 'type': 'directory'}],
[task 2020-08-12T15:14:20.538Z]  'chain-of-trust': True,
[task 2020-08-12T15:14:20.538Z]  'command': ['/builds/worker/bin/run-task',
[task 2020-08-12T15:14:20.538Z]              '--fetch-hgfingerprint',
[task 2020-08-12T15:14:20.538Z]              '--',
[task 2020-08-12T15:14:20.538Z]              '/builds/worker/bin/fetch-content',
[task 2020-08-12T15:14:20.538Z]              'static-url',
[task 2020-08-12T15:14:20.538Z]              '--sha256',
[task 2020-08-12T15:14:20.538Z]              '0ba273b748b872117a4b230c776bbd73550398da164025a735c28a16c0224397',
[task 2020-08-12T15:14:20.538Z]              '--size',
[task 2020-08-12T15:14:20.538Z]              '4433793',
[task 2020-08-12T15:14:20.538Z]              'https://github.com/mozilla/perf-automation/releases/download/wasm-misc-v1/wasm-misc-c55c3c7690b2.zip',
[task 2020-08-12T15:14:20.538Z]              '/builds/worker/artifacts/wasm-misc.zip'],
[task 2020-08-12T15:14:20.538Z]  'docker-image': {'in-tree': 'fetch'},
[task 2020-08-12T15:14:20.538Z]  'docker-in-docker': False,
[task 2020-08-12T15:14:20.538Z]  'env': {'MOZ_SCM_LEVEL': '3', 'UPLOAD_DIR': '/builds/worker/artifacts'},
[task 2020-08-12T15:14:20.538Z]  'implementation': 'docker-worker',
[task 2020-08-12T15:14:20.538Z]  'loopback-audio': False,
[task 2020-08-12T15:14:20.538Z]  'loopback-video': False,
[task 2020-08-12T15:14:20.538Z]  'max-run-time': 900,
[task 2020-08-12T15:14:20.538Z]  'os': 'linux',
[task 2020-08-12T15:14:20.538Z]  'privileged': False,
[task 2020-08-12T15:14:20.538Z]  'taskcluster-proxy': False,
[task 2020-08-12T15:14:20.538Z]  'volumes': []}
[taskcluster 2020-08-12 15:14:20.971Z] === Task Finished ===
[taskcluster 2020-08-12 15:14:21.122Z] Artifact "public/docker-contexts" not found at "/builds/worker/checkouts/gecko/docker-contexts"
[taskcluster 2020-08-12 15:14:21.222Z] Unsuccessful task run with exit code: 1 completed in 20.15 seconds

Backed out changeset d244e80dc826 (Bug 1649987) for causing builds to fail as exceptions.

Backout link: https://hg.mozilla.org/integration/autoland/rev/7c40291ffb228bc771ed91f38c59f913dc9d2179

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&tochange=7c40291ffb228bc771ed91f38c59f913dc9d2179&fromchange=d244e80dc826e3e2b380e0db2c335082b7634109&selectedTaskRun=S_MmOmQ1Ttm0q-ciVoAObA.0

Failure logs:

• Valgrind: https://firefox-ci-tc.services.mozilla.com/tasks/UjfDtJkbS3ePXgTcvB-7hg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FUjfDtJkbS3ePXgTcvB-7hg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L14
• Repackage: https://firefox-ci-tc.services.mozilla.com/tasks/S_MmOmQ1Ttm0q-ciVoAObA/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FS_MmOmQ1Ttm0q-ciVoAObA%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L13

I think Joel is taking over this work.

Set default task expiry for artifacts

Backed out in:
https://hg.mozilla.org/integration/autoland/rev/157db696462d8a98905d0f8697088aa97cb6e08f

There are at least two problems it causes:

• It breaks some tasks on try because the task expiration is before the artifact expiration, which is not allowed (see https://firefox-ci-tc.services.mozilla.com/tasks/ewN8YES1TuaiALpT1qLXhw/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FewN8YES1TuaiALpT1qLXhw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log)
• It will subtly break things later on because some tasks that are accessed subsequently via the index, like toolchains, docker images, etc. are set ti expire later than their artifact. So they will stay in the index, and tasks using those toolchains, docker images, etc. will fail to download those because the artifacts will be gone. See the task definition of this docker image I triggered on the landing (https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NuMUoI9wRKCMny6HoMtqng) vs. on the backout (https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RTcdSOo1RlOinLFVaa1cSQ)

With this patch, will we still keep test logs for longer than 12 weeks?

we would keep test logs for 1 month. Is there a use case for keeping logs longer? IIRC activedata ingests logs within 2 weeks (typically 2 hours, but assuming outage, restart, etc.)

ActiveData only keeps a limited amount of data about tests (12 weeks), for test selection we need a larger timespan (at least for the errorsummary.log files).

We wrote this doc last year:
https://docs.google.com/document/d/1QC2pj5Y1aK95SdA2nHCSmRrT8aXz9BjZVZarElvdZz0/edit

https://hg.mozilla.org/mozilla-central/rev/971139c802a7
https://hg.mozilla.org/mozilla-central/rev/925838f86a80
https://hg.mozilla.org/mozilla-central/rev/77c769965781

Backed out for causing Bug 1787098.

Push with failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&fromchange=2895842928e98ccf141664d1c03033acebe36706&tochange=ed1f1140d8bd7dca6d3d4cd06ff824b1021d2957&searchStr=static%2Canalysis&selectedTaskRun=PCU-GRTRQaShRMWacnbGPA.0

Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/PCU-GRTRQaShRMWacnbGPA/runs/1/logs/public/logs/live.log

Backout link: https://hg.mozilla.org/mozilla-central/rev/ed1f1140d8bd7dca6d3d4cd06ff824b1021d2957

Depends on D155401

Depends on D156012

Depends on D156013

Depends on D156014

Depends on D156015

Depends on D156016

https://hg.mozilla.org/mozilla-central/rev/5d903d89092c