https://treeherder.mozilla.org/logviewer.html#?job_id=11934248&repo=mozilla-inbound task failed with 403 errors https://tools.taskcluster.net/task-inspector/#45HwZbdbQrSwcappBwpcSg/0

So as I understand, the problem here is that the test runs are failing to download the artifacts that they need, but the more concerning thing is that the build that creates the artifacts is erroring on the upload of said artifacts but still showing as completed. We should be failing if the artifact upload fails. I'm not sure what part of the system owns this but it looks like it might be docker-worker [taskcluster] Artifact "public/build" failed to upload "artifacts/target.linux-x86_64.tar.bz2" error: Error: read ECONNR ESET localhost:~/taskcluster/docker-worker $ git grep 'failed to upload' lib/features/artifacts.js: 'Artifact "%s" failed to upload "%s" error code: %s', lib/features/artifacts.js: 'Artifact "%s" failed to upload "%s" error: %s', I suspect that we want to make uploadArtifact know if any of the uploads failed to happen, and throw if there are any failures.

We haven't historically failed tasks which had an artifact upload failure, but we should have been. We haven't really seen this too much because the errors were being eaten up by something else causing the task to fail in other ways. Because of recent changes made to correctly capture these errors (Thanks to EggyLv999), we will not see more tasks not failing because of this. Definitely should be changed to finish uploading all artifacts and fail if any one of those fails to upload. Also we should be retrying the upload.

I assume this is docker-worker specific. Note: workers can create error artifacts for missing files, in which case the queue returns 403.

Worker does create the error artifact if the artifact is not in the docker container, but the issue is that we don't fail the task if that artifact is not present. In this case a downstream task relies on an artifact in the parent task, which isn't there and causes the downstream task to fail. Really that downstream task shouldn't have been scheduled at all.

@garndt, Agree. But the error artifact explains why it's a 403 and not a 404. But yeah, the point of this bug is resolving tasks as failed if artifacts are missing.

this happened again this morning, seems this is becomming a at least once-a-day problem :(

:Tomcat A fix [1] was merged this morning that will be rolling out to the workers shortly that hopefully will help with this and at least prevent the downstream tasks from running if the artifact wasn't uploaded by the build task. [1] https://github.com/taskcluster/docker-worker/commit/d667661d00db48d4c0e0c99bc315d27e2383e68b

Tasks that cannot upload artifacts because the connection was reset will fail. Reopen if this still is an issue.