Once build artifacts are in s3 (bug 1100624), we'll want to have testers pull data from s3 instead of ftp. The initial target is to support trunk & try builds.

This still needs the dependent bug to land and stick in mozharness before it can land, but it seems to work fine on try. I've just duplicated the annoying property_conditions for now. This set will be able to replace the ones in m-c/build/gen_mach_buildprops.py. The other sets in mozharness (MakeUploadOutputParser) and buildbotcustom (parse_make_upload) can hopefully be removed as things are converted mach+mozharness and take advantage of the S3 uploading. I think I would prefer avoiding this silly pattern matching to determine the property type though, and somehow expose that from the build system into mach_build_properties.json - if you have any thoughts on that let me know. In the try push I've just launched a single test and talos job to verify that the files are pulled from S3 instead of ftp.m.o: try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c0735a40931

Comment on attachment 8560689 [details] [diff] [review] s3-tests Review of attachment 8560689 [details] [diff] [review]: ----------------------------------------------------------------- this is awesome. I see you took my painful logic in MakeUploadOutputParser and made it better! Though I agree it is still a pain. maybe you would be interested in talking with myself and nick next week about improving that: https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3

(In reply to Jordan Lund (:jlund) from comment #2) > this is awesome. I see you took my painful logic in MakeUploadOutputParser > and made it better! Though I agree it is still a pain. maybe you would be > interested in talking with myself and nick next week about improving that: > https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3 Definitely - let me know when you guys are meeting and I'll join in.

:jonasfj recommended waiting until the TC queue rewrite is complete (bug 1121293) before starting to use the files from s3 for testers & other things.

We'll also want to spruce up the python client error handling before switching testers over. I hit an issue where a tests.zip file wasn't uploaded correctly. We suspect that the upload died part way through, and then retries failed since the file-handle wasn't reset (bug 1133876). Even if a retry did succeed, it looks like it would have retried several times unnecessarily (bug 1133856).

Comment on attachment 8560689 [details] [diff] [review] s3-tests https://hg.mozilla.org/build/mozharness/rev/32f80f4d499e

bump mozharness rev, this time with a production revision. Note I haven't tried this one on try, but the previous default rev looks good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=53caf322caad

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment on attachment 8581914 [details] [diff] [review] 0001-Bug-1117960-bump-mozharness.json-to-rev-93f17b170f43.patch Review of attachment 8581914 [details] [diff] [review]: ----------------------------------------------------------------- thanks for changing this to a prod rev :)

bump mozharness rev: https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f