We've hit three download errors while working on Firefox 41.0.2, so filing in case it's a canary in the traffic mines. eg: Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/update/linux-i686/pa-IN/firefox-39.0.complete.mar to firefox-39.0.pa-IN.complete.mar retry: Calling <function downloadUpdateIgnore404 at 0x2bb19b0> with args: ('ftp.mozilla.org', 'firefox', '39.0', 6, 'linux', 'pa-IN'), kwargs: {}, attempt #2 Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/update/linux-i686/pa-IN/firefox-39.0.complete.mar to firefox-39.0.pa-IN.complete.mar The retry indicates an error occurred, although we don't have debug logging enabled so don't have any specifics. The second request succeeded, but returned 17235968 bytes instead of 48374718. The file has a modification time of 17:45 Pacific today; the request came from an AWS instance in us-east-1. The other two were * http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar, returning 9617408 of 49031170 at 17:53. * http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar, returning 17530880 of 48016923 at 17:32. Do we have any logs at the nginx or CDN layers to determine what happened ?

We recently swapped from http://stage.mozilla.org to http://ftp.mozilla.org, but I think that ends up at the same place and probably isn't relevant.

Cloudfront log entries for these requests: > 2015-10-15 00:37:21 SFO9 61252 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar 200 - Python-urllib/2.6 - - Error 0JNjzXcGtXH28COzzkYDvR9k5_zWV-34QYBjaFxcW4g4_W7bPKZxzw== >ftp.mozilla.orghttp 231 330.082 - - - Miss 2015-10-15 00:32:53 SFO9 17531494 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar 200 - Python-urllib/2.6 - - Hit JrTtHSkzXrkiIZjxUbrD5B-KUIVP56XpFflTFiQLUd6pv4HnF90CfA== ftp.mozilla.org http 231 1.735 - - - Hit > 2015-10-15 00:57:27 SFO9 576731 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar 200 - Python-urllib/2.6 - - Error 2XGMP2IKsY23HYyyuiwi1fEtiK24yra8-IMN_ZGVwJ5EFeDEId0Ubg== ftp.mozilla.org http 228 330.056 - - - Miss 2015-10-15 00:53:00 SFO9 9618022 63.245.214.82 GET > d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar 200 - Python-urllib/2.6 - - Hit DBaKQ2wPUQWmmgbUQivwnXUcO6b0XPK0RhNmZALdnkryDyc81D6Thw== ftp.mozilla.org http 228 1.963 - - - Hit

Yikes, ugly formatting. Attaching.

It's hard to tell exactly what happened, but it looks like there was some Error between the origin (zeus lb) and cloudfront. Probably not worth debugging since we are moving to S3 in a few days.

Log format is described here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#BasicDistributionFileFormat

We did another release without hitting this, and we're migrating tomorrow, so that sounds fair.

We see an identical behavior a couple of days now for our firefox-ui-tests. We fail to run the tests at all because the downloaded installer file is invalid. See bug 1219934 for more details. Shall we reopen this bug or file a new one for netops given that it seems to be an issue only for scl3 -> cloudfront?

We shouldn't reopen this for netops, new clean bugs work best there.

Jeremy suggest to use bug 1170832 for that underlying problem.