We should change our test machinery to use https to download test/build files from ftp rather than plain http. Ideally we can roll this out slowly so WebOps can monitor load as we do this. We're hoping that we'll be able to have traffic between AWS and ftp.m.o go over the public internet in order to reduce load on the ipsec tunnel, which requires that we use https to guarantee file integrity.

(In reply to Chris AtLee [:catlee] from comment #0) > We're hoping that we'll be able to have traffic between AWS and ftp.m.o go > over the public internet in order to reduce load on the ipsec tunnel Presuming this will this help with bug 957502?

(In reply to Ed Morley [:edmorley UTC+0] from comment #1) > (In reply to Chris AtLee [:catlee] from comment #0) > > We're hoping that we'll be able to have traffic between AWS and ftp.m.o go > > over the public internet in order to reduce load on the ipsec tunnel > > Presuming this will this help with bug 957502? that's the hope!

This will switch over all post_upload-based sendchanges/triggers over to https://ftp.m.o, which isn't "rolling out slowly". Open to other ideas..? I didn't touch the candidates url because it looks like stage.m.o doesn't have https enabled.

Stop clobbering hg share dirs on differences of http vs https. Untested, but what could possibly go wrong?

Scope creep!

Comment on attachment 8366753 [details] [diff] [review] post_upload_https.diff [13:35] <catlee> aki: for bug 960571 you may want to wait to get the new hostname from bug 964486 [13:35] <aki> catlee: ok. we use that to explicitly change netflows? [13:36] <catlee> yes

Comment on attachment 8366753 [details] [diff] [review] post_upload_https.diff Er..

Landed https://hg.mozilla.org/build/mozharness/rev/8ac0f103b3de to switch traffic from http://ftp to https://ftp-ssl for now.

(In reply to Chris AtLee [:catlee] from comment #9) > Landed https://hg.mozilla.org/build/mozharness/rev/8ac0f103b3de to switch > traffic from http://ftp to https://ftp-ssl for now. We still need public IPs for EC2, and routing table updates before this traffic will go over the public network.

Comment on attachment 8366913 [details] [diff] [review] post_upload_https.diff I had wondered if we could set up the Apache config on http://ftp.m.o to redirect to https://ftp-ssl in some random way, but with a knob to control the proportion of the time it happens. Just for our machines preferably, which might get difficult when we're off the tunnel and the IP making the request isn't in 10.x.y.z any more. Could do something similar thing in mozharness, with much more direct control.

Modifying the request I mean, pretty sure I saw a patch today that does that, just with some random() thrown in.

Use https://hg-ssl.m.o for hg access if we don't think we can switch everything over to https://hg in a reasonable timeframe.

Comment on attachment 8366764 [details] [diff] [review] hgtool.diff Hm, in a way we *don't* want this patch if we're switching to https. We want all the http:// clones to go away.

It begins.

braindump : http://hg.mozilla.org/build/braindump/rev/f84539717a09

Comment on attachment 8368856 [details] [diff] [review] autoland https://hg.mozilla.org/build/autoland/rev/6d9e50722946

Comment on attachment 8368850 [details] [diff] [review] mozilla-inbound-talos Review of attachment 8368850 [details] [diff] [review]: ----------------------------------------------------------------- I'm not overly familiar with this code, but I've taken a look at how it's used, and it seems like this should work fine.

with 100% less hg-internal.

Comment on attachment 8368896 [details] [diff] [review] tools https://hg.mozilla.org/build/tools/rev/935bb5461c1d

Comment on attachment 8368838 [details] [diff] [review] b2g-inbound-https Review of attachment 8368838 [details] [diff] [review]: ----------------------------------------------------------------- and catlee saw that it was good

Comment on attachment 8368868 [details] [diff] [review] buildbotcustom Review of attachment 8368868 [details] [diff] [review]: ----------------------------------------------------------------- do the tests work? I'm assuming twisted's getPage handles SSL ok

http://hg.mozilla -> https://hg.mozilla.org removal of hg-internal http://ftp.mozilla.org -> https://ftp-ssl.mozilla.org

Comment on attachment 8368838 [details] [diff] [review] b2g-inbound-https https://hg.mozilla.org/integration/b2g-inbound/rev/dfd275183653

Comment on attachment 8366764 [details] [diff] [review] hgtool.diff Obsoleting due to comment 15.

Comment on attachment 8368866 [details] [diff] [review] buildbot-configs WCPGW? :)

Comment on attachment 8368911 [details] [diff] [review] mozharness https://hg.mozilla.org/build/mozharness/rev/ce76df196bb1

Comment on attachment 8368868 [details] [diff] [review] buildbotcustom https://hg.mozilla.org/build/buildbotcustom/rev/066534659fe8

Comment on attachment 8368866 [details] [diff] [review] buildbot-configs https://hg.mozilla.org/build/buildbot-configs/rev/7720318925fd

Comment on attachment 8368850 [details] [diff] [review] mozilla-inbound-talos https://hg.mozilla.org/integration/mozilla-inbound/rev/1aa0a5f405fa

Comment on attachment 8368942 [details] [diff] [review] buildapi https://hg.mozilla.org/build/buildapi/rev/73dbae9cb3db

Comment on attachment 8368951 [details] [diff] [review] cloud-tools https://hg.mozilla.org/build/cloud-tools/rev/432b3e0b983f

Comment on attachment 8368838 [details] [diff] [review] b2g-inbound-https [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 957502 User impact if declined: More tree closures due to infrastructure load on the tunnel. We likely will not be able to redirect hg.mozilla.org traffic off the tunnel. Testing completed (on m-c, etc.): Landed on b2g-inbound. Risk to taking this patch (and alternatives if risky): Could cause some build bustage, but we should catch it relatively quickly. String or IDL/UUID changes made by this patch: None.

Comment on attachment 8368850 [details] [diff] [review] mozilla-inbound-talos [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 957502 User impact if declined: More tree closures due to infrastructure load on the tunnel. We likely will not be able to redirect hg.mozilla.org traffic off the tunnel. Testing completed (on m-c, etc.): Landed on mozilla-inbound. Risk to taking this patch (and alternatives if risky): Could cause some build bustage, but we should catch it relatively quickly. String or IDL/UUID changes made by this patch: None.

These are [hopefully all] the harder-to-find ones.

Switch over to ftp-ssl and remove mirror urls. This requires the buildbotcustom2 patch, or test-masters.sh dies on the mirror url removal.

Comment on attachment 8368850 [details] [diff] [review] mozilla-inbound-talos We can probably use a=testing.

Comment on attachment 8368998 [details] [diff] [review] buildbotcustom2 https://hg.mozilla.org/build/buildbotcustom/rev/5f7ccb6c3034

(In reply to Aki Sasaki [:aki] from comment #51) > Comment on attachment 8368999 [details] [diff] [review] > buildbot-configs2 > > https://hg.mozilla.org/build/cloud-tools/rev/432b3e0b983f Er, https://hg.mozilla.org/build/buildbot-configs/rev/374f2d94035e

Comment on attachment 8368934 [details] [diff] [review] partner-repacks https://hg.mozilla.org/build/partner-repacks/rev/1987c727dcec

Comment on attachment 8368953 [details] [diff] [review] mozpool Review of attachment 8368953 [details] [diff] [review]: ----------------------------------------------------------------- This is comments, docs, and a human-readable link in setup.py, but I've no problem with it. It won't be necessary to ship a new version.

Comment on attachment 8368953 [details] [diff] [review] mozpool https://hg.mozilla.org/build/mozpool/rev/ca7e4096de0e

Comment on attachment 8368838 [details] [diff] [review] b2g-inbound-https let's get this in before merge.

Uplifted to aurora: https://hg.mozilla.org/releases/mozilla-aurora/rev/9737529dc0e2 https://hg.mozilla.org/releases/mozilla-aurora/rev/60803c5e3151 RyanVM has the other trees like a boss.

https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/4abbfef147cb https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/1592ce49929c https://hg.mozilla.org/releases/mozilla-b2g18/rev/6221852afb99 https://hg.mozilla.org/releases/mozilla-b2g18/rev/65371561a833 https://hg.mozilla.org/releases/mozilla-esr24/rev/fbfcffa000a6 I'll take care of the v1.1hd branch (Helix config) when I merge b2g18 there next.

Comment on attachment 8368929 [details] [diff] [review] puppet https://hg.mozilla.org/build/puppet/rev/fc81396fae40

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/6221852afb99 https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/65371561a833 With the Helix change included in the merge commit.

a mozharness patch has been merged into production :)

Comment on attachment 8366913 [details] [diff] [review] post_upload_https.diff Sending files/etc/post_upload.ini Transmitting file data . Committed revision 81800.

http://hg.mozilla.org/build/mozharness/rev/eba30402acca landed to fix the wget ssl issue on 28.0b1. I transplanted to default: http://hg.mozilla.org/build/mozharness/rev/618b1d93dca7 then backed it out on default: http://hg.mozilla.org/build/mozharness/rev/fd61adbd8703

Merged mozharness; running a single locale nightly to test.

make wget-en-US works ok without this patch, since bug 967452's patch landed. Resolving this bug!

Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

(In reply to Aki Sasaki [:aki] from comment #67) > Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf IT rolled this out. Hitting https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c9 though; backing out post_upload.py

Comment on attachment 8366913 [details] [diff] [review] post_upload_https.diff Backed out: Sending files/etc/post_upload.ini Transmitting file data . Committed revision 81811.

OSX Jetpack also hit issues. We need new wgets or smarter uses of wget. https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c10

Reopening for post_upload.ini. We need to fix wget on foopies and osx, at the least, before this can reland.

Also windows wget for jetpack. Yay

Comment on attachment 8370971 [details] [diff] [review] fix_l10n https://hg.mozilla.org/build/buildbot-configs/rev/e7b3168ad5a2

https://hg.mozilla.org/build/buildbot-configs/rev/e7b3168ad5a2 -> is in production

(In reply to Aki Sasaki [:aki] from comment #67) > Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf In production :)

My localized nightly just updated to 30, so we have now l10n central builds.

(In reply to Francesco Lodolo [:flod] from comment #77) > My localized nightly just updated to 30, so we have now l10n central builds. Great! I assume you're on Mac, because the Windows nightly hasn't finished yet. Windows l10n builds should be getting updates again shortly after it does though.

(In reply to Ben Hearsum [:bhearsum] from comment #78) > (In reply to Francesco Lodolo [:flod] from comment #77) > > My localized nightly just updated to 30, so we have now l10n central builds. > > Great! I assume you're on Mac, because the Windows nightly hasn't finished > yet. Windows l10n builds should be getting updates again shortly after it > does though. Windows seems to be working now too. Eg: https://aus4.mozilla.org/update/3/Firefox/14.0a1/20120222174716/WINNT_x86-msvc/de/nightly/default/default/default/update.xml

It would have been great if this change got more attention from other teams. I haven't seen any notification for it. As result our test automation for Mozmill was totally broken the whole last week given that pulsetranslator tried to grab the details via HTTP but not HTTPS. :( I know that there are most likely dozen of tools involved here, which you might not all know. But especially because of that it would be kinda helpful to get information upfront, so that enough time exists to get tools updated. Can we make sure to do that in the future? Thanks.

Sorry about that; it didn't even cross my mind. However, it's not really clear what will and will not break external tools. Is https the main thing you're concerned about when changed, or are there other things?

Well, this time it was the HTTPS change. The update from Jgriffin for pulsetranslator made it work again. What I think could be helpful is to make an announcement in the future and cc the tools list, so people working on different tools for automation are aware of upcoming changes, which might break the current workflow.

(In reply to Henrik Skupin (:whimboo) from comment #82) > Well, this time it was the HTTPS change. The update from Jgriffin for > pulsetranslator made it work again. What I think could be helpful is to make > an announcement in the future and cc the tools list, so people working on > different tools for automation are aware of upcoming changes, which might > break the current workflow. In this particular case we were working quickly to fix tree closing issues. There was a blog post made on the 5th though: http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html

This appears to have also broken Telemetry submissions since it bubbled up into the Telemetry Payload via the HISTOGRAMS_FILE_VERSION constant (which in turn comes from the "getSourceRepo" function in config/makefiles/rcs.mk) It might be worth checking other uses of getSourceRepo to see if it's likely to cause any other problems, as well as possibly updating the comment in rcs.mk to indicate that the URI can be https.

Unassigning, should someone want to take this bug while I'm out. Aiui we need to land post_upload.py again once we deal with the blocking bugs.