mcMerge's code is currently hosted within TBPL's repository: https://hg.mozilla.org/webtools/tbpl/file/ae716547090e/mcmerge We should probably make sure it finds a new home when TBPL is decomissioned.

Yeah we'll need to do this - however this doesn't block bug 1030636 since that's not about eol (see bug 1030636 comment 2). Switching off tbpl is several steps after bug 1030636.

I'm not sure what we should do with mcMerge - we could either: * move it into treeherder as-is for now, and work on integrating it better in the future * move it into version-control-tools and have new hosting set up to point to there

(In reply to Ed Morley [:edmorley] from comment #2) > I'm not sure what we should do with mcMerge - we could either: > * move it into treeherder as-is for now, and work on integrating it better > in the future > * move it into version-control-tools and have new hosting set up to point to > there Whichever we do, it will simplify the move greatly if we can remove the only server side component to mcMerge (particularly since it's PHP, which may not even be set up on the treeherder boxes), which can only be done if bug 774766 is fixed.

I would eventually like mcMerge to be driven automatically merely by pushing to a repository. Unless some other arrangement is made, my guess is that replacing mcMerge with something more automated will make its way onto the Developer Services team roadmap for 2015.

I'm not sure if all of the functionality of mcMerge can be replaced with a tool that has no interactive mode - at least not whilst we're clumsily scraping commit messages and relying on a human to check the interpretation & handle things like security bugs, closing bugs, the myriad of B2G status flags etc. (All things we should be fixing, but will take a while). However something that manages to do so would be very welcome :-) That said, this bug is needed shorter term than a solution to the above would be available, since it would hold up the switching off of tbpl.m.o. However fortunately this bug will be pretty easy to fix, now that hg.m.o returns the correct access control headers - since I'll just fix bug 1072923 (and so make mcMerge static content only) and sling it up on Github pages to keep us going until another tool replaces it.

Yeah, some manual functionality is still needed. Once Autoland is in place and driving 99% of landings, Autoland effectively becomes mcMerge. I agree it is a security nightmare to work out. But we'll get there. We need to: we can't continue having humans doing work machines can do more efficiently.

Agree totally :-)

So couple of decisions to make: 1) Are we definitely putting this on Github (I'm leaning this way) vs say the version control tools repo? 2) Are we going to host it using Github pages or not (they support HTTPS, but forcing it is tricky unless we use Cloudflare or similar). I'd like to if possible, just because it won't require using Chief or managing some box somewhere (for what is a fairly simple client side only app). Presuming the answer to #1 is yes, #2 for now only really affects the Git branch name (master vs gh-pages). Either way, these are the steps that will be required to export the mcmerge commits into their own git repo: 1) Setup: git clone http://repo.or.cz/r/fast-export.git 2) Break out mcmerge commits to their own Hg repo (and rewrite paths so it's at the repo root): echo include mcmerge > ~filemap echo rename mcmerge . >> ~filemap hg convert --filemap ~filemap tbpl mcmerge-hg 3) Checking to see if author mapping required when converting to Git: cd mcmerge-hg hg log | grep user: | sort | uniq | sed 's/user: *//' > ../authors cd .. -> Inspect output to see if any changes required. (None needed in this case, so this step can be omitted). 4) Convert the standalone Hg repo to a Git repo, whose branch is named gh-pages: git init mcmerge cd mcmerge ../fast-export/hg-fast-export.sh -r ../mcmerge-hg -M gh-pages git checkout HEAD 5) <Add a remote and push>

Adding these too so I don't lose them: https://me.net.nz/blog/github-pages-secure-with-cloudflare/ https://konklone.com/post/github-pages-now-supports-https-so-use-it

This is now blocked on making a decision on where to host this. Some considerations: * As a quick reminder, mcMerge is client-side only, with Bugzilla credentials only ever being posted to Bugzilla's bzapi compatibility layer over HTTPS. The kicker being that if someone MITM'd the mcMerge JS assets they could obviously do whatever they like with the credentials. * Only a half-dozen people use mcMerge, so the audience with respects to encouraging people to only ever accessing mcMerge via Treeherder's HTTPS URL links/HTTPS bookmarks (option 1) is fairly small. * We could always consider making mcMerge work only with a bot account, so people get in the habit of using the credentials for that, and not their personal (potentially security bug permission granted) bugzilla account (as they do at present), making option 1 less of a big deal. The added advantage is that it solves some existing problems we have: mcMerge activity cluttering up people's personal Bugzilla activity/profile, plus other devs not realising a bug comment/closure was performed by a script ("Hey you closed the wrong bug, why did you do that!?"). Choices: Option 1: github pages... * HTTPS is supported, but we have no way of forcing people visit the HTTPS version over the HTTP one, either than a MITM-able client side redirect. * That said, Treeherder (where the vast majority of accesses come from) will link to the HTTPS version, and the redirect would at least ensure people bookmark or copy/paste/share the HTTPS URL. * Motivation: Extremely simple, to set up & deploy. I don't have to file bugs to request someone from IT deploy it - we just push to the gh-pages branch on the new github repo. Option 2: github pages with cloudflare CDN in front... * HTTPS can be forced on the cloudflare protected custom domain, however the connection between github and cloudflare is susceptible, for the reasons mentioned at https://me.net.nz/blog/github-pages-secure-with-cloudflare/ * For this reason I think it's inferior to option 1. Option 3: Hosted by IT/in some AWS instance with an SSL cert * Ideally we'd have deployment done on a cron, so we can still just check into the github repo and be done with it. (Chief is much better than the previous, but it's still an unnecessary extra step IMO). * I'm probably just put off this one because I'm over-estimating the hassle based on previous experience. Happy to be convinced otherwise :-) Thoughts?

Kendall, I don't suppose you could suggest who I should discuss comment 11 with - in particular the ease with which we could prop up Mozilla hosting for this extremely low usage client-side only webapp, that just needs HTTPS for MITM protection? Thanks :-)

I'm still an AWS noob, else I'd stand it up there (I still could, it just wouldn't be fast). WebOps could probably make this happen fairly quickly on the generic cluster, which would alleviate having to spin up a VM somewhere. Let me throw jakem under the bus and see what he has to say. :-D

As much as I'd love to make mcMerge a service that runs automatically and is derived from events on the repository server, I don't think I'll have time to do that until at least Q2.

Yeah the scope for this bug is purely "move mcMerge to new hosting", even if comment 14 is the desired future end state.

(In reply to Kendall Libby [:fubar] from comment #13) > I'm still an AWS noob, else I'd stand it up there (I still could, it just > wouldn't be fast). WebOps could probably make this happen fairly quickly on > the generic cluster, which would alleviate having to spin up a VM somewhere. > > Let me throw jakem under the bus and see what he has to say. :-D Jake, any idea when you might be able to let us know our options/next steps? :-)

Laura, any luck with this? (following on from the email) Also seems like we can go with a AWS/VM solution and have a pretty simple deploy by pushing to the Git repo, using Git 2.3's new feature: https://github.com/blog/1957-git-2-3-has-been-released

Laura asked me to set this up in AWS, but I forgot to come back and find the bug.

(In reply to Kendall Libby [:fubar] from comment #18) > Laura asked me to set this up in AWS, but I forgot to come back and find the > bug. Would it be helpful for me to file another bug for the ops side of this? If so, in which component? :-) With TBPL EOL looming, we'll need somewhere to put mcMerge soon.

I've broken the "we need somewhere to host" part out to webops bug 1140889.

I'm not sure if mcMerge found a new repo home, but I'd be happy to have it in version-control-tools. We already have the test infrastructure in place to provide Docker containers for running Bugzilla and Selenium tests so we can verify end-to-end behavior works as expected.

That's a very kind offer, but I think it might be a bit heavyweight given the inactivity of mcmerge and how much would be involved in getting sensible test data mocked up (well, more than I have time for at the moment anyway). That and I can't get the version-control-tools environment to work properly (bug 1110345), the jenkins run for it takes hours & I'm really starting to like Git. I previously created https://github.com/mozilla/mcMerge but I'm waiting on the new hosting to be ready before running my conversion script locally, so the old and new repos don't get out of sync.

mcMerge has now been extracted from the TBPL repo, converted to git & pushed to: https://github.com/mozilla/mcMerge I won't delete it from the TBPL repo yet - but we should no longer make changes to mcMerge there. -- [~/src]$ echo include mcmerge > ~filemap [~/src]$ echo rename mcmerge . >> ~filemap [~/src]$ hg convert --filemap ~filemap tbpl mcmerge-hg initializing destination mcmerge-hg repository scanning source... sorting... converting... 1514 Initial import. <snip> 0 Bug 911273 - Add the submit button to the top of the page as well. r=edmorley [~/src]$ cd mcmerge-hg [~/src/mcmerge-hg]$ hg log | grep user: | sort | uniq | sed 's/user: *//' > ../authors [~/src/mcmerge-hg]$ cd .. [~/src]$ git init mcMerge Initialized empty Git repository in C:/Users/Ed/src/mcMerge/.git/ [~/src]$ cd mcMerge/ [~/src/mcMerge]$ ../fast-export/hg-fast-export.sh -r ../mcmerge-hg -M master master: Exporting full revision 1/171 with 46/0/0 added/changed/removed files master: Exporting simple delta revision 2/171 with 0/1/0 added/changed/removed files <snip> master: Exporting simple delta revision 171/171 with 0/2/0 added/changed/removed files Issued 171 commands C:\Program Files (x86)\Git\libexec\git-core\git-fast-import.exe statistics: --------------------------------------------------------------------- Alloc'd objects: 5000 Total objects: 841 ( 35 duplicates ) blobs : 312 ( 25 duplicates 138 deltas of 310 attempts) trees : 358 ( 10 duplicates 308 deltas of 331 attempts) commits: 171 ( 0 duplicates 0 deltas of 0 attempts) tags : 0 ( 0 duplicates 0 deltas of 0 attempts) Total branches: 1 ( 1 loads ) marks: 1024 ( 171 unique ) atoms: 70 Memory total: 2294 KiB pools: 2098 KiB objects: 195 KiB --------------------------------------------------------------------- pack_report: getpagesize() = 65536 pack_report: core.packedGitWindowSize = 33554432 pack_report: core.packedGitLimit = 268435456 pack_report: pack_used_ctr = 527 pack_report: pack_mmap_calls = 328 pack_report: pack_open_windows = 1 / 1 pack_report: pack_mapped = 990321 / 990321 --------------------------------------------------------------------- [~/src/mcMerge]$ git checkout HEAD [~/src/mcMerge]$ git remote add origin git@github.com:mozilla/mcMerge.git [~/src/mcMerge]$ git push -u origin master Counting objects: 841, done. Delta compression using up to 8 threads. Compressing objects: 100% (386/386), done. Writing objects: 100% (841/841), 966.82 KiB | 0 bytes/s, done. Total 841 (delta 446), reused 841 (delta 446) To git@github.com:mozilla/mcMerge.git * [new branch] master -> master Branch master set up to track remote branch master from origin.

This is pretty much complete now after much blood, sweat, tears & forking. We're having to use my fork of the buildpack for now, until upstream accept my PRs. I can't quite decide whether using Stackato like this (as an overkill, but unavoidable alternative to force-https github pages) is awful or awesome, but hey - it works haha. To run the instance follow the Mozilla Stackato setup docs then |stackato push --no-prompt| from the repo root. However once I deploy it live, it should need no further intervention, since the cron will update it. Temporary dev (allizom) instance for testing: https://mcmerge.paas.allizom.org/ The http URL 301s to https :-) Deployed revision info available at: https://mcmerge.paas.allizom.org/deploy.txt (updates on 5 min cron). The allizom paas sites have a wildcard ssl cert; for mozilla.org we'll need to decide re the "shall we rename mcMerge" email first, to avoid churn.

We want to EOL TBPL at the end of the month, but I'm on PTO from Monday until 1st April (back 2nd). So we don't have to rush the decision around choosing a new name for mcMerge, and given that we already have the Github repo set up with the current name (since it was created before the name change was raised) - let's just use the allizom instance with the old name for now (https://mcmerge.paas.allizom.org/) - and we can rename (and create a proper X.mozilla.org) later.

Comment on attachment 8579065 [details] Add configuration files for Stackato I've just landed this as-is, since (a) it works, (b) there isn't really anyone who understands this stuff who can review, (c) my PTO ends after TBPL will be EOLed, (d) this stuff is easy to revise later. https://github.com/mozilla/mcMerge/commit/cfeedd2b7398ecf4a69bc4926c2eb90f5f212208 I'll file other bugs for: * Moving to X.mozilla.org rather than allizom or X.pass.mozilla.org * Updating links in Treeherder * Renaming mcMerge * Updating the readme with info on the deployment (the comments at the top of stackato.yml are pretty comprehensive and will be fine for now) For mcMerge, for now use: https://mcmerge.paas.allizom.org/ To check how the auto-deploy is running, see: https://mcmerge.paas.allizom.org/deploy.txt

mcMerge removed from the TBPL repo: https://hg.mozilla.org/webtools/tbpl/rev/5507d424a283

All done here apart from: 1) Deploy treeherder master to production (I've just updated stage, looks fine) 2) Deploy tbpl default to production (to remove mcmerge from there) Will close this bug once those are done.

I've run merges tonight from https://mcmerge.paas.allizom.org/ and it all appears to be working, for whatever that's worth...

(In reply to Wes Kocher (:KWierso) from comment #29) > I've run merges tonight from https://mcmerge.paas.allizom.org/ and it all > appears to be working, for whatever that's worth... Awesome, thank you :-)

(In reply to Ed Morley (Away 23rd March -> 1st April) [:edmorley] from comment #28) > All done here apart from: > 1) Deploy treeherder master to production (I've just updated stage, looks > fine) Done a couple of days ago. > 2) Deploy tbpl default to production (to remove mcmerge from there) Done now. Treeherder now links to mcmerge.pass.allizom.o, and anyone using the old links will still get redirected correctly thanks to the rules added in bug 1145713. Bug 1145836 will handle things like: picking a different name for mcMerge, switching to NAME.mozilla.org rather than something on *.pass.* or anything on allizom etc.

Switched back to the upstream buildpack, now that the changes in my fork have been accepted/merged: https://github.com/mozilla/mcMerge/commit/05184f7dd4f0b848c31eb07b76e741bce7f2ecbe