Closed
Bug 992368
Opened 11 years ago
Closed 10 years ago
Create process for upgrading Mercurial on a schedule
Categories
(Developer Services :: Mercurial: hg.mozilla.org, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bkero, Assigned: bkero)
References
(Blocks 1 open bug)
Details
(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/706] )
Attachments
(2 files, 1 obsolete file)
There is a demand for new versions of Mercurial on the hg.mozilla.org infrastructure due to stability, performance, and new features. It has been demonstrated that the server and client versions of Mercurial need not be the same. During operations both sides will negotiate capabilities that they both support. To accomplish this we'll need to define a schedule this operates on. Recommendations include following the Mercurial release schedule, which is every 3 months. This is long enough for new versions to stabliize and short enough that we shouldn't run into upgrade issues once the process has been determined. We'll also need a set of QA requirements from Release Engineering that we can automate to minimize the amount of manual QA work that needs to be done for each release. IT already has the capabilities to roll our own RPMs for new versions, and has done so for all recent Mercurial upgrades.
Comment 1•11 years ago
|
||
Mercurial's release schedule is at http://mercurial.selenic.com/wiki/TimeBasedReleasePlan tl;dr major releases Feb, May, Aug, and Nov 1. Minor (typically bugfix and minor issue only) releases every month. Given the rapid rate of development on Mercurial as of late and the efficiency gains and bug fixes those entail, I second shadowing the 3 month releases. If we want to mitigate risk, we would deploy the 1st or 2nd minor release after each major version. That limits our exposure to bugs introduced in major releases and lets others do most of the testing for us. mpm: I'm curious if you have any recommendations for server operators. Currently Mozilla is running 2.5.4.
Flags: needinfo?(mpm)
Comment 2•11 years ago
|
||
Facebook internally deploys Mercurial's development tip to servers on a roughly weekly basis with basically no disruption. So I think you'll be just fine with just about any upgrade schedule. Just don't deploy on Friday evening.
Flags: needinfo?(mpm)
Updated•11 years ago
|
Updated•10 years ago
|
Assignee: server-ops-devservices → hwine
Updated•10 years ago
|
Blocks: try-tracker
Updated•10 years ago
|
Component: Server Operations: Developer Services → Mercurial: hg.mozilla.org
Product: mozilla.org → Developer Services
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/243]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/243] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/706] [kanban:engops:https://kanbanize.com/ctrl_board/6/243]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/706] [kanban:engops:https://kanbanize.com/ctrl_board/6/243] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/706]
Assignee | ||
Comment 3•10 years ago
|
||
The work that's gone into version-control-tools (especially create-test-environment and run-mercurial-tests.py) in addition to the documentation about building and deploying package updates has taken us much closer to updating on a regular cycle. Is this enough to mark this as done, or should we create some more material relating to the scheduling process for how the upgrades should take place?
Flags: needinfo?(hwine)
Comment 4•10 years ago
|
||
We have the pieces now, we just need the actual steps and order to stitch it together. E.g. something like: - add Hg version X to test harness (see XXXXX) - get the following test suites to pass: - run-mercurial-tests.py ... - ... - build packages - deploy to staging - run tests YYYY - schedule cutover time with sheriffs - cutover & run post cutover validations - push commit to m-i (e.g. this would have caught last TCW issue) - verify with sheriffs that system look good - declare victory That will give us a place to keep "lessons learned" (probably in the form of additional post cutover checks) as we proceed. While the traditional place for this is in mana, it may make sense to have the several documented in a public place, as community members could perform those steps. (I'd be for having the entire checklist public, with just a pointer in mana, but will defer to group decision.) Greg - would version-control-tools/docs/moz-upgrade.rst be a reasonable spot for the checklist?
Flags: needinfo?(hwine) → needinfo?(gps)
Comment 5•10 years ago
|
||
Yeah, throwing some docs under /docs/ sounds like a plan. I was thinking we may want to create an entire directory/section for hg.mozilla.org/git.mozilla.org/etc implementation and admin notes.
Flags: needinfo?(gps)
Assignee | ||
Comment 6•10 years ago
|
||
I've attached a preliminary version of a document for how to upgrade Mercurial safely and successfully. Let me know what you think.
Attachment #8533928 -
Flags: review?(hwine)
Attachment #8533928 -
Flags: review?(gps)
Comment 7•10 years ago
|
||
Comment on attachment 8533928 [details] Mercurial upgrade instructions >Webheads >^^^^^^^^ > ... >Repeat this procedure until all webheads have been upgraded. The step to change each node back from "draining" to active is missing - following the steps as is will leave zero webheads active :-)
Comment 8•10 years ago
|
||
Comment on attachment 8533928 [details]
Mercurial upgrade instructions
Nice!
A couple of questions I'd like to see answered - only the first one is a blocker:
1. I don't see an explanation for "Re-add and confirm correctness".
2. The yum process as explained to be was to always use "yum-wrapper" if it exists.
3. For the coordination step, advance notification via CAB is preferred - what you have is minimum for an emergency up/down grade. (This can be expanded later.)
Attachment #8533928 -
Flags: review?(hwine) → review+
Comment 9•10 years ago
|
||
Please indent code blocks like so: .. code-block:: shell $ command Generally 3 space indent is used in RST. For the testing against a specific hg, you don't need to hack up test-requirements.txt. Instead: $ ./run-mercurial-tests --with-hg /path/to/hg We should probably automated RPM building as part of the Jenkins job...
Comment 11•10 years ago
|
||
Comment on attachment 8533928 [details]
Mercurial upgrade instructions
I went ahead and pushed this (attributed to bkero) into v-c-t. I made some minor formatting changes to make things play nicer with Sphinx.
Feel free to make any subsequent changes directly in source control.
Attachment #8533928 -
Flags: review?(gps) → review+
Comment 12•10 years ago
|
||
ni :bkero - once comment 8 is addressed, this is ready to resolve fixed
Assignee: hwine → bkero
Flags: needinfo?(bkero)
Assignee | ||
Comment 13•10 years ago
|
||
/r/6703 - Bug 992368: docs: update server documentation Pull down this commit: hg pull -r 9140d9450ef4d25318b45495f18e6f516ff08960 https://reviewboard-hg.mozilla.org/version-control-tools/
Attachment #8589355 -
Flags: review?(hwine)
Attachment #8589355 -
Flags: review?(gps)
Comment 14•10 years ago
|
||
Comment on attachment 8589355 [details] MozReview Request: bz://992368/bkero https://reviewboard.mozilla.org/r/6701/#review5643 lgtm - thanks!
Attachment #8589355 -
Flags: review?(hwine) → review+
Updated•10 years ago
|
Attachment #8589355 -
Flags: review?(gps) → review+
Comment 15•10 years ago
|
||
Comment on attachment 8589355 [details] MozReview Request: bz://992368/bkero https://reviewboard.mozilla.org/r/6701/#review5645 Ship It!
Assignee | ||
Comment 16•10 years ago
|
||
Merged in http://hg.mozilla.org/hgcustom/version-control-tools/rev/b1d9e9814cde
Flags: needinfo?(bkero)
Assignee | ||
Updated•10 years ago
|
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 17•9 years ago
|
||
Attachment #8589355 -
Attachment is obsolete: true
Attachment #8618115 -
Flags: review+
Assignee | ||
Comment 18•9 years ago
|
||
You need to log in
before you can comment on or make changes to this bug.
Description
•