Closed Bug 1331697 Opened 8 years ago Closed 8 years ago

Mercurial extension to synchronize changesets between subdirectories in different repos

Categories

(Developer Services :: General, task)

Product:
Developer Services
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gps, Assigned: gps)

References

Details

Attachments

(2 files)

overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);
(deleted), text/x-review-board-request
glob
: review+
Details
vcssync: command for performing hg repo overlaying (bug 1331697);
(deleted), text/x-review-board-request
glob
: review+
Details
The goal of this bug is to write Mercurial commands that make it easy to manage a monorepo with projects in sub-directories by providing a mechanism to import and export changes within those sub-directories from and to repositories having only the history of this sub-directories. Think of it as a poor man's "narrow clone." The underlying goal is to support project/sub-directory-isolated development in Git. There are 2 scenarios we want to support: 1) Replay changesets from a source repository into a sub-directory of a destination repository 2) Replay changesets from a sub-directory of a source repository into a destination repository I have code for #1 mostly written. Just needs more tests and some polish. I may or may not get to #2 in this bug, as it isn't in the critical path for bug 1322769.
FWIW, I started to refactor the code to follow the model of `git subtree`, which is basically what I aim to eventually build. However, I realized after about an hour or so that it was going to be a significant time sink. I didn't want to delay the servo syncing any more. So for now we have a standalone `hg overlay` command as opposed to say `hg subtreepull`.
Comment on attachment 8828223 [details] overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); https://reviewboard.mozilla.org/r/105688/#review109532 this looks great as always. ::: hgext/overlay/__init__.py:45 (Diff revision 2) > + assert prefix.endswith('/') > + > + sourceman = sourcectx.manifest() > + destman = destctx.manifest() > + > + sourcefiles = set(sourceman.iterkeys()) do we care about python3 support here? iterkeys() won't work on py3. ::: hgext/overlay/__init__.py:269 (Diff revision 2) > + """Mirror a source repository into the .hg directory of another.""" > + u = util.url(url) > + if u.islocal(): > + raise error.Abort(_('source repo cannot be local')) > + > + path = store.encodefilename(url.replace('://', '_')).replace('/', '_') the protocol should probably be excluded from the path; switching protocols for the same repo shouldn't be treated as a different repo. ::: hgext/overlay/tests/test-overlay-basic.t:104 (Diff revision 2) > + new file mode 100644 > + --- /dev/null > + +++ b/foo > + @@ -0,0 +1,1 @@ > + +foo > + i'd like to see the test co tip and ensure the --into directory was created correctly.
Attachment #8828223 - Flags: review?(glob) → review-
Comment on attachment 8828223 [details] overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); https://reviewboard.mozilla.org/r/105688/#review109532 > do we care about python3 support here? > iterkeys() won't work on py3. Not yet since Mercurial itself doesn't work on Python 3. Although in this case, the manifest class implements `iterkeys()` IIRC so it isn't an issue :) > the protocol should probably be excluded from the path; switching protocols for the same repo shouldn't be treated as a different repo. Good idea.
Comment on attachment 8828223 [details] overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); https://reviewboard.mozilla.org/r/105688/#review109532 > i'd like to see the test co tip and ensure the --into directory was created correctly. This comment doesn't parse :/
Blocks: 1337173
No longer blocks: 1322769
Comment on attachment 8828223 [details] overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); https://reviewboard.mozilla.org/r/105688/#review109532 > This comment doesn't parse :/ sorry about that. how about.. i'd like to see the test execute "hg co tip", and then ensure the directory specified by --into was created with the correct name and location.
glob: I'm probably not going to finish the Ansible work tonight, but the idea is to invoke the CLI process for doing the overlay after every push to https://hg.mozilla.org/projects/converted-servo-linear and periodically (on a systemd timer) as a fallback. The result will then get pushed somewhere, likely ssh://hg.mozilla.org/integration/autoland until we have somewhere better. If trees are open, it should "just work" and commits to github.com/servo/servo will be linearized, converted to Mercurial, pushed to https://hg.mozilla.org/projects/converted-servo-linear then overlayed into the tip of a Firefox repo and pushed somewhere. The overlay operation should never fail if the code in this patch is working properly and we keep the servo/ directory of Firefox repos read-only outside of this automated process. The only thing that should fail is the push operation (due to e.g. tree closures). The current recourse for that is to wait for the timer to retry the overlay. I think that's good enough for a MVP. The code is definitely reusable outside the context of Servo (this was intentional). It should also be possible to fold the new code in mozvcssync/overlay.py into Autoland and/or the eventual Servo landing tool with minimal effort. I'll try to linger on IRC tonight in case you want to chat about things.
Comment on attachment 8828223 [details] overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); https://reviewboard.mozilla.org/r/105688/#review112260
Attachment #8828223 - Flags: review?(glob) → review+
Comment on attachment 8835234 [details] vcssync: command for performing hg repo overlaying (bug 1331697); https://reviewboard.mozilla.org/r/110922/#review112518
Attachment #8835234 - Flags: review?(glob) → review+
Pushed by gszorc@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/0ff7e64be514 overlay: command for overlaying a repo into a subdirectory of another ; r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/a355883e91b7 vcssync: command for performing hg repo overlaying ; r=glob
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: