Closed Bug 1317525 Opened 8 years ago Closed 8 years ago

stop-gap VCS "syncing" service from mozilla-central and servo/servo to the incubator/stylo repo

Categories

(Developer Services :: Servo VCS Sync, defect, P1)

Product:
Developer Services
Component:
Servo VCS Sync
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cpeterson, Assigned: gps)

References

Details

Attachments

(1 file)

vcssync: synchronize upstream changes to stylo repository (bug 1317525)
(deleted), text/x-review-board-request
Details
The goals here are to: 1. merge upstream changes from mozilla-central and servo/servo into the incubator/stylo repository more frequently 2. and have better visibility into regressions via more frequent CI runs
This is on me.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Component: Build Config → General
Product: Core → Developer Services
Here's my proposal for the stop gap: * We establish a new head on the stylo repository to track "upstream" * The new head will be labeled with a bookmark. Tentatively named "upstream." If using cinnabar, that bookmark should get converted to a Git branch automatically. * When there is a push to mozilla-central, mozilla-central is merged into the "upstream" head/bookmark and the result is pushed, triggering CI. * Where there is a push to servo/servo@master, the contents of the servo/ directory in the "upstream" head/bookmark are replaced by the contents from servo/servo@master and committed and pushed, triggering CI. * Periodically, a Stylo developer can merge a suitable commit from the "upstream" head into the main development head. bholley: is this proposal satisfactory?
Flags: needinfo?(bobbyholley)
I think we'll need to regenerate bindings in there, but generally I think this sounds great! CCing the rest of the team. Anyone else have thoughts here?
Flags: needinfo?(bobbyholley)
Where exactly do we need to regenerate bindings? And how does the work to integrate rust-bindgen into servo upstream's build system change our needs to check bindings into the tree and regenerate them?
(In reply to Gregory Szorc [:gps] from comment #5) > Where exactly do we need to regenerate bindings? Any time Servo gets paired up with a different rev of m-c. So pretty much on every push to 'upstream'. > And how does the work to integrate rust-bindgen into servo upstream's build > system change our needs to check bindings into the tree and regenerate them? The bindings need into servo/servo until servo/servo CI is able to run builds that generate them automatically (so presumably post-autolander).
> check bindings into the tree and regenerate them? We need to check bindings into the tree as long as Servo CI doesn't do m-c builds (i.e. until the autolander magic happens) But when doing automatic CI on the m-c side we should use build-time bindgen.
(In reply to Manish Goregaokar [:manishearth] from comment #7) > But when doing automatic CI on the m-c side we should use build-time bindgen. That's a good point. If we land support-for build-time bindgen, we don't need any of the incubator machinery to regenerate bindings.
No longer blocks: 1294883
Depends on: 1170600
I've got everything in comment #3 working locally. That excludes bindgen foo for now. I need to work out where to run this from and what SSH key to use. I'll likely stand up a cheap EC2 instance in the devservices AWS account.
Depends on: 1318177
This sounds great, thanks Greg. It does seem like build-time bindgen will simplify things a fair bit. Without it, this syncing service would need to do a build with the new m-c/servo pair (at least until the "export" phase of the build is done, i.e. when headers have been generated in the objdir), run the bindgen script, check in its changes, then do a real build. As Manish says in comment 7, we still need to update the checked-in bindings in servo/servo whenever the main development head of incubator/stylo moves. Is there any way we can make that process more automated? Maybe we could we have the syncing service push a Servo branch to GitHub somewhere that includes the updated bindings, which the human could make a pull request out of when deciding to update the incubator/stylo development head?
What happens when the automated mozilla-central merge fails with conflicts? We have some local changes in the branch, like updated taskcluster files or test descriptions, and these do sometimes conflict. (Maybe we could move those changes to different files so they're less likely to conflict.)
(In reply to Cameron McCormack (:heycam) from comment #12) > What happens when the automated mozilla-central merge fails with conflicts? > We have some local changes in the branch, like updated taskcluster files or > test descriptions, and these do sometimes conflict. (Maybe we could move > those changes to different files so they're less likely to conflict.) That's the point of having multiple heads. The "upstream" head will be pristine and there should be 0 merge conflicts because there is no drift. However, there could be merge conflicts when merging "upstream" into the other head. Only a human can resolve those.
Of course, to have accurate CI results, the "upstream" head needs to be running servo/stylo automation. That means we'll want to get the automation changes to support this landed in mozilla-central. We'll likely want to limit those to running on commits that have a "servo" root directory so they don't waste automation resources running outside of stylo and stylo-try.
I have the syncing running on my local machine and configured to push to https://hg.mozilla.org/users/gszorc_mozilla.com/stylo in case anyone wants to have a look. I'll likely add some polish tomorrow and hopefully get this submitted for review by EOD.
(In reply to Gregory Szorc [:gps] from comment #14) > Of course, to have accurate CI results, the "upstream" head needs to be > running servo/stylo automation. That means we'll want to get the automation > changes to support this landed in mozilla-central. We'll likely want to > limit those to running on commits that have a "servo" root directory so they > don't waste automation resources running outside of stylo and stylo-try. I don't follow this last part, can you elaborate? For servo changes, we can cull automation for anything that doesn't touch ports/geckolib or components/style. But we'll need to run on every m-c change.
FWIW, build-time bindgen work is currently on-going in https://github.com/servo/servo/pull/14244. That requires mach to pass in MOZ_OBJDIR via environment variable, which I think should be trivial to add to gecko's build system.
(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #16) > (In reply to Gregory Szorc [:gps] from comment #14) > > Of course, to have accurate CI results, the "upstream" head needs to be > > running servo/stylo automation. That means we'll want to get the automation > > changes to support this landed in mozilla-central. We'll likely want to > > limit those to running on commits that have a "servo" root directory so they > > don't waste automation resources running outside of stylo and stylo-try. > > I don't follow this last part, can you elaborate? There is code currently unique to the stylo repository that hacks up CI configs to run the automation you want. Those hacks need to make their way to mozilla-central. But we likely want to change the code such that things are only running/special if running on the "stylo" or "stylo-try" repository. Since I'm a huge proponent of not keying behavior off repository names (the names don't mean anything: the content does), I think we should key that behavior off the presence (or lack thereof) of a "servo" directory in the revision being built.
Depends on: 1318200
Oh I see - sounds good. Getting local modifications out of our tree sounds great.
I think I have this running in production on a one-off EC2 instance [that only I can access]. But I need to wait for a push to mozilla-central and servo/servo to verify the daemon is working. In the interim, I manually ran some commands and pushed a bunch of updates to the "upstream" bookmark on the stylo repo. As part of that, I discovered that a change to mozilla-central broke stylo builds. Down the rabbit hole I go...
This is definitely running in production. I've seen several servo commits and a handful of mozilla-central merges go through. The automation on the "upstream" bookmark is busted. I'll need to continue iterating on that. (I've definitely spent more time on the Firefox automation side of this problem than the VCS side.)
Blocks: stylo-central
This is done. Commits to servo/servo are now automatically landing in https://hg.mozilla.org/integration/autoland. Latency looks to be ~20s under normal conditions. More info at https://mozilla-version-control-tools.readthedocs.io/en/latest/vcssync/servo.html and in an email I sent around to people most impacted by this.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: General → Servo VCS Sync
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: