Closed
Bug 592060
Opened 14 years ago
Closed 14 years ago
Create custom hgpoller for shadow-central that can poll https:// pushlog
Categories
(Release Engineering :: General, defect, P3)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: lsblakk, Assigned: catlee)
References
Details
(Whiteboard: [sg:nse])
Attachments
(2 files, 2 obsolete files)
(deleted),
patch
|
bhearsum
:
review+
nthomas
:
review+
catlee
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
nthomas
:
review+
catlee
:
checked-in+
|
Details | Diff | Splinter Review |
This bug is to track the following changes to shadow-central:
Take out the hgHost for shadow-central (which has been throwing errors): http://hg.mozilla.org/build/buildbot-configs/file/5aeb32b3f87e/mozilla/config.py#l729
Create an hg commit hook that will do a sendchange to production-master01:9009 when a new change is submitted to the private repo.
When this is done we should be able to (still) do force builds on shadow-central repo as well as see builds triggered by shadow-central checkins
Comment 1•14 years ago
|
||
Is the one catlee is working on (or at least thinking about), or is that an alternate approach tracked in a different bug?
Reporter | ||
Updated•14 years ago
|
Summary: Take out shadow-central hgpoller and create an hg commit hook that does sendchange → Create custom hgpoller for shadow-central that can poll https:// pushlog
Reporter | ||
Updated•14 years ago
|
Assignee: lsblakk → catlee
Assignee | ||
Updated•14 years ago
|
Priority: -- → P3
Comment 2•14 years ago
|
||
I can't see "depends on" bug 598681 -- is that an IT request to deploy this? If so, how is it coming?
Reporter | ||
Comment 3•14 years ago
|
||
(In reply to comment #2)
> I can't see "depends on" bug 598681 -- is that an IT request to deploy this? If
> so, how is it coming?
Catlee can only get so far in staging trying to test an https custom poller, he needs an account with credentials so we can poll https://hgpvt.m.o
repositories to see if there are new changes that require building and test that we're actually getting the builds to trigger properly.
Reporter | ||
Comment 4•14 years ago
|
||
(In reply to comment #1)
> Is the one catlee is working on (or at least thinking about), or is that an
> alternate approach tracked in a different bug?
And yes, it's an IT request to get that LDAP account set up.
Assignee | ||
Comment 5•14 years ago
|
||
Attachment #482964 -
Flags: feedback?(nrthomas)
Attachment #482964 -
Flags: feedback?
Assignee | ||
Updated•14 years ago
|
Attachment #482964 -
Flags: feedback? → feedback?(bhearsum)
Comment 6•14 years ago
|
||
Comment on attachment 482964 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
>diff --git a/bin/hgpoller.py b/bin/hgpoller.py
>+ if branch_state['last_run'] + interval > time.time():
>+ log.debug("Skipping %s, too soon since last run", branch)
>+ return
I think this is a comparison between a smallish integer (300) and an epoch timestamp. Otherwise looks good.
Attachment #482964 -
Flags: feedback?(nrthomas) → feedback+
Comment 7•14 years ago
|
||
Comment on attachment 482964 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
>+def sendchange(master, branch, change):
>+ log.info("Sendchange %s to %s on branch %s", change['changeset'], master, branch)
>+ cmd = ['buildbot', 'sendchange',
>+ '--master', master,
>+ '--branch', branch,
>+ '--comments', change['comments'],
>+ '--revision', change['changeset'],
>+ '--user', change['author'],
>+ '--when', str(change['updated']),
>+ ]
>+ cmd.extend(change['files'])
>+ subprocess.check_call(cmd)
Might be better to run this through retry_command for better chance of success.
Just want to make sure I understand how this will be run, too. It looks like we'll be providing an initial config file with basic information, branch information, etc. The poller is then going to use this file to store state? And is it running through cron?
Attachment #482964 -
Flags: feedback?(bhearsum) → feedback+
Assignee | ||
Comment 8•14 years ago
|
||
(In reply to comment #6)
> Comment on attachment 482964 [details] [diff] [review]
> external buildbot poller that supports checking certs for https urls
>
> >diff --git a/bin/hgpoller.py b/bin/hgpoller.py
> >+ if branch_state['last_run'] + interval > time.time():
> >+ log.debug("Skipping %s, too soon since last run", branch)
> >+ return
>
> I think this is a comparison between a smallish integer (300) and an epoch
> timestamp. Otherwise looks good.
Not sure what you mean here...That the operator precedence is wrong?
Assignee | ||
Comment 9•14 years ago
|
||
(In reply to comment #7)
> Comment on attachment 482964 [details] [diff] [review]
> external buildbot poller that supports checking certs for https urls
>
> >+def sendchange(master, branch, change):
> >+ log.info("Sendchange %s to %s on branch %s", change['changeset'], master, branch)
> >+ cmd = ['buildbot', 'sendchange',
> >+ '--master', master,
> >+ '--branch', branch,
> >+ '--comments', change['comments'],
> >+ '--revision', change['changeset'],
> >+ '--user', change['author'],
> >+ '--when', str(change['updated']),
> >+ ]
> >+ cmd.extend(change['files'])
> >+ subprocess.check_call(cmd)
>
> Might be better to run this through retry_command for better chance of success
Good idea
> Just want to make sure I understand how this will be run, too. It looks like
> we'll be providing an initial config file with basic information, branch
> information, etc. The poller is then going to use this file to store state? And
> is it running through cron?
The state is stored in an auxiliary file, specified by the 'state_file' option in the config (defaults to 'state.json'). This file gets updated by the script, and the original config file is untouched. The state file stores the times we last polled for data, as well as the last known changeset for the repo.
Assignee | ||
Comment 10•14 years ago
|
||
And yes, running through cron for now. We could set something up in buildbot to run this command too. SubprocessPoller?
Comment 11•14 years ago
|
||
(In reply to comment #10)
> And yes, running through cron for now. We could set something up in buildbot
> to run this command too. SubprocessPoller?
I'm not really picky, was just wondering.
Comment 12•14 years ago
|
||
(In reply to comment #6)
> >+ if branch_state['last_run'] + interval > time.time():
> I think this is a comparison between a smallish integer (300) and an epoch
> timestamp. Otherwise looks good.
Gah, I just parsed it wrong in my head. Maybe it's easier to read if written as
if time.time() > branch_state['last_run'] + interval
though.
Assignee | ||
Comment 13•14 years ago
|
||
Addresses your comments.
'retry.py' needs to be in $PATH somewhere. I was thinking of symlinking it from ~/bin to the tools checkout the masters have?
Attachment #482964 -
Attachment is obsolete: true
Attachment #485303 -
Flags: review?(nrthomas)
Attachment #485303 -
Flags: review?(bhearsum)
Comment 14•14 years ago
|
||
Comment on attachment 485303 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
>+ cmd = ['retry.py', '-r', '5', '-s', '5', '-t', '1800',
>+ '--stdout-regexp', 'change sent successfully']
...
>+ subprocess.check_call(cmd)
This script looks OK for polling shadow-central, but do you think this blocking call will be OK if we end up polling multiple repositories ? I'm concerned that a 30 minute timeout on each sendchange attempt will delay changes from other branches. Perhaps it's better to fail quickly and not update the tip in the state, then try again next time the poller runs.
Updated•14 years ago
|
Attachment #485303 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 15•14 years ago
|
||
(In reply to comment #14)
> Comment on attachment 485303 [details] [diff] [review]
> external buildbot poller that supports checking certs for https urls
>
> >+ cmd = ['retry.py', '-r', '5', '-s', '5', '-t', '1800',
> >+ '--stdout-regexp', 'change sent successfully']
> ...
> >+ subprocess.check_call(cmd)
>
> This script looks OK for polling shadow-central, but do you think this blocking
> call will be OK if we end up polling multiple repositories ? I'm concerned that
> a 30 minute timeout on each sendchange attempt will delay changes from other
> branches. Perhaps it's better to fail quickly and not update the tip in the
> state, then try again next time the poller runs.
Sure, I can change it to what...30 seconds instead?
Assignee | ||
Comment 16•14 years ago
|
||
Differences from previous:
* change sendchange timeout to 30s
* fix unicode encoding problems on comments and authors
* change log levels so it's less noisy by default
* add branch configuration parameter, default_branch_only, which defaults to true. If true, only pushes on the default branch will be used. If false, all pushes will be used. We could use this for the try branch.
* actually ignore off-default pushes when default_branch_only is set
Attachment #485303 -
Attachment is obsolete: true
Attachment #485756 -
Flags: review?(nrthomas)
Attachment #485756 -
Flags: review?(bhearsum)
Attachment #485303 -
Flags: review?(nrthomas)
Comment 17•14 years ago
|
||
Comment on attachment 485756 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
Thanks for the summary of the differences, looks good.
Attachment #485756 -
Flags: review?(bhearsum) → review+
Comment 18•14 years ago
|
||
Comment on attachment 485756 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
Looks good.
Attachment #485756 -
Flags: review?(nrthomas) → review+
Assignee | ||
Comment 19•14 years ago
|
||
Comment on attachment 485756 [details] [diff] [review]
external buildbot poller that supports checking certs for https urls
changeset: 1026:140e3c0e22a6
Attachment #485756 -
Flags: checked-in+
Assignee | ||
Comment 20•14 years ago
|
||
Set up on pm01 in ~/poller. The pushlog is empty currently, so haven't been able to verify.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 21•14 years ago
|
||
Attachment #486454 -
Flags: review?(nrthomas)
Updated•14 years ago
|
Attachment #486454 -
Flags: review?(nrthomas) → review+
Assignee | ||
Updated•14 years ago
|
Attachment #486454 -
Flags: checked-in+
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Updated•9 years ago
|
Group: core-security → core-security-release
Updated•9 years ago
|
Group: core-security-release
You need to log in
before you can comment on or make changes to this bug.
Description
•