OF switched in bug 1026557, now time to do bzcache: https://github.com/jonallengriffin/bzcache/blob/master/bzcache/config.py#L2

The PR includes two commits: 1) Remove a straggler from bug 869652's conversion to using a central config.py rather than hardcoded values. 2) Switch to the native bzapi compatibility layer. I'm happy with the deployment of this on brasstacks - however I guess we'll also need to release a new version on pypi? (for local orangefactor development etc)

Comment on attachment 8450879 [details] Switch to the native bzapi compatibility layer & remove hard-coded URL Merged as https://github.com/jonallengriffin/bzcache/commit/e684dbde9c148632a9a446f5b676927891546f1b

(In reply to Ed Morley [:edmorley UTC+0] from comment #1) > I'm happy with the deployment of this on brasstacks - however I guess we'll > also need to release a new version on pypi? (for local orangefactor > development etc) It looks like we've never published bzcache on PyPI before, but I just did so with a new version number: https://pypi.python.org/pypi/bzcache/0.1.1

[webtools@brasstacks1.dmz.scl3 ~]$ cd ~/apps/orangefactor/src/bzcache/ [webtools@brasstacks1.dmz.scl3 bzcache]$ git pull -v remote: Counting objects: 12, done. remote: Compressing objects: 100% (12/12), done. remote: Total 12 (delta 1), reused 0 (delta 0) Unpacking objects: 100% (12/12), done. From https://github.com/jonallengriffin/bzcache 672c864..553bbf5 master -> origin/master Updating 672c864..553bbf5 Fast-forward bzcache/bzload.py | 3 ++- bzcache/config.py | 2 +- setup.py | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) [webtools@brasstacks1.dmz.scl3 bzcache]$ python setup.py build <snip> [webtools@brasstacks1.dmz.scl3 bzcache]$ python setup.py bdist <snip> [webtools@brasstacks1.dmz.scl3 bzcache]$ crontab -l | grep bzcache 0 0,4,8,12,16,18,20 * * * /home/webtools/apps/orangefactor/bin/python /home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py [webtools@brasstacks1.dmz.scl3 bzcache]$ cat /home/webtools/apps/orangefactor/src/bzcache/bzcache/config.py DEFAULT_ES_SERVER = 'elasticsearch-zlb.webapp.scl3.mozilla.com:9200' DEFAULT_BZAPI_SERVER = 'https://bugzilla.mozilla.org/bzapi/' DEFAULT_OF_SERVER = 'http://brasstacks.mozilla.com/orangefactor/api/'

However, when testing this bz_cache_refresh.py seems to timeout - which if I kill early, gives: (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py ♥Traceback (most recent call last): File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 56, in <module> main(options) File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 23, in main bzcache.index_bugs_by_keyword('intermittent-failure') File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bzcache.py", line 54, in index_bugs_by_keyword jsonurl = urllib.urlopen(apiURL) File "/usr/lib64/python2.6/urllib.py", line 86, in urlopen return opener.open(url) File "/usr/lib64/python2.6/urllib.py", line 207, in open return getattr(self, name)(url) File "/usr/lib64/python2.6/urllib.py", line 444, in open_https errcode, errmsg, headers = h.getreply() File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply response = self._conn.getresponse() File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 391, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status line = self.fp.readline() File "/usr/lib64/python2.6/socket.py", line 433, in readline data = recv(1) File "/usr/lib64/python2.6/ssl.py", line 215, in recv return self.read(buflen) File "/usr/lib64/python2.6/ssl.py", line 136, in read return self._sslobj.read(len) KeyboardInterrupt Or if I leave it for a while: (orangefactor)[webtools@brasstacks1.dmz.scl3 ~]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py Traceback (most recent call last): File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 56, in <module> main(options) File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 23, in main bzcache.index_bugs_by_keyword('intermittent-failure') File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bzcache.py", line 63, in index_bugs_by_keyword bug['whiteboard'], KeyError: 'whiteboard' Switching back to the old bzapi endpoint works: (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ git checkout 09c7ee6687a923241fd4fd6a9590224cbef349c3 <snip> HEAD is now at 09c7ee6... Bug 1034524 - bzload.py should use the config.py bzapi URL rather than hard-coding; r=jgriffin (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py 2014-07-08, 06:05:23 - 989240 added, status: UNCONFIRMED, id: 989240 2014-07-08, 06:05:23 - 961107 added, status: NEW, id: 961107 2014-07-08, 06:05:23 - 928677 added, status: NEW, id: 928677 2014-07-08, 06:05:23 - 984274 added, status: NEW, id: 984274 ... We're now using something like: https://bugzilla.mozilla.org/bzapi/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09 Compared to the previous: https://api-dev.bugzilla.mozilla.org/latest/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09 It appears that the new bzapi compatibility layer: a) Takes 76s instead of 6s to return the same results b) Only returns the 'whiteboard' field if it is non-empty, which doesn't match bzapi behaviour. I'll file deps for these, and for now roll back bzcache on brasstacks to 09c7ee6687a923241fd4fd6a9590224cbef349c3 so the cron can continue to work.

(In reply to Ed Morley [:edmorley UTC+0] from comment #5) > It appears that the new bzapi compatibility layer: > a) Takes 76s instead of 6s to return the same results Bug 1035804 > b) Only returns the 'whiteboard' field if it is non-empty, which doesn't > match bzapi behaviour. Bug 1035799

Given that bug 1035804 is apparently going to be difficult to solve, I recommend switching that query over to the native REST API if it's not too difficult.

If switching from bzapi to native rest, we'd need to change... Config.py: https://github.com/jonallengriffin/bzcache/blob/master/bzcache/config.py#L2 Then the queries that would need changing: https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzcache.py#L50 def index_bugs_by_keyword(self, keyword): # only look at bugs that have been updated in the last 6 months ago = datetime.date.today() - datetime.timedelta(days=180) apiURL = self.bzapi_server + "bug?keywords=%s&include_fields=id,summary,status,whiteboard&changed_after=%s" % (keyword, ago.strftime('%Y-%m-%d')) jsonurl = urllib.urlopen(apiURL) buginfo = jsonurl.read() https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzcache.py#L66 def _get_bugzilla_data(self, bugid_array): buginfo = {} retVal = {} apiURL = (self.bzapi_server + "bug?id=" + ','.join(bugid_array) + "&include_fields=id,summary,status,whiteboard") jsonurl = urllib.urlopen(apiURL) buginfo = jsonurl.read() https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzload.py def _load_component(self, product, component, status): status.insert(0, '') now = datetime.datetime.now() - datetime.timedelta(days=180) url = '%sbug?product=%s&component=%s%s&include_fields=id,summary,status&changed_after=%s' % \ (self.bzurl, product, urllib.quote(component), '&status='.join(status), now.strftime('%Y-%m-%d')) data = self._load_json_url(url)

Well, we *could* mix the two to limit the number of changes, either through a second config variable, or changing the existing one to just be the URL to bmo (to which we would internally append 'bzapi' or 'rest'). Not sure that's worth it, though, since there aren't many queries to change to switch fully over to native REST.

With bug 1035799 fixed, the Bugzilla query works, it just takes ~80s instead of ~7s. However, the actual insertion into ES still takes much longer than either of these (several minutes), so it really doesn't impact the overall runtime as much as one would expect. Seeing as the script doesn't timeout, and bug 1035804 is still looking a long ways out, I've decided to land this anyway. [webtools@brasstacks1.dmz.scl3 ~]$ cd ~/apps/orangefactor/src/bzcache/ [webtools@brasstacks1.dmz.scl3 bzcache]$ source /home/webtools/apps/orangefactor/bin/activate (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ git checkout master Previous HEAD position was 09c7ee6... Bug 1034524 - bzload.py should use the config.py bzapi URL rather than hard-coding; r=jgriffin Switched to branch 'master' (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ git pull Already up-to-date. (orangefactor)[webtools@brasstacks1.dmz.scl3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py No handlers could be found for logger "pyes.urllib3.connectionpool" 2014-09-16, 01:49:50 - 453969 added, status: NEW, id: 453969 2014-09-16, 01:49:50 - 482073 added, status: RESOLVED, id: 482073 2014-09-16, 01:49:50 - 492821 added, status: RESOLVED, id: 492821 2014-09-16, 01:49:50 - 493692 added, status: NEW, id: 493692 2014-09-16, 01:49:50 - 497884 added, status: RESOLVED, id: 497884 2014-09-16, 01:49:50 - 501383 added, status: RESOLVED, id: 501383 <snip> 2014-09-16, 01:53:29 - 1067555 added, status: NEW, id: 1067555 2014-09-16, 01:53:29 - 1067559 added, status: NEW, id: 1067559 2014-09-16, 01:53:29 - 1067568 added, status: NEW, id: 1067568 2014-09-16, 01:53:29 - 1067569 added, status: NEW, id: 1067569 2014-09-16, 01:53:29 - 1067727 added, status: NEW, id: 1067727 2014-09-16, 01:53:29 - 1067762 added, status: NEW, id: 1067762 2014-09-16 01:53:29.445910 http://brasstacks.mozilla.com/orangefactor/api/bybug?startday=2014-07-18&endday=2014-09-16&tree=All