I was looking at the mailer logs and for some reason it tries to fetch bug details for an empty bug ID. eg: Running in production mode. Getting top bugs... Getting bug data from /bybug?tree=trunk&startday=2015-08-11&endday=2015-08-17. Getting JSON from http://localhost/orangefactor/api/bybug?tree=trunk&startday=2015-08-11&endday=2015-08-17. Getting bug details... Getting bug data from /bugdetails?bugid=. Getting JSON from http://localhost/orangefactor/api/bugdetails?bugid=. HTTPError 500 fetching http://localhost/orangefactor/api/bugdetails?bugid=: internal server error Traceback (most recent call last): File "woo_mailer.py", line 293, in <module> main() File "woo_mailer.py", line 209, in main bugs = tb.bug_details(map(lambda x: x[0], top)) File "woo_mailer.py", line 81, in bug_details jdata = self.get_bugdata('/bugdetails?bugid=%s' % ','.join(buglist)) File "woo_mailer.py", line 106, in get_bugdata jdata = self.get_json(url_path) File "woo_mailer.py", line 95, in get_json resp.raise_for_status() File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/requests/models.py", line 851, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error

We're now getting this in the logs: Run started at: Tue Sep 15 08:00:01 PDT 2015 Running in production mode. Getting top bugs... Traceback (most recent call last): File "woo_mailer.py", line 297, in <module> main() File "woo_mailer.py", line 210, in main top = tb.top_bugs()[:10] File "woo_mailer.py", line 71, in top_bugs jdata = self.get_bybug() File "woo_mailer.py", line 111, in get_bybug return self.get_bugdata('/bybug?tree=trunk&startday=%s&endday=%s' % (self.startday, self.endday)) File "woo_mailer.py", line 102, in get_bugdata print 'Getting bug data...' % url_path TypeError: not all arguments converted during string formatting Which is a regression from: https://hg.mozilla.org/automation/orangefactor/rev/d34b238ff750 Which I've fixed in: https://hg.mozilla.org/automation/orangefactor/rev/b582aa8a40e8

Ok so the reason we even made the request that resulted in an HTTP 500 in comment 0 was due to top_bugs() returning an empty list, which can only happen if in the response from '/bybug?tree=trunk&startday=%s&endday=%s' contained no dates in `jdata['oranges'].values()`. Without more to go on it's hard to tell why that happened. I'll add some handling so if we hit that again, we'll have more info in the logs. As for the 500, that's: Traceback (most recent call last): File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/web/application.py", line 237, in process return self.handle() File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/web/application.py", line 228, in handle return self._delegate(fn, self.fvars, args) File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/web/application.py", line 409, in _delegate return handle_class(cls) File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/web/application.py", line 385, in handle_class return tocall(*args) File "/home/Ed/src/orangefactor/server/handlers.py", line 193, in GET results = json.dumps(self._GET(params, body)) File "/home/Ed/src/orangefactor/server/handlers.py", line 431, in _GET data['bugs'] = bzcache.get_bugs(list(bugids)) File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/bzcache/bzcache.py", line 111, in get_bugs bzbugs = self._get_bugzilla_data(list(bugset)) File "/home/Ed/.virtualenvs/of/lib/python2.7/site-packages/bzcache/bzcache.py", line 80, in _get_bugzilla_data bugdict = json.loads(buginfo)['bugs'] KeyError: 'bugs' Which just needs a fix in handlers.py I believe we've hit this before but just haven't known (perhaps it's the elusive OrangeFactor = 0.0 issue), since until we switched to use the python requests library, we ignored 500s etc.

https://hg.mozilla.org/automation/orangefactor/rev/b58df1eb842b https://hg.mozilla.org/automation/orangefactor/rev/e54d3fc6809a