While testing patches for bug 1353509, I ran into an exception which can only be caused during a retrigger after a test manifest change has removed a test. 2017-05-23 11:20:56,839 22376 autophone PulseMonitorThread autophone ERROR AutophonePulseMonitor Exception Traceback (most recent call last): File "/mozilla/projects/autophone/src/bclary-autophone/autophonepulsemonitor.py", line 219, in listen connection.drain_events(timeout=self.timeout) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events return self.transport.drain_events(self.connection, **kwargs) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events return connection.drain_events(**kwargs) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/connection.py", line 464, in drain_events return self.blocking_read(timeout) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/connection.py", line 469, in blocking_read return self.on_inbound_frame(frame) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/method_framing.py", line 88, in on_frame callback(channel, msg.frame_method, msg.frame_args, msg) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/connection.py", line 473, in on_inbound_method method_sig, payload, content, File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method listener(*args) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/amqp/channel.py", line 1613, in _on_basic_deliver fun(msg) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/kombu/messaging.py", line 623, in _receive_callback return on_m(message) if on_m else self.receive(decoded, message) File "/mozilla/projects/.virtualenvs/autophone/lib/python2.7/site-packages/kombu/messaging.py", line 589, in receive [callback(body, message) for callback in callbacks] File "/mozilla/projects/autophone/src/bclary-autophone/autophonepulsemonitor.py", line 272, in handle_message self.handle_jobaction(data, message) File "/mozilla/projects/autophone/src/bclary-autophone/autophonepulsemonitor.py", line 361, in handle_jobaction self.jobaction_callback(jobaction_data) File "autophone.py", line 1060, in on_jobaction json.dumps(job_action, sort_keys=True, indent=4)) File "/usr/lib64/python2.7/json/__init__.py", line 251, in dumps sort_keys=sort_keys, **kw).encode(obj) File "/usr/lib64/python2.7/json/encoder.py", line 209, in encode chunks = list(chunks) File "/usr/lib64/python2.7/json/encoder.py", line 434, in _iterencode for chunk in _iterencode_dict(o, _current_indent_level): File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict for chunk in chunks: File "/usr/lib64/python2.7/json/encoder.py", line 442, in _iterencode o = _default(o) File "/usr/lib64/python2.7/json/encoder.py", line 184, in default raise TypeError(repr(o) + " is not JSON serializable") TypeError: datetime.datetime(2017, 5, 19, 15, 38, 10, tzinfo=<UTC>) is not JSON serializable This occurs in autophone.py's on_jobaction when a test is not found. This is in a debug logging call so it is not necessary to treat it as json. It does abnormally terminate the AutophonePulseMonitor's listen method however it is not clear to me that it is the root cause of the hang in bug 1353509.

Comment on attachment 8870904 [details] [diff] [review] bug-1367199-job-action-logging-v1.patch Review of attachment 8870904 [details] [diff] [review]: ----------------------------------------------------------------- nice