The current approach is to keep the full "raw" stream (I will call this the "landfill" stream) in one location, and keep the cleaned and validated stream (I will call this the "published" stream) in another location. The published stream simply discards invalid messages (and keeps error counts for monitoring purposes). If you want to investigate errors, you currently need to go back to the landfill stream and find the message(s) you want. I propose that we add a third "errors" stream. This stream would include the messages that are currently discarded in the published stream. If there are any messages that we explicitly don't care about, these can continue to be discarded and not become part of the errors stream. Messages in the error stream will be in a similar form to the landfill stream, with the addition of a field for error type, and any other useful metadata we can glean before processing failed (uncompressed payload size, update channel, etc). The categorized error messages will be easier to investigate (and reprocess if necessary) than the landfill stream. We would also need to change the monitoring to look at the errors stream instead of counting failures (dropped messages) in the published stream.

> We would also need to change the monitoring to look at the errors stream > instead of counting failures (dropped messages) in the published stream. Actually in addition to, if we are still allowing discards