Closed
Bug 1137747
Opened 10 years ago
Closed 10 years ago
Keep an "errors only" stream for telemetry data.
Categories
(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)
Cloud Services Graveyard
Metrics: Pipeline
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mreid, Assigned: trink)
References
Details
The current approach is to keep the full "raw" stream (I will call this the "landfill" stream) in one location, and keep the cleaned and validated stream (I will call this the "published" stream) in another location.
The published stream simply discards invalid messages (and keeps error counts for monitoring purposes).
If you want to investigate errors, you currently need to go back to the landfill stream and find the message(s) you want.
I propose that we add a third "errors" stream. This stream would include the messages that are currently discarded in the published stream. If there are any messages that we explicitly don't care about, these can continue to be discarded and not become part of the errors stream.
Messages in the error stream will be in a similar form to the landfill stream, with the addition of a field for error type, and any other useful metadata we can glean before processing failed (uncompressed payload size, update channel, etc).
The categorized error messages will be easier to investigate (and reprocess if necessary) than the landfill stream.
We would also need to change the monitoring to look at the errors stream instead of counting failures (dropped messages) in the published stream.
Assignee | ||
Comment 1•10 years ago
|
||
> We would also need to change the monitoring to look at the errors stream
> instead of counting failures (dropped messages) in the published stream.
Actually in addition to, if we are still allowing discards
Reporter | ||
Updated•10 years ago
|
Assignee: nobody → mreid
OS: Mac OS X → All
Priority: -- → P1
Hardware: x86 → All
Reporter | ||
Updated•10 years ago
|
Assignee: mreid → nobody
Updated•10 years ago
|
Priority: P1 → P2
Assignee | ||
Updated•10 years ago
|
Assignee: nobody → mtrinkala
Status: NEW → ASSIGNED
Assignee | ||
Comment 2•10 years ago
|
||
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Product: Cloud Services → Cloud Services Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•