Closed Bug 1388025 Opened 7 years ago Closed 6 years ago

Get rid of crash_aggregates in favor of error_aggregates

Categories

(Data Platform and Tools :: General, task, P2)

Product:
Data Platform and Tools
Component:
General
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mdoglio, Assigned: wlach)

References

Details

(Whiteboard: [DataPlatform])

Attachments

(1 file)

Link to GitHub pull-request: https://github.com/mozilla/telemetry-batch-view/pull/530
(deleted), text/x-github-pull-request
Details
The Mission Control dataset (error_aggregates) schema is a superset of what crash_aggregates offers. Also, its latency is <1h vs up to 24h. We should get rid of crash_aggregates and save both machine and human time.
Is that (error_aggregates) coming from main pings, or is it using crash pings? I'm wondering if we should focus on using error_aggregates for everything dealing with crashes or if we should be creating a new crash_aggregates based on crash pings.
It's using both, like crash_aggregates does. I like the idea of having a crash ping-only aggregates dataset. I would be happy to work on that if it becomes a priority.
With 55 in release, we finally have the full capability to correlate crash reports with crash pings (if applicable). So if we could get this dataset up and running, we could start looking at ping-only data on all of release, even segmented by process type. I don't know who would determine the priority of this, but it would be very useful to get dashboards hooked up to this data as we lead up to 57.
(In reply to Mauro Doglio [:mdoglio] from comment #2) > It's using both, like crash_aggregates does. I like the idea of having a > crash ping-only aggregates dataset. I would be happy to work on that if it > becomes a priority. We could use direct-to-parquet for this dataset. It doesn't require parsing the entire payload, you can have it output a subset of fields to parquet, which in this case would probably just be that dimensional information (version, app, build, etc.).
It sounds like we can: 1- get rid of crash_aggregates 2- use crash_summary (which is already in parquet) to derive new datasets (like the one for crash reports correlations that :ddurst is talking about).
Depends on: 1382236
Priority: -- → P3
Moved, per https://bugzilla.mozilla.org/show_bug.cgi?id=1453996
Component: Datasets: Crash Aggregates → Datasets: General
Assignee: nobody → akomarzewski
Priority: P3 → P2
Whiteboard: [DataPlatform]

I'll take this.

Assignee: akomarzewski → wlachance

First step is going through redash to find queries still scheduled against this dataset. There were quite a few, but most were obviously out of date / not relevant any more. Except for some that were scheduled/run by hkirschner@mozilla.com and rtestard@mozilla.com, cdenizet@mozilla.com.

Romain, Harald, Calixte: can you let me know if it's ok to stop scheduling runs of these queries? If the data is still useful, can I help you migrate to error_aggregates?

https://docs.telemetry.mozilla.org/datasets/streaming/error_aggregates/reference.html

The schema is fairly similar so it shouldn't take long to update your queries if needed.

rtestard@mozilla.com:

Release_OK 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/14107/source
ESR 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/51957/source
Release 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/4655/source

hkirschner@mozilla.com:

Firefox Beta Crash Rates By Version/Build, v2 https://sql.telemetry.mozilla.org/queries/2856/source
Usage Hour ratio e10s/non-e10s https://sql.telemetry.mozilla.org/queries/972/source
Release Crash Rate by Activity Date https://sql.telemetry.mozilla.org/queries/331/source
Firefox Aurora Crash Rates By Version/Build https://sql.telemetry.mozilla.org/queries/999/source
Firefox Crash Rates by Release/Date https://sql.telemetry.mozilla.org/queries/184/source
Firefox Watershed Beta 44.0b1 https://sql.telemetry.mozilla.org/queries/469/source
[deprecated] Firefox Beta Crash Rates By Version/Build, v1 https://sql.telemetry.mozilla.org/queries/207/source
Beta Usage Hour ratio e10s/non-e10s https://sql.telemetry.mozilla.org/queries/1415/source

cdenizet@mozilla.com:

Daily usage khours for Firefox release, beta, aurora and nightly: https://sql.telemetry.mozilla.org/queries/346/source

Flags: needinfo?(rtestard)
Flags: needinfo?(hkirschner)
Flags: needinfo?(cdenizet)

OK for my queries, the ones mentioned above won't run anymore.

Flags: needinfo?(rtestard)

I just killed mine.

Flags: needinfo?(cdenizet)

OK for mine, health.graphics doesn't use them anymore.

Flags: needinfo?(hkirschner)

Thanks all! Harald: I took the liberty of unscheduling your queries for you, since you weren't using them anymore. :)

I will go ahead with the subsequent steps in the deprecation process.

Type: enhancement → task
Depends on: 1541148

This dataset has been unscheduled from airflow and removed from hive metastore, next step per our deprecation policy is to wait a week and make sure nothing breaks.

All done!

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Datasets: General → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: