There are some fields we'd like added to the derived parquet dataset: * channel - this was an omission when I initially created the data set. * sqlerror recording - this is a recent addition to the ping * device list - this is a recent addition to the ping. The change will be made in https://github.com/mozilla/telemetry-batch-view via a pull request, but I'm opening this but for our own tracking purposes.

FTR: * The original PR for the sync ping is https://github.com/mozilla/telemetry-batch-view/pull/114 - while it has been merged, it might help for background about what needs to be done. * Some doc tweaks are at https://github.com/mozilla/telemetry-batch-view/pull/137 - hopefully these will be merged into master soon

Even though this will make the bug significantly more effort, I think we should also do bug 1301389 (or at least the back-end bits) as part of this - the issue is that once we start recording "events", we will want to know what user (and maybe devices) were involved. Currently that will involve inspecting one of the "sync" elements - but that might not be accurate as there is a tiny risk the UID in the first/last ping isn't the same one involved in the event. As we later include other events, we might not even have a single "sync" record to look into. Designing this and implementing it in the back-end means we would then be capable of changing the ping format in the client without risk of losing pings.

So just to be clear, this doesn't seem to include getting the validator information in?

(In reply to Thom Chiovoloni [:tcsc] from comment #3) > So just to be clear, this doesn't seem to include getting the validator > information in? No - but we do need to do that, either in this bug or another. And to be clear, when I said "even though this will make the bug significantly more effort" I didn't necessarily mean this specific bug, more this "effort" - we should split and/or combine these various bits into as many or as few bugs makes sense.

It's not clear to me what we want to do in the case where those entries are not consistent across syncs... Or do you just mean holding off on that decision (which probably we'd want to do on the client anyway), and adding backend support for pings that happen to come in with those fields outside of the syncs array.

I'm thinking that only the ping format changes where the UID and DID is stored, but the format in parquet remains the same. Thus, SyncView should be able to handle the data being in either the existing or "new" formats. We can then land the client changes which emit pings in the new format. Does that make sense and answer the question?

Closing this since it's merged. I had been keeping it open since the data it outputs was not yet readable in ReDash, but that is now being tracked by bug 1329023.