Georg Fritzsche [:gfritzsche] Assignee
	Description • 9 years ago

We landed some investigative Telemetry client probes to track client side misbehavior.

We need to follow up on some basic checks with them.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 1 • 9 years ago

* bug 1190302 - no data yet for TELEMETRY_SESSIONDATA_FAILED_*
* bug 1186955 - no data yet for TELEMETRY_PING_SIZE_EXCEEDED_*, TELEMETRY_DISCARDED_*_SIZE_MB
* bug 1186492 - oddly, no data yet for TELEMETRY_PING_EVICTED_FOR_SERVER_ERRORS, i expected to see some before bug 1186955 landed due to oversized pings

We do have data for bug 1187340, bug 1168835:
* pending ping load failures: http://bit.ly/1EambFu
* pending ping parse failures: http://bit.ly/1IPaytb
* pending ping sizes: http://bit.ly/1DwtUmi

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 2 • 9 years ago

(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> TELEMETRY_PING_EVICTED_FOR_SERVER_ERRORS, i expected to see some before bug
> 1186955 landed due to oversized pings

Local testing shows that the server correctly returns 4xx errors (411 for a 4mb payload) and the client handles this correctly, so we may just not have had any submissions yet or before bug 1186955.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 3 • 9 years ago

(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> We do have data for bug 1187340, bug 1168835:
> * pending ping load failures: http://bit.ly/1EambFu
> * pending ping parse failures: http://bit.ly/1IPaytb
> * pending ping sizes: http://bit.ly/1DwtUmi

There is some interesting data here:
* We do have a lot of parse failures, i'm not sure yet what to make of this.
  However, we can track this in analysis to see if that explains the missing pings.
* We have very few actual disk load failures, 2 of 3 submissions have very high load failure counts.
  This seems to point to us repeatedly trying to load from disk after failures from the ping send task.
  I think we have to prioritize bug 1189425 to avoid clients getting stuck.
* The pending ping size distribution shows that we have a long tail of pretty big pings.
  88.86% are <1MB, a further 10.56% are 1MB<=size<2MB. That means we will get a lot of pings evicted
  soon, as bug 1186955 introduces a 1MB ping size limit.
  We can't uplift that bug in this form and probably have to consider increasing that limit to 2MB temporarily.
  as bug 1186955 introduced a 1MB limit.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 4 • 9 years ago

(In reply to Georg Fritzsche [:gfritzsche] from comment #3)
> * The pending ping size distribution shows that we have a long tail of
> pretty big pings.
>   88.86% are <1MB, a further 10.56% are 1MB<=size<2MB. That means we will
> get a lot of pings evicted
>   soon, as bug 1186955 introduces a 1MB ping size limit.
>   We can't uplift that bug in this form and probably have to consider
> increasing that limit to 2MB temporarily.
>   as bug 1186955 introduced a 1MB limit.

Worth noting that this only measures the sizes of persisted pings on disk at startup, so this may be biased.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 5 • 9 years ago

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #3)
> > * The pending ping size distribution shows that we have a long tail of
> > pretty big pings.
> >   88.86% are <1MB, a further 10.56% are 1MB<=size<2MB. That means we will
> > get a lot of pings evicted
> >   soon, as bug 1186955 introduces a 1MB ping size limit.
> >   We can't uplift that bug in this form and probably have to consider
> > increasing that limit to 2MB temporarily.
> >   as bug 1186955 introduced a 1MB limit.
> 
> Worth noting that this only measures the sizes of persisted pings on disk at
> startup, so this may be biased.

This is actually the size of the whole pending ping directory, so not immediately indicating brokenness.

The individual ping size is in TELEMETRY_DISCARDED_*_SIZE_MB and those have no data yet.

	Mark Reid [:mreid]
	Comment 6 • 9 years ago

(In reply to Georg Fritzsche [:gfritzsche] from comment #5)
> The individual ping size is in TELEMETRY_DISCARDED_*_SIZE_MB and those have
> no data yet.

Is this measuring the compressed size (as stored on disk) or the raw uncompressed payload size?

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 7 • 9 years ago

(In reply to Mark Reid [:mreid] from comment #6)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #5)
> > The individual ping size is in TELEMETRY_DISCARDED_*_SIZE_MB and those have
> > no data yet.
> 
> Is this measuring the compressed size (as stored on disk) or the raw
> uncompressed payload size?

Depends:
* TELEMETRY_DISCARDED_PENDING_PINGS_SIZE_MB - pending ping on-disk size
* TELEMETRY_DISCARDED_ARCHIVED_PINGS_SIZE_MB - ditto for archived
* TELEMETRY_DISCARDED_SEND_PINGS_SIZE_MB - ping size after serializing, before compression & sending to the server

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 8 • 9 years ago

They measurements here seem ok except pending ping parse failures. That issue is tracked in bug 1201045.