I'd like to start collecting Test Pilot data using Telemetry's submitExternalPing() function. I haven't done this before so I might be missing info below. I'd like to use a type of 'testpilot'. I'm planning on enabling both the clientID and the environment. Within the payload I'm expecting something that looks like: > {"tests": > {"universal_search": > {"last_enabled": 1457462200, > "last_disabled": 1457461100, > "features": {} > }, > "page_shot": > {"last_enabled": 1457462200, > "last_disabled": null, > "features": {"featureone": "on"} > } > }, > "agent": "User Agent String", > "uid": 1000, > "version": 1 > } Is that example specific enough? Is there more data you need from me? I'm hoping to analyze the data from Redshift. Full metrics plan at https://github.com/mozilla/testpilot/blob/master/docs/README-METRICS.md

See also: bug 1255184

I ran the above through jsonschema.net and got this: { "$schema": "http://json-schema.org/draft-04/schema#", "id": "http://jsonschema.net", "type": "object", "properties": { "tests": { "id": "http://jsonschema.net/tests", "type": "object", "properties": { "addon_id": { "id": "http://jsonschema.net/tests/addon_id", "type": "object", "properties": { "last_enabled": { "id": "http://jsonschema.net/tests/addon_id/last_enabled", "type": "integer" }, "last_disabled": { "id": "http://jsonschema.net/tests/addon_id/last_disabled", "type": "null" }, "features": { "id": "http://jsonschema.net/tests/addon_id/features", "type": "object", "properties": { "featureone": { "id": "http://jsonschema.net/tests/addon_id/features/featureone", "type": "string" } } } }, "required": [ "last_enabled", "last_disabled", "features" ] } }, "required": [ "addon_id" ] }, "agent": { "id": "http://jsonschema.net/agent", "type": "string" }, "uid": { "id": "http://jsonschema.net/uid", "type": "integer" }, "version": { "id": "http://jsonschema.net/version", "type": "integer" } }, "required": [ "tests", "agent", "uid", "version" ] }

Client-side using submitExternalPing() sounds good. Beware that this (currently) means that you won't get pings send from users that opted out of Healthreport. Is that a problem? If yes, can Test Pilot notify about that or do we need to change this? Some other points: * We need to add documentation: * either in-tree, similar to e.g. the heartbeat ping: https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/docs/heartbeat-ping.rst * Or, if that lives completely in an add-on you could link an alternative documentation from here: https://wiki.mozilla.org/Telemetry/PingTypes * This will need data collection review: https://wiki.mozilla.org/Firefox/Data_Collection * I'm not sure if the JSON structure lends itself well to RedShift or other exports, rmiller or others would be better to comment on that.

(In reply to Georg Fritzsche [:gfritzsche] from comment #3) > Client-side using submitExternalPing() sounds good. > Beware that this (currently) means that you won't get pings send from users > that opted out of Healthreport. > Is that a problem? > If yes, can Test Pilot notify about that or do we need to change this? That's understood. We're under the impression that is a very small percentage and if we think it affects us in the future we would make it a requirement to opt-in to it to use Test Pilot. Not something we're worried about currently though. > > Some other points: > * We need to add documentation: > * either in-tree, similar to e.g. the heartbeat ping: > > https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/ > docs/heartbeat-ping.rst > * Or, if that lives completely in an add-on you could link an alternative > documentation from here: > https://wiki.mozilla.org/Telemetry/PingTypes Is that something I would write a patch for? I heard that was actively maintained so was skipping it. I don't have external docs, but I'd write them if that's helpful. > * This will need data collection review: > https://wiki.mozilla.org/Firefox/Data_Collection Thanks for the link. I'll a?bsmedberg here. Ben: we've been working with Marshall on the privacy team and Rebecca for data consulting. I'm happy to answer any questions you have. > * I'm not sure if the JSON structure lends itself well to RedShift or other > exports, rmiller or others would be better to comment on that. I've been working with Rob on this. In https://github.com/mozilla/testpilot/issues/479 it sounds like we're going to need a `testpilot` ping as well as `testpilottest` pings with a less defined payload. Can we do both in this bug or should I file a separate bug? Thanks!

We'll pack the sprint tomorrow, and try to make ingesting these telemetry pings a P1. We should file separate bugs for any tasks in addition to ingestion (documentation, etc.)

(In reply to Wil Clouser [:clouserw] from comment #4) > > Some other points: > > * We need to add documentation: > > * either in-tree, similar to e.g. the heartbeat ping: > > > > https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/ > > docs/heartbeat-ping.rst > > * Or, if that lives completely in an add-on you could link an alternative > > documentation from here: > > https://wiki.mozilla.org/Telemetry/PingTypes > > Is that something I would write a patch for? I heard that was actively > maintained so was skipping it. I don't have external docs, but I'd write > them if that's helpful. You'd write a patch for the in-tree docs or we add a link to the wiki - given that this is an addon, external docs seem fine, we can take that elsewhere.

Slightly tweaked version of the schema from Comment 2

(In reply to Georg Fritzsche [:gfritzsche] from comment #6) > (In reply to Wil Clouser [:clouserw] from comment #4) > > > Some other points: > > > * We need to add documentation: > > > * either in-tree, similar to e.g. the heartbeat ping: > > > > > > https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/ > > > docs/heartbeat-ping.rst > > > * Or, if that lives completely in an add-on you could link an alternative > > > documentation from here: > > > https://wiki.mozilla.org/Telemetry/PingTypes > > > > Is that something I would write a patch for? I heard that was actively > > maintained so was skipping it. I don't have external docs, but I'd write > > them if that's helpful. > > You'd write a patch for the in-tree docs or we add a link to the wiki - > given that this is an addon, external docs seem fine, we can take that > elsewhere. Ok. https://github.com/mozilla/testpilot/blob/master/docs/README-METRICS.md#data-collection-details is our planning document detailing the current schemas. Let's link to that?

I am reviewing this: https://github.com/mozilla/testpilot/blob/master/docs/README-METRICS.md#data-collection-details The following things need to be clarified: * What is the test pilot user ID? How is it generated and stored? Is it deleted when users uninstall test pilot? * I strongly recommend collecting the telemetry environment instead of the user agent. It is the standard environment we're going to use for all correlation analysis. Where will the specifics of the feature flags for each experiment be documented? I think it's important that those are documented in detail. I'm working with Rebecca and nchapman to have somebody who is more intimately involved with testpilot to serve as a data-steward peer for those requests so that I don't have to ask a bunch of silly questions and be higher latency for you. As for the "testpilottest" ping type, I don't know whether we should just have a separate custom ping type for each experiment: e.g. testpilot-foobar-action. The advantages of that are that it's easy to keep them separate and use a separate schema for each one. The disadvantages are that we'd need to add each one specifically and that is currently a manual process. I expect we'll be automating that in the future. If you are expecting the total incoming ping volume across all experiments to be low (less than 10k per day) then it really doesn't matter. If it's higher, then separating them out is more advisable. It's not specified in this doc: by default retention for these pings is 180 days. If you expect something different, we need to discuss that.

(In reply to Benjamin Smedberg [:bsmedberg] from comment #9) > I am reviewing this: > > https://github.com/mozilla/testpilot/blob/master/docs/README-METRICS.md#data- > collection-details Thanks Ben! > The following things need to be clarified: > > * What is the test pilot user ID? How is it generated and stored? Is it > deleted when users uninstall test pilot? It's an incrementing integer primary key in the database. It's not deleted when a user uninstalls test pilot. We're still discussing what the story for leaving Test Pilot is. Current thinking is to anonymize the data we have for the user in the db. I added a note at the very bottom of the document linked above. The Test Pilot user ID is *not* recorded with Telemetry, so we aren't able to identify exact users from telemetry data (on purpose). It is recorded with our regular server logs. > * I strongly recommend collecting the telemetry environment instead of the > user agent. It is the standard environment we're going to use for all > correlation analysis. I'm collecting that as well. Since we're doing experimental stuff having the user agent available seemed prudent in case it gave us any hints if we started getting strange feedback from people. Do you have concerns beyond correlation (which will be possible with the regular environment)? > Where will the specifics of the feature flags for each experiment be > documented? I think it's important that those are documented in detail. Each experiment will be documenting their own metrics and feature flags in a similar looking document as to what is above. For example, an incomplete and in-development one is at https://github.com/chuckharmston/universal-search/blob/18-metrics-plan/docs/metrics.md (but that experiment has no feature flags). > I'm working with Rebecca and nchapman to have somebody who is more > intimately involved with testpilot to serve as a data-steward peer for those > requests so that I don't have to ask a bunch of silly questions and be > higher latency for you. Thanks. Let me know if I can help. > As for the "testpilottest" ping type, I don't know whether we should just > have a separate custom ping type for each experiment: e.g. > testpilot-foobar-action. > > The advantages of that are that it's easy to keep them separate and use a > separate schema for each one. The disadvantages are that we'd need to add > each one specifically and that is currently a manual process. I expect we'll > be automating that in the future. If you are expecting the total incoming > ping volume across all experiments to be low (less than 10k per day) then it > really doesn't matter. If it's higher, then separating them out is more > advisable. That's a good number to know. Since we're first starting it's hard to predict final numbers but I suspect we'll have a mix - some will be under and some will be over that number. Is it a waste if we make separate ping types for traffic smaller than 10k/day? Is a mix advisable or just making things complicated? I like the idea of separate ping types, although if that's the final suggestion here I have 3 more of these bugs to file and get looked at, like, today. :) > It's not specified in this doc: by default retention for these pings is 180 > days. If you expect something different, we need to discuss that. This is good to know. I'll confirm with the group and also add a note to the main readme-metrics.md document. Thanks for all your advice, Ben!

(In reply to Benjamin Smedberg [:bsmedberg] from comment #9) > Where will the specifics of the feature flags for each experiment be > documented? I think it's important that those are documented in detail. Just to chime with some detail on this one: Still working out how feature flags will work, but currently they're a part of the data model for each individual experiment [1]. We could add a view that lists out all the defined features for an experiment - say, included in or linked from the experiment details page. Would that count as a form of self-documentation? FWIW, we also have a spot for describing what we plan to measure for each experiment. [1] https://github.com/mozilla/testpilot/blob/master/testpilot/experiments/models.py#L125

Re: the ping types, I don't think schema is a huge issue here. I've checked and there's no problem using a single ping type for all tests with a nested `testdata` JSON object that contains only the data for that particular test. I can't speak to how hard it would be to tease out one test's set of pings from those of any others, however, will inquire and report back. In any event, yes, there are plans to automate the ability to ingest new ping types, but we don't yet have a schedule for that, and the manual process is done by an already overloaded ops team, so I'd lean towards caution.

(In reply to Wil Clouser [:clouserw] from comment #10) > (In reply to Benjamin Smedberg [:bsmedberg] from comment #9) > > As for the "testpilottest" ping type, I don't know whether we should just > > have a separate custom ping type for each experiment: e.g. > > testpilot-foobar-action. > > > > The advantages of that are that it's easy to keep them separate and use a > > separate schema for each one. The disadvantages are that we'd need to add > > each one specifically and that is currently a manual process. I expect we'll > > be automating that in the future. If you are expecting the total incoming > > ping volume across all experiments to be low (less than 10k per day) then it > > really doesn't matter. If it's higher, then separating them out is more > > advisable. > > That's a good number to know. Since we're first starting it's hard to > predict final numbers but I suspect we'll have a mix - some will be under > and some will be over that number. Is it a waste if we make separate ping > types for traffic smaller than 10k/day? Is a mix advisable or just making > things complicated? I like the idea of separate ping types, although if > that's the final suggestion here I have 3 more of these bugs to file and get > looked at, like, today. :) I don't like the idea of a separate document type per experiment. I much prefer to have a doctype for a *class* of documents, with some other field to identify individual experiments. It's true that we should consider the volume of submissions, and think about splitting things up if the volume gets too high, but I would set the bar one or two orders of magnitude higher than 10k/day.

If that's the recommendation, I'm all for it. Especially since the automated set-up isn't available yet. Thanks.

Wil asked on IRC whether I could give a generic data review to proceed with ingesting this data. Having reviewed https://github.com/mozilla/testpilot/blob/master/docs/README-METRICS.md again and noting that the user ID is no longer mentioned, I think this is fine. We'll consider this the data doc of record. We should figure out a way to either document the various experiment-specific payloads directly here or cross-reference them so that people can see it.

I've submitted this PR to add the two testpilot doc types: https://github.com/mozilla-services/puppet-config/pull/1909

12:00 PM <@mreid> relud: ya, that sounds like it has some work for your team. I think you can consider "1255182" to be done.

The above PR has been merged and deployed (confirmed by :relud). Submission of 'testpilot' or 'testpilottest' documents will now be split out into their own category for easy retrieval via analysis.telemetry.mozilla.org. I filed bug 1264018 for capturing the attached JSON Schema, since that work doesn't block testpilot development or deployment.

:mreid Can I assume that data can be retrieved in a.t.m.o via get_pings and thus we can try some basic template reports for computing some of the key test pilot metrics?

Yes, testpilot data is now accessible via get_pings.