RDF update manifests have been emitting deprecation/removal warnings for a couple of releases now, and it's getting to be about time to remove them. We should add telemetry to try to get an idea of how often they're still used first, though. If they still have a lot of usage, we may need to do some outreach before removing support.

Comment on attachment 8974589 [details] Bug 1460336: Add telmetry for update RDF usage. data-r=chutten For data review: 1. What questions will you answer with this data? We are intending to remove support for the RDF update manifest format. Before we do so, we need to establish how much usage it currently has for updates served outside of AMO, and ideally get a list of extension IDs which are still using that format so their authors can be contacted. 2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Removing support for the RDF update manifest format while it still has significant usage may harm users who may lose access to important updates. We want to make sure usage of the old format is minimal before we remove support. 3. What alternative methods did you consider to answer these questions? Why were they not sufficient? There are not other plausible mechanisms to collect this data. 4. Can current instrumentation answer these questions? No. 5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki. Both measurements are category 1: - A histogram of the updates served with each format, RDF and JSON. This data will be collected on all channels. - A keyed histogram of extension IDs which serve updates using the RDF format. This data will be used to contact extensions authors who still serve a large number of updates using the RDF format, if necessary. 6. How long will this data be collected? Until the 66 release cycle, or earlier if we get clear data that the RDF update format is clearly unused before then. 7. What populations will you measure? All populations for the type of updates served. Non-release for the extension IDs which serve RDF update manifests. 8. If this data collection is default on, what is the opt-out mechanism for users? Standard telemetry opt-out. 9. Please provide a general description of how you will analyze this data. The update type histogram will be used to give us a broad idea of the relative usage of RDF and JSON update manifests. The RDF update ID histogram will allow us to contact specific extension authors who are using RDF update manifests to give them a chance to migrate before the functionality is removed. 10. Where do you intend to share the results of your analysis? Nowhere outside of the standard telemetry dashboards. We may use the detailed analysis of RDF update pings to contact specific extension authors.

Comment on attachment 8974589 [details] Bug 1460336: Add telmetry for update RDF usage. data-r=chutten https://reviewboard.mozilla.org/r/242936/#review248870 With my Telemetry hat on, I have a few comments about the patch ::: toolkit/components/telemetry/Histograms.json (Diff revision 1) > "description": "The amount of time taken to update the accessibility tree (ms)", > "kind" : "exponential", > "high": 60000, > "n_buckets": 50 > }, > - "ADDON_CONTENT_POLICY_SHIM_BLOCKING_LOADING_MS": { The removal of old, expired addon probes is lovely. Can we split it to its own commit? ::: toolkit/components/telemetry/Histograms.json:13586 (Diff revision 1) > + "bug_numbers": [1460336], > + "expires_in_version": "66", > + "kind": "enumerated", > + "n_values": 3, > + "releaseChannelCollection": "opt-out", > + "description": "The type of update manifest served for a given update request. (1=JSON, 2=RDF)" Instead of enumerated, did you consider "categorical"? The tooling is nicer (you get axis labels on TMO, and auto-expansion if you need more labels later (up to 50)), and it is generally considered superior to enumerated histograms in most cases. ::: toolkit/components/telemetry/Histograms.json:13593 (Diff revision 1) > + "EXTENSION_RDF_UPDATES": { > + "record_in_processes": ["main"], > + "alert_emails": ["addons-dev-internal@mozilla.com"], > + "bug_numbers": [1460336], > + "expires_in_version": "66", > + "kind": "exponential", For recording counts of things keyed by strings we now recommend keyed uint scalars: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/scalars.html#keyed-scalars "exponential" histograms are designed to record many different numbers. In this case the code only records "1"s.

DATA COLLECTION REVIEW RESPONSE: Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? Yes. Standard Telemetry mechanisms apply. Is there a control mechanism that allows the user to turn the data collection on and off? Yes. Standard Telemetry mechanisms apply. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under? ** Category 1: Technical Data Is the data collection request for default-on or default-off? default-on Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)? Nothing new. It does include addon ids which are already collected within Firefox and are public on AMO. Is the data collection covered by the existing Firefox privacy notice? Yes. Does there need to be a check-in in the future to determine whether to renew the data? Yes. :kmag, please ensure a bug is filed to run an analysis on the results and determine if they should be extended or removed. ---- Result: datareview+

Comment on attachment 8974589 [details] Bug 1460336: Add telmetry for update RDF usage. data-r=chutten https://reviewboard.mozilla.org/r/242936/#review248936 The implementation itself it pretty simple. For helping us understand how widely used rdf updates are though, a couple of things that might be helpful are: - keeping track of how many update requests were sent to AMO versus to some self-hosted update URL (I'm assuming that the vast majority of update requests go to AMO and hence are now JSON, but what we'd really like to know is what fraction of self-hosted updates are affected) - keeping track of failed requests. If we just look at successful requests we don't get an accurate measure of what fraction of total update requests are affected

Comment on attachment 8974589 [details] Bug 1460336: Add telmetry for update RDF usage. data-r=chutten https://reviewboard.mozilla.org/r/242936/#review251834

https://hg.mozilla.org/integration/mozilla-inbound/rev/ecbe25557836cfd3e85a48d79f3d861e25d2527d Bug 1460336: Add telmetry for update RDF usage. r=aswan,chutten data-r=chutten

Is manual testing required on this bug? If Yes, please provide some STR and the proper webextension(if required), if No set the “qe-verify-“ flag.

I verified this myself.

So, apparently the 6 months of data we've collected here so far are completely useless, since the patch that switched AMO to serving JSON update manifests was quietly backed out for the sake of Seamonkey and never re-landed. Jorge, what's going on here? Why was that change reverted for the sake of a tier 3 platform without anyone on the Firefox side being notified?

As far as I can tell, AMO is serving JSON updates to all clients that support it. The initial PR was reverted and a Firefox-only PR replaced it (https://github.com/mozilla/addons-server/pull/8271). I checked for updates on my Nightly and this is the kind of request I'm seeing: https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=effin-birds@jorgev&version=1.0&maxAppVersion=*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=65.0a1&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=65.0a1&updateType=97&compatMode=normal mat, please confirm.

This is correct. In addition, we also removed RDF support entirely in September since Seamonkey support was moved to different site outside of AMO (https://github.com/mozilla/addons-server/pull/9380)

OK, it looks like I misread the histogram. We're still getting RDF updates for some extensions, but they're mostly JSON now. Thanks.