The out of date system addon will be submitting telemetry data[1]. We should analyze this data to answer the following questions: 1. Do users continue to run Firefox 44 after having seen the notification bar? 2. Do users continue to run Firefox 44 after having clicked the button to download the latest version? 3. Do users get used to seeing the notification bar and don't click the button? By correlating telemetry IDs with main ping telemetry, it would also be great to see how many people (percentage or total number of users) end up on the most recent version of Firefox. [1] https://github.com/mozilla/outofdate-notifications-system-addon/blob/master/README.md

Chris, I was planning to start on this and I'm very happy to see you take this bug. Thanks!

Actually, something just came up, so if you want it you can have it :)

I was asking around about how to get this data from telemetry since is is a custom histogram. Is it possible from the telemetry dashboard and where is it located so I can query it directly? Since I will be working with telemetry anyways I don't mind taking this but that info would be a big help.

More than being just a custom histogram, it uses a custom ping type. That requires custom analysis via analysis.telemetry.mozilla.org, as there's nothing built to analyse or graph this type of ping. get_pings(doc_type="outofdate-notifications-system-addon", ...) ^ will get you the pings, but won't tell you the other things about the client that you need to answer the questions (well, not without joining to another batch of pings)... this could get tricky. If you aren't experienced with fiddling with get_pings on atmo, maybe it would be best to not take this on as your first rodeo.

Even though Chris and I *just* talked about this yesterday, I feel like other things are a greater priority now (for you, Chris, sorry). I'm sure Robert can handle it, after getting some pointers. (Probably good to spread some of this knowledge around, anyway, as well.) If there are real trouble spots, just call them out.

Thanks Chris and David. I've worked with get_pings before and though I'm sure it will be tricky I can deal.

Chris, the following isn't returning any data though the query without doc_type so it uses the default does have a count. pings = get_pings(sc, app="Firefox", channel="release", version="44.0", doc_type="outofdate-notifications-system-addon") ping.count() 0 I also tried the following with the same result pings = get_pings(sc, app="Firefox", channel="release", doc_type="outofdate-notifications-system-addon") ping.count() 0 pings = get_pings(sc, doc_type="outofdate-notifications-system-addon") ping.count() 0 Any ideas?

I think you need to specify a date range.

From what I can tell from the code[1], you don't need to specify anything and you'll just get the entire Telemetry dataset. Very unwieldy, but not empty. ...I have this faint memory of... you have to use doc_type="other" to get all "non-recognized" types, then you need to do a filter on the resultant RDD for the actual "type" field contained in the record. Let me double-check this on #telemetry. [1]: https://github.com/mozilla/python_moztelemetry/blob/master/moztelemetry/spark.py#L77

I was right. :mreid confirmed on #telemetry that you need to get_pings(doc_type="OTHER", ...) and then filter(lambda p: p["type"] == "outofdate-notifications-system-addon")

Interesting since bug 1292493 supposedly moved it from out of the other bucket. :(

I confirmed the above for hypothetical docs of type "foo-bar" :) "outofdate-notifications-system-addon" is one of the whitelisted types, so it should work with the specific type. You may need to replace "-" with "_", I will check the code.

I confirmed that replacing "-" with "_" worked. So, doc_type="outofdate_notifications_system_addon"

I filed bug 1306049 to avoid needing that piece of extra knowledge of system internals.

Just some summary data I was able to extract tonight. Still working on joining this data with the latest client ping to check if the client has upgraded, etc. Only Firefox 44.0 and records with a clientId Total 468024 Started 412 Shown 467976 Clicked 211672 Clicked | Client | Count | Count | --------+--------+ 0 | 256352 | 1 | 130950 | 2 | 39812 | 3 | 16197 | 4 | 7781 | 5 | 4404 | 6+ | 12528 |

Updated since I refreshed the data to add the last submissionDate. Only Firefox 44.0 and records with a clientId Total 473456 Started 418 Shown 473406 Clicked 214077 Not Clicked = 259379 Clicked = 214077 259379 + 214077 = 473456 which is Total The values in the Shown and Clicked columns are the number of unique client ID's that have reported the associated event the number of times in the Count column. Count | Shown | Clicked | -------+---------+---------+ 0 | 50 | 259379 | 1 | 119000 | 132423 | 2 | 66584 | 40194 | 3 | 43902 | 16443 | 4 | 31428 | 7858 | 5 | 23643 | 4461 | 6+ | 188849 | 12698 | 100+ | 9277 | 180 | 1000+ | 97 | 1 | 2000+ | 40 | 0 | Max clicked 1046 - clientId=u'e7adb3c7-9743-4e42-886f-e84eba09a14b', started=0, shown=8939, clicked=1046, submissionDate=u'20160930' Max shown 11614 - clientId=u'28e51d31-370d-4878-a3d0-277ca4ec0f8f', started=0, shown=11614, clicked=362, submissionDate=u'20160930' 50 client ID's that have shown=0 and clicked>0 as well as started=0 1 has clicked=22 2 have clicked=2 The remainder have clicked=1 Examples clientId=u'390c8fcc-bfe7-4ed6-995f-c5534c35b26c', started=0, shown=0, clicked=22, submissionDate=u'20160929' clientId=u'e1a6a813-62b3-462f-b1c3-7d7e20593184', started=0, shown=0, clicked=2, submissionDate=u'20160906' clientId=u'f047ae3e-f758-4027-87b4-b14b62b39119', started=0, shown=0, clicked=1, submissionDate=u'20160811' Perhaps there is some sort of race condition causing payload/event for started to be overwritten by shown which happens soon after the ping for started and shown is being overwritten if clicked is sent soon after it as well? Since shown and clicked are what I care about I'm not going to investigate this unless I find reason to do so since the discrepancy between clicked without shown is extremely small.

The report in comment #21 is definitely bogus. I'll look into it more next week.

I've got a much more accurate report almost finished and will post it sometime today.

Since this is looking at 44, and 45 *would* be the new line, I'm wondering if we shouldn't look at these metrics periodically (weekly?) after each new system add-on update. Both the rate of change and the impact would be interesting -- even more so if we weren't updating the system add-on until 2-4 weeks after the release, as we'd be running updates to version N and then N+1 in the course of one state of the system add-on.

(In reply to David Durst [:ddurst] from comment #26) > Since this is looking at 44, and 45 *would* be the new line, I'm wondering > if we shouldn't look at these metrics periodically (weekly?) after each new > system add-on update. Both the rate of change and the impact would be > interesting -- even more so if we weren't updating the system add-on until > 2-4 weeks after the release, as we'd be running updates to version N and > then N+1 in the course of one state of the system add-on. Agreed and weekly should be fine. Now that I have the detailed data in the reports in a consistent state it should be possible to come up with a dashboard design for a subset of this data.

I found the problem with the updated queries I used when processing the entire out of date add-on data... results below. I ran the report for the entire out of date add-on data joined by the client ID with the MainSummary records last Firefox version with a submission date greater than 8/9/2016 which is the date that the out of date add-on was deployed. Last version records from the MainSummary parquet (I should have optimized this for client ID's that are only in the out of date add-ons data) Total 343466090 Out of date add-on records with a client Id Total : 3851896 Started: 2236 Shown : 3851594 Clicked: 1686827 Count | Shown | Clicked | -------+---------+---------+ 0 | 302 | 2165069 | 1 | 1044410 | 1119898 | 2 | 597486 | 304935 | 3 | 387674 | 112946 | 4 | 274377 | 51966 | 5 | 202213 | 27349 | 6+ | 1345434 | 69733 | 100+ | 51390 | 947 | 1000+ | 350 | 11 | 2000+ | 121 | 2 | Out of date add-on records with a client Id that also have a record in the MainSummary parquet Total : 3829395 (99.42% of all records) Started: 2236 (100% of all started records) Shown : 3829105 (99.42% of all shown records) Clicked: 1678772 (99.52% of all clicked records) Count | Shown | Clicked | -------+---------+---------+ 0 | 290 | 2150623 | 1 | 1032667 | 1113358 | 2 | 592959 | 303882 | 3 | 385462 | 112671 | 4 | 273132 | 51871 | 5 | 201456 | 27302 | 6+ | 1343429 | 69688 | 100+ | 51388 | 947 | 1000+ | 350 | 11 | 2000+ | 121 | 2 | Out of date add-on client ID's that also have a record in the MainSummary parquet Total: 3829395 Out of date add-on client ID's Firefox version distribution of latest version from the MainSummary parquet | Total | Version | Count | Percent | --------+---------+---------+ 39.0.3 | 1 | 0.00% | 40.0.0 | 1 | 0.00% | 40.0.3 | 2 | 0.00% | 41.0.0 | 11 | 0.00% | 41.0.1 | 9 | 0.00% | 41.0.2 | 16 | 0.00% | 42.0.0 | 589 | 0.02% | 43.0.0 | 31 | 0.00% | 43.0.1 | 2721 | 0.07% | 43.0.2 | 115 | 0.00% | 43.0.3 | 167 | 0.00% | 43.0.4 | 1007 | 0.03% | 44.0.0 | 293206 | 7.66% | 44.0.1 | 86172 | 2.25% | 44.0.2 | 1853212 | 48.39% | 44.0.3 | 8 | 0.00% | 45.0.0 | 864 | 0.02% | 45.0.1 | 32967 | 0.86% | 45.0.2 | 27222 | 0.71% | 45.0a1 | 1 | 0.00% | 46.0.0 | 1012 | 0.03% | 46.0.1 | 44531 | 1.16% | 47.0.0 | 25203 | 0.66% | 47.0.1 | 308095 | 8.05% | 48.0.0 | 78648 | 2.05% | 48.0.1 | 74659 | 1.95% | 48.0.2 | 593026 | 15.49% | 49.0.0 | 43142 | 1.13% | 49.0.1 | 362757 | 9.47% | Firefox 48.0 was released on 08/02/2016 and the out of date add-on was released on 08/10/2016 so the Firefox clients that are reporting a version of anything before Firefox 48.0 was released most likely updated using application update though it is possible that they intentionally navigated away from the page opened by the add-on and selected a version other than a latest version. Out of date add-on client ID's with latest version of 48 and greater from the MainSummary parquet | Total | Count | Percent | ---------+---------+ 1152232 | 30.09% | Out of date add-on client ID's with clicked greater than 0 Firefox version distribution of latest version from the MainSummary parquet Clicked Total: 1678772 | Clicked | Total | Version | Count | Percent | Percent | --------+---------+---------+---------+ 41.0.0 | 8 | 0.00% | 0.00% | 41.0.1 | 5 | 0.00% | 0.00% | 41.0.2 | 8 | 0.00% | 0.00% | 42.0.0 | 294 | 0.02% | 0.01% | 43.0.0 | 22 | 0.00% | 0.00% | 43.0.1 | 1196 | 0.07% | 0.03% | 43.0.2 | 59 | 0.00% | 0.00% | 43.0.3 | 71 | 0.00% | 0.00% | 43.0.4 | 357 | 0.02% | 0.01% | 44.0.0 | 93978 | 5.60% | 2.45% | 44.0.1 | 27374 | 1.63% | 0.71% | 44.0.2 | 537772 | 32.03% | 14.04% | 44.0.3 | 2 | 0.00% | 0.00% | 45.0.0 | 333 | 0.02% | 0.01% | 45.0.1 | 8902 | 0.53% | 0.23% | 45.0.2 | 7171 | 0.43% | 0.19% | 46.0.0 | 318 | 0.02% | 0.01% | 46.0.1 | 10966 | 0.65% | 0.29% | 47.0.0 | 7535 | 0.45% | 0.20% | 47.0.1 | 110726 | 6.60% | 2.89% | 48.0.0 | 72567 | 4.32% | 1.89% | 48.0.1 | 66894 | 3.98% | 1.75% | 48.0.2 | 439018 | 26.15% | 11.46% | 49.0.0 | 37215 | 2.22% | 0.97% | 49.0.1 | 255981 | 15.25% | 6.68% | Out of date add-on client ID's with clicked greater than 0 with latest version of 48 and greater from the MainSummary parquet | Clicked | Total | Count | Percent | Percent | --------+---------+---------+ 871675 | 51.92% | 22.76% | Out of date add-on client ID's with clicked equal to 0 Firefox version distribution of latest version from the MainSummary parquet Not Clicked Total: 2150623 | Not Clicked | Total | Version | Count | Percent | Percent | --------+---------+-------------+---------+ 39.0.3 | 1 | 0.00% | 0.00% | 40.0.0 | 1 | 0.00% | 0.00% | 40.0.3 | 2 | 0.00% | 0.00% | 41.0.0 | 3 | 0.00% | 0.00% | 41.0.1 | 4 | 0.00% | 0.00% | 41.0.2 | 8 | 0.00% | 0.00% | 42.0.0 | 295 | 0.01% | 0.01% | 43.0.0 | 9 | 0.00% | 0.00% | 43.0.1 | 1525 | 0.07% | 0.04% | 43.0.2 | 56 | 0.00% | 0.00% | 43.0.3 | 96 | 0.00% | 0.00% | 43.0.4 | 650 | 0.03% | 0.02% | 44.0.0 | 199228 | 9.26% | 5.20% | 44.0.1 | 58798 | 2.73% | 1.54% | 44.0.2 | 1315440 | 61.17% | 34.35% | 44.0.3 | 6 | 0.00% | 0.00% | 45.0.0 | 531 | 0.02% | 0.01% | 45.0.1 | 24065 | 1.12% | 0.63% | 45.0.2 | 20051 | 0.93% | 0.52% | 45.0a1 | 1 | 0.00% | 0.00% | 46.0.0 | 694 | 0.03% | 0.02% | 46.0.1 | 33565 | 1.56% | 0.88% | 47.0.0 | 17668 | 0.82% | 0.46% | 47.0.1 | 197369 | 9.18% | 5.15% | 48.0.0 | 6081 | 0.28% | 0.16% | 48.0.1 | 7765 | 0.36% | 0.20% | 48.0.2 | 154008 | 7.16% | 4.02% | 49.0.0 | 5927 | 0.28% | 0.15% | 49.0.1 | 106776 | 4.96% | 2.79% | Out of date add-on client ID's with clicked equal to 0 with latest version of 48 and greater from the MainSummary parquet | Not Clicked | Total | Count | Percent | Percent | --------+-------------+---------+ 280557 | 13.05% | 7.33% | Comparing the Total Percent for clients that clicked the button in the out of date add-on and are now running Firefox 48 or greater it is possible to see that the add-on had an affect on getting users to the latest version when compared to clients that did not report the clicked event. Specifically | Clicked | Total | Count | Percent | Percent | --------+---------+---------+ 871675 | 51.92% | 22.76% | when compared to | Not Clicked | Total | Count | Percent | Percent | --------+-------------+---------+ 280557 | 13.05% | 7.33% | When comparing all Firefox clients that had a record in both the out of date add-ons data source and the MainSummary data source it appears that at a minimum approximately 15.43% more clients out of all clients upgraded due to the add-on and at maximum 22.76% upgraded out of all clients. The actual percentage is likely somewhere in between these two percentages. The percentage of clients that clicked the button in the out of date add-on that are now running Firefox 48 or greater (51.92% of the clicked population) when compared to the percentage of clients that did not click the button that are now running Firefox 48 or greater which (13.05% of the not clicked population) provides confidence that the out of date add-on lessens the number of orphaned users. To measure success I think it is better to measure the change in percentage of all clients to measure the overall affect as was done in the previous paragraph.

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #27) > (In reply to David Durst [:ddurst] from comment #26) > > Since this is looking at 44, and 45 *would* be the new line, I'm wondering > > if we shouldn't look at these metrics periodically (weekly?) after each new > > system add-on update. Both the rate of change and the impact would be > > interesting -- even more so if we weren't updating the system add-on until > > 2-4 weeks after the release, as we'd be running updates to version N and > > then N+1 in the course of one state of the system add-on. > Agreed and weekly should be fine. Now that I have the detailed data in the > reports in a consistent state it should be possible to come up with a > dashboard design for a subset of this data. I also think this should be accomplished in a new bug.

I took into account clients that had multiple Firefox versions in the add-on data. Here is the Firefox version distribution for the same Firefox version for client ID's and multiple Firefox versions for client ID's from all pings in the out of date add-on data. Count of client ID's by Firefox version with the same Firefox version Version | Total | ---------+---------+ 44.0.0 | 483163 | 44.0.1 | 141490 | 44.0.2 | 3223632 | 45.0.1 | 1 | 45.0a1 | 1 | 47.0.1 | 3 | 48.0.0 | 1 | 48.0.2 | 3 | 49.0.0 | 1 | 49.0.1 | 1 | Count of client ID's by Firefox versions with a different Firefox version Minimum | Maximum | Version | Version | Total | ---------+---------+---------+ 43.0.1 | 44.0.1 | 1 | 43.0.1 | 44.0.2 | 2 | 44.0.0 | 44.0.1 | 147 | 44.0.0 | 44.0.2 | 2580 | 44.0.0 | 47.0.1 | 5 | 44.0.0 | 48.0.2 | 3 | 44.0.0 | 49.0.0 | 2 | 44.0.0 | 49.0.1 | 4 | 44.0.1 | 44.0.2 | 736 | 44.0.1 | 45.0.2 | 1 | 44.0.1 | 47.0.1 | 1 | 44.0.1 | 48.0.2 | 1 | 44.0.1 | 49.0.0 | 1 | 44.0.1 | 49.0.1 | 2 | 44.0.2 | 45.0.2 | 1 | 44.0.2 | 46.0.1 | 2 | 44.0.2 | 47.0.0 | 1 | 44.0.2 | 47.0.1 | 17 | 44.0.2 | 48.0.0 | 10 | 44.0.2 | 48.0.1 | 14 | 44.0.2 | 48.0.2 | 35 | 44.0.2 | 49.0.0 | 10 | 44.0.2 | 49.0.1 | 24 |

This notebook creates the parquet used to report this data. The data sources it collects the data from are the outofdate_notifications_system_addon pings and the MainSummary parquet.

Just prints out the results in the notebook

Note: the notebook for creating the parquet took around 1 hour and 45 minutes to run using 4 clusters. It has a fairly complex join operation to get the latest record from the MainSummary parquet in order to get the last Firefox version reported.

Report is in comment #28 Resolving as fixed.

Since there have been questions about the effectiveness of the websense system add-on I took a look at the count of client ID's that have sent a ping for this add-on and the count of client ID's reporting 44.0, 44.0.1, and 44.0.2 as the highest version in the MainSummary parquet. Client ID's reporting 44.0, 44.0.1, and 44.0.2 8167840 Client ID's in the out of date add-on datasource 4478979 This is around 55% of the clients which is the uptake Felipe has seen with the websense add-on.