It would be interesting to know how people use the app manager: - how many failed connections? - how many click on Help - how many projects are managed - how often does the user update his app - ...

Filter on 86b7095e-2bd0-499e-a704-d00f2524aeef / PAUL STOP SETTING QA CONTACT TO THE DEVTOOLS COMPONENT'S WATCHERS EMAIL FOR BUGS YOU FILE :)

I'm lost. I have no idea how to record these data. First, I use toolOpened and toolClose. How do I check it actually works? Then, I want to track: - what runtimes have been discovered (type + number) (Simulators, USB Runtimes, Wifi Runtimes, localRuntime, remoteRuntime) - number of successful runtime connections (with type) - number of failed runtime connections (with type) - how long is the runtime connected - how often the toolbox is open per connection - how often the user click "play" per connection How do I do that? Who can help me?

Mike has done the majority of the telemetry work, and Jordan ( freshly needinfo'd ) has done some for Web Audio and Canvas tools.

For the timing stuff follow the comments in browser/devtools/shared/telemetry.js For counting things: 1. Create DEVTOOLS_XXX_OPENED_BOOLEAN in toolkit/components/telemetry/Histograms.json (search for boolean to see how). 2. let Telemetry = require("devtools/shared/telemetry") 3. let telemetry = new Telemetry(); 4. telemetry.log("DEVTOOLS_XXX_OPENED_BOOLEAN", true); // Do this every time you want to increment the number. For logging numbers like how often the toolbox is open per connection... do the same as above but use "kind": "exponential" in the histogram. See https://developer.mozilla.org/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe to see which types to use. To check that it is working look at how I tested it: /Users/mratcliffe/Desktop/fx-team/browser/devtools/shared/test/browser_telemetry_sidebar.js // I monkeypatched the log method.

I think Mike addressed all the issues, but ping me if anything else is unclear!

Paul, I am making progress here, but I have some questions about what you want to track. (In reply to Paul Rouget [:paul] (slow to respond. Ping me on IRC) from comment #2) > - what runtimes have been discovered (type + number) (Simulators, USB > Runtimes, Wifi Runtimes, localRuntime, remoteRuntime) The available runtimes can change at any time when devices are unplugged / replugged. WiFi devices only show up when you scan explicitly, etc. So, when should data be recorded? Each time the runtime list changes (this sounds like it would be very noisy data)? Only once per opening of WebIDE? More generally, what do we hope to learn from this particular data point? I am not sure I see the value here... It *does* seem useful to know what runtime types are used (USB, WiFi, etc.), but we'll have that from the successful / failed connection results you also suggested. > - how often the toolbox is open per connection So if I: * Connect * Open toolbox * Close toolbox * Open toolbox * Disconnect We'd record "2", because the toolbox was opened twice? How is that useful? > - how often the user click "play" per connection Same question here as above: how is this data point useful?

(In reply to J. Ryan Stinnett [:jryans] from comment #6) > > - what runtimes have been discovered (type + number) (Simulators, USB > > Runtimes, Wifi Runtimes, localRuntime, remoteRuntime) > > The available runtimes can change at any time when devices are unplugged / > replugged. WiFi devices only show up when you scan explicitly, etc. So, > when should data be recorded? Each time the runtime list changes (this > sounds like it would be very noisy data)? Only once per opening of WebIDE? > > More generally, what do we hope to learn from this particular data point? I > am not sure I see the value here... You're right. Let's ignore that. > It *does* seem useful to know what > runtime types are used (USB, WiFi, etc.), but we'll have that from the > successful / failed connection results you also suggested. > > > - how often the toolbox is open per connection > > So if I: > > * Connect > * Open toolbox > * Close toolbox > * Open toolbox > * Disconnect > > We'd record "2", because the toolbox was opened twice? How is that useful? > > > - how often the user click "play" per connection > > Same question here as above: how is this data point useful? Both for toolbox and play, I'd like to know if people use WebIDE only to push app, or push app and iterate on the code, or push app, debug and iterate on the code. Maybe people will use WebIDE only to push an app sometimes, maybe they will iterate on the code and not use the toolbox at all. But all of that can happen in a follow up bug.

Paul, I believe this captures the values you're looking for. You can test it interactively at about:telemetry, and filter on "webide". (You have to refresh this tab to get updated histograms after you perform actions.) Mike, I refactored the telemetry module slightly to expose the timing functionality more generally. Can you check this over, and also see if I've set up the probes correctly?

Try: https://tbpl.mozilla.org/?tree=Try&rev=9d6b90f627bf

Fixed test failures. Try: https://tbpl.mozilla.org/?tree=Try&rev=0ce2b99d173d

Comment on attachment 8469006 [details] [diff] [review] Telemetry for WebIDE (v2) Review of attachment 8469006 [details] [diff] [review]: ----------------------------------------------------------------- Nice patch, r+.

Comment on attachment 8469006 [details] [diff] [review] Telemetry for WebIDE (v2) This covers enough actions. Can't wait to see the results. Thanks a lot for taking over this patch Ryan.

This was causing timeouts that frequently spilled into other tests as shown in the log below. Backed out. https://hg.mozilla.org/mozilla-central/rev/2a621e79e6b9 https://tbpl.mozilla.org/php/getParsedLog.php?id=45563460&tree=Mozilla-Central

Tried to make the new test more failure-resistant. Also updated some other tests to check the |DebuggerServer| state. Try: https://tbpl.mozilla.org/?tree=Try&rev=b59b76ddff84

green.

Backed out for new mochitest-other failures. https://hg.mozilla.org/integration/fx-team/rev/ed15a6020e93 https://tbpl.mozilla.org/php/getParsedLog.php?id=45771560&tree=Fx-Team

It fails on Linux 32 PGO only... :( I'll try to investigate this more tomorrow.

I am still looking into this via more try debugging. Adding more logging made the error go away... :(

I am still blocked on intermittent test issues here. When I add more logs to debug the issue, the tests start passing. :( My most recent push[1] at least caused consistent failures on Windows, but I cannot repro this locally. Paul, can you spot any signs of racy behavior here? I am not seeing it myself. From past debugging, when it does fail, it seems to get as far as adding the test app with Cmds.importPackagedApp before hanging and hitting a timeout. [1]: https://tbpl.mozilla.org/?tree=Try&rev=510e08287517

https://tbpl.mozilla.org/php/getParsedLog.php?id=46276352&tree=Try 09:46:38 INFO - 187 INFO TEST-OK | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_runtime.html | took 863ms 09:46:39 INFO - 188 INFO TEST-START | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_telemetry.html 09:46:39 INFO - 189 INFO Closing WebIDE 09:46:39 INFO - 190 INFO WebIDE closed 09:46:39 INFO - 191 INFO opening WebIDE 09:46:40 INFO - 192 INFO WebIDE open I can't get why we are seeing "Closing WebIDE" first. Given my reading of test_telemetry.html, we are calling cycleWebIDE first which immediatly calls openWebIDE. Any idea why/how we end up trying to close WebIDE first? Could it be just some randomization of logs?

Also, if you have a rebased patch, that may help ;)

Here's a rebased version. Note that in past, failures mainly appeared on Linux PGO, which is not run on Try by default. To make Try do this, you have to add "mk_add_options MOZ_PGO=1" to the "build/mozconfig.common.override" file. It seems to be some kind of interaction with a previous WebIDE test. Sprinkling many requestCompleteLog and dump / print calls in the past seemed to suggest it would hang inside the "importPackagedApp" method, but once I added further logging calls, the problem vanished.

I tried to push just test_runtime.html to try, but I'm not sure I rebased correctly (I had conflict with my recent patches... [autoselect and app front refactoring]): https://tbpl.mozilla.org/?tree=Try&rev=563d83bd3430 That ended up failing with the same error with the whole patch: https://tbpl.mozilla.org/?tree=Try&rev=53c46ebf6e7d

Here's a branch with my patch, as requested: https://github.com/jryans/gecko-dev/tree/wi-telemetry-reg

I got a green try with all the patch rebased: https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5 Is it only intermittent? Could it be that we fixed something?

Comment on attachment 8491555 [details] [diff] [review] Telemetry for WebIDE (v5) Review of attachment 8491555 [details] [diff] [review]: ----------------------------------------------------------------- I launched a bunch of try runs at various times, especially the last one during pacific time wake up. https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5 Looks like it has been fixed by some recent change. That doesn't surprise me that much as we fixed various tiny issues recently. The last patch doesn't apply with very last changes I made to webide, here is a rebase of it if that can help: https://github.com/ochameau/mozilla-central/commit/c08302783b8f36103dc548fe876249137e704251 https://github.com/ochameau/mozilla-central/commit/7ef9524c42732e0f30768116c2c439fffd0a7d97 Could you try to rebase it and push it yourself to confirm it is really green?

Thanks Alex for your help! Both regular and PGO try are green (Mac PGO failure is expected): Non-PGO: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d1765a288884 PGO: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9e5f09c50656