Create python scripts to generate Places DBs with various characteristics such as "many visits within the same domain", "visits across many domains", "many tags", "many bookmarks", etc. Create JS bookmarklet/console script to harvest statistics from Places db.

This is working on my machine, posting so adw can get it running as well. We may yets need to create a VirtualEnv for automated usage

First draft JS to generate stats on the user's Places database. Copy and paste into your JS console. Outputs stats to the console in JSON for easy parsing later.

With source code comments about each piece collected and dates output as nice strings.

Now computes livemark container and livemark child counts, per Dietrich's email.

nice, some comment on the receiving page legend is vertical while results are horizontal, for every column could be available a sort of tooltip with a description and the query we run. and column headers should be always visible also with many entries

temp tables handled correctly.

Now with an alert on completion!

Doesn't phone home anymore. To be used with new site.

we should wrap it up in a ubiquity command

WIP #2 added more information about the command in the "preview", updated the url to https://places-stats.mozilla.com

I see that you're getting information about all the add-ons a user has installed, but you're not collecting which add-ons are actually enabled or not. I think you should probably get that data, too. Breakpad collects that, for instance.

(In reply to comment #13) Yeah, two reasons for that: 1) I couldn't figure out how to do it. :\ If anyone knows, holla back. 2) People can disable an extension after using it for a long time. I imagined we would come across a scenario like this: We get some stats that bear the signature of a certain add-on. The user(s) has that add-on installed but disabled. So we totally ignore the disabled status. Or, not even that. We come across some stats with weird profiles. Which extension is the cause? Some are disabled, but maybe they were enabled last week. So again we assume they were each enabled at one point and ignore disabled status.

(In reply to comment #14) > 1) I couldn't figure out how to do it. :\ If anyone knows, holla back. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/src/nsExtensionManager.js.in#2617 http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/src/nsExtensionManager.js.in#3848

(In reply to comment #14) I'm in support of (2). If it's installed, it had it's impact on the db at some point likely.

Updated the python generator code - added glue to allow https fetch of JSON stats data live from the https://places-stats.mozilla.com/stats/ site

According to sdwilsh: Add moz_keywords Add moz_inputhistory to the data generator

added keywords support, next inputhistory

I plan on setting up my mac mini to run the generator nightly, perhaps pushing the resulting places.sqlite to the intranet.

generate.sh

Need to add a date updater script to use daily to keep the data in the db "fresh"

IMHO advising users to paste code into the error console's command line isn't the best of ideas, more in bug 491243 comment 11.

Cleaned up date handling and added ability to update all dates in the generated db via builddb/increment_dates.py - relies on existing env vars of the generate script. Not Python 2.6 compatible due to timeout bug in httplib2

Forgot to update the places schema to the current 3.5 version

The generated places dbs should be places_generated_max.sqlite, places_generated_avg.sqlite, places_generated_min.sqlite The command line args look like this python builddb/generator.py -i avg python builddb/generator.py -i min python builddb/generator.py -i max

average generation failed due to floating point number to int type coersion.

Now we know when the generator/date increment script was last run - so the date_increment.py will calculate how many days to roll the dates forward.

Zipped it for cvs users:)

I wonder if writing the queries to a file, and reading that file as a single transaction would really speed up the initial creation step? Would the overhead of writing to the file then reading back the file and executing all of the inserts be slower than the current implementation via django's ORM?

adding dev-doc-needed, please create a wiki page to explain how to use it and which prerequisites you need, i spent 1 hour to discover all prerequisites needed on ubuntu (django, httpLib2, simpleJson python packages are needed, plus a bunch of env vars). It is finally working, but was not straight-forward as i was expecting :) also maybe a script to setup the env variables would help if this is goingto be a general purpose db generator.

can this be closed?

This still needs to be landed somewhere, as far as I know?

alice over irc said it could be checked into the talos repo, if it needs any tweaks I can look it over first.

Comment on attachment 453234 [details] [diff] [review] [checked in]add generator code to buildfarm/utils This looks like it's already been used a few times. Don't think it makes sense for me to review it. Welcome to stampy town.

Comment on attachment 453234 [details] [diff] [review] [checked in]add generator code to buildfarm/utils Looks good to me. I assume this is exactly the code you are currently using in production?

Comment on attachment 453234 [details] [diff] [review] [checked in]add generator code to buildfarm/utils changeset: 650:455404252ab3