We might want to cover: - IndexedDB - Cookies - LocalStorage - DataStores - AppCache Minimum required for V1 will be: - IndexedDB - LocalStorage - DataStores V1 can be read-only.

This is important for Gaia developers (IndexedDB is used a lot in Gaia apps and soon Gaia will rely on the DataStore API). Myself and Optimizer will take a look at this.

I wrote some user stories to cover the basics last quarter: https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user-stories.md Ideally each user story would be covered as a bug, and the bugs grouped by a whiteboard tag or marked as blocking this bug - I have no strong preference. Also ( ideally ) if you have additional use cases not covered by the user stories, please add them to the user stories doc.

(In reply to Paul Rouget [:paul] from comment #0) > We might want to cover: > - IndexedDB > - Cookies > - LocalStorage > - DataStores > - AppCache That looks like a great list. > Minimum required for V1 will be: > - IndexedDB > - LocalStorage > - DataStores I think AppCache is maybe more important than DataStores for most web developers. We have the gcli comand but when we show that to developers they complain it isn't discoverable, and they would prefer a more GUI-oriented tool ( possible with the commands exposed ). Do you see having a cli-oriented query tool like the GCLI / Appcache commands? I would assume some apps would load in a lot of data, and a more mysql/cli interface might be preferable for some users even for querying. > V1 can be read-only. Sure. For a V2+ where you can write/delete/modify data, developers might want to be able to import data from a json file or similar. Do you have any ideas about the additional import/export functionality people expect from data tools?

Another question, regarding integration with the rest of the toolbox: do we see this as being a separate additional tool? The toolbox is crowded already. Chrome has the resources tab which includes all page assets, but has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick for input.

(In reply to Jeff Griffiths (:canuckistani) from comment #3) > I wrote some user stories to cover the basics last quarter: > > https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user- > stories.md That's very useful. Thanks. > Ideally each user story would be covered as a bug, and the bugs grouped by a > whiteboard tag or marked as blocking this bug - I have no strong preference. > Also ( ideally ) if you have additional use cases not covered by the user > stories, please add them to the user stories doc. Yes. This bug is a meta bug. (In reply to Jeff Griffiths (:canuckistani) from comment #4) > I think AppCache is maybe more important than DataStores for most web > developers. You're right. App Cache is a nightmare, and we should support that in the V1. DataStore is important too for apps. It will become a central piece of Gaia starting very soon. > Do you see having a cli-oriented query tool like the GCLI / Appcache > commands? I would assume some apps would load in a lot of data, and a more > mysql/cli interface might be preferable for some users even for querying. We designed a DataBase API. IndexedDB. So if we ever implement a cli to request data, I think it should be based on the synchronous IndexedDB API. > > V1 can be read-only. > > Sure. For a V2+ where you can write/delete/modify data, developers might > want to be able to import data from a json file or similar. Do you have any > ideas about the additional import/export functionality people expect from > data tools? Not yet. > Another question, regarding integration with the rest of the toolbox: do we > see this as being a separate additional tool? The toolbox is crowded > already. Chrome has the resources tab which includes all page assets, but > has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick > for input. Yes. An additional panel. I don't know how we will scale yet. I think we can address this issues later. But indeed, at some point, we probably want to have a "Resource" tab where we'd get the Data Inspector, + all the files used by the pages.

Some of my thoughts : - For web devs, Cookies might be the most requested things (if not the most important) - IndexedDB inspection might really not be that useful for web devs. I know FxOS uses it very much, but on web, IndexedDB is mainly used to store offline data like images, textures, maps etc. - We definitely need one extra panel for this tool as a whole. Its true that Style Editor and Debugger can be merged (but that is an another discussion). - AppCache is not only a nightmare for web devs, but for us too. It "might" be a bottleneck for V1 ;) [This is just an assumption]

(In reply to Girish Sharma [:Optimizer] from comment #8) > - For web devs, Cookies might be the most requested things (if not the most > important) It's also very simple. So let's do it. > - IndexedDB inspection might really not be that useful for web devs. I know > FxOS uses it very much, but on web, IndexedDB is mainly used to store > offline data like images, textures, maps etc. This is the main DB used in Firefox OS. We need to support it. > - AppCache is not only a nightmare for web devs, but for us too. It "might" > be a bottleneck for V1 ;) [This is just an assumption] It's not hard to inspect: See `about:cache?device=offline` and: nsIApplicationCache nsIApplicationCacheNamespace nsIApplicationCacheContainer nsIApplicationCacheChannel nsIApplicationCacheService nsIDOMOfflineResourceList https://developer.mozilla.org/en-US/docs/Application_cache_implementation_overview

http://mxr.mozilla.org/mozilla-central/source/netwerk/cache2/nsICacheStorageService.idl#39 http://mxr.mozilla.org/mozilla-central/source/netwerk/cache2/nsICacheStorage.idl#86

(In reply to Paul Rouget [:paul] from comment #9) > (In reply to Girish Sharma [:Optimizer] from comment #8) > > - For web devs, Cookies might be the most requested things (if not the most > > important) > > It's also very simple. So let's do it. +1. > > - IndexedDB inspection might really not be that useful for web devs. I know > > FxOS uses it very much, but on web, IndexedDB is mainly used to store > > offline data like images, textures, maps etc. > > This is the main DB used in Firefox OS. We need to support it. I agree we need to support it, I think for anyone to take this tool seriously we need to provide basic access in V1. To me for each item ( appcache, localstorage, indexedDB ) the prioritization could look like this: V1: let developers view the data! In some case, maybe allow basic operations like flushing appcache. When we announce, we also show a roadmap for proposed enhancements and ask for feedback. V2 -> Vn: let the developers do appropriate actions for each type of data. Get feedback from developers based on V1 to inform V2+. Developers might want to do all sorts of things like: - save binary blobs from indexedDB to the filesystem - edit cookies or localstorage in a property editor similar to the manifest editor in App Manager - delete data based on a query, or other command-line interactions We should see what they ask for and weigh those requests. Fair question: should V1 be an add-on?

(In reply to Jeff Griffiths (:canuckistani) from comment #11) > Fair question: should V1 be an add-on? I like that. Would it be possible to design the whole tool as an addon, and then, ship it as a builtin feature? How would we do that? What about tests? Is the devtools code ready for that? Apparently, you are working with Irakli on a devtools API, would we use this API? Actors will have to be built in though.

(In reply to Paul Rouget [:paul] from comment #12) > (In reply to Jeff Griffiths (:canuckistani) from comment #11) > > Fair question: should V1 be an add-on? > > I like that. Would it be possible to design the whole tool as an addon, and > then, ship it as a builtin feature? How would we do that? What about tests? > Is the devtools code ready for that? I don't think we're ready for that. > Apparently, you are working with Irakli on a devtools API, would we use this > API? Down the road, for a tool that is maybe less core to what developers expect from us. > Actors will have to be built in though. I've been thinking a lot more about the tools as client-server apps. I think the really cool thing will be when add-ons are created that re-mix our existing server capabilities in interesting ways? For this tool in particular consider my add-on comment as idle distraction. We know we need this tool and it shouldn't bee too much of a mystery how to build it. Let's just build it!

Can we please call this the "storage" tool instead of the "resource" tool? I feel very strongly about this. Everything is a resource; that term is incredibly vague. I think that "storage" better describes what we are shooting for. I agree, this should be a new panel, but we only need one panel for all storage mediums. I think everyone is on the same page here, right? I'm imagining a sidebar with the storage type you want to inspect (cookies, appcache, indexeddb, etc) that has its own view when selected with more details that are specific to that type of storage. It would be really cool if we had an api to add new types of storage types that addons could use. That way, libs like lawchair[0], pouchdb[1], jstorage[2], store.js[3], or whatever the cool kids are using these days could provide addons for inspecting/displaying/editing their specific storage stuff. Definitely not a v1 thing, but I think it is something to keep in the back of our minds while architecting the code. [0] http://brian.io/lawnchair/ [1] http://pouchdb.com/ [2] http://www.jstorage.info/ [3] https://github.com/marcuswestin/store.js

I agree with Nick. A problem with the term "resource" is that this also includes every files used on the page, including .html files, .js files, and image files. As evidence for this, if you look at Chrome's "resource" tab, you'll find this all-encompassing monstrosity that includes essentially a file explorer (of the above types) combined with stuff like local storage, indexeddb, etc. Things that don't really belong in the same tool.

Sure, "storage" works for me. And all in one panel. Like Chrome.

(In reply to Paul Rouget [:paul] from comment #17) > Sure, "storage" works for me. And all in one panel. Like Chrome. Like Chrome in that it's one panel. Not like Chrome in that we don't like the "Frames" section and we don't need the WebSQL section. Also most websites are just going to be using one or maybe 2 of the storage options, the tree presentation rather assumes that you'll be have many of them to switch between. I'd be interested to see just the detail pane with everything on. Empty storage systems would be included with a simple "No Cookies" type message.

For the record, we have been looking at the option of generating the whole HTML report on the server side. But we'll need to be more granular than that as a DB can hold several dozens of Mb.

I just talked to Paul, we think the absolutely critical bits for v1 can be reduced to: Storage Types: * DataStores ( for FxOS & Apps developers ) * IndexedDB * Localstorage - Data will be read-only. - We will need some sort of rudimentary search / filtering V2+ then looks like this: * Cookies * Appcache * Session data * Read / write access * Search improvements * follow-ups based on user feedback

Also, let's not forget that Indexdb and DataStore will soon be accessible from workers: https://bugzilla.mozilla.org/show_bug.cgi?id=701634 https://bugzilla.mozilla.org/show_bug.cgi?id=916196 https://bugzilla.mozilla.org/show_bug.cgi?id=916204

(In reply to Paul Rouget [:paul] from comment #22) > Also, let's not forget that Indexdb and DataStore will soon be accessible > from workers: Is there a bug for being able to connect a toolbox to a worker? The Gaia team really wants this.

(In reply to Jeff Griffiths (:canuckistani) from comment #23) > (In reply to Paul Rouget [:paul] from comment #22) > > Also, let's not forget that Indexdb and DataStore will soon be accessible > > from workers: > > Is there a bug for being able to connect a toolbox to a worker? The Gaia > team really wants this. There's bug 757133.

DataStores use IndexDB on the backend.

(In reply to Paul Rouget [:paul] from comment #25) > DataStores use IndexDB on the backend. Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore access. I suspect that read-write is less work than AppCache & DataStores (please correct me!), so maybe you can look into that for v1 again (though I agree that read access is far more important than write access).

(In reply to Florian Bender from comment #26) > (In reply to Paul Rouget [:paul] from comment #25) > > DataStores use IndexDB on the backend. > > Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore > access. > > I suspect that read-write is less work than AppCache & DataStores (please > correct me!), so maybe you can look into that for v1 again (though I agree > that read access is far more important than write access). Write/Delete is not a big task as per the UI, but protocol wise and backend wise, it is not that straight forward. Specially given that we still do not have all the apis we require. This is not even in its prototype stage. So we will see what happens :)

As for priorities, shouldn't cookies go alongside localStorage in v1 because, you know, lots of code out there uses that old thingie... My 2 cents.

(In reply to bobi from comment #29) > As for priorities, shouldn't cookies go alongside localStorage in v1 > because, you know, lots of code out there uses that old thingie... > > My 2 cents. Please see the dependent bugs. The actor already has cookies and local storage.

Just to give a bit more info from the Web Compatibility front. More and more Web sites are using localStorage capabilities for storing information about Web browser profiles. There is no easy way for people in Web compatibility to modify and test the values (except through the console). This interface would be very useful in our work. Also see http://www.opera.com/dragonfly/documentation/storage/

mass component move . filter on #MassComponentMoveStorageInspector

Do we still keep this bug open ?

Ping, should that bug be closed now? Sebastian

Yup, everything else should be follow-ups, would like to see a tracking bug for planned improvements.