There are a number of small improvements we want to make to the published data layout, so we might as well co-ordinate and deploy them together. - Switch from lzma to xz for compression - Use "/" as a consistent dimension separator instead of the current mish-mash of "/" and "." - Swap the location of submission date and build id in the file path As an example, in the old bucket, we have a file: saved_session/Firefox/nightly/27.0a1/20130918030202.20131010.v2.log.a95d4ba999de4c24a3ecf3964cb463b7.lzma We would want to download it, uncompress it, recompress using xz, then upload it as saved_session/Firefox/nightly/27.0a1/20131010/20130918030202.v2.log.a95d4ba999de4c24a3ecf3964cb463b7.xz

We can do this on spot nodes using the analysis framework once I'm done writing it... which I hope I am soon :) btw, while we are doing this we really should move it to a bucket in us-west-2, instead of having it in US Standard. This would give some nice consistency guarantees and who knows maybe even better performance.

I actually ran into weird consistency issues w/ using US-standard. I tried setting the region to us-west-2 and it told me to use us-east-1. So definitely a good idea to pick a specific region. Also, would this be a good opportunity to combine the files as well as convert them xz compression?

Convert to xz compression is planned here. Scope of this bug: * Convert to xz compression * Swap submission date and build-id in file-prefixes * Separate dimensions in file-prefixes by slash * Move to a bucket in us-west-2 The analysis framework would be suitable for this task, and this should allow us to do this on spot instances :) This would also be a great stress test of the analysis framework. We could include combining in this task, if one of two things happend: 1) We don't use the analysis framework (bad idea - lot of work) 2) We switch to using a database for the analysis job queue. 2) Is required as current analysis framework uses SQS, which limits the number of files we can put in to a single task. Personally, I think we should setup the database and discuss elsewhere if we really need to combine anymore. Either, way I don't think we should add combining to the scope of this bug.

With respect to choice of region, I've so far just said us-west-2 because that's where we run all instances right now. So in the interest of not being the person who said let's move all data to wrong region :) would taras, please confirm that we want all of our data in us-west-2 ?

(In reply to Ben Wong [:mostlygeek] from comment #2) > Also, would this be a good opportunity to combine the files as well as > convert them xz compression? One benefit of switching to xz compression is that files can be combined without having to decompress/recompress - they can just be concatenated. This will make it much less work (cpu-wise) to do the combine phase. I think the "combine" thing should be handled separately too - it would make it much more difficult to verify that everything worked correctly for the rest of the conversion. If we were going to have to decompress/recompress again to do the combine phase, I would want to do it at the same time too, but since we don't have to redo that work there's not a lot to be gained by doing compression here too.

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #4) > With respect to choice of region, I've so far just said us-west-2 because > that's where we run all instances right now. All the configuration files for provisioning EC2 nodes specify a "region" parameter, and they're all currently set to "us-west-2". Whatever region we decide to use for S3 should be set explicitly, and should also be used for EC2 (at least for the HTTP server nodes and for the "process incoming" workers). See: https://github.com/mreid-moz/telemetry-server/blob/master/provisioning/aws/aws_telemetry_server_config.prod.json

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #4) > With respect to choice of region, I've so far just said us-west-2 because > that's where we run all instances right now. > > So in the interest of not being the person who said let's move all data to > wrong region :) > would taras, please confirm that we want all of our data in us-west-2 ? I'm leaving that up to Benson. He'll be oping it.

Use us-west-2. Never had any issues with this region before.

(In reply to Ben Wong [:mostlygeek] from comment #8) > Use us-west-2. Never had any issues with this region before. at least not that I can remember.

We would also like to split out MetroFirefox to a separate folder at the "appName" level (per bug 940422).

Hi Mark, just wanted to confirm that our metro frontend appName gets set back to "MetroFirefox" at some point so that we can be differentiated from desktop? Right now it's set to "Firefox" in our .ini file : http://mxr.mozilla.org/mozilla-central/source/browser/metro/metroapp.ini.in#4 Thanks!

Note bug 967126 which we think may be a simple addition to TelemetryPing.jsm to set our appName back to MetroFirefox

We are definitely seeing some Telemetry pings with "MetroFirefox" as the appName. I'll check to see if there are submissions from nightly builds after Nov 19th that still include MetroFirefox as the appName.

We have MetroFirefox submissions from nightly builds up to and including build 20131128030201, but nothing after that. Moving this discussion over to bug 967126 since it's not really related to this bug.

There hasn't been a compelling enough reason to spend time on this, so closing this out.