Use standard lz4 file format instead of the non-standard jsonlz4/mozlz4
Categories
(Toolkit :: General, defect, P5)
Tracking
()
People
(Reporter: 1utyu4+43h7ypa654vuo, Unassigned)
References
Details
Updated•9 years ago
|
Comment 2•9 years ago
|
||
Updated•9 years ago
|
Comment 3•9 years ago
|
||
Comment 4•9 years ago
|
||
Comment hidden (offtopic) |
Comment 6•9 years ago
|
||
Comment hidden (offtopic) |
Comment 8•8 years ago
|
||
Comment 9•8 years ago
|
||
Comment 10•8 years ago
|
||
Comment 11•8 years ago
|
||
Comment 12•7 years ago
|
||
Comment 13•7 years ago
|
||
Comment 15•7 years ago
|
||
Comment 16•7 years ago
|
||
Comment 17•5 years ago
|
||
Another utility, for people trying to work around this: https://gist.github.com/Tblue/62ff47bef7f894e92ed5
What I'm wondering now - why compress at all? For me, and I guess for most other users, this file doesn't even reach a Megabyte. Yes, lz4 can compress it to around 10%, but who cares at that size?
Comment 18•5 years ago
|
||
It does less I/O when storing and retrieving on disk.
Comment 19•4 years ago
|
||
Ok from a quick benchmark, I'd zuggest switching to zstd? It seems to compress/decompress slightly faster than LZ4 with triple the compression ratio...
(I unpacked a ~4MiB upgrade.jsonlz4 I had in my profile and duplicated its contents random times, than ran multitime with mozlz4 (python3 implemented, that may skew the results), zstd and pigz commands packing and unpacking the test file.. on a tmpfs, with BOINC paused on my hexacore machine)
652M test.json
123M test.json.gz
144M test.json.lz4
43M test.json.zst
Compressing:
++ multitime -v zstd -kf test.json
===> Executing zstd -kf test.json
===> multitime results
1: zstd -kf test.json
Mean Std.Dev. Min Median Max
real 1.266 0.000 1.266 1.266 1.266
user 1.363 0.000 1.363 1.363 1.363
sys 0.178 0.000 0.178 0.178 0.178
++ multitime -v sh -c 'mozlz4 -c < '\''test.json'\'' > '\''test.json'\''.lz4'
===> Executing sh -c "mozlz4 -c < 'test.json' > 'test.json'.lz4"
===> multitime results
1: sh -c "mozlz4 -c < 'test.json' > 'test.json'.lz4"
Mean Std.Dev. Min Median Max
real 1.353 0.000 1.353 1.353 1.353
user 0.852 0.000 0.852 0.852 0.852
sys 0.497 0.000 0.497 0.497 0.497
++ multitime -v pigz -kf test.json
===> Executing pigz -kf test.json
===> multitime results
1: pigz -kf test.json
Mean Std.Dev. Min Median Max
real 2.010 0.000 2.010 2.010 2.010
user 22.482 0.000 22.482 22.482 22.482
sys 0.454 0.000 0.454 0.454 0.454
Decompression:
++ multitime -v zstd -dc test.json.zst
===> Executing zstd -dc test.json.zst
===> multitime results
1: zstd -dc test.json.zst
Mean Std.Dev. Min Median Max
real 0.368 0.000 0.368 0.368 0.368
user 0.363 0.000 0.363 0.363 0.363
sys 0.004 0.000 0.004 0.004 0.004
++ multitime -v sh -c 'mozlz4 -d < test.json.lz4 > /dev/null'
===> Executing sh -c "mozlz4 -d < test.json.lz4 > /dev/null"
===> multitime results
1: sh -c "mozlz4 -d < test.json.lz4 > /dev/null"
Mean Std.Dev. Min Median Max
real 0.863 0.000 0.863 0.863 0.863
user 0.408 0.000 0.408 0.408 0.408
sys 0.455 0.000 0.455 0.455 0.455
++ multitime -v pigz -dc test.json.gz
===> Executing pigz -dc test.json.gz
===> multitime results
1: pigz -dc test.json.gz
Mean Std.Dev. Min Median Max
real 1.467 0.000 1.467 1.467 1.467
user 2.184 0.000 2.184 2.184 2.184
sys 0.174 0.000 0.174 0.174 0.174
Comment 20•3 years ago
|
||
For anyone coming across this needing to decompress these files:
/*
NOTE: BEFORE RUNNING THIS SCRIPT, CHECK THIS SETTING:
Type or paste about:config into the address bar and press Enter
Click the button promising to be careful
In the search box type devt and pause while Firefox filters the list
If devtools.chrome.enabled is false, double-click it to toggle to true
Paste this entire script into the command line at the bottom of the Browser Console (Windows: Ctrl+Shift+j)
Then press Enter to run the script. A file picker should promptly open.
*/
async function convert() {
// Set up file chooser
var fp = Components.classes["@mozilla.org/filepicker;1"]
.createInstance(Components.interfaces.nsIFilePicker);
fp.init(window, "Open File", Components.interfaces.nsIFilePicker.modeOpen);
fp.appendFilter("Bookmark Backup Files", "*.jsonlz4");
var result = await new Promise(resolve => fp.open(resolve));
// Call file choose, proceed if a file was chosen
if (result == Components.interfaces.nsIFilePicker.returnOK) { var file = fp.file;
// Check that file can be used
if (file.exists() && file.isFile() && file.isReadable()) {
var oldfile = fp.file.path;
// Construct output file name
var newfile = oldfile.replace(".jsonlz4", "_converted.json");
// See: http://forums.mozillazine.org/viewtopic.php?p=14111285#p14111285
var {utils:Cu} = Components;
Cu.import("resource://gre/modules/osfile.jsm");
var jsonString = await OS.File.read(oldfile,{ compression: "lz4" });
// console.log(jsonString);
OS.File.writeAtomic(newfile, jsonString);
}
}
}
convert()
Updated•2 years ago
|
Comment 21•2 years ago
|
||
The severity field for this bug is relatively low, S3. However, the bug has 25 votes and 52 CCs.
:mossop, could you consider increasing the bug severity?
For more information, please visit auto_nag documentation.
Comment 22•2 years ago
|
||
The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.
Comment hidden (advocacy) |
Comment 24•1 years ago
|
||
I don't think there is any value in posting further comments in support of this, we're on board with doing it, we just don't currently have the resources available to do it. If someone wants to work on implementing it then we would review a patch.
Description
•