I'm working on mitigating shutdown hangs and I noticed that some of the shutdown hangs are caused by slow GetBodyUsage method. I'm going to add a check there that will abort the directory traversal if we are shutting down, but the storage initialization time can be significantly affected by this too. So it would be great if someone could investigate if we can eliminate these empty sub directories.

Here are some statistics from a storage/default directory:
bash-3.2$ find . | wc -l
15992
bash-3.2$ find . -type d | wc -l
7935
bash-3.2$ find . -type d -empty | wc -l
2802
bash-3.2$ find . -type d -empty | grep morgue | wc -l
2145

So if we fix this bug we can expect approximately 14% perf improvement.

Tom, please take a look when you have time.

Sure

Depends on D46586

One note, we might want to remove existing empty directories as part of a minor storage upgrade.

(In reply to Jan Varga [:janv] from comment #5)

One note, we might want to remove existing empty directories as part of a minor storage upgrade.

I remember I did that on another bug (not including empty directories on DOM Cache), but with the patch here we could achieve that easily. I'll try to find that.

(moving to at least a bit better component)

Ah, sorry.

(In reply to Tom Tung [:tt, :ttung] from comment #6)

(In reply to Jan Varga [:janv] from comment #5)

One note, we might want to remove existing empty directories as part of a minor storage upgrade.

I remember I did that on another bug (not including empty directories on DOM Cache), but with the patch here we could achieve that easily. I'll try to find that.

It's bug 1576593

How bug 1576593 relates to this ?

Sorry wrong bug number; I meant this one bug 1557289

Depends on D47451

Note: I landed the first two patches since the implementation removes empty directories while accessing the cache origin

For the rest two patches.

https://hg.mozilla.org/mozilla-central/rev/fbfdd8ee1b42
https://hg.mozilla.org/mozilla-central/rev/89df3c99bf3b

Backed out 2 changesets (bug 1581067) for xpcshell failures at dom/cache/test/xpcshell/test_empty_directories.js

Backout: https://hg.mozilla.org/integration/autoland/rev/5d83a502d367de1332ceb234275416f315328340

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=e7a90d8f615d7902d127febd00e5c2231cbef5ee

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=269026867&repo=autoland&lineNumber=2582

[task 2019-09-30T11:26:33.643Z] 11:26:33 INFO - TEST-START | dom/cache/test/xpcshell/test_empty_directories.js
[task 2019-09-30T11:26:33.965Z] 11:26:33 WARNING - TEST-UNEXPECTED-FAIL | dom/cache/test/xpcshell/test_empty_directories.js | xpcshell return code: 0
[task 2019-09-30T11:26:33.966Z] 11:26:33 INFO - TEST-INFO took 319ms
[task 2019-09-30T11:26:33.966Z] 11:26:33 INFO - >>>>>>>
[task 2019-09-30T11:26:33.967Z] 11:26:33 INFO - PID 13221 | [13221, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2632
[task 2019-09-30T11:26:33.968Z] 11:26:33 INFO - PID 13221 | Couldn't convert chrome URL: chrome://branding/locale/brand.properties
[task 2019-09-30T11:26:33.968Z] 11:26:33 INFO - PID 13221 | [13221, Main Thread] WARNING: NSS will be initialized without a profile directory. Some things may not work as expected.: file /builds/worker/workspace/build/src/security/manager/ssl/nsNSSComponent.cpp, line 1342
[task 2019-09-30T11:26:33.969Z] 11:26:33 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-09-30T11:26:33.970Z] 11:26:33 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-09-30T11:26:33.970Z] 11:26:33 INFO - "Setting up environment"
[task 2019-09-30T11:26:33.971Z] 11:26:33 INFO - PID 13221 | [13221, Main Thread] WARNING: '!window', file /builds/worker/workspace/build/src/dom/cache/CacheStorage.cpp, line 575
[task 2019-09-30T11:26:33.971Z] 11:26:33 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-09-30T11:26:33.972Z] 11:26:33 INFO - running event loop
[task 2019-09-30T11:26:33.973Z] 11:26:33 INFO - PID 13221 | [13221, Main Thread] WARNING: Could not get the program name for a cubeb stream.: 'NS_SUCCEEDED(rv)', file /builds/worker/workspace/build/src/dom/media/CubebUtils.cpp, line 384
[task 2019-09-30T11:26:33.973Z] 11:26:33 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
[task 2019-09-30T11:26:33.973Z] 11:26:33 INFO - "Test 0 - InitOrigin shouldn't leave an empty directoy"
[task 2019-09-30T11:26:33.974Z] 11:26:33 INFO - PID 13221 | [13221, Main Thread] WARNING: '!window', file /builds/worker/workspace/build/src/dom/cache/CacheStorage.cpp, line 575

(In reply to Cristina Coroiu [:ccoroiu] from comment #19)
Sorry for this, but I cannot reproduce it on my desktop (ubuntu). I might need to log more info on try

Just discussed with Jan if we need this on a storage upgrade. We don't need to clear the empty cache directories on a storage upgrade, because we would trigger InitOrigin once build-id changes and that would remove the empty morgue directory.

So, the nsID value is not reliable for the test I wrote. The body directory is the last two number of the nsID (or some other way to hash the value). There are two body directories/files and I assumed the first one always collides with the cache.put(testURL). And it actually work on my mac and ubuntu, but the later one collides with that on the try server.

I should just ensure that the morgue folder won't be empty.

I rewrote the test and will ask for the review again if the try goes green
try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea3d9a12b816f6fdc3333713c64441dc8edb0d58

IIUC, P5 should fix the bug 1584746

Hope this is the late try push https://treeherder.mozilla.org/#/jobs?repo=try&revision=9df55b310f0fdc1dd68f177fe86cc01e35bc834b

https://hg.mozilla.org/mozilla-central/rev/d34125d9dba3
https://hg.mozilla.org/mozilla-central/rev/0a3fd8b7865a
https://hg.mozilla.org/mozilla-central/rev/74d480f1bfa8
https://hg.mozilla.org/mozilla-central/rev/d87f43669456

https://hg.mozilla.org/mozilla-central/rev/d34125d9dba3
https://hg.mozilla.org/mozilla-central/rev/0a3fd8b7865a
https://hg.mozilla.org/mozilla-central/rev/74d480f1bfa8
https://hg.mozilla.org/mozilla-central/rev/d87f43669456