Profile folder grows too big (over 20gb) unexpectedly. I have also experienced this issue with previous nightly versions. Chances are this is happening after an auto update.

and which file in the profile folder is growing that big ? I'm using the nightly builds on windows and my profile folder is "only" 403MB

Seems that I have not kept that information, I'll update next time I observe this again. my profile folder is usually much smaller, like 80MB. But it happened a few times to grow that big, maybe I wouldn't have noticed if it didn't almost fill up my disk. ps. thanks for looking into this!

Forgot to also add that I didn't need to manually remove anything. Just closed and reopened the browser and boom back to normal size.

Thanks Alco for your reply, I will close this as Resolved WFM based on comment 2. Please feel free to reopen it if you encounter this issue in the future.

Just encountered this again. the file that grows that big is "places.sqlite-wal". Any ideas?

(In reply to alco from comment #5) > Just encountered this again. the file that grows that big is > "places.sqlite-wal". Any ideas? Please attach a log from about:support

Also, please run the Places database integrity check from about:support

places.sqlite database is sane. Also, here is a log from about:support. Is that ok? Application Basics ------------------ Name: Firefox Version: 62.0a1 Build ID: 20180526100113 Update Channel: nightly User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0 OS: Linux 4.16.11-300.fc28.x86_64 Multiprocess Windows: 1/1 (Enabled by default) Web Content Processes: 4/4 Enterprise Policies: Inactive Google Key: Found Mozilla Location Service Key: Found Safe Mode: false Crash Reports for the Last 3 Days --------------------------------- All Crash Reports Nightly Features -------------------------------------- Name: Activity Stream Version: 2018.05.24.1266-fdda2139 ID: activity-stream@mozilla.org Name: Application Update Service Helper Version: 2.0 ID: aushelper@mozilla.org Name: Firefox Screenshots Version: 32.1.0 ID: screenshots@mozilla.org Name: Follow-on Search Telemetry Version: 0.9.7 ID: followonsearch@mozilla.com Name: Form Autofill Version: 1.0 ID: formautofill@mozilla.org Name: Photon onboarding Version: 1.0 ID: onboarding@mozilla.org Name: Pocket Version: 1.0.5 ID: firefox@getpocket.com Name: Web Compat Version: 2.0 ID: webcompat@mozilla.org Name: WebCompat Reporter Version: 1.0.0 ID: webcompat-reporter@mozilla.org Extensions ---------- Name: ColorZilla Version: 3.3 Enabled: true ID: {6AC85730-7D0F-4de0-B3FA-21142DD85326} Name: HTTPS Everywhere Version: 2018.4.11 Enabled: true ID: https-everywhere@eff.org Name: Privacy Badger Version: 2018.5.10 Enabled: true ID: jid1-MnnxcxisBPnSXQ@jetpack Name: uBlock Origin Version: 1.16.8 Enabled: true ID: uBlock0@raymondhill.net Name: Greasemonkey Version: 4.3 Enabled: false ID: {e4a8a97b-f2ed-450b-b12d-ee082ba24781} Security Software ----------------- Type: Type: Type: Graphics -------- Features Compositing: Basic Asynchronous Pan/Zoom: wheel input enabled; scrollbar drag enabled; keyboard enabled; autoscroll enabled WebGL 1 Driver WSI Info: GLX 1.4 GLX_VENDOR(client): Mesa Project and SGI GLX_VENDOR(server): SGI Extensions: GLX_ARB_create_context GLX_ARB_create_context_profile GLX_ARB_fbconfig_float GLX_ARB_framebuffer_sRGB GLX_ARB_get_proc_address GLX_ARB_multisample GLX_EXT_buffer_age GLX_EXT_create_context_es2_profile GLX_EXT_create_context_es_profile GLX_EXT_fbconfig_packed_float GLX_EXT_framebuffer_sRGB GLX_EXT_import_context GLX_EXT_texture_from_pixmap GLX_EXT_visual_info GLX_EXT_visual_rating GLX_INTEL_swap_event GLX_MESA_copy_sub_buffer GLX_MESA_multithread_makecurrent GLX_MESA_query_renderer GLX_MESA_swap_control GLX_OML_swap_method GLX_OML_sync_control GLX_SGIS_multisample GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGIX_visual_select_group GLX_SGI_make_current_read GLX_SGI_swap_control GLX_SGI_video_sync WebGL 1 Driver Renderer: Intel Open Source Technology Center -- Mesa DRI Intel(R) Ironlake Mobile WebGL 1 Driver Version: 2.1 Mesa 18.0.2 WebGL 1 Driver Extensions: GL_3DFX_texture_compression_FXT1 GL_AMD_seamless_cubemap_per_texture GL_AMD_shader_trinary_minmax GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_APPLE_object_purgeable GL_APPLE_packed_pixels GL_ARB_ES2_compatibility GL_ARB_arrays_of_arrays GL_ARB_buffer_storage GL_ARB_clear_buffer_object GL_ARB_clear_texture GL_ARB_clip_control GL_ARB_color_buffer_float GL_ARB_compressed_texture_pixel_storage GL_ARB_copy_buffer GL_ARB_copy_image GL_ARB_debug_output GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex GL_ARB_draw_instanced GL_ARB_explicit_attrib_location GL_ARB_explicit_uniform_location GL_ARB_fragment_coord_conventions GL_ARB_fragment_program GL_ARB_fragment_program_shadow GL_ARB_fragment_shader GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_get_program_binary GL_ARB_get_texture_sub_image GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_multisample GL_ARB_multitexture GL_ARB_occlusion_query GL_ARB_occlusion_query2 GL_ARB_pixel_buffer_object GL_ARB_point_parameters GL_ARB_point_sprite GL_ARB_polygon_offset_clamp GL_ARB_program_interface_query GL_ARB_provoking_vertex GL_ARB_robustness GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_objects GL_ARB_shader_texture_lod GL_ARB_shading_language_100 GL_ARB_shading_language_packing GL_ARB_shadow GL_ARB_sync GL_ARB_texture_barrier GL_ARB_texture_border_clamp GL_ARB_texture_compression GL_ARB_texture_compression_rgtc GL_ARB_texture_cube_map GL_ARB_texture_env_add GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_filter_anisotropic GL_ARB_texture_float GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_mirrored_repeat GL_ARB_texture_non_power_of_two GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_storage GL_ARB_texture_swizzle GL_ARB_transpose_matrix GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_program GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_window_pos GL_ATI_blend_equation_separate GL_ATI_draw_buffers GL_ATI_separate_stencil GL_ATI_texture_env_combine3 GL_ATI_texture_float GL_EXT_abgr GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_equation_separate GL_EXT_blend_func_separate GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_compiled_vertex_array GL_EXT_copy_texture GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_draw_range_elements GL_EXT_fog_coord GL_EXT_framebuffer_blit GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_gpu_program_parameters GL_EXT_multi_draw_arrays GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_packed_pixels GL_EXT_pixel_buffer_object GL_EXT_point_parameters GL_EXT_polygon_offset GL_EXT_polygon_offset_clamp GL_EXT_provoking_vertex GL_EXT_rescale_normal GL_EXT_secondary_color GL_EXT_separate_specular_color GL_EXT_shadow_funcs GL_EXT_stencil_two_side GL_EXT_stencil_wrap GL_EXT_subtexture GL_EXT_texture GL_EXT_texture3D GL_EXT_texture_array GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_cube_map GL_EXT_texture_edge_clamp GL_EXT_texture_env_add GL_EXT_texture_env_combine GL_EXT_texture_env_dot3 GL_EXT_texture_filter_anisotropic GL_EXT_texture_integer GL_EXT_texture_lod_bias GL_EXT_texture_object GL_EXT_texture_rectangle GL_EXT_texture_sRGB GL_EXT_texture_sRGB_decode GL_EXT_texture_shared_exponent GL_EXT_texture_snorm GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_vertex_array GL_EXT_vertex_array_bgra GL_IBM_multimode_draw_arrays GL_IBM_rasterpos_clip GL_IBM_texture_mirrored_repeat GL_INGR_blend_func_separate GL_KHR_blend_equation_advanced GL_KHR_context_flush_control GL_KHR_debug GL_KHR_no_error GL_KHR_robustness GL_MESA_pack_invert GL_MESA_texture_signed_rgba GL_MESA_window_pos GL_NV_blend_square GL_NV_conditional_render GL_NV_depth_clamp GL_NV_light_max_exponent GL_NV_packed_depth_stencil GL_NV_primitive_restart GL_NV_texgen_reflection GL_NV_texture_barrier GL_NV_texture_env_combine4 GL_NV_texture_rectangle GL_OES_EGL_image GL_OES_read_format GL_S3_s3tc GL_SGIS_generate_mipmap GL_SGIS_texture_border_clamp GL_SGIS_texture_edge_clamp GL_SGIS_texture_lod GL_SUN_multi_draw_arrays WebGL 1 Extensions: ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_frag_depth EXT_sRGB EXT_shader_texture_lod EXT_texture_filter_anisotropic OES_element_index_uint OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_s3tc WEBGL_compressed_texture_s3tc_srgb WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context WebGL 2 Driver WSI Info: - WebGL 2 Driver Renderer: WebGL creation failed: * Error during native OpenGL init. * Exhausted GL driver caps. * Exhausted GL driver options. WebGL 2 Driver Version: - WebGL 2 Driver Extensions: - WebGL 2 Extensions: - GPU #1 Active: Yes Description: Intel Open Source Technology Center -- Mesa DRI Intel(R) Ironlake Mobile Vendor ID: Intel Open Source Technology Center Device ID: Mesa DRI Intel(R) Ironlake Mobile Driver Version: 2.1 Mesa 18.0.2 Diagnostics AzureCanvasAccelerated: 0 AzureCanvasBackend: skia AzureContentBackend: skia AzureFallbackCanvasBackend: none CairoUseXRender: 0 Device Reset: Trigger Device Reset Decision Log HW_COMPOSITING: blocked by default: Acceleration blocked by platform OPENGL_COMPOSITING: unavailable by default: Hardware compositing is disabled WEBRENDER: opt-in by default: WebRender is an opt-in feature unavailable by runtime: Hardware compositing is disabled OMTP: disabled by default: Disabled by default Media ----- Audio Backend: remote Max Channels: 2 Preferred Sample Rate: 44100 Output Devices Name: Group Built-in Audio Analog Stereo: /devices/pci0000:00/0000:00:1b.0/sound/card0 LifeChat LX-3000 Headset Analog Stereo: /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/sound/card1 Input Devices Name: Group Monitor of Built-in Audio Analog Stereo: /devices/pci0000:00/0000:00:1b.0/sound/card0 Monitor of LifeChat LX-3000 Headset Analog Stereo: /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/sound/card1 LifeChat LX-3000 Headset Analog Mono: /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/sound/card1 Important Modified Preferences ------------------------------ accessibility.typeaheadfind.flashBar: 0 browser.cache.disk.capacity: 51200 browser.cache.disk.filesystem_reported: 1 browser.cache.disk.smart_size.enabled: false browser.cache.disk.smart_size.first_run: false browser.cache.disk.smart_size.use_old_max: false browser.cache.frecency_experiment: 1 browser.places.smartBookmarksVersion: 8 browser.search.suggest.enabled: false browser.sessionstore.upgradeBackup.latestBuildID: 20180526100113 browser.startup.homepage_override.buildID: 20180526100113 browser.startup.homepage_override.mstone: 62.0a1 browser.tabs.remote.autostart.2: true browser.urlbar.daysBeforeHidingSuggestionsPrompt: 0 browser.urlbar.lastSuggestionsPromptDate: 20170815 browser.urlbar.matchBuckets: general:5,suggestion:Infinity browser.urlbar.placeholderName: DuckDuckGo browser.urlbar.timesBeforeHidingSuggestionsHint: 0 dom.apps.lastUpdate.buildID: 20160919122928 dom.apps.lastUpdate.mstone: 49.0 dom.apps.reset-permissions: true dom.forms.autocomplete.formautofill: true extensions.formautofill.creditCards.enabled: false extensions.formautofill.creditCards.used: 2 extensions.formautofill.firstTimeUse: false extensions.lastAppVersion: 62.0a1 font.internaluseonly.changed: false media.gmp-gmpopenh264.abi: x86_64-gcc3 media.gmp-gmpopenh264.lastUpdate: 1526929871 media.gmp-gmpopenh264.version: 1.7.1 media.gmp-manager.buildID: 20180526100113 media.gmp-manager.lastCheck: 1527339327 media.gmp.storage.version.observed: 1 media.webrtc.debug.log_file: /tmp/WebRTC.log network.cookie.lifetimePolicy: 2 network.cookie.prefsMigrated: true network.dns.disablePrefetch: true network.http.speculative-parallel-limit: 0 network.predictor.cleaned-up: true network.predictor.enabled: false network.prefetch-next: false places.database.lastMaintenance: 1527016985 places.database.migrateV48Frecencies: true places.history.expiration.transient_current_max_pages: 112348 plugin.disable_full_page_plugin_for_types: application/pdf plugins.ctprollout.cohort: user-changed-from-test plugins.ctprollout.cohortSample: 0.513439 print.print_bgcolor: false print.print_bgimages: false print.print_duplex: 0 print.print_evenpages: true print.print_margin_bottom: 0.5 print.print_margin_left: 0.5 print.print_margin_right: 0.5 print.print_margin_top: 0.5 print.print_oddpages: true print.print_orientation: 0 print.print_page_delay: 50 print.print_paper_data: 0 print.print_paper_height: 11.00 print.print_paper_name: na_letter print.print_paper_size_unit: 0 print.print_paper_width: 8.50 print.print_scaling: 1.00 print.print_shrink_to_fit: true print.print_to_file: false print.print_unwriteable_margin_bottom: 56 print.print_unwriteable_margin_left: 25 print.print_unwriteable_margin_right: 25 print.print_unwriteable_margin_top: 25 privacy.cpd.offlineApps: true privacy.cpd.siteSettings: true privacy.donottrackheader.enabled: true privacy.sanitize.pending: [] privacy.sanitize.timeSpan: 0 privacy.trackingprotection.enabled: true privacy.trackingprotection.introCount: 20 security.disable_button.openCertManager: false security.sandbox.content.tempDirSuffix: 2a9bfd61-0674-496a-a22b-37361901a592 services.sync.declinedEngines: services.sync.engine.addresses.available: true services.sync.engine.creditcards.available: true signon.importedFromSqlite: true signon.rememberSignons: false storage.vacuum.last.index: 1 storage.vacuum.last.places.sqlite: 1525064467 Important Locked Preferences ---------------------------- Places Database --------------- JavaScript ---------- Incremental GC: true Accessibility ------------- Activated: false Prevent Accessibility: 0 Library Versions ---------------- NSPR Expected minimum version: 4.19 Version in use: 4.19 NSS Expected minimum version: 3.38 Beta Version in use: 3.38 Beta NSSSMIME Expected minimum version: 3.38 Beta Version in use: 3.38 Beta NSSSSL Expected minimum version: 3.38 Beta Version in use: 3.38 Beta NSSUTIL Expected minimum version: 3.38 Beta Version in use: 3.38 Beta Sandbox ------- Seccomp-BPF (System Call Filtering): true Seccomp Thread Synchronization: true User Namespaces: true Content Process Sandboxing: true Media Plugin Sandboxing: true Content Process Sandbox Level: 4 Effective Content Process Sandbox Level: 4 Rejected System Calls --------------------- Internationalization & Localization ----------------------------------- Application Settings Requested Locales: ["en-US"] Available Locales: ["en-US"] App Locales: ["en-US"] Regional Preferences: ["en-US"] Default Locale: "en-US" Operating System System Locales: ["en-US"] Regional Preferences: ["en-US"]

Interesting, places.database.migrateV48Frecencies: true, this pref should disappear once the frecencies migration is complete, that means for some reason it doesn't complete in your profile. There's some apparent (but not detected) corruption that likely causes this migration to grow the WAL journal. I fear the only way we have to debug this may be to privately check your database, if you willing to share it through private e-mail. Though, by looking at the code I think I have a couple ideas to make this better, for example by avoiding looping in cpp during a read transaction, we can rather loop in SQL.

I'm not sure if this will completely solve the problem, but it could. Currently what we do is opening a read transaction, then converting it to a r/w transaction and running 200 statements inside it. This transaction is large and may include a bunch of other stuff happening in the meanwhile. The patch makes us open a single write transaction doing 400 items at a time.

Thank you very much for your support. I removed the database from the profile folder in order to let it regenerate and places.database.migrateV48Frecencies disappeared as it should. Not sure why it reached that state before. I'll keep an eye open in case this happens again.

Comment on attachment 8981157 [details] Bug 1463017 - Migrate origins frecencies more efficiently. https://reviewboard.mozilla.org/r/247260/#review253724 Nice. This reminds me that test_current_from_v47.js only runs with a small number of moz_places (22 looks like), so it doesn't trigger the chunking at all. We should probably update it to do that? Thanks for removing the unnecessary backslashes too. I had #defines on the brain after working in nsPlacesTriggers.h a lot. We/I should probably fix all those remaining lines that your patch here doesn't.

hm, it may be expensive to test chunking with 400 entries... not sure it's worth it.

Pushed by mak77@bonardo.net: https://hg.mozilla.org/integration/autoland/rev/eaaf06745eee Migrate origins frecencies more efficiently. r=adw

Hi Marco, based off your earlier correspondence, is there any suggested STR you might have in order for us to replicate/verify this particular bug? Thanks.

(In reply to Grover Wimberly IV [:Grover-QA] from comment #19) > Hi Marco, based off your earlier correspondence, is there any suggested STR > you might have in order for us to replicate/verify this particular bug? > Thanks. As we discussed yesterday, it may not be trivial to reproduce/verify. You need a large places.sqlite (mine is around 70MB for example) copied into the test profile, open Firefox and set places.database.migrateV48Frecencies to true (create bool pref), then close Firefox. Then you must run a query on places.sqlite through the sqlite3 cli (downloadable from sqlite.org) like "UPDATE moz_origins SET frecency = -1" and close the cli. Keep the profile folder open. Restarting Firefox should then start the process of recalculating all the frecencies in moz_origins. While this happens you can monitor the profile folder and check the -wal journal file size. The previous code was less space efficient, though it's possible there were other coincidental causes in the profile where this bug happened we don't know about. In that case you may not be able to see a meaningful different.

(In reply to alco from comment #0) > Profile folder grows too big (over 20gb) unexpectedly. I have also > experienced this issue with previous nightly versions. Chances are this is > happening after an auto update. Are you still encountering this issue in newer versions of Nightly?

(In reply to Grover Wimberly IV [:Grover-QA] from comment #21) > (In reply to alco from comment #0) > > Profile folder grows too big (over 20gb) unexpectedly. I have also > > experienced this issue with previous nightly versions. Chances are this is > > happening after an auto update. > > Are you still encountering this issue in newer versions of Nightly? No, I have not encountered this issue with newer nightly versions. A valid use case, for my profile, was to just let the browser open overnight. And the profile folder was growing that big every time. I have not managed to reproduce with newer versions.

I did not manage to reproduce this bug using the STR from comment 20. I've tried on an older Nightly build (2018-05-20) running Windows 10 x64 and Ubuntu 16.04 x64. In my case the -wal journal file (places.sqlite-wal) had approximately 2MB in the profile folder after the Firefox was restarted, and per my discussion on Slack with Mak, the wal file size expected to be much bigger in an affected environment. I've used a large places.sqlite around 71MB but my understanding is that the bug involves many variables and may not happen on all systems. Hi alco, I've seen that you already verified this issue on Nightly 62, would you mind help us verifying this on Beta 62 as well (since you report the original issue), just to make sure that everything works as expected in that build too. You can download Firefox 62 Beta from here: https://www.mozilla.org/en-US/firefox/channel/desktop/

(In reply to Ciprian Georgiu [:ciprian_georgiu], Release Desktop QA from comment #23) > I did not manage to reproduce this bug using the STR from comment 20. I've > tried on an older Nightly build (2018-05-20) running Windows 10 x64 and > Ubuntu 16.04 x64. > > In my case the -wal journal file (places.sqlite-wal) had approximately 2MB > in the profile folder after the Firefox was restarted, and per my discussion > on Slack with Mak, the wal file size expected to be much bigger in an > affected environment. I've used a large places.sqlite around 71MB but my > understanding is that the bug involves many variables and may not happen on > all systems. > > Hi alco, I've seen that you already verified this issue on Nightly 62, would > you mind help us verifying this on Beta 62 as well (since you report the > original issue), just to make sure that everything works as expected in that > build too. > > You can download Firefox 62 Beta from here: > https://www.mozilla.org/en-US/firefox/channel/desktop/ I have not managed to reproduce with beta62 but I'll certainly update if I notice anything!

Thanks alco! Closing this bug as verfied fixed per comment 24.