We want to do a deploy for socorro tagged 2023.03.28.

It consists of the following:

34b0528: Bug 1821641 - Ignore functions used to store crash annotations, manipulate POD o (gabrielesvelto)
6f8cea5: Bug 1821453 - Ignore array iterator methods and the std::all_of() function when (gabrielesvelto)
469c093: Update dependencies (willkg)
72f20cc: bug 1529342: add mozilla settings (willkg)
6ad0410: bug 1529342: overhaul env vars, implement Settings, update sqs code (willkg)
aa13bd4: bug 1529342: move MetricsCounter into new statsd directory (willkg)
89ffa34: bug 1529342: remove configman from fs crashstorage (willkg)
b30ea37: bug 1529342: remove configman from elasticsearch (willkg)
36ad396: bug 1529342: remove configman from boto code (willkg)
ebc07d5: bug 1529342: remove configman from processor app (willkg)
e283c2f: bug 1529342: update env vars in test.sh script (willkg)
1edbc36: bug 1529342: remove configman from verifyprocessed and archivescraper (willkg)
36a3f4f: bug 1529342: remove configman from webapp (willkg)
11c449d: bug 1529342: clean up webapp settings (willkg)
e62877c: bug 1529342: improve sqs and s3 docstrings (willkg)
19a794c: bug 1529342: change bucket_name to bucket (willkg)
56c1b71: bug 1529342: fix env vars (willkg)
17d617e: bug 1529342: fix processor so it runs and processes (willkg)
d3e49bb: bug 1529342: fixes for python formatting (willkg)
536f422: bug 1529342: reimplment timing/incr metrics for saving processed crashes (willkg)
5275010: Fix logging to interpolate lazily (willkg)
7ce1406: bug 1529342: update remove_field and permadelete_crash_data (willkg)
393e187: bug 1529342: remove symbol_cache_manager (willkg)
1c3aea1: bug 1529342: remove obsolete tests and fix test configuration (willkg)
a74f2fe: bug 1529342: update socorro lib tests (willkg)
0e71747: bug 1529342: update socorro processor tests (willkg)
33dffff: bug 1529342: update fs crashstorage tests (willkg)
4ec5077: bug 1529342: update sqs tests (willkg)
c8028ff: bug 1529342: update crashqueue and crashstorage base tests (willkg)
d727e09: bug 1529342: update processor and rules tests (willkg)
ee7885f: bug 1529342: update boto crashstorage tests (willkg)
8c06816: bug 1529342: fix settings to handle nested overrides (willkg)
f52c72b: bug 1529342: update elasticsearch tests (willkg)
0e296ac: bug 1529342: rename BotoHelper to S3Helper (willkg)
062c9b5: bug 1529342: update webapp/crashstats/crashstats/ tests (willkg)
2828026: bug 1529342: update webapp/crashstats/api/ tests (willkg)
c5d71c7: bug 1529342: update webapp/crashstats/signature/ tests (willkg)
461c350: bug 1529342: update webapp/crashstats/supersearch/ tests (willkg)
4485f1e: bug 1529342: update webapp/crashstats/topcrashers/tests (willkg)
bad1a83: bug 1529342: remove errant print functions (willkg)
142a06a: bug 1529342: remove configman dependency (willkg)
67f5857: bug 1698682: run processor with honcho (willkg)
ff84336: bug 1673493: rewrite cache manager (willkg)
ee55e85: bug 1529342: remove sentry bits from upload_telemetry_schema (willkg)
0951364: bug 1529342: fix errant comment (willkg)
b20adb4: bug 1529342: remove vestigial STATSD_CLIENT things (willkg)
ef397f7: bug 1529342: document and test Settings class (willkg)
d8646d0: bug 1529342: fix minor elasticsearch crashstorage issues (willkg)
b0820e8: bug 1529342: switch to an f-string (willkg)
7364452: bug 1529342: remove errant print statement (willkg)
ad9c705: bug 1529342: use fresh crash ids in boto crashstorage tests (willkg)
505fe38: bug 1529342: adjust crash destination settings (willkg)
3b8b6cb: bug 1529342: fix issues with elasticsearch tests (willkg)
e1780fd: bug 1529342: simplify crash destination settings usage in webapp (willkg)
50c3aef: bug 1529342: fix supersearch tests (willkg)
f79e670: bug 1529342: tweak some webapp tests (willkg)
b88a90f: bug 1529342: add run_postdeploy.sh script (willkg)
5e63617: bug 1529342: add documentation for configuration (willkg)
fb53297: bug 1529342: switch AWS_ENDPOINT_URL to LOCAL_DEV_AWS_ENDPOINT_URL (willkg)
ee1ceba: bug 1529342: handle empty string not being a valid int (willkg)
191371e: bug 1822648: handle race condition in CREATE event (willkg)
a7a3b6b: bug 1822651: add handling for IN_Q_OVERFLOW (willkg)
2a3f54f: bug 1822648: improve filenotfound resiliency and switch to & (willkg)
faf4079: bug 1529342: switch to datadogmetrics backend (willkg)
d6d5614: bug 1529342: switch from ACCESS to OPEN; minor improvements (willkg)
0e82596: bug 1529342: make adding watches more resilient to errors (willkg)
1a123fc: bug 1529342: make removing watches more resilient; set default to 40gb (willkg)
9cf0805: Bug 1822931 - Fix constant reference qualifiers in signatures (gabrielesvelto)
b9544da: bug 1529342: update everett, fix symbols_cache_max_size (willkg)
7b2c858: bug 1529342: fix honcho logging (willkg)
584ff7c: Update datadog to 0.45.0 (willkg)
778407e: Update system test checklist (willkg)
7091c7b: bug 1529342: fix possible race condition when adding directories (willkg)
ed6e77d: bug 1529342: add exception logging for add_watch fails (willkg)
01c7cb7: bug 1529342: implement pruning empty directories in cache manager (willkg)
a1f1191: Update README (willkg)
973a291: Update django-cors-headers to 3.14.0 (willkg)
e1cfd23: bug 1529342: fix remove_watch keyerror (willkg)

This one pushes out the configuration overhaul for Socorro. It's been baking on stage for a week and a half. All issues that have come up so far have been addressed.

Tagged 2023.03.28:

tag 2023.03.28
Tagger: Will Kahn-Greene <willkg@mozilla.com>
Date:   Tue Mar 28 13:41:09 2023 -0400

Tag 2023.03.28 (bug #1824987)

`34b0528`: Bug 1821641 - Ignore functions used to store crash annotations, manipulate POD o (gabrielesvelto)
`6f8cea5`: Bug 1821453 - Ignore array iterator methods and the std::all_of() function when  (gabrielesvelto)
`469c093`: Update dependencies (willkg)
`72f20cc`: bug 1529342: add mozilla settings (willkg)
`6ad0410`: bug 1529342: overhaul env vars, implement Settings, update sqs code (willkg)
`aa13bd4`: bug 1529342: move MetricsCounter into new statsd directory (willkg)
`89ffa34`: bug 1529342: remove configman from fs crashstorage (willkg)
`b30ea37`: bug 1529342: remove configman from elasticsearch (willkg)
`36ad396`: bug 1529342: remove configman from boto code (willkg)
`ebc07d5`: bug 1529342: remove configman from processor app (willkg)
`e283c2f`: bug 1529342: update env vars in test.sh script (willkg)
`1edbc36`: bug 1529342: remove configman from verifyprocessed and archivescraper (willkg)
`36a3f4f`: bug 1529342: remove configman from webapp (willkg)
`11c449d`: bug 1529342: clean up webapp settings (willkg)
`e62877c`: bug 1529342: improve sqs and s3 docstrings (willkg)
`19a794c`: bug 1529342: change bucket_name to bucket (willkg)
`56c1b71`: bug 1529342: fix env vars (willkg)
`17d617e`: bug 1529342: fix processor so it runs and processes (willkg)
`d3e49bb`: bug 1529342: fixes for python formatting (willkg)
`536f422`: bug 1529342: reimplment timing/incr metrics for saving processed crashes (willkg)
`5275010`: Fix logging to interpolate lazily (willkg)
`7ce1406`: bug 1529342: update remove_field and permadelete_crash_data (willkg)
`393e187`: bug 1529342: remove symbol_cache_manager (willkg)
`1c3aea1`: bug 1529342: remove obsolete tests and fix test configuration (willkg)
`a74f2fe`: bug 1529342: update socorro lib tests (willkg)
`0e71747`: bug 1529342: update socorro processor tests (willkg)
`33dffff`: bug 1529342: update fs crashstorage tests (willkg)
`4ec5077`: bug 1529342: update sqs tests (willkg)
`c8028ff`: bug 1529342: update crashqueue and crashstorage base tests (willkg)
`d727e09`: bug 1529342: update processor and rules tests (willkg)
`ee7885f`: bug 1529342: update boto crashstorage tests (willkg)
`8c06816`: bug 1529342: fix settings to handle nested overrides (willkg)
`f52c72b`: bug 1529342: update elasticsearch tests (willkg)
`0e296ac`: bug 1529342: rename BotoHelper to S3Helper (willkg)
`062c9b5`: bug 1529342: update webapp/crashstats/crashstats/ tests (willkg)
`2828026`: bug 1529342: update webapp/crashstats/api/ tests (willkg)
`c5d71c7`: bug 1529342: update webapp/crashstats/signature/ tests (willkg)
`461c350`: bug 1529342: update webapp/crashstats/supersearch/ tests (willkg)
`4485f1e`: bug 1529342: update webapp/crashstats/topcrashers/tests (willkg)
`bad1a83`: bug 1529342: remove errant print functions (willkg)
`142a06a`: bug 1529342: remove configman dependency (willkg)
`67f5857`: bug 1698682: run processor with honcho (willkg)
`ff84336`: bug 1673493: rewrite cache manager (willkg)
`ee55e85`: bug 1529342: remove sentry bits from upload_telemetry_schema (willkg)
`0951364`: bug 1529342: fix errant comment (willkg)
`b20adb4`: bug 1529342: remove vestigial STATSD_CLIENT things (willkg)
`ef397f7`: bug 1529342: document and test Settings class (willkg)
`d8646d0`: bug 1529342: fix minor elasticsearch crashstorage issues (willkg)
`b0820e8`: bug 1529342: switch to an f-string (willkg)
`7364452`: bug 1529342: remove errant print statement (willkg)
`ad9c705`: bug 1529342: use fresh crash ids in boto crashstorage tests (willkg)
`505fe38`: bug 1529342: adjust crash destination settings (willkg)
`3b8b6cb`: bug 1529342: fix issues with elasticsearch tests (willkg)
`e1780fd`: bug 1529342: simplify crash destination settings usage in webapp (willkg)
`50c3aef`: bug 1529342: fix supersearch tests (willkg)
`f79e670`: bug 1529342: tweak some webapp tests (willkg)
`b88a90f`: bug 1529342: add run_postdeploy.sh script (willkg)
`5e63617`: bug 1529342: add documentation for configuration (willkg)
`fb53297`: bug 1529342: switch AWS_ENDPOINT_URL to LOCAL_DEV_AWS_ENDPOINT_URL (willkg)
`ee1ceba`: bug 1529342: handle empty string not being a valid int (willkg)
`191371e`: bug 1822648: handle race condition in CREATE event (willkg)
`a7a3b6b`: bug 1822651: add handling for IN_Q_OVERFLOW (willkg)
`2a3f54f`: bug 1822648: improve filenotfound resiliency and switch to & (willkg)
`faf4079`: bug 1529342: switch to datadogmetrics backend (willkg)
`d6d5614`: bug 1529342: switch from ACCESS to OPEN; minor improvements (willkg)
`0e82596`: bug 1529342: make adding watches more resilient to errors (willkg)
`1a123fc`: bug 1529342: make removing watches more resilient; set default to 40gb (willkg)
`9cf0805`: Bug 1822931 - Fix constant reference qualifiers in signatures (gabrielesvelto)
`b9544da`: bug 1529342: update everett, fix symbols_cache_max_size (willkg)
`7b2c858`: bug 1529342: fix honcho logging (willkg)
`584ff7c`: Update datadog to 0.45.0 (willkg)
`778407e`: Update system test checklist (willkg)
`7091c7b`: bug 1529342: fix possible race condition when adding directories (willkg)
`ed6e77d`: bug 1529342: add exception logging for add_watch fails (willkg)
`01c7cb7`: bug 1529342: implement pruning empty directories in cache manager (willkg)
`a1f1191`: Update README (willkg)
`973a291`: Update django-cors-headers to 3.14.0 (willkg)
`e1cfd23`: bug 1529342: fix remove_watch keyerror (willkg)

Deploy bug #1824987
-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEzY3I6/E2Vh8o9nPoGL3RpvBm7BEFAmQjJrUACgkQGL3RpvBm
7BHyCggApG0YHeOc+UeGBxmrWmCpvOWKchhyrM/9qcfc/GG0t1lrHV115M1uoxrB
pluogWEx8+dVkSSCY/s9l39ftX3L+LspZrJpIe1MFm/fhAhwbUzDZid/EWQNNs8B
LitMGGvSsdIz6KGMBT6XATKQsLPCjgARDQT0N0Pw5CZI8FTOHRDuZy+WEwateuDm
AJh5oOgoSsIzghkDSoEMKEDPmt4yPZMYyrlIScuYcn1L+vQoWsrYveNJYw6wl7qp
X497+M6hZjffokamOONFgJPmUb22tT/+sG4S6zC+g6qF40XfzcxVnW6rim+VaoPp
Wwb6IciC+tcOQ+jBxhVnMw+B+jXnMQ==
=8Wmw
-----END PGP SIGNATURE-----

commit e1cfd237d21d18bad104b19a9a98f9dfb1eeefd6
Author: Will Kahn-Greene <willkg@mozilla.com>
Date:   Fri Mar 24 22:32:40 2023 -0400

    bug 1529342: fix remove_watch keyerror

I updated the dashboard with the new metrics and watched everything for a while. It looks good. Marking as FIXED.