In Socorro, crontabber, the processor, and parts of the webapp use configman to configure components. configman has allowed Socorro a level of flexibility that makes it much easier to develop, maintain, and use.

However, configman has some downsides. One is that it's very complex and the library doesn't currently have active maintainers. That's mostly fine since it's static. configman doesn't generate documentation making it easier to figure out configuration. It does log configuration at startup, but learning about how things are configured by reading logs requires many iterations. configman doesn't aid in debugging configuration problems. Often you get odd errors and have to have a working knowledge of what you're tinkering with to know where to look. configman variables are lowercase and period-delimited which means they're not valid Bash variables which makes working with them painful in shell scripts.

When I built Antenna, I wrote a configuration library called Everett which is inspired by configman and python-decouple. Everett has fewer features than configman, but it supports auto-documentation of components, has better configuration debugging facilities, and works fine in shell scripts.

For these reasons, I want to switch from configman to Everett.

This bug covers that.

Grabbing this to look into now. If we were using Everett, then it'd be a little easier to finish up the processor/webapp side of the Pub/Sub changes.

willkg merged PR #4828: "bug 1529342: redo logging in socorro apps" in 6105d50.

willkg merged PR #4829: "bug 1529342: remove sighup handling" in a02c482.

willkg merged PR #4832: "bug 1529342: reduce crontabber some more" in 3b10bb7.

willkg merged PR #4833: "bug 1529342 - remove change default, executor_identity, ThreadedTaskManagerWithConfigSetup" in 36c0cdb.

willkg merged PR #4835: "bug 1529342: redo Elasticsearch-related configuration" in c5417e0.

Not going to work on this anymore any time soon.

Making this block the processor rewrite. The current processor architecture is inextricably tied to configman and DotDict. We need to untie those and it might be easier to do it as a separate first step.

docker-compose v2 doesn't work with configman variables:

unexpected character "." in variable name near "resource.logging.level=DEBUG

Grabbing this to work on now since it blocks switching to honcho, too.

willkg merged PR #6339: "bug 1529342: more configman-related cleanup" in 9ca6cb8.

willkg merged PR #6340: "bug 1529342: removing waiting_func" in b2aa0a0.

willkg merged PR #6342: "bug 1529342: fix temporary path value" in 7454ee2.

I think this will fix stage deploys which are broken.

willkg merged PR #6345: "bug 1529342: remove context_tools" in f672720.

willkg merged PR #6346: "bug 1529342: fix symbol cache manager race" in e930541.

willkg merged PR #6350: "bug 1529342: remove tests and fix docstrings for things that will be changed" in f813c1f.

willkg merged PR #6351: "bug 1529342: more removals" in 4626020.

This (and the previous PR) chip away at the monster PR I've got that removes configman from Socorro.

willkg merged PR #6347: "bug 1529342: convert from configman to everett" in fb53297.

Next steps:

1. this will autodeploy to stage
2. validate everything on stage: processor, crontabber, webapp
3. update dashboards to track new metrics keys
4. let it bake over the weekend

While doing that, fix or write follow-up bugs for later for anything that's awry.

Problem: Connecting to SQS fails because it's got the wrong url.

https://mozilla.sentry.io/issues/4008048714/?project=6700122

I need to set the region in the configuration. I'll do that in an infra pr.

Problem: run_migrations.sh and the webapp can't start because the webapp uses socorro/mozilla_settings.py and that requires several settings for the processor to be valid ints, but the way the infrastructure code is set up, those values are the empty string which isn't a valid int. The webapp doesn't need the processor configuration, so this is clunky.

https://mozilla.sentry.io/issues/4008048358/?project=6700122

I think the way I'm going to deal with this is to change the parser to an int_or_none so it handles things that don't evaluate to int and makes them None.

willkg merged PR #6369: "bug 1529342: handle empty string not being a valid int" in ee1ceba.

This will autodeploy the sqs region configuration fix as well as a first pass at an int_or_none fix. I wrote up issue 205 in everett for a better fix. Once I fix that in everett and push an update, we can switch to that.

willkg merged PR #6373: "bug 1529342: switch to datadogmetrics backend" in faf4079.

willkg merged PR #6375: "bug 1529342: switch from ACCESS to OPEN; minor improvements" in d6d5614.

willkg merged PR #6376: "bug 1529342: make adding watches more resilient to errors" in 1a123fc.

willkg merged PR #6381: "bug 1529342: fix possible race condition when adding directories" in 01c7cb7.

This should fix the most recent cache manager problem where it needed to prune empty directories so as to free up slots in the inotify watch list. After this, I think we're good to go.

willkg merged PR #6384: "bug 1529342: fix remove_watch keyerror" in e1cfd23.

I've been working on this for a little over 4 years.

I pushed the last round of changes out today with bug #1824987. I updated the graphs in the production dashboard. Marking as FIXED.