I configured the production consul cluster, and nodes can join via the ELB, but they drop off quickly because the clients needs to be able to connect directly to the servers and they can't right now in prod.

Stage isn't working properly either (same errant behaviour). According to a post on the Consul googlegroup[0], ports 830[01]/tcp|udp need to be accessible by *all* members of the cluster (servers and clients), which currently isn't the case in our infra. [0] https://groups.google.com/forum/#!topic/consul-tool/2cGsecnqLeE

I added ingress rules for the aforementioned ports/protocols for an existing (stage) role group and then applied it via Terraform. Once the security group was updated I manually triggered a consul join <stage_consul_elb> and observed the logs on both the client and one of the servers. The client joined and the logs looked good; after a couple of hours the client was still joined. It would appear that this is the (a?) solution to the problem.

Needed a little bit more, opening up UDP egress seems to do the trick: https://github.com/mozilla/socorro-infra/commit/60fba1106a1c76a53e7ea9636c7132367e8dbdd0