We've been closing the trees daily, specially, when cycle_data starts doing clean ups.
For now let's increase the storage another 350GB which should give us another ~1,000 IOPS (Storage rate $0.23 per GB-month -> 350 * .23 = $81).

If the instance type can be change so that IOPS increases without increasing the database size, that would be best. Also, maybe we can just pay for more IOPS?

I don't see why we wouldn't pay for it, keeping things running is priority #1, hopefully in Q1 we can find a way to reduce it back down.

https://bugzilla.mozilla.org/show_bug.cgi?id=1566204#c3

dividehex 11:30 AM
(my 2 cents)  I think it is pretty clear we should up the IOPs.  The only question is do we up it by uping the gp2 size or do we up it to piops and eat the cost.  I personally think we should move to piops since it actually designed for IO intensive workloads, which is what we have here.
Armen (armenzg) 11:30 AM
@dividehex I agree with you
fubar:spiral_calendar_pad: 11:30 AM
wfm
dividehex 11:30 AM
I'm not the decision maker on that though
^^ that guy is :simple_smile:
fubar:spiral_calendar_pad: 11:31 AM
shipit
dividehex 11:31 AM
makes it so

Terraform will perform the following actions:

  ~ aws_db_instance.treeherder-prod-rds
      iops:         "0" => "7000"
      storage_type: "gp2" => "io1"

Tue, 19 Nov 2019 19:49:49 GMT	Finished preparing for modification of storage
Tue, 19 Nov 2019 19:53:53 GMT	Applying modification to allocated storage
Tue, 19 Nov 2019 19:56:58 GMT	Finished applying modification to allocated storage

Prod is currently in the Storage-optimization state.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html

Thank you!
Things look a bit better overnight.