We need to test if this will work. We can change dev/staging/whatever to test this out.

Sheeri, I'm pretty sure this is not the component you're looking for.

Let us know which table you want to try first and we should just try it on dev or stage. We will just need to coordinate with krupa and other devs so as not to disrupt QA. I think the only python code that we'll need to fix is for translations (don't ask): https://github.com/mozilla/zamboni/blob/master/apps/translations/models.py#L69 other than that, I can't think of any Python code that will break. We'll find out.

Can we try this out tomorrow (Thursday) morning? What about Sunday or Monday? (I know it's a US holiday)

We'll need to modify the translations model to work with this which will need dev time. This is a requirement for the milestone after next but not our next one (on the dev side of things) - is this time sensitive because you have hardware you've got on trial and need to give back or just because we need to get it done?

Just 'cause we need to get it done. The hardware we had was a different vendor and it's long gone, we're not going with them.

We can try for next week, will have to see how the end of this week shakes out

[18:13:55] <kumar> we have that one table that selects next id + 1 to do an insert My recommendation - to do an insert: SELECT next id + @@global.auto_increment_increment

I'll try this on Monday and see if anything breaks! Sheeri, can I go ahead and use this global setting or do you need to turn it on in -dev first?

Kumar - right now auto_increment_increment is set to 1. Let's do this - try it Monday, with the current auto_increment_increment, see if the program still works (it should, but basically we're testing that the variable @@global.auto_increment_increment does the right thing). Then, after it works, I'll change the setting of auto_increment_increment to 2, and we can verify again that it still works.

This is on -dev and has been working fine it seems. Sheeri, let me know when you want to try bumping it and I can test the translations.

[14:57:03] <sheeri> there's a problem tho [14:57:19] <sheeri> it's a global parameter, and there are other dbs on there: [14:57:20] <sheeri> | addons_allizom_org | [14:57:20] <sheeri> | addons_dev_allizom_org | [14:57:20] <sheeri> | builder_dev | [14:57:20] <sheeri> | builder_stage | [14:57:33] <sheeri> so changing it for dev would change it for stage too [14:57:36] <sheeri> and for builder

Also fwiw: [14:55:20] <kumar> I see addons_dev_allizom_org on db-zlb-rw.stage.addons.phx1.mozilla.com

Done! mysql> set global auto_increment_increment=2; Query OK, 0 rows affected (0.00 sec)

I did a quick spot check and translations are working fine and the id is incrementing by two as expected. Once we bang on it a little more this seems like it's a safe change.

FWIW this is on dev *and* stage. Let me know when you're ready to take on the staging server architecture that we can failover on (there's a separate heirarchy for that).

ok, that can be done in a separate bug. In response to this original bug, staggered auto increments looks like it will work.

Per QA we are letting this bake in -dev a few more days so I'm reverting the commit while we do a prod push today. The fix: https://github.com/mozilla/zamboni/commit/4a4fdd43f0d9a8211d4c1fe5870821b40a25d94c The revert https://github.com/mozilla/zamboni/commit/04c1caeae4c15297bc742b84280b94063c3d9e17

The prod release has been pushed so I'm putting this back on -dev https://github.com/mozilla/zamboni/commit/cac83768f446184095268d0b0e0e87efe8e85d54

our contractors have been testing this for a week and have found no issues with translations.

Yay!

Note that this is on dev *and* stage, and if it works we can put it into production whenever you want us to, and enable automatic failover.

(In reply to Sheeri Cabral [:sheeri] from comment #21) > Note that this is on dev *and* stage, and if it works we can put it into > production whenever you want us to, and enable automatic failover. How about putting it in production today after we push the code that switches us to the setting? The testing has been done already. Our push is at 2pm PST

Just updating - sounds good to me!

reopening, this is mine - I changed auto_increment_increment to 2 on all machines, and 1 or 2 depending on the machine in production in scl3 and phx1. I need to put these into puppet so the changes persist across restarts.

How's this going?

Hasn't been done yet, but the current configs are set, so the only problems are if a machine restarts.

Is there an ETA for this?

ETA is this week, barring tons of fires preventing the work from getting done.

Adding a link to bug 717638 - we're almost there, but we might spill over into next week.

Giving to Dustin - this entails making variables for 2 options in the /etc/my.cnf: auto_increment_increment auto_increment_offset They should default to not being set in /etc/my.cnf, and they should be specifically configured for the addons databases to be: auto_increment_increment=2 and auto_increment_offset either 1 or 2, depending on what's currently set in the running configs (mysql> SHOW GLOBAL VARIABLES LIKE 'auto_increment_offset';)

So, these db's have been restarted - at least twice - in the last few weeks. Luckily, not the masters - at least, not by me. Puppet only knows of addons DB's in phx1, and they're all running with an increment of 1. Hopefully scl3 has the increment of 2 so we're not duplicating id's or anything. Incidentally, the options are already available in the puppet module, so they'll just need to be enabled -- and whatever hosts are in scl3 should be added to puppet! So I can set this up, but clearly I lack some context, and it may already be broken. Pls advise :)

AMO's not in slc3, because scl3 is not sas70 compliant, so our failover is to the cloud (not built yet, in tickets). let's have a vidyo meeting tomorrow, and I can explain how the auto_increment_increment/offset thing works. (e.g. the algorithm to make up good values).

I understand that part. I don't understand how you configured servers in scl3 if there are no servers in scl3 :)

Please clarify the current status of dev/stage/prod. Which are set to 1?

addons{1..7}.db.phx1 and addons{1..5}.stage.db.phx1 are set to 1. I don't know of any other addons servers.

Wil, we're updating the configs today, and after that we'll give you a full report.

So the plan is to set the odd servers to 1 and even servers to 2 across staging and prod, *except* addons7.db.phx1 (prod) is 2. I'll set this in puppet and live.

OK. All increments are 2. Offset is 1 on addons1.db.phx1 addons3.db.phx1 addons5.db.phx1 addons1.stage.db.phx1 (current stage master) addons3.stage.db.phx1 addons5.stage.db.phx1 and 2 on addons2.db.phx1 addons4.db.phx1 addons6.db.phx1 addons7.db.phx1 (current prod master) addons2.stage.db.phx1 addons4.stage.db.phx1 The same settings are enforced from puppet.