Broken out from bug 1130303. It appears that the cycle-data task is a bit too aggressive at present. We should look into reducing the chunk size and/or reworking the queries. We could also look into using the active_status field to separate the task into a "mark" and "delete" phase - though I have no idea whether that would help, given we already avoid cascade deletes via the way we delete rows, plus DBs and me are still a bit http://i0.kym-cdn.com/photos/images/original/000/234/739/fa5.jpg Adding bug 1126943 as a dep, since reducing the size of the object store will speed up those deletes, plus the approach in that bug is likely to use independent criteria for deleting rows, rather than specifying a large list of IDs (which my gut says may be faster).

To stop the pain here, I'm going to try reducing the chunk size. Bug 1130303 will look into more significant changes later on.

Commit pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/12d991e204e19e430d6619d696c4e8e97e9f1ab4 Bug 1130355 - Data cycling: Reduce chunk size to avoid timeouts/alerts The current chunk size of 100 translates to 100 pushes, which can be as many as 40,000 jobs, which can result in deletes of 100,000+ rows at a time from the artifact tables. This blocks other DB activity & also means the cycle-data task sometimes hits: OperationalError: (2006, 'MySQL server has gone away')

So this has landed and fixes the "MySQL server has gone away" + makes the data-cycling less aggressive (since we wait 2 secs between each query, so with smaller queries, it's spread out more). However deleting from the objectstore is still slower than most of the other tables, but I think we should handle that in bug 1140349 / bug 1126943. Tomcat - let me know if you see any more lag - the original issue should be less of an issue now.