Closed
Bug 1130355
Opened 10 years ago
Closed 10 years ago
Make the cycle-data task less aggressive, so it does not block process-objects for 25 mins
Categories
(Tree Management :: Treeherder, defect, P1)
Tree Management
Treeherder
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Assigned: emorley)
References
Details
Attachments
(1 file)
Broken out from bug 1130303.
It appears that the cycle-data task is a bit too aggressive at present.
We should look into reducing the chunk size and/or reworking the queries. We could also look into using the active_status field to separate the task into a "mark" and "delete" phase - though I have no idea whether that would help, given we already avoid cascade deletes via the way we delete rows, plus DBs and me are still a bit http://i0.kym-cdn.com/photos/images/original/000/234/739/fa5.jpg
Adding bug 1126943 as a dep, since reducing the size of the object store will speed up those deletes, plus the approach in that bug is likely to use independent criteria for deleting rows, rather than specifying a large list of IDs (which my gut says may be faster).
Assignee | ||
Comment 1•10 years ago
|
||
To stop the pain here, I'm going to try reducing the chunk size. Bug 1130303 will look into more significant changes later on.
Assignee: nobody → emorley
Assignee | ||
Comment 2•10 years ago
|
||
Attachment #8571905 -
Flags: review?(cdawson)
Assignee | ||
Updated•10 years ago
|
Status: NEW → ASSIGNED
Updated•10 years ago
|
Attachment #8571905 -
Flags: review?(cdawson) → review+
Comment 3•10 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder-service
https://github.com/mozilla/treeherder-service/commit/12d991e204e19e430d6619d696c4e8e97e9f1ab4
Bug 1130355 - Data cycling: Reduce chunk size to avoid timeouts/alerts
The current chunk size of 100 translates to 100 pushes, which can be as
many as 40,000 jobs, which can result in deletes of 100,000+ rows at a
time from the artifact tables. This blocks other DB activity & also
means the cycle-data task sometimes hits:
OperationalError: (2006, 'MySQL server has gone away')
Assignee | ||
Comment 4•10 years ago
|
||
So this has landed and fixes the "MySQL server has gone away" + makes the data-cycling less aggressive (since we wait 2 secs between each query, so with smaller queries, it's spread out more).
However deleting from the objectstore is still slower than most of the other tables, but I think we should handle that in bug 1140349 / bug 1126943.
Tomcat - let me know if you see any more lag - the original issue should be less of an issue now.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Summary: The cycle-data task should not block process-objects for 25 mins → Make the cycle-data task less aggressive, so it does not block process-objects for 25 mins
You need to log in
before you can comment on or make changes to this bug.
Description
•