Closed
Bug 1088920
Opened 10 years ago
Closed 8 years ago
Scheduler: Allow for more tasks in a task-graph
Categories
(Taskcluster Graveyard :: Scheduler, defect)
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: jonasfj, Assigned: jonasfj)
References
Details
Leaf tasks in a task-graph are all referenced as required by the task-graph root node.
As azure table storage values are limited to 64k (and items to 1MB) we will run into issues if there are 1200 leaf nodes in a task-graph.
The solution is to:
A) Encode slugids as binary 128 bit ints, that gives us 4k under a single key
B) Use multiple keys to the list of leaf taskIds
4k might be enough, if not we can use 4 keys that'll take 5 keys, use 320 KB of space and give us up to 20k tasks.
Assignee | ||
Comment 1•10 years ago
|
||
I mixed up my tabs and the discussion took place on bug 1088916, conclusion:
support 24k taskIds using a binary array...
Assignee | ||
Comment 2•10 years ago
|
||
So while implementing test for the new entities wrapper... I realize that, this might be more complicated than initially thought.
My thinking was that we would have a large list of taskIds 4k+ in a single azure table entity.
And we can, but the idea with this list is that taskIds are removed as tasks are completed.
ie. the list is used to track state, but as it turns out... If we have 4k current editor to a single azure table storage entity, then our optimistic concurrency model might just break.
Anyways, we'll implement it this way for now. And hope all tasks don't finish at the same time. Later, we probably need to engineer around this issue by using multiple azure table entities to track state. If we need 4k tasks in a task-graph.
Assignee | ||
Updated•10 years ago
|
Assignee: nobody → jopsen
Updated•9 years ago
|
Component: TaskCluster → Scheduler
Product: Testing → Taskcluster
Assignee | ||
Comment 3•8 years ago
|
||
scheduler is now deprecated in favor of task-groups and task.dependencies.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Updated•6 years ago
|
Product: Taskcluster → Taskcluster Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•