Closed Bug 1088920 Opened 10 years ago Closed 8 years ago

Scheduler: Allow for more tasks in a task-graph

Categories

(Taskcluster Graveyard :: Scheduler, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jonasfj, Assigned: jonasfj)

References

Details

Leaf tasks in a task-graph are all referenced as required by the task-graph root node. As azure table storage values are limited to 64k (and items to 1MB) we will run into issues if there are 1200 leaf nodes in a task-graph. The solution is to: A) Encode slugids as binary 128 bit ints, that gives us 4k under a single key B) Use multiple keys to the list of leaf taskIds 4k might be enough, if not we can use 4 keys that'll take 5 keys, use 320 KB of space and give us up to 20k tasks.
I mixed up my tabs and the discussion took place on bug 1088916, conclusion: support 24k taskIds using a binary array...
So while implementing test for the new entities wrapper... I realize that, this might be more complicated than initially thought. My thinking was that we would have a large list of taskIds 4k+ in a single azure table entity. And we can, but the idea with this list is that taskIds are removed as tasks are completed. ie. the list is used to track state, but as it turns out... If we have 4k current editor to a single azure table storage entity, then our optimistic concurrency model might just break. Anyways, we'll implement it this way for now. And hope all tasks don't finish at the same time. Later, we probably need to engineer around this issue by using multiple azure table entities to track state. If we need 4k tasks in a task-graph.
Assignee: nobody → jopsen
Depends on: 1124839
Component: TaskCluster → Scheduler
Product: Testing → Taskcluster
scheduler is now deprecated in favor of task-groups and task.dependencies.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.