Closed
Bug 1263955
Opened 9 years ago
Closed 8 years ago
Presto should limit the amount of memory used to allow other things to run on the instance
Categories
(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)
Cloud Services Graveyard
Metrics: Pipeline
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: azhang, Assigned: robotblake)
References
Details
(Whiteboard: [SvcOps])
When a query is running in the Presto machine [1], it uses almost all of the available memory. When this happens, Parquet2Hive doesn't have enough, and it fails with:
> Failure to parse dataset, 'NoneType' object has no attribute 'group'
What we probably want to do is to limit the memory used by Presto.
[1]: hadoop@ec2-54-218-5-112.us-west-2.compute.amazonaws.com
Reporter | ||
Updated•9 years ago
|
Flags: needinfo?(whd)
Comment 1•9 years ago
|
||
The current Presto configuration uses a large chunk of the available memory on that instance [1]. The celery queue of re:dash can also consume a large amount of memory and I am not sure if that's bounded somehow. When that happens nearly no memory is left for other processes.
We should deploy the re:dash service on its own independent instance; that would also make it easier to redeploy Presto with a new configuration.
[1] https://github.com/vitillo/emr-bootstrap-presto/blob/master/ansible/files/telemetry.sh#L54
Updated•9 years ago
|
Whiteboard: [SvcOps]
Comment 2•9 years ago
|
||
reassigning to Travis in light of whd's availability.
Flags: needinfo?(tblow)
Updated•9 years ago
|
Assignee: nobody → bimsland
Points: --- → 2
Priority: -- → P2
Assignee | ||
Comment 3•8 years ago
|
||
This will be taken care of by https://bugzilla.mozilla.org/show_bug.cgi?id=1279147
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Updated•8 years ago
|
Flags: needinfo?(whd)
Updated•8 years ago
|
Flags: needinfo?(tblow)
Updated•6 years ago
|
Product: Cloud Services → Cloud Services Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•