Closed
Bug 1269781
Opened 8 years ago
Closed 8 years ago
Use centralized metastore for Hive
Categories
(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)
Cloud Services Graveyard
Metrics: Pipeline
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: rvitillo, Assigned: whd)
References
Details
(Whiteboard: [SvcOps])
Attachments
(1 file)
|
(deleted),
text/x-github-pull-request
|
Details |
We should have a centralized Hive metastore so that we can avoid importing the tables during the bootstrap process of the Spark clusters. This will also allow us to easily update the metastore when a new partition is added on S3 by a scheduled job.
|
Reporter | |
Updated•8 years ago
|
Whiteboard: [SvcOps]
|
||
Updated•8 years ago
|
Points: --- → 2
Priority: -- → P2
|
|
||
Comment 1•8 years ago
|
||
Blake being out this sprint, will pack into next sprint as a P1.
|
||
Updated•8 years ago
|
Priority: P2 → P1
|
|
||
Updated•8 years ago
|
Assignee: nobody → bimsland
|
|
||
Comment 2•8 years ago
|
||
There is now a centralized metastore available that is being used by Presto, the config changes now just need to get added to / tested with Spark.
|
|
Reporter | |
Comment 3•8 years ago
|
||
Blake, could you please send a PR with the changes you have made to emr-bootstrap-presto?
Flags: needinfo?(bimsland)
|
|
||
Comment 5•8 years ago
|
||
The patch contained both EMR upgrade and metastore, need to split the patch as the version of Presto with EMR 4.7.2 has some issues that will take longer to resolve.
|
||
Comment 6•8 years ago
|
||
|
|
||
Updated•8 years ago
|
Attachment #8806519 -
Flags: review?(rvitillo)
|
|
||
Updated•8 years ago
|
Attachment #8806519 -
Flags: review?(rvitillo)
|
|
||
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
|
|
Reporter | |
Comment 7•8 years ago
|
||
I am reopening this at it appears that all our EMR clusters (our two Presto ones & Spark) are using all different metastores.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
|
|
Reporter | |
Updated•8 years ago
|
Severity: normal → critical
|
||
Updated•8 years ago
|
Assignee: bimsland → whd
|
Assignee | |
Comment 8•8 years ago
|
||
We're using the same metastore for the main presto cluster and spark instances now. The other spark cluster should be going away by the end of the month, so I'm marking this as fixed.
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
|
||
Updated•6 years ago
|
Product: Cloud Services → Cloud Services Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•