Compunnel
Senior \/ Lead Hadoop Engineer
Compunnel, Charlotte, North Carolina, United States, 28245
Job Summary
We are seeking a highly experienced Senior / Lead Hadoop Engineer to lead complex technology initiatives focused on maintaining platform stability, automating monitoring, and providing 24/7 support for Hadoop clusters. The ideal candidate will have extensive experience in Hadoop administration, Unix scripting, and incident management. This role involves managing and optimizing the Hadoop ecosystem, coordinating with platform tenants and vendors, and ensuring platform resilience through backup, recovery, and patch management.
Key Responsibilities
Platform Stability and Incident Management:
• Lead complex technology initiatives focused on Hadoop platform stability and automation.
• Manage and troubleshoot platform incidents, serving as part of a 24/7 platform support team.
• Assess the impact of platform issues, prioritize triaging efforts, and partner with tenants, developers, and vendors.
• Perform root cause analysis of incidents and implement permanent solutions.
• Communicate progress and status updates to stakeholders and leadership.
Hadoop Cluster Monitoring and Troubleshooting:
• Monitor Hadoop cluster health and performance using tools such as Prometheus, Splunk, and Thousand Eyes.
• Automate monitoring opportunities and fine-tune alert systems for optimal performance.
• Use Autosys or similar scheduling tools to manage platform activities.
• Troubleshoot tenant-reported failures by analyzing logs to identify root causes.
Hadoop Configuration and Tuning:
• Periodically configure and tune Hadoop components (HDFS, YARN, Hive, HBase, Spark, HPOS).
• Schedule and track change requests for implementation.
• Collaborate with tenants and operating system administrators to manage patches and upgrades.
Backup, Recovery, and Patching Management:
• Ensure platform resilience through robust data backup and recovery strategies.
• Execute failover to disaster recovery sites and manage fallback processes.
• Plan and perform upgrades and patch management of Hadoop components.
• Validate cluster health post-patching and communicate with stakeholders.
• Identify automation opportunities to enhance patching processes.
Documentation and Reporting:
• Maintain detailed documentation of cluster configurations, troubleshooting processes, and procedures.
• Generate reports on cluster usage, performance metrics, and capacity utilization.
Collaboration and Support:
• Collaborate with platform tenants, development teams, and Hadoop vendors to troubleshoot complex issues.
• Work independently and in coordination with key technical experts to ensure production stability.
Required Qualifications:
• 8+ years of experience in technology incident and problem management.
• 4+ years of experience using monitoring tools (Prometheus, Splunk, SiteScope, Thousand Eyes).
• 6+ years of experience with Unix scripting.
• 2+ years of experience in Hadoop administration or a similar role.
• Strong knowledge of the Hadoop ecosystem and its components (Hive, HBase, Spark, HPOS).
• Experience with Autosys or a similar job scheduling tool.
Preferred Qualifications:
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills for working with cross-functional teams.
• Ability to work both independently and as part of a collaborative team.
Education:
Bachelors Degree
We are seeking a highly experienced Senior / Lead Hadoop Engineer to lead complex technology initiatives focused on maintaining platform stability, automating monitoring, and providing 24/7 support for Hadoop clusters. The ideal candidate will have extensive experience in Hadoop administration, Unix scripting, and incident management. This role involves managing and optimizing the Hadoop ecosystem, coordinating with platform tenants and vendors, and ensuring platform resilience through backup, recovery, and patch management.
Key Responsibilities
Platform Stability and Incident Management:
• Lead complex technology initiatives focused on Hadoop platform stability and automation.
• Manage and troubleshoot platform incidents, serving as part of a 24/7 platform support team.
• Assess the impact of platform issues, prioritize triaging efforts, and partner with tenants, developers, and vendors.
• Perform root cause analysis of incidents and implement permanent solutions.
• Communicate progress and status updates to stakeholders and leadership.
Hadoop Cluster Monitoring and Troubleshooting:
• Monitor Hadoop cluster health and performance using tools such as Prometheus, Splunk, and Thousand Eyes.
• Automate monitoring opportunities and fine-tune alert systems for optimal performance.
• Use Autosys or similar scheduling tools to manage platform activities.
• Troubleshoot tenant-reported failures by analyzing logs to identify root causes.
Hadoop Configuration and Tuning:
• Periodically configure and tune Hadoop components (HDFS, YARN, Hive, HBase, Spark, HPOS).
• Schedule and track change requests for implementation.
• Collaborate with tenants and operating system administrators to manage patches and upgrades.
Backup, Recovery, and Patching Management:
• Ensure platform resilience through robust data backup and recovery strategies.
• Execute failover to disaster recovery sites and manage fallback processes.
• Plan and perform upgrades and patch management of Hadoop components.
• Validate cluster health post-patching and communicate with stakeholders.
• Identify automation opportunities to enhance patching processes.
Documentation and Reporting:
• Maintain detailed documentation of cluster configurations, troubleshooting processes, and procedures.
• Generate reports on cluster usage, performance metrics, and capacity utilization.
Collaboration and Support:
• Collaborate with platform tenants, development teams, and Hadoop vendors to troubleshoot complex issues.
• Work independently and in coordination with key technical experts to ensure production stability.
Required Qualifications:
• 8+ years of experience in technology incident and problem management.
• 4+ years of experience using monitoring tools (Prometheus, Splunk, SiteScope, Thousand Eyes).
• 6+ years of experience with Unix scripting.
• 2+ years of experience in Hadoop administration or a similar role.
• Strong knowledge of the Hadoop ecosystem and its components (Hive, HBase, Spark, HPOS).
• Experience with Autosys or a similar job scheduling tool.
Preferred Qualifications:
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills for working with cross-functional teams.
• Ability to work both independently and as part of a collaborative team.
Education:
Bachelors Degree