Compunnel
Job Summary:
We are seeking a Senior Data Engineer to design, develop, and maintain big data applications using the latest open-source technologies.
The ideal candidate will have extensive experience in GCP, data pipeline development, and distributed data processing.
This role also requires mentorship, project leadership, and collaboration with cross-functional teams to deliver high-quality data solutions.
Key Responsibilities:
Big Data Development & Engineering: Design and develop big data applications using Apache Hive, Spark, and Kafka. Automate workflows using Apache Airflow. Build data pipelines and optimize data processing workflows. Develop logical and physical data models for big data platforms. Provide ongoing maintenance, enhancements, and on-call support for existing systems. GCP Expertise & Cloud Data Engineering:
Build data pipelines in GCP using GCP Dataproc, GCS, and BigQuery. Leverage cloud-native tools to optimize performance, scalability, and cost. Implement best practices for cloud data processing and storage. Leadership & Project Execution:
Lead daily standups, design reviews, and backlog grooming using JIRA. Mentor and guide junior engineers on best practices and development standards. Act as the point of contact for assigned business domains, ensuring alignment with business needs. Work collaboratively with cross-functional teams, including product managers, architects, and DevOps teams. Code Quality & Best Practices:
Follow Gitflow development practices and version control using BitBucket, JIRA, and Confluence. Implement CI/CD pipelines using Jenkins, Bamboo, or TFS. Ensure coding standards, performance optimization, and code reviews are consistently followed. Conduct unit testing and validate test results against product specifications. Required Qualifications:
10+ years of experience in data engineering and data warehouse development. 6+ years of hands-on experience with Hadoop, Hive, Spark, Airflow, or similar workflow orchestration solutions. 4+ years of recent experience working with GCP. Strong proficiency in Python, Java, Scala, and scripting languages (Perl, Shell, etc.). Experience in schema design and data modeling for data lakes and RDBMS platforms. Expertise in processing and managing large datasets (multi-TB/PB scale). Hands-on experience with Kubernetes, Docker, and containerized environments. Experience with Agile methodologies, Scrum/Kanban development practices. Strong analytical, problem-solving, and troubleshooting skills. Excellent verbal and written communication skills. Preferred Qualifications:
Experience with CI/CD tools (Bamboo, Jenkins, or TFS). Familiarity with Atlassian products (BitBucket, JIRA, Confluence). Knowledge of Gitflow branching strategies. Experience in test-driven development and automated testing frameworks. Background in domain-specific knowledge and customer engagement.
Education:
Bachelors Degree
We are seeking a Senior Data Engineer to design, develop, and maintain big data applications using the latest open-source technologies.
The ideal candidate will have extensive experience in GCP, data pipeline development, and distributed data processing.
This role also requires mentorship, project leadership, and collaboration with cross-functional teams to deliver high-quality data solutions.
Key Responsibilities:
Big Data Development & Engineering: Design and develop big data applications using Apache Hive, Spark, and Kafka. Automate workflows using Apache Airflow. Build data pipelines and optimize data processing workflows. Develop logical and physical data models for big data platforms. Provide ongoing maintenance, enhancements, and on-call support for existing systems. GCP Expertise & Cloud Data Engineering:
Build data pipelines in GCP using GCP Dataproc, GCS, and BigQuery. Leverage cloud-native tools to optimize performance, scalability, and cost. Implement best practices for cloud data processing and storage. Leadership & Project Execution:
Lead daily standups, design reviews, and backlog grooming using JIRA. Mentor and guide junior engineers on best practices and development standards. Act as the point of contact for assigned business domains, ensuring alignment with business needs. Work collaboratively with cross-functional teams, including product managers, architects, and DevOps teams. Code Quality & Best Practices:
Follow Gitflow development practices and version control using BitBucket, JIRA, and Confluence. Implement CI/CD pipelines using Jenkins, Bamboo, or TFS. Ensure coding standards, performance optimization, and code reviews are consistently followed. Conduct unit testing and validate test results against product specifications. Required Qualifications:
10+ years of experience in data engineering and data warehouse development. 6+ years of hands-on experience with Hadoop, Hive, Spark, Airflow, or similar workflow orchestration solutions. 4+ years of recent experience working with GCP. Strong proficiency in Python, Java, Scala, and scripting languages (Perl, Shell, etc.). Experience in schema design and data modeling for data lakes and RDBMS platforms. Expertise in processing and managing large datasets (multi-TB/PB scale). Hands-on experience with Kubernetes, Docker, and containerized environments. Experience with Agile methodologies, Scrum/Kanban development practices. Strong analytical, problem-solving, and troubleshooting skills. Excellent verbal and written communication skills. Preferred Qualifications:
Experience with CI/CD tools (Bamboo, Jenkins, or TFS). Familiarity with Atlassian products (BitBucket, JIRA, Confluence). Knowledge of Gitflow branching strategies. Experience in test-driven development and automated testing frameworks. Background in domain-specific knowledge and customer engagement.
Education:
Bachelors Degree